Re: TeXLive merge into FreeBSD ports tree - FreeBSD project idea
Hi all, Summary of below. I have started an effort to get TeXLive into the FreeBSD ports. See github.com/DragonSA/texlive for details. Volunteers welcome. On Sunday, 17 June 2012 22:04:15 David Schultz wrote: On Sun, Jun 17, 2012, Jan Henrik Sylvester wrote: Even with a knob instead of checking if print/texlive-core is installed, it would put a lot of mess into the ports tree. Some maintainers will not agree to introduce these conditions, if there is no general agreement that we want to transition to TeXLive that way. teTeX hasn't been updated since 2006 and the maintainer of teTeX has discontinued development and recommends using TeXLive. Undoubtably updating the TeXLive will break some things but the same could be said about teTeX if an update existed. Are there any people who don't want to migrate to TeXLive? As far as I remember, both romain@ and hrs@ have stated that they do not want both teTeX and TeXLive in the tree concurrently. Looking through the list of bundled packages for TeXLive it will be quite difficult to have both TeXLive and teTeX in the tree concurrently. There are a group of support ports, such as xdvik and ptex that require special integration into either teTeX or TeXLive. To switch between either will require reinstalling many dependencies, not just the TeXLive or teTeX ports. With that said, it is not impossible nor, I think, will it impose a big maintenance burden. In that case, it sounds like using TeXLive in FreeBSD will be a bit tricky until someone steps in and does all the work required for integration. Unfortunately I'm a bit time-constrained for the next few months, but I do use TeXLive and would be happy to test any proposed patches. I have started working on just this, getting TeXLive into FreeBSD ports. It is still early days and I have only two monolithic ports, with texlive-base requiring lots of love. I believe my work is in a state ready for initial testing (verifying the ports) and some feedback. Some open questions are: - how to split texlive-texmf (OpenBSD has an approach) - how to handle bundled software that has existing (sometimes outdated) ports Also, I am looking for a sponsor: someone who will commit the ports when they are ready, and to provide feedback so that the ports may get to a ready state. For details about the project, and to get the ports, please see: github.com/DragonSA/texlive Regards P.S. I am not subscribed to freebsd-ports@ so please CC me. signature.asc Description: This is a digitally signed message part.
Re: rtld optimizations
On Wednesday 26 January 2011 06:49:11 Alexander Kabaev wrote: On Tue, 25 Jan 2011 21:40:42 -0500 Mark Saad nones...@longcount.org wrote: Hello Hackers The NetBSD folks have a nice improvement with the rtld-elf subsystem, known as Negative Symbol Cache . http://blog.netbsd.org/tnf/entry/netbsd_runtime_linker_gains_negative Roy Marples roy@ has a simple write up of the change. I took the basic idea from FreeBSD, but improved the performance drastically. Basically, the huge win is by caching both breadth and depth of the needed/weak symbol lookup. Easiest to think of a,b,c,d as a matrix and FreeBSD just cache a row where we cache both rows and columns. Has anyone looked into porting the changes back to FreeBSD ? The improvement on load time for things like firefox, openoffice, and java is huge on NetBSD. It looks like this change could improve load times on FreeBSD in the same ways. This is a second time someone posts this to public mailing list and curiously enough is a second time it suggested that someone else is to do the investigation. From the quick look, the commit in question is more or less a direct rip-off of Donelists we had for ages and as such is completely over-hyped. The only extra quirk that said commit does is an optimization of a dlsym() call, which is hardly ever in critical performance path. Said optimization is trivial and easy to try. Here you have it: http://people.freebsd.org/~kan/rtld-symlook-depth.diff Since it only applies to dlsym, it only affects programs that are heavy plugin users, which I suppose is the category OpenOffice and firefox both fall into. Care to do some benchmarks with and without the patch and report the results? I frankly doubt that you'll see any noticeable difference compared to our stock rtld's performance. I benchmarked the impact said patch has on the boot-time of my system. I timed the boot-time to when KDE launches autostart programs and once all programs have loaded (I run a few extra programs, such as amarok). The latter measure requires human action thus it has extra, human, variance in its measure. I tried an older version of rtld (about 2 months old), current version of rtld and the new (patched) rtld. I ran each test three times. There was little variance in the tests and I am confident that there is no difference between the different rtld versions and my boot-time. Here is a summary of my boot times (in seconds). First measure is when KDE autostarts programs, the latter is when I determined when all programs had launched. rtld-old: 69 96 rtld: 69 94 rtld-new: 69 94 Please note that kernel boot time is approximately 10 seconds and kdm is delayed by about 10 seconds thus 20 seconds can be removed from above numbers to determine non-kernel boot wall-time. I would like to add that the blog entry claims a substantial improvement for some use cases. Is it not worth to optimise these fringe cases as one mans fringe case is another mans normal case (or woman as one prefers)? signature.asc Description: This is a digitally signed message part.
python + build ports = failure (threading problem)
Hi, I've found a problem in FreeBSD's implementation for threads, pth does not have this problem. The following test case consistantly reproduces the problem: --- start --- #! /usr/bin/env python from subprocess import Popen, PIPE from threading import Lock, Thread def runner(): process = Popen((make, -C, /usr/ports/graphics/kdegraphics4, clean, all), close_fds=True) process.wait() # If runner called directly then port compiles correctly, otherwise it stalls Thread(target=runner).start() --- end --- The port stalls with snip/ grep: writing output: Broken pipe grep: writing output: Broken pipe grep: writing output: Broken pipe - found === Configuring for kdegraphics-4.5.3 /bin/mkdir -p /usr/ports/graphics/kdegraphics4/work/kdegraphics-4.5.3/build end/ The grep errors appear to be mostly harmless, `ps` shows: snip/ 17648 4 D 0:00.03 python test.py 17649 4 D 0:00.08 make -C /usr/ports/graphics/kdegraphics4 clean all 17974 4 D 0:00.03 [cmake] 17978 4 Z 0:00.03 defunct snip/ I suspect any port using cmake will have this problem (unconfirmed) and if memory serves me correct then building lang/perl causes the same problem (although the stall happens much later on in the build). Using WITH_PTH when building python fixes the problem so I suspect libthr as the cause of the problem. I hope someone finds this information useful. David P.S. Should a PR be filed for this? P.S.S. It is possible that one of pythons test cases can reproduce this... signature.asc Description: This is a digitally signed message part.
Re: [RFR] mtree improvements (take 2)
On Friday 25 June 2010 22:12:42 David Naylor wrote:
On Friday 25 June 2010 20:01:42 Garrett Cooper wrote:
On Fri, Jun 25, 2010 at 9:52 AM, David Naylor naylor.b.da...@gmail.com
wrote:
Hi,
I've created a patch that increases the performance of mtree. This is
of particular use during a port install. In an extreme case I have
experienced a ~20% increase [1].
For a full discussion see PR bin/143732. This arose out of [2] where I
experienced the increase.
For your convenience I have attached the patch.
Please review this patch and if it is acceptable, commit it.
Regards,
David
1] http://markmail.org/message/iju3l6hyv7s7emrb
2] http://markmail.org/message/gfztjpszl5dozzii
I've included some more changes. I do not think -[uU] needs to be set when
skipping vwalk. I also move sflag so it will get called even if eflag is
set (bug in my patch).
I'm not sure sflag will produce the same results as check() and vwalk() may
call compare() in different orders? Will this impact on crc?
I've also fixed a buffer overflow bug in mtree, there is an example of an mtree
file that triggers the bug. Nothing interesting happens.
All comments are welcome.
David
P.S. does anyone know of a test suite for mtree?
/set uid=1001 gid=1001 mode=0755
. type=dir
this_is_a_very_long_directory_name_that_is_exactly_128_characts_long_requiring_8_to_exceed_the_maximum_path_length_of_1024_byte
type=dir
this_is_a_very_long_directory_name_that_is_exactly_128_characts_long_requiring_8_to_exceed_the_maximum_path_length_of_1024_byte
type=dir
this_is_a_very_long_directory_name_that_is_exactly_128_characts_long_requiring_8_to_exceed_the_maximum_path_length_of_1024_byte
type=dir
this_is_a_very_long_directory_name_that_is_exactly_128_characts_long_requiring_8_to_exceed_the_maximum_path_length_of_1024_byte
type=dir
this_is_a_very_long_directory_name_that_is_exactly_128_characts_long_requiring_8_to_exceed_the_maximum_path_length_of_1024_byte
type=dir
this_is_a_very_long_directory_name_that_is_exactly_128_characts_long_requiring_8_to_exceed_the_maximum_path_length_of_1024_byte
type=dir
this_is_a_very_long_directory_name_that_is_exactly_128_characts_long_requiring_8_to_exceed_the_maximum_path_length_of_1024_byte
type=dir
this_is_a_very_long_directory_name_that_is_exactly_128_characts_long_requiring_8_to_exceed_the_maximum_path_length_of_1024_byte
type=dir
and_a_file_to_tip_the_scale
..
..
..
..
..
..
..
..
..
--- /usr/src/usr.sbin/mtree/verify.c 2007-12-07 14:22:38.0 +0200
+++ verify.c 2010-08-30 22:32:32.0 +0200
@@ -47,20 +47,35 @@
#include mtree.h
#include extern.h
+/*
+ * Error returned when buffer overflows, cannot be 2 as that is reserved for
+ * MISMATCHEXIT.
+ */
+#define BUFFEROVERFLOWEXIT 3
+
static NODE *root;
static char path[MAXPATHLEN];
-static void miss(NODE *, char *);
+static int miss(NODE *, char *, size_t);
+static int check(NODE *, char *, size_t);
static int vwalk(void);
int
mtree_verifyspec(FILE *fi)
{
- int rval;
+ int rval = 0;
root = mtree_readspec(fi);
- rval = vwalk();
- miss(root, path);
+ /* No need to walk tree if we are ignoring extra files. */
+ if (!eflag)
+ rval = vwalk();
+ *path = '\0';
+ rval |= miss(root, path, MAXPATHLEN);
+
+ /* Called here to make sure vwalk() and check() have been called */
+ if (sflag)
+ warnx(%s checksum: %lu, fullpath, (unsigned long)crc_total);
+
return (rval);
}
@@ -135,40 +150,96 @@
if (ep)
continue;
extra:
- if (!eflag) {
- (void)printf(%s extra, RP(p));
- if (rflag) {
-if ((S_ISDIR(p-fts_statp-st_mode)
-? rmdir : unlink)(p-fts_accpath)) {
- (void)printf(, not removed: %s,
- strerror(errno));
-} else
- (void)printf(, removed);
- }
- (void)putchar('\n');
+ (void)printf(%s extra, RP(p));
+ if (rflag) {
+ if ((S_ISDIR(p-fts_statp-st_mode)
+ ? rmdir : unlink)(p-fts_accpath)) {
+(void)printf(, not removed: %s,
+strerror(errno));
+ } else
+(void)printf(, removed);
}
+ (void)putchar('\n');
(void)fts_set(t, p, FTS_SKIP);
}
(void)fts_close(t);
- if (sflag)
- warnx(%s checksum: %lu, fullpath, (unsigned long)crc_total);
return (rval);
}
-static void
-miss(NODE *p, char *tail)
+static int
+check(NODE *p, char *tail, size_t tail_len)
+{
+ FTSENT fts;
+ struct stat fts_stat;
+
+ if (strlcpy(tail, p-name, tail_len) = tail_len)
+ return (BUFFEROVERFLOWEXIT);
+
+ /*
+ * It is assumed that compare() only requires fts_accpath and fts_statp
+ * fields in the FTSENT structure.
+ */
+ fts.fts_accpath = path;
+ fts.fts_statp = fts_stat;
+
+ if (ftsoptions FTS_LOGICAL) {
+ if (stat(path, fts.fts_statp) || lstat(path, fts.fts_statp))
+ return (0);
+ } else if (lstat(path, fts.fts_statp))
+ return (0);
+
+ p-flags
[RFC] mtree improvements
Hi,
I've created a patch that increases the performance of mtree. This is of
particular use during a port install. In an extreme case I have experienced a
~20% increase [1].
For a full discussion see PR bin/143732. This arose out of [2] where I
experienced the increase.
For your convenience I have attached the patch.
Please review this patch and if it is acceptable, commit it.
Regards,
David
1] http://markmail.org/message/iju3l6hyv7s7emrb
2] http://markmail.org/message/gfztjpszl5dozzii
--- /usr/src/usr.sbin/mtree/verify.c 2010-02-07 15:07:28.0 +0200
+++ verify.c 2010-02-07 15:04:10.0 +0200
@@ -50,17 +50,23 @@
static NODE *root;
static char path[MAXPATHLEN];
-static void miss(NODE *, char *);
+static int miss(NODE *, char *);
+static int check(NODE *, char *);
static int vwalk(void);
int
mtree_verifyspec(FILE *fi)
{
- int rval;
+ int rval = 0;
root = mtree_readspec(fi);
- rval = vwalk();
- miss(root, path);
+ /*
+ * No need to walk entire tree if we are only updating the structure
+ * and extra files are ignored.
+ */
+ if (!(uflag eflag))
+ rval = vwalk();
+ rval |= miss(root, path);
return (rval);
}
@@ -155,15 +161,47 @@
return (rval);
}
-static void
+static int
+check(NODE *p, char *tail)
+{
+ FTSENT fts;
+ struct stat fts_stat;
+
+ strcpy(tail, p-name);
+
+ /*
+ * It is assumed that compare() only requires fts_accpath and fts_statp
+ * fields in the FTSENT structure.
+ */
+ fts.fts_accpath = path;
+ fts.fts_statp = fts_stat;
+
+ if (stat(path, fts.fts_statp))
+ return (0);
+
+ p-flags |= F_VISIT;
+ if ((p-flags F_NOCHANGE) == 0 compare(p-name, p, fts))
+ return (MISMATCHEXIT);
+ else
+ return (0);
+
+ /*
+ * tail is not restored to '\0' as the next time tail (or path) is used
+ * is with a strcpy (thus overriding the '\0'). See +19 lines below.
+ */
+}
+
+static int
miss(NODE *p, char *tail)
{
int create;
char *tp;
const char *type, *what;
- int serr;
+ int serr, rval = 0;
for (; p; p = p-next) {
+ if (uflag eflag)
+ rval |= check(p, tail);
if (p-flags F_OPT !(p-flags F_VISIT))
continue;
if (p-type != F_DIR (dflag || p-flags F_VISIT))
@@ -256,4 +294,5 @@
(void)printf(%s: file flags not set: %s\n,
path, strerror(errno));
}
+ return (rval);
}
signature.asc
Description: This is a digitally signed message part.
Re: [RFC] mtree improvements
On Friday 25 June 2010 20:01:42 Garrett Cooper wrote:
On Fri, Jun 25, 2010 at 9:52 AM, David Naylor naylor.b.da...@gmail.com
wrote:
Hi,
I've created a patch that increases the performance of mtree. This is of
particular use during a port install. In an extreme case I have
experienced a ~20% increase [1].
For a full discussion see PR bin/143732. This arose out of [2] where I
experienced the increase.
For your convenience I have attached the patch.
Please review this patch and if it is acceptable, commit it.
Regards,
David
1] http://markmail.org/message/iju3l6hyv7s7emrb
2] http://markmail.org/message/gfztjpszl5dozzii
Hmm... this has other interesting applications other than just ports,
but unfortunately pkg_install won't really feel as much of a
performance boost (because it uses mtree -e -U when +MTREE exists in
the package).
Comments follow.
Thanks!
-Garrett
I've included some more changes. I do not think -[uU] needs to be set when
skipping vwalk. I also move sflag so it will get called even if eflag is set
(bug in my patch).
I'm not sure sflag will produce the same results as check() and vwalk() may
call compare() in different orders? Will this impact on crc?
--- /usr/src/usr.sbin/mtree/verify.c 2010-02-07 15:07:28.0 +0200
+++ verify.c 2010-02-07 15:04:10.0 +0200
@@ -50,17 +50,23 @@
static NODE *root;
static char path[MAXPATHLEN];
-static void miss(NODE *, char *);
+static int miss(NODE *, char *);
+static int check(NODE *, char *);
static int vwalk(void);
int
mtree_verifyspec(FILE *fi)
{
- int rval;
+ int rval = 0;
root = mtree_readspec(fi);
- rval = vwalk();
- miss(root, path);
+ /*
+ * No need to walk entire tree if we are only updating the structure
+ * and extra files are ignored.
+ */
+ if (!(uflag eflag))
+ rval = vwalk();
gcooper This is where the performance boost is coming from as you're
not walking the directory tree, correct?
Yes, if an update is requested without reporting extra files. In some cases
with a well populated /usr/local (and a slowish HDD) it could take some time.
I think it is possible to even use this optimisation if uflag is not set?
+ rval |= miss(root, path);
return (rval);
}
@@ -155,15 +161,47 @@
return (rval);
}
-static void
+static int
+check(NODE *p, char *tail)
+{
+ FTSENT fts;
+ struct stat fts_stat;
+
+ strcpy(tail, p-name);
gcooper Dangerous. Please use strlcpy with appropriate bounds.
This is the same code used in miss.c:171.
It appears the code assumes that a path a mtree file describes will not exceed
MAXPATHLEN and does not check to see if the buffer overflows. I could create a
patch to fix that. Perhaps a separate patch?
+ /*
+ * It is assumed that compare() only requires fts_accpath and fts_statp
+ * fields in the FTSENT structure.
+ */
+ fts.fts_accpath = path;
+ fts.fts_statp = fts_stat;
+
+ if (stat(path, fts.fts_statp))
+ return (0);
gcooper What about symlink functionality? lstat(2)?
Fixed. This should now be consistent with fts_read.
+ p-flags |= F_VISIT;
+ if ((p-flags F_NOCHANGE) == 0 compare(p-name, p, fts))
+ return (MISMATCHEXIT);
+ else
+ return (0);
+
+ /*
+ * tail is not restored to '\0' as the next time tail (or path) is used
+ * is with a strcpy (thus overriding the '\0'). See +19 lines below.
+ */
+}
+
+static int
miss(NODE *p, char *tail)
{
int create;
char *tp;
const char *type, *what;
- int serr;
+ int serr, rval = 0;
gcooper This isn't correct as per-style(9). Please do:
gcooper
gcooper int rval = 0;
gcooper int serr;
gcooper
gcooper This reduces diff churn and is more style(9)-istically correct.
I didn't know that. Good point.
for (; p; p = p-next) {
+ if (uflag eflag)
+ rval |= check(p, tail);
if (p-flags F_OPT !(p-flags F_VISIT))
continue;
if (p-type != F_DIR (dflag || p-flags F_VISIT))
@@ -256,4 +294,5 @@
(void)printf(%s: file flags not set: %s\n,
path, strerror(errno));
}
+ return (rval);
}
--- mtree/verify.c 2010-05-27 13:47:20.0 +0200
+++ verify.c 2010-06-25 22:04:40.0 +0200
@@ -50,17 +50,25 @@
static NODE *root;
static char path[MAXPATHLEN];
-static void miss(NODE *, char *);
+static int miss(NODE *, char *);
+static int check(NODE *, char *);
static int vwalk(void);
int
mtree_verifyspec(FILE *fi)
{
- int rval;
+ int rval = 0;
root = mtree_readspec(fi);
- rval = vwalk();
- miss(root, path);
+ /* No need to walk tree if we are ignoring extra files. */
+ if (!eflag)
+ rval = vwalk();
+ rval |= miss(root, path
Re: [Proof of Concept] Stacked unionfs based 'tinderbox'
Hi,
Apologies for the delayed reply.
On Thursday 25 February 2010 15:53:03 Ulrich Spörlein wrote:
On Thu, 25.02.2010 at 10:08:15 +0200, David Naylor wrote:
Hi,
As some may have noticed on -current I have been working on using
stacked unionfs to implement a 'tinderbox' type build system. I have
successfully used the scripts to build x11/xorg (and have compared the
results to using the traditional approach using pkg_add). The build
system is stable except for one nasty corner case: deadlocks.
When I did this a couple of years ago, the major problems were failing
chdir(2) calls during ports build, etc.
I'm hoping time and -current will solve this problem as it is beyond my
ability to fix.
To setup a compatible test environment requires:
- recompile the kernel with `options KSTACK_PAGES=32`, otherwise the
kernel will panic with a double fault. WITNESS options results in
substantial performance degradation.
- patch mtree (see below) [optional]
- create the appropriate chroot environment (and reboot) [see below
for corner case]
A performance bottleneck in mtree was identified. This resulted in
mtree (as run by port install) consuming ~20% of the build time. See
bin/143732 for a patch and further details.
Good work!
The normal tinderbox approach takes ~80% more time to install compared to
the quick and dirty approach. The stacked unionfs approach takes ~170%
more time (an increase of ~50% over the tinderbox approach). Some
performance gains can be had if one uses memory backed storage (vs HDD
in this case).
Please explain: what is the quick and dirty approach and which one is
faster now?
The quick and dirty is `make -C /usr/ports/x11/xorg install clean`. The
stacked unionfs is still the slowest (even with a 20% improvement from
patching mtree).
If one is interested in performance in building ports in a sandbox then the
tinderbox route is the way to go.
As your scripts did not make it through, perhaps you can upload them to
the wiki? What I did back then was using a clean base system as the
underlying unionfs store to avoid re-generating the clean base over and
over again. Nowadays, a ZFS clone would probably be the way to go.
I've attached the scripts with a .txt suffix. This should hopefully make it
past mailman. To which wiki page do you refer? You are welcome to add the
scripts there.
I skipped the base system and only unionfs /usr/local. I never cleaned the
base system. I figured that was the easiest to implement when the script runs
in a chroot. An area for further improvements?
I'm not sure if a recursive approach is feasible here, as you can have
only one underlying unionfs mount. But special casing, e.g., perl may
still give a massive speedup. So for each port that has perl as
dependancy, you would not pull in the clean base + pkg_add perl, but
instead grab the clean-base+perl directory as an underlying unionfs.
Have you done any speed comparisons of running programs (such as perl) from a
unionfs. If the disk cache is big enough I don't think there will be a big
performance hit? It is certainly a viable approach.
Regards,
David
#!/bin/sh
BUILDDIR=${BUILDDIR:-/usr/build}
LOCALBASE=${LOCALBASE:-/usr/local}
PORTSDIR=${PORTSDIR:-/usr/ports}
PORT_DBDIR=${PKG_DBDIR:-$BUILDDIR/db_ports}
PKG_DBDIR=${PKG_DBDIR:-$LOCALBASE/db_pkg}
PACKAGES=${PACKAGES:-$BUILDDIR/packages}
ENV=env LOCALBASE=$LOCALBASE PORTSDIR=$PORTSDIR PORT_DBDIR=$PORT_DBDIR
PKG_DBDIR=$PKG_DBDIR PACKAGES=$PACKAGES
MAKE=$ENV make
PKG_ADD=$ENV pkg_add
PKG_DELETE=$ENV pkg_delete
set -e
mkdir -p $BUILDDIR $PACKAGES
port2name() {
echo $1 | sed 's|[/.-]|_|g'
}
port2pkg() {
local pkg_name=
local port=
port=$1; shift
eval pkg_name=PKG$(port2name $port)
eval pkg=\$$pkg_name
if [ -z $pkg ]
then
pkg=$($MAKE -C $port -V PKGNAME)
eval $pkg_name=$pkg
fi
}
depends() {
local depend=
local depends_name=
local _deps=
local name=
local port=
local type
type=$1
port=$2
eval depends_name=DEPEND_${type}_$(port2name $port)
eval deps=\\$$depends_name\
if [ -z $deps ]
then
echo Getting $type dependancies for $port /dev/stderr
if [ $type = build ]
then
depend_list=$($MAKE -C $port -V EXTRACT_DEPENDS -V BUILD_DEPENDS -V
LIB_DEPENDS -V RUN_DEPENDS)
else
depend_list=$($MAKE -C $port -V LIB_DEPENDS -V RUN_DEPENDS)
fi
for depend in $depend_list
do
name=$(echo $depend | cut -f 2 -d ':')
depends runtime $name
_deps=$_deps $deps $name
done
deps=
for depend in $_deps
do
if [ -z `echo $deps | grep $depend ` ]
then
deps=$deps$depend
fi
done
[ `echo $deps | tr ' ' '\n' | sort` = `echo $deps | tr ' ' '\n' | sort
-u` ]
depends_name=$depends_name
eval $depends_name=\$deps \
fi
}
run() {
set +e
trap echo Terminating... INT TERM EXIT
$@
_status=$?
[ $status
[Proof of Concept] Stacked unionfs based 'tinderbox'
Hi, As some may have noticed on -current I have been working on using stacked unionfs to implement a 'tinderbox' type build system. I have successfully used the scripts to build x11/xorg (and have compared the results to using the traditional approach using pkg_add). The build system is stable except for one nasty corner case: deadlocks. To setup a compatible test environment requires: - recompile the kernel with `options KSTACK_PAGES=32`, otherwise the kernel will panic with a double fault. WITNESS options results in substantial performance degradation. - patch mtree (see below) [optional] - create the appropriate chroot environment (and reboot) [see below for corner case] A performance bottleneck in mtree was identified. This resulted in mtree (as run by port install) consuming ~20% of the build time. See bin/143732 for a patch and further details. Here are my build times (in the chroot): # time make -C /usr/ports/x11/xorg install clean 2397.42 real 2228.35 user 1151.00 sys # time ./ports-union-builder.sh (old mtree) 8123.25 real 2280.53 user 6319.77 sys # time ./ports-union-builder.sh (new mtree) 6456.11 real 2272.07 user 5778.74 sys # time ./ports-tinder-builder.sh (new mtree) 4270.82 real 2961.88 user 1636.27 sys The new mtree is the mtree with the above mentioned patch applied. ports-tinder-builder is an approximate to how the read tinderbox works. The normal tinderbox approach takes ~80% more time to install compared to the quick and dirty approach. The stacked unionfs approach takes ~170% more time (an increase of ~50% over the tinderbox approach). Some performance gains can be had if one uses memory backed storage (vs HDD in this case). This approach should have application where wearing of the underlying storage (such as SDD) is of concern. Further investigation suggests that stacking unionfs scales exponentially, not linearly. This may be an area for further improvements. The corner case: sometimes the directories involved experience a deadlock. Any IO under those directories stall the program. I have found that deleting files in the chroot prior to running the unionfs script reliably causes the deadlock. Since WITNESS had significant performance impact the only data into the deadlocks come from procstat. Below are a variety of procstat -kk calls for deadlocked programs: 68217 100119 pkg-config -mi_switch+0x16f sleepq_wait+0x3b __lockmgr_args+0x641 ffs_lock+0x8c VOP_LOCK1_APV+0x46 unionfs_lock+0x28a VOP_LOCK1_APV+0x46 unionfs_lock+0x161 VOP_LOCK1_APV+0x46 unionfs_lock+0x161 VOP_LOCK1_APV+0x46 unionfs_lock+0x161 VOP_LOCK1_APV+0x46 unionfs_lock+0x161 VOP_LOCK1_APV+0x46 unionfs_lock+0x161 VOP_LOCK1_APV+0x46 unionfs_lock+0x161 6574 100201 ls -mi_switch+0x16f sleepq_wait+0x3b __lockmgr_args+0x654 ffs_lock+0x8c VOP_LOCK1_APV+0x46 _vn_lock+0x44 vget+0x67 cache_lookup+0x4f8 vfs_cache_lookup+0xad VOP_LOOKUP_APV+0x40 lookup+0x44d namei+0x4ec kern_statat_vnhook+0x82 kern_statat+0x15 lstat+0x22 syscall+0x1e7 Xfast_syscall+0xe1 Is anyone able to fix the deadlocking issue? Regards, David. P.S. This has been tested under 8 and 9. P.P.S. The scripts used are attached ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: [Proof of Concept] Stacked unionfs based 'tinderbox'
On 25 February 2010 15:53, Ulrich Spörlein u...@freebsd.org wrote: On Thu, 25.02.2010 at 10:08:15 +0200, David Naylor wrote: Hi, As some may have noticed on -current I have been working on using stacked unionfs to implement a 'tinderbox' type build system. I have successfully used the scripts to build x11/xorg (and have compared the results to using the traditional approach using pkg_add). The build system is stable except for one nasty corner case: deadlocks. When I did this a couple of years ago, the major problems were failing chdir(2) calls during ports build, etc. To setup a compatible test environment requires: - recompile the kernel with `options KSTACK_PAGES=32`, otherwise the kernel will panic with a double fault. WITNESS options results in substantial performance degradation. - patch mtree (see below) [optional] - create the appropriate chroot environment (and reboot) [see below for corner case] A performance bottleneck in mtree was identified. This resulted in mtree (as run by port install) consuming ~20% of the build time. See bin/143732 for a patch and further details. Good work! The normal tinderbox approach takes ~80% more time to install compared to the quick and dirty approach. The stacked unionfs approach takes ~170% more time (an increase of ~50% over the tinderbox approach). Some performance gains can be had if one uses memory backed storage (vs HDD in this case). Please explain: what is the quick and dirty approach and which one is faster now? The quick and dirty is `make -C /usr/ports/x11/xorg install clean`. The stacked unionfs is still the slowest (even with a 20% improvement from patching mtree). As your scripts did not make it through, perhaps you can upload them to the wiki? What I did back then was using a clean base system as the underlying unionfs store to avoid re-generating the clean base over and over again. Nowadays, a ZFS clone would probably be the way to go. I'm not sure if a recursive approach is feasible here, as you can have only one underlying unionfs mount. But special casing, e.g., perl may still give a massive speedup. So for each port that has perl as dependancy, you would not pull in the clean base + pkg_add perl, but instead grab the clean-base+perl directory as an underlying unionfs. Cheers Uli ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: compiling root filesystem into kernel (preferably tmpfs root filesystem)
On Thursday 09 April 2009 08:34:22 Travis Daygale wrote: David, thank you for the great information! Yes, I would appreciate seeing the scripts and hearing about the other method you outline. Yes, you understand what I want to achieve exactly. Please see attached for the scripts. There is now a README file, that *should* explain everything, I hope. If you are running FreeBSD 7 then usbbt.sh needs to be edited (newfs and du have options only supported by -current). I see what you're saying about not needing to put it in the kernel file, though for a variety of reasons, I do prefer a single file in the end. Although testing is made much easier by having it as a module (and not having to rebuild the kernel every time). And I did find this after posting (didn't see it on previous searches, though I invested a lot of time looking before I posted):http://lists.freebsd.org/pipermail/freebsd-hackers/2006-November/01 8662.html I have built a root image that I put in the kernel as described in the Nov 2006 post. My UFS root image consists of /sbin/init, where init is a statically compiled C program that just spits out Hello world and sleeps, this binary runs fine under FBSD. At this point, I have the kernel booting but it panics because it says it can't find init Hmmm... I believe (haven't had time to test) that it is finding root? The kernel complains when there is no root (and asks for one). Here is a hint: / grep init_ /boot/defaults/loader.conf #init_path=/sbin/init:/sbin/oinit:/sbin/init.bak:/rescue/init:/stand/sysinstall #init_shell=/bin/sh # The shell binary used by init(8). #init_script= # Initial script to run by init(8) before chrooting. #init_chroot= # Directory for init(8) to chroot into. If you need further help on this please provide more information (the output produced by the kernel, for starters). Not sure though what loader args I might need to be providing? Could it be a /dev issue (though I'm not needing sh, etc., since my init is not a real init)? Still figuring this out. The kernel automatically mounts /dev/ for you (just make sure there is a mount point). Just to point out. There are two other scripts (that I know of) that also produce live images: freesbie and nanobsd. Neither do RAM based images (again, as far as I know). Sam: I've cc'ed you so you can have a look at my scripts, if you are interested. I'll send you an e-mail once I've read through nanobsd carefully. Regards signature.asc Description: This is a digitally signed message part.
Re: compiling root filesystem into kernel (preferably tmpfs root filesystem)
On Saturday 04 April 2009 21:52:14 Travis Daygale wrote: In both the loader and kernel compiling doc, I see snippets of information like this: #Make space in the kernel for a root filesystem on a md device.options MD_ROOT_SIZE=10 boot_dfltrootInstructs the kernel to mount the statically compiled-in root file system. Yes, you can compile a fs image into the kernel. This however will be static and if you want editing then will need to use unionfs with mdmfs. tmpfs cannot be used for this as it does not yet (to my knowledge) support unionfs. My question is, how does one compile a root filesystem into the FreeBSD kernel? Personally I wouldn't recommend taking the approach you want to do. There is simply an easier way. Just load the fs image as a kernel module (sort of). You get the same effect with more flexibility. (I'll explain below). When mounted, I want this root filesystem to run entirely in memory with no other backing store (not even a readonly flash disc nor other backing media such as DVD/CD). This is do-able. I've created a CD that ejects it self when loaded completely. (I thought it was cool :-)) The standard FreeBSD DVD install disc uses just such a root? (Though seems to rely heavily on the rescue binaries being on a read only filesystem backed by the install DVD?) Can't comment, haven't used the FreeBSD CD/DVD's for years (since 6.0) I'm still trying to reverse engineer how that was done, without much luck. Is there a place/documentation I should be finding? PicoBSD, NanoBSD, NFS root diskless systems... all tantalizing close, but not the same thing (read only roots backed by media other than memory). The root filesystem I'm wanting would presumably be in some conceptual sense similar to initramfs in Linux land, if that helps explain what I'm trying to achieve. I'll give you a quick tutorial below (if you need further help please let me know). In fact I have a Linux distribution which consists of a single giant kernel image and when boot, runs entirely in memory, the kernel in fact can't read filesystems other than tmpfs because no filesystems are compiled in. I think you are getting some concepts confused here. It appears all of this won't be possible in FreeBSD (looks like ufs is required) but it appears I can get close to this. Indeed, I'd love a way for the root filesystem in FreeBSD to be of type tmpfs, again similar to what is possible on the Linux side, though I'm much less concerned with the type of filesystem (it just needs to be compiled into the FreeBSD kernel and needs to be a memory backed filesystem when it mounts, no other backing store). Thanks in advance! Ok, onto my explanation: my understanding is that you want to have some type of FreeBSD based system that is loaded completely into RAM. Once loaded (at boot time) this system should have no reliance on any medium (other than RAM). This system, once loaded, should behave the same as if it were backed by a hard drive (except the statefullness after reboots). i.e. the filesystem should be editable. This is of course very possible. STAGE 1: The filesystem In order to have the system in memory one needs a delivery method. As mentioned before this is achieved using a MD device. MD's can have three types of backing, a vnode (aka file, on a CD/DVD or hard drive), or memory (purely in memory, AFAIK no swapping out) and swap (same as memory except my get swapped out). Ignoring the subtle difference between memory and swap, swap is better. Technically the forth is preload but this is the same as memory but done by the loader. See md(4) for further details. Now, MD just imitates a hard drive, one still needs the data to put there. Any filesystem will suite this purpose. My preference is UFS but ISO9660 works just as easily (other options are not so easy but still do-able). Now, to create the filesystem, just install your system into a folder. e.g. # su - # mkdir /tmp/world # cd src; make world kernel distribution DESTDIR=/tmp/world # cp /path/to/packages /tmp/world/tmp # chroot /tmp/world sh -c 'cd /tmp ; pkg_add *' # rm -rf /tmp/world/tmp/* # cat /tmp/world/etc/fstab _EOF proc/proc procfs rw 0 0 tmpfs /tmptmpfs rw 0 0 _EOF * Now, edit /tmp/world as you require to make it work as you want (the easiest way is to create a Flash stick [as per my script], edit the system live and then copy all changes across). * STAGE 2: The filesystem image *** For UFS *** # makefs /tmp/world.ufs /tmp/world # MDDEV=$(mdconfig -a -t vnode /tmp/world.ufs) # tunefs -L ROOTFS /dev/$MDDEV # mdconfig -d -u $MDDEV *** For CD9660 *** # mkisofs -quiet -sysid FREEBSD -rock -untranslated-filenames -max-iso9660-filenames -iso-level 4 -volid DragonBSD -o $WORKDIR/DragonBSD.iso -volid DragonBSD -o /tmp/world.iso /tmp/world Now, since these
Project Ideas and a question
Hi, The question first: init does allow a chroot before booting the system however does it allow the first device to be unmounted and use the new chroot as the root device. If it does how can that be achieved. My motivation for this: Allow a USB device or CDROM to boot the system, then init running of the removable device to execute a script that will allow the system to be setup (such as configure devices, setup gdbe and geli, etc) then when init chroot's it unmounts the removable device and allows that device to be physically removed. I have some project ideas (due to lack of technical skills I can not pursue them at this time but that is no reason not to share :-). If someone thinks an idea is a good one could you please add it to the appropriate location (the volunteer projects page???). My ideas: 1) Automatic module loading. Create a discovery system that upon identifying hardware that a module supports, loads the module. This would probably be a user-land implementation? Motivation: Additional ease of use (especially with sound) 2) Automatic kernel customisation. A tool that builds a custom kernel with all the devices for the current system builtin and with everything not needed removed. Motivation: Take the hard work out of building a custom kernel] 3) If question above is not implemented than add to idea list... Feedback is welcome. Thank you for listening to me. David ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Nvidia amd64
On Friday 28 September 2007 10:21, you wrote: Remko Lodder wrote: Ok, let's see how we can fund this project. First of all the interested developers should assess the bulk of work to do and how long it will take them to complete it. Then we can think about how many people are interested in opening their wallet to see it come into being. Lets do it the other way around ,lets see what people are willinoneg to pay for this, so that the person applying for this knows what he can expect. Pay by feature instead of pay by time. I'm in, as are at least other two posters in this thread. I think that if someone set up a web page for the bounty and spread the news to other forums a fair number of subscriptions will come up. What we need to know is : what is the bulk of work needed? Can't anyone sketch a simple plan with the tasks and the hours/man for each? Hi Has anyone contacted the FreeBSD Foundation. They would be the best to provide this bounty service and could most effectively invest and utilise the bounties for the greater benefit of the comunity while the project is under work. (And they are most likely to be the most trusted people to handle the project and the money). There was a KDE project, Kontact (if I remember correctly), that had implemented a similar facility. (And of course there is the Ubuntu equivalent). May I also point out that this, specifically with regards to nVidia/amd64 and generally, can only work if capable hackers are willing to work on the project(s). Generally with volunteers money is not a critical issue to begin with and thus may not be a strong enough incentive and interest is. I may be jumping the gun here but if this bounty principle takes of then enthusiasts could earn a descant wage and provide a viable financial community around FreeBSD (with businesses more willing to use FreeBSD knowing that requested features [and form of support] could be purchased). I am also keenly awaiting a nvidia/amd64 driver and am willing to donate what I can to the good cause. David pgpTxphtett5W.pgp Description: PGP signature
FreeBSD and Information handling
Hi, I was wondering if there was any tool available in freebsd for handling information, specifically configuration files, package information and such. I know all information is available via text files however in order to access the information the program might have to hard code the retreval system (not being able to use a shared library). So: 1) What tools are there for accessing information (i.e. libraries)? 2) If so is there any unification of the interface? 3) If no to any of the above, is there a need for such tools and should it be a unified interface (not just specific subsets)? David pgpO5c5MBTP0T.pgp Description: PGP signature
fetch: Resume errors, help!
Hi,
I have been having a very annoying problem with fetch and the ports distfiles
fetching. Living in South Africa internet access is not what it could be, as
such I have to use a proxy to access the internet. On to the problem:
When fetch tries to resume from an ftp site that does not support resuming (or
fetch lacks the ability in these specific cases, or the proxy + fetch
faulted) it appends to the file but restarts the transfer from 0. This
results in a file that is too big but does contain a complete copy of the
file. A solution is simple:
es = Expected size of the file
as = Actual size of the file
f = File name
# dd if=f of=f~ bs=$((as - es)) skip=1
# mv f~ f
This is very inconvenient and a problem when trying to download files in
batch. One gets interrupted and stops everything else. As such I looked
into fetch.c and tried to fix the problem. See below for my patch. A
problem is that how to handle the -R flag (no overwrite file flag). I have
three solutions:
1) Ignore flag and overwrite the file anyway
2) Do not overwrite and abort
3) Specify above behaviour with an additional flag
Given that the ports and having many mirrors it is possible to find a site
that does support resuming, and thus will save in bandwidth (important for
me). However it is possible that there are no alternatives or all sites do
not support resume.
In the patch the commented out section is solution 2, the implemented version
is solution 1.
Any advice will be welcome, once a satisfactory solution has been achieved I
will submit a PR with the patch?
Thank you
David
P.S. I've been using FreeBSD for a year or two however I am very new at
writing patches for it, this will be my first so all help and mentoring will
be very welcome.
Patch file:
--- usr.bin/fetch/fetch.c.orig 2007-06-07 09:17:18.0 +0200
+++ usr.bin/fetch/fetch.c 2007-06-07 09:18:39.0 +0200
@@ -510,8 +510,36 @@
(intmax_t)sb.st_size, (intmax_t)us.size);
goto failure;
}
+// /* check fetched offset is as expected */
+// if (url-offset != sb.st_size) {
+// if (R_flag) {
+// /* unable to resume */
+// warnx(%s: unexpected offset,
+// expected %jd bytes, got %jd
+// bytes, (intmax_t)sb.st_size,
+// (intmax_t)url-offset);
+// goto failure_keep;
+// } else if (url-offset != 0)
+// /* if not we need to restart from scratch */
+// goto restart;
+// }
+// if ((of = fopen(path, a)) == NULL) {
+// warn(%s: fopen(), path);
+// goto failure;
+// }
+// /* check feteched offset is either as expected or 0 */
+ if (url-offset != sb.st_size || url-offset != 0)
+ /* if not we need to restart from scratch */
+ goto restart;
+ /*
+* NOTE: R_flag is ignored and local file is over
+* written *not* deleted as it is the only way
+* to resume transfer of this file from this
+* URL.
+*/
/* we got it, open local file */
- if ((of = fopen(path, a)) == NULL) {
+ if ((of = url-offset == 0 ? fopen(path, w) :
+ fopen(path, a)) == NULL) {
warn(%s: fopen(), path);
goto failure;
}
@@ -536,6 +564,7 @@
goto success;
}
+restart:
if (of == NULL) {
/*
* We don't yet have an output file; either this is a
pgp1L79Bm48wE.pgp
Description: PGP signature
Re: Looking for speed increases in make index and pkg_version for ports
On Monday 28 May 2007 03:43, you wrote: Maybe I should look at the inner workings of cmake and gmake. Maybe they have some good ideas. However having looked through the source code of make, and also looking at the cvs logs, it does seem to be well written. The only possibility I see of making it go a lot faster is a complete redesign, e.g. my just in time idea for processing variables. Stephen Just in time (jit), if I remember correctly, is a term used by java interpreters which compile the byte code into machine code!!! Perhaps this could be developed for makefile's, especially bsd.*.mk. This, I think, could be done in two ways: 1) Develop the bsd.*.mk files in C and link it in with make, or 2) Use the makefiles as source to compile into machine code (passibly via C-ASM). The machine code could be created on demand, or cached and only updated if the source makefile changes. I am not sure if this could work or even if it will have any significant speed increase. However if method 2 does work it has the potential to radically increase the speed of ports _while_ maintaining the flexability. All that will be needed is an API for the machine code and a compiler??? David pgp6Gma7Ai2zD.pgp Description: PGP signature
DPS Initial Ideas
Hi, Thank you all for your responses, it has given me much to think about. I guess there is consenses that there is room for improvement in the current pkg system. Attached are some of my initial ideas about what is required and expected in any (and all future) package systems. Since I am going away this weekend I have had limited time to work on this document and as such it is in early stages of development. My objective is to get a clear understanding and target of what is required for this new package system. I am looking at a hybrid approach to storing the package metadata, a combination of SQLite and compressed text files. I am hoping to create a situation where if either gets corrupted it can be created from the other. Further more, any changes to the text files (even without recompressing them) will be intergrated back into the database. This will allow administrators to fiddle around with the text files without having to touch the database. (( Another idea I have is the ability for the whole package data storage to be recreated from the repository data if all stored data got destroyed. This will occur though a process of inspection of the file system. This could be extended to allow all none package files to be combined into a special package with the correct dependencies allowing a system to be restored without a single error... Any thoughts? )) I would also like to create package groupings, where by many packages share the same package file. This will allow easy distribution of ports such as the Xorg 7.2, reducing the 400 or so packages to only a few physical files that can then be installed and managed individually. Ultimately I would like this project to be compatible with the current pkg system (allowing easy transition from one to another), proper integration into the ports system and possibly into the pkgsrc system, but the future will only be revealed in time. All feedback is welcome, I do have some questions to ask: 1) What would random access of a package be required for and how often will/does it occur? 2) I have chosen XML for storing the data. Is there any practical alternative (Please keep in mind that multiple packages metadata could be stored in a single file) I apologies if the document is too dense any too cryptic, all ideas are welcome. Once again thank you all for your feedback and have a good weekend. David pgpzOf156zhK7.pgp Description: PGP signature
New FreeBSD package system (a.k.a. Daemon Package System (dps))
Dear Jordan Recently I stumbled across a document you wrote in 2001, entitled FreeBSD installation and package tools, past, present and future. I find FreeBSD appealing and I would like to contribute it its success, and as your article describes, the installation and packaging system is lacking. Since the installation system is being tackled under a SoC project I am hoping to give the packaging system a go. I was hoping you could help me with an update about the situation with pkg. I have searched the FreeBSD mailing lists and have found little information on the package system. Once I have a (much more) complete understanding of the packaging system (and providing there is work to be done) I would like to write up a proposal to solve the problems, and perhaps provide some innovating new capabilities. After that I will gladly contribute what I can to this (possible) project and hopefully further and improve FreeBSD. Any assistance or information you can give I will be greatly appreciate. I look forward to your reply. David pgp7JOsxLJgHm.pgp Description: PGP signature
