Re: svn commit: r242102 - in head: contrib/bmake usr.bin/bmake
[A bit delayed] On 2012-Nov-15 02:41:22 -0600, Matthew D. Fuller fulle...@over-yonder.net wrote: On Thu, Nov 15, 2012 at 06:07:27PM +1100 I heard the voice of Bruce Evans, and lo! it spake thus: Maybe malloc() would be faster with MALLOC_PRODUCTION. I use /etc/malloc.conf - aj locally. freefall doesn't have /etc/malloc.conf. MALLOC_OPTIONS no longer works, and MALLOC_CONF is too large for me to understand, so I don't know how to turn off non-production features dynamically. FWIW, I suspected MALLOC_PRODUCTION in some massive slowdowns I see from time to time on my -CURRENT box. Most vividly around running portupgrade-related bits (portupgrade, pkgdb, etc). I got annoyed by it enough a few weeks ago to write up a quickie to bang malloc() around and see what happened. Attached, just for grins. Not defining MALLOC_PRODUCTION causes quite significant startup costs for any process that uses malloc() at all. In this case, jemalloc verifies that memory obtained from the kernel is zero-filled, causing (thousands of unnecessary) page faults. For a short-running process that doesn't allocate all the space in the jemalloc arenas, this dominates the runtime - in the case of /bin/echo, I measured 87 page faults with MALLOC_PRODUCTION and 2133 without. -- Peter Jeremy pgpCEY8jxCTmb.pgp Description: PGP signature
Re: svn commit: r242102 - in head: contrib/bmake usr.bin/bmake
On Thu, Nov 15, 2012 at 06:07:27PM +1100 I heard the voice of Bruce Evans, and lo! it spake thus: Maybe malloc() would be faster with MALLOC_PRODUCTION. I use /etc/malloc.conf - aj locally. freefall doesn't have /etc/malloc.conf. MALLOC_OPTIONS no longer works, and MALLOC_CONF is too large for me to understand, so I don't know how to turn off non-production features dynamically. FWIW, I suspected MALLOC_PRODUCTION in some massive slowdowns I see from time to time on my -CURRENT box. Most vividly around running portupgrade-related bits (portupgrade, pkgdb, etc). I got annoyed by it enough a few weeks ago to write up a quickie to bang malloc() around and see what happened. Attached, just for grins. It allocates a big array, frees every other element, allocates another, frees every other, hops back and forth filling in the holes, then hopes around free'ing everything. Doesn't put anything in the pages, just allocates, so it doesn't call on the VM subsystem for much. i.e., just beat the snot out of malloc ;) Here are some quick comparative runs on two systems I have here locally, one running stable/9 (late Jun), the other head (mid Oct). Same binary on both, so it's not a compiler difference either. stable/9: 1.594u 0.142s 0:01.73 100.0%5+2767k 0+0io 0pf+0w 1.586u 0.157s 0:01.74 99.4% 5+2792k 0+0io 0pf+0w 1.567u 0.156s 0:01.72 99.4% 5+2786k 0+0io 0pf+0w 1.583u 0.211s 0:01.79 100.0%5+2783k 0+0io 0pf+0w 1.599u 0.126s 0:01.72 99.4% 5+2786k 0+0io 0pf+0w head: 12.748u 0.165s 0:12.91 99.9%5+167k 0+0io 0pf+0w 12.755u 0.157s 0:12.91 99.9%5+167k 0+0io 0pf+0w 12.778u 0.157s 0:12.94 99.8%5+167k 0+0io 0pf+0w 12.657u 0.204s 0:12.86 99.9%5+167k 0+0io 0pf+0w 12.733u 0.173s 0:12.98 99.3%5+167k 0+0io 0pf+0w Neither system is entirely quiescent, but they're both largely idle. The CPU's are from the same model series, though the -CURRENT box is 500MHz faster, and has faster memory. So, yeah, it's a safe bet that non-_PRODUCTION malloc() can hurt you quite a lot when you're malloc()-heavy. -- Matthew Fuller (MF4839) | fulle...@over-yonder.net Systems/Network Administrator | http://www.over-yonder.net/~fullermd/ On the Internet, nobody can hear you scream. ___ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org
Re: svn commit: r242102 - in head: contrib/bmake usr.bin/bmake
I aim to import bmake-20121109 soon (once I sort out svn 1.7.7 bug), will loook at this as well. On Wed, 14 Nov 2012 19:27:49 +, Chris Rees writes: I was kicking the tires on this and noticed bmake is dynamically linked. Can you change it to being statically linked? ___ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org
Re: svn commit: r242102 - in head: contrib/bmake usr.bin/bmake
On Thu, Oct 25, 2012 at 11:18:06PM +, Simon J. Gerraty wrote: Log: Merge bmake-20121010 Hi Simon, I was kicking the tires on this and noticed bmake is dynamically linked. Can you change it to being statically linked? This issue most recently came up in freebsd-current. See thread pieces http://lists.freebsd.org/pipermail/freebsd-current/2012-April/033460.html http://lists.freebsd.org/pipermail/freebsd-current/2012-April/033472.html http://lists.freebsd.org/pipermail/freebsd-current/2012-April/033473.html -- -- David (obr...@freebsd.org) ___ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org
Re: svn commit: r242102 - in head: contrib/bmake usr.bin/bmake
On Wed, Nov 14, 2012 at 09:28:23AM -0800, David O'Brien wrote: On Thu, Oct 25, 2012 at 11:18:06PM +, Simon J. Gerraty wrote: Log: Merge bmake-20121010 Hi Simon, I was kicking the tires on this and noticed bmake is dynamically linked. Can you change it to being statically linked? This issue most recently came up in freebsd-current. See thread pieces http://lists.freebsd.org/pipermail/freebsd-current/2012-April/033460.html http://lists.freebsd.org/pipermail/freebsd-current/2012-April/033472.html http://lists.freebsd.org/pipermail/freebsd-current/2012-April/033473.html As you see, I prefer to not introduce new statically linked binaries into base. If, for unfortunate turns of events, bmake is changed to be statically linked, please obey WITH_SHARED_TOOLCHAIN. pgpb9jNngLb8L.pgp Description: PGP signature
Re: svn commit: r242102 - in head: contrib/bmake usr.bin/bmake
On 14 Nov 2012 18:49, Konstantin Belousov kostik...@gmail.com wrote: On Wed, Nov 14, 2012 at 09:28:23AM -0800, David O'Brien wrote: On Thu, Oct 25, 2012 at 11:18:06PM +, Simon J. Gerraty wrote: Log: Merge bmake-20121010 Hi Simon, I was kicking the tires on this and noticed bmake is dynamically linked. Can you change it to being statically linked? This issue most recently came up in freebsd-current. See thread pieces http://lists.freebsd.org/pipermail/freebsd-current/2012-April/033460.html http://lists.freebsd.org/pipermail/freebsd-current/2012-April/033472.html http://lists.freebsd.org/pipermail/freebsd-current/2012-April/033473.html As you see, I prefer to not introduce new statically linked binaries into base. If, for unfortunate turns of events, bmake is changed to be statically linked, please obey WITH_SHARED_TOOLCHAIN. Or a /rescue/bmake for when speed is a concern would also be acceptable. Chris ___ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org
Re: svn commit: r242102 - in head: contrib/bmake usr.bin/bmake
On Wed, 14 Nov 2012, Chris Rees wrote:
On 14 Nov 2012 18:49, Konstantin Belousov kostik...@gmail.com wrote:
On Wed, Nov 14, 2012 at 09:28:23AM -0800, David O'Brien wrote:
On Thu, Oct 25, 2012 at 11:18:06PM +, Simon J. Gerraty wrote:
Log:
Merge bmake-20121010
Hi Simon,
I was kicking the tires on this and noticed bmake is dynamically linked.
Can you change it to being statically linked?
This issue most recently came up in freebsd-current. See thread pieces
http://lists.freebsd.org/pipermail/freebsd-current/2012-April/033460.html
http://lists.freebsd.org/pipermail/freebsd-current/2012-April/033472.html
http://lists.freebsd.org/pipermail/freebsd-current/2012-April/033473.html
As you see, I prefer to not introduce new statically linked binaries into
base.
If, for unfortunate turns of events, bmake is changed to be statically
linked,
please obey WITH_SHARED_TOOLCHAIN.
Or a /rescue/bmake for when speed is a concern would also be acceptable.
Yes, the big rescue executable is probably even better than dynamic linkage
for pessimizing speeds. Sizes on freefall now:
%text data bss dec hex filename
% 130265 19889992 142245 22ba5 /bin/sh
% 5256762133964 2220464 7611190 742336 /rescue/sh
% -r--r--r-- 1 root wheel 3738610 Nov 11 06:48 /usr/lib/libc.a
The dynamically-linked /bin/sh is deceptively small, although it is larger
than the statically linked /bin/sh in FreeBSD-1 for few new features.
When executed, it expands to 16.5MB with 10MB RSS. I don't know how much
of that is malloc bloat that wouldn't need to be copied on fork, but it
is a lot just to map. /rescue/sh starts at 5MB and expands to 15.5MB with
9.25MB when executed. So it is slightly smaller, and its slowness is
determined by its non-locality. Perhaps its non-locality is not as good
for pessimization as libc's.
I don't use dynamic linkage of course. /bin/sh is bloated by static
linkage (or rather libc) in the FreeBSD-~5.2 that I usually run:
textdata bss dec hex filename
6496238192 64056 721871 b03cf /bin/sh
but this only expands to 864K with 580K RSS when executed. This can be
forked a little faster than 10MB RSS. In practice the timings for
time whatever/sh -c 'for i in $(jot 1000 1); do echo -n; done'
are:
freefall /bin/sh:6.93 real 1.69 user 5.16 sys
freefall /rescue/sh: 6.86 real 1.65 user 5.13 sys
local/bin/sh:0.21 real 0.01 user 0.18 sys
freefall:
FreeBSD 10.0-CURRENT #4 r242881M: Sun Nov 11 05:30:05 UTC 2012
r...@freefall.freebsd.org:/usr/obj/usr/src/sys/FREEFALL amd64
CPU: Intel(R) Xeon(R) CPU X5650 @ 2.67GHz (2666.82-MHz K8-class CPU)
Origin = GenuineIntel Id = 0x206c2 Family = 0x6 Model = 0x2c Stepping =
2
local:
FreeBSD 5.2-CURRENT #4395: Sun Apr 8 12:15:03 EST 2012
b...@besplex.bde.org:/c/obj/usr/src/sys/compile/BESPLEX.fw
...
CPU: AMD Athlon(tm) 64 Processor 3200+ (2010.05-MHz 686-class CPU)
Origin = AuthenticAMD Id = 0xf48 Stepping = 8
freefall may be pessimized by INVARIANTS. It is pessimized by /bin/echo
being dynamically linked. Normally shells use builtin echo so the speed
of /bin/echo is unimportant. There is also some strangeness in the timing
for /bin/echo specifically. Changing 'echo -n' to
'/bin/rm -f /etc/nonesuch' or /usr/bin/true reduces the times on freefall
by almost a factor of 2, although rm is larger and has to do more:
freefall:
textdata bss dec hex filename
2661 540 83209 c89 /bin/echo
11026 884 152 120622f1e /bin/rm
1420 484 81912 778 /usr/bin/true
(all dynamically linked to libc only. truss verifies that rm does a little
more).
freefall /bin/shecho: 6.93 real 1.69 user 5.16 sys
freefall /bin/shrm: 3.83 real 0.91 user 2.84 sys
freefall /bin/shtrue: 3.68 real 0.75 user 2.85 sys
freefall /rescue/sh echo: 6.86 real 1.65 user 5.13 sys
freefall /rescue/sh rm: 3.69 real 0.83 user 2.78 sys
freefall /rescue/sh true: 3.67 real 0.85 user 2.74 sys
local/bin/shecho: 0.21 real 0.01 user 0.18 sys
local/bin/shrm: 0.22 real 0.02 user 0.19 sys
local/bin/shtrue: 0.18 real 0.01 user 0.17 sys
local:
textdata bss dec hex filename
11926 60 768 1275431d2 /bin/echo
3807586752 61772 449282 6db02 /bin/rm
1639 40 6042283 8eb /usr/bin/true
(all statically linked. I managed to debloat crtso and libc enough for
/usr/bin/true to be small. The sources for /bin/echo are excessively
optimized for space in the executable -- they have contortions to avoid
using printf. But this is useless in -current, since crtso and libc
drag in printf, so that the null program int main(){} has size:
freefall (amd64):
textdata bss dec hex filename
316370 12156 55184 383710 5dade null-static
1452 484 81944
svn commit: r242102 - in head: contrib/bmake usr.bin/bmake
Author: sjg
Date: Thu Oct 25 23:18:05 2012
New Revision: 242102
URL: http://svn.freebsd.org/changeset/base/242102
Log:
Merge bmake-20121010
Approved by: marcel (mentor)
Modified:
head/contrib/bmake/ChangeLog
head/contrib/bmake/Makefile.in
head/contrib/bmake/bmake.1
head/contrib/bmake/bmake.cat1
head/contrib/bmake/bsd.after-import.mk
head/contrib/bmake/compat.c
head/contrib/bmake/make.1
head/usr.bin/bmake/Makefile
Directory Properties:
head/contrib/bmake/ (props changed)
Modified: head/contrib/bmake/ChangeLog
==
--- head/contrib/bmake/ChangeLogThu Oct 25 22:54:29 2012
(r242101)
+++ head/contrib/bmake/ChangeLogThu Oct 25 23:18:05 2012
(r242102)
@@ -1,3 +1,27 @@
+2012-10-10 Simon J. Gerraty s...@bad.crufty.net
+
+ * Makefile.in (MAKE_VERSION): 20121010
+ o protect syntax that only bmake parses correctly.
+ o remove auto setting of FORCE_MACHINE, use configure's
+ --with-force-machine=whatever if that is desired.
+
+2012-10-08 Simon J. Gerraty s...@bad.crufty.net
+
+ * Makefile.in: do not lose history from make.1 when generating bmake.1
+
+2012-10-07 Simon J. Gerraty s...@bad.crufty.net
+
+ * Makefile.in (MAKE_VERSION): 20121007
+ Merge with NetBSD make, pick up
+ o compat.c: ignore empty commands - same as jobs mode.
+ o make.1: document meta chars that cause use of shell
+
+2012-09-11 Simon J. Gerraty s...@bad.crufty.net
+
+ * Makefile.in (MAKE_VERSION): bump version to 20120911
+ * bsd.after-import.mk: include Makefile.inc early and allow it to
+ override PROG
+
2012-08-31 Simon J. Gerraty s...@bad.crufty.net
* Makefile.in (MAKE_VERSION): bump version to 20120831
Modified: head/contrib/bmake/Makefile.in
==
--- head/contrib/bmake/Makefile.in Thu Oct 25 22:54:29 2012
(r242101)
+++ head/contrib/bmake/Makefile.in Thu Oct 25 23:18:05 2012
(r242102)
@@ -1,7 +1,7 @@
# $NetBSD: Makefile,v 1.56 2012/05/30 21:54:23 sjg Exp $
# @(#)Makefile5.2 (Berkeley) 12/28/90
-# $Id: Makefile.in,v 1.170 2012/08/31 06:46:22 sjg Exp $
+# $Id: Makefile.in,v 1.174 2012/10/10 18:46:24 sjg Exp $
PROG= bmake
SRCS= arch.c buf.c compat.c cond.c dir.c for.c hash.c job.c main.c \
@@ -21,7 +21,7 @@ srcdir= @srcdir@
CC?= @CC@
# Base version on src date
-MAKE_VERSION= 20120831
+MAKE_VERSION= 20121010
MACHINE=@machine@
MACHINE_ARCH=@machine_arch@
DEFAULT_SYS_PATH = @default_sys_path@
@@ -71,10 +71,9 @@ SUBDIR= PSD.doc
.endif
.endif
+.if defined(.PARSEDIR)
+# we cannot rely on anything but bmake to parse this correctly.
.if empty(isBSD44:M${OS})
-# XXX not sure if we still want this given that configure
-# lets us force or not the definition of MACHINE.
-CFLAGS_main.o+= -DFORCE_MACHINE=\${MACHINE}\
MANTARGET=cat
INSTALL?=${srcdir}/install-sh
.if (${MACHINE} == sun386)
@@ -85,7 +84,7 @@ SRCS+= sigcompat.c
CFLAGS+= -DSIGNAL_FLAGS=SA_RESTART
.endif
.endif
-.if defined(.PARSEDIR)
+
.if make(obj) || make(clean)
SUBDIR+= unit-tests
.endif
@@ -104,14 +103,18 @@ EXTRACT_MAN=no
MAN=${PROG}.1
.if (${PROG} != make)
-${MAN}:make.1
- @echo making ${PROG}.1
- @sed -e 's/^.Nx/NetBSD/' -e '/^.Nm/s/make/${PROG}/' -e '/^.Sh
HISTORY/,$$d' ${srcdir}/make.1 $@
- @(echo .Sh HISTORY; \
- echo .Nm; \
+my.history: ${MAKEFILE}
+ @(echo .Nm; \
echo is derived from NetBSD; \
echo .Xr make 1 .; \
- echo It uses autoconf to facilitate portability to other platforms.)
$@
+ echo It uses autoconf to facilitate portability to other platforms.; \
+ echo .Pp) $@
+
+${MAN}:make.1 my.history
+ @echo making ${PROG}.1
+ @sed -e 's/^.Nx/NetBSD/' -e '/^.Nm/s/make/${PROG}/' \
+ -e '/^.Sh HISTORY/rmy.history' \
+ -e '/^.Sh HISTORY/,$$s,^.Nm,make,' ${.CURDIR}/make.1 $@
.endif
Modified: head/contrib/bmake/bmake.1
==
--- head/contrib/bmake/bmake.1 Thu Oct 25 22:54:29 2012(r242101)
+++ head/contrib/bmake/bmake.1 Thu Oct 25 23:18:05 2012(r242102)
@@ -1,4 +1,4 @@
-.\$NetBSD: make.1,v 1.206 2012/08/30 22:35:37 wiz Exp $
+.\$NetBSD: make.1,v 1.209 2012/10/08 15:09:48 christos Exp $
.\
.\ Copyright (c) 1990, 1993
.\The Regents of the University of California. All rights reserved.
@@ -29,7 +29,7 @@
.\
.\from: @(#)make.18.4 (Berkeley) 3/19/94
.\
-.Dd August 30, 2012
+.Dd October 8, 2012
.Dt MAKE 1
.Os
.Sh NAME
@@ -2042,6 +2042,13 @@ or
To be compatible with Makefiles that do this, one can use
.Fl B
to disable this behavior.
+.Pp
+In compatibility mode, each command is run in a separate process.
+If the command contains any shell meta
