Re: "make delete-old" performance.
On Thu, May 17, 2012 at 02:13:40PM +0200, Dimitry Andric wrote:
> On 2012-05-17 05:18, b. f. wrote:...
> > The slowdown is probably due - at least in part - to two factors:
> > - the list of files to be checked for removal has grown substantially,
> > because missing entries for old knobs and new entries for new knobs
> > have been added; and
> > - a new (and slower) method of checking was added in:
> > http://svnweb.FreeBSD.org/base?view=revision&revision=220255
> > because the old method broke down with the size of the new lists of files.
> Hm, maybe it would have been better to fix make, so it can accept
> arbitrarily long lists, without segfaulting? There's such a thing as
> malloc(), and I simply don't believe any of those lists could be larger
> than a few hundred kilobytes.
Alternatively, make could be fixed so that the original code works.
Although an invocation like
sh -c 'for file in VERY_LONG_LIST; do something; done'
will bump into {ARG_MAX}, the shell itself does not have a fixed
limitation so longer command lines can be written to a temporary file
and passed to sh that way.
In some cases (such as with -j), make always uses a temporary file,
slowing things down and obscuring ps output.
At the cost of needing the temporary file named a bit longer, it is
better to pass the pathname to sh rather than feeding the script on
standard input: this avoids interfering with terminal input and is
potentially faster.
The code currently in Makefile.inc1 can probably be sped up by passing
the output of the make -V command to something like
xargs sh -c 'for file do rm -i "${DESTDIR}/${file}"; done' sh
instead of the xargs -n1 | while read file; do ...; done loop.
(Note the second "sh" at the end, which serves as a value for $0 so all
strings from xargs become positional parameters.)
--
Jilles Tjoelker
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: "make delete-old" performance.
On 2012-05-17 05:18, b. f. wrote:... > The slowdown is probably due - at least in part - to two factors: > > - the list of files to be checked for removal has grown substantially, > because missing entries for old knobs and new entries for new knobs > have been added; and > > - a new (and slower) method of checking was added in: > http://svnweb.FreeBSD.org/base?view=revision&revision=220255 > because the old method broke down with the size of the new lists of files. Hm, maybe it would have been better to fix make, so it can accept arbitrarily long lists, without segfaulting? There's such a thing as malloc(), and I simply don't believe any of those lists could be larger than a few hundred kilobytes. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: "make delete-old" performance.
> I recently ran "make delete-old" on a -current box and felt it was > rather slow. That prompted me to do some more careful experiments. > > On one box where I have both 8-stable and 9-stable available, there > was a ~30x slowdown (based on 5 runs, ignoring the first). I don't > have a -current world on that box so I can't directly compare but on > another pair of fairly similar boxes, I get a ~180x slowdown between > 8-stable and -current (and that figure is probably optimistic since > the -current box was idle whereas the 8-stable box was fairly busy). > > I realise that "make delete-old" isn't something you nede to do every > day but going from sub-second to multi-minute duration is quite > noticable. Can anyone suggest what has caused the change? The slowdown is probably due - at least in part - to two factors: - the list of files to be checked for removal has grown substantially, because missing entries for old knobs and new entries for new knobs have been added; and - a new (and slower) method of checking was added in: http://svnweb.FreeBSD.org/base?view=revision&revision=220255 because the old method broke down with the size of the new lists of files. Some changes could be made to lessen the affect of the latter. b. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: "make delete-old" performance.
On 2012-May-16 18:11:32 -0700, Devin Teske wrote: >Right now, I believe the most useful comparison between systems is >(assuming UFS is in play) the output of "tunefs -p" for the >filesystem that the slowness is appearing on. These systems all run ZFS and apart from the first run, there doesn't seem to be any disk activity at all. It looks like the kernel is the bottleneck. >SoftUpdates (and whether it's enabled or disabled) can play a huge >difference in how fast file-deletions are. I've already successfully run "make delete-old" so there are no actual file deletions. This is all just looking for files that aren't present. -- Peter Jeremy pgpI6smYwen8A.pgp Description: PGP signature
Re: "make delete-old" performance.
On May 16, 2012, at 4:06 PM, Peter Jeremy wrote: > I recently ran "make delete-old" on a -current box and felt it was > rather slow. That prompted me to do some more careful experiments. > > On one box where I have both 8-stable and 9-stable available, there > was a ~30x slowdown (based on 5 runs, ignoring the first). I don't > have a -current world on that box so I can't directly compare but on > another pair of fairly similar boxes, I get a ~180x slowdown between > 8-stable and -current (and that figure is probably optimistic since > the -current box was idle whereas the 8-stable box was fairly busy). > > I realise that "make delete-old" isn't something you nede to do every > day but going from sub-second to multi-minute duration is quite > noticable. Can anyone suggest what has caused the change? > > -- > Peter Jeremy Right now, I believe the most useful comparison between systems is (assuming UFS is in play) the output of "tunefs -p" for the filesystem that the slowness is appearing on. SoftUpdates (and whether it's enabled or disabled) can play a huge difference in how fast file-deletions are. Also note that SU+J may be interesting if-set in 9 (while not available in 8). -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
"make delete-old" performance.
I recently ran "make delete-old" on a -current box and felt it was rather slow. That prompted me to do some more careful experiments. On one box where I have both 8-stable and 9-stable available, there was a ~30x slowdown (based on 5 runs, ignoring the first). I don't have a -current world on that box so I can't directly compare but on another pair of fairly similar boxes, I get a ~180x slowdown between 8-stable and -current (and that figure is probably optimistic since the -current box was idle whereas the 8-stable box was fairly busy). I realise that "make delete-old" isn't something you nede to do every day but going from sub-second to multi-minute duration is quite noticable. Can anyone suggest what has caused the change? -- Peter Jeremy pgpedlZA6ISMi.pgp Description: PGP signature
