freebsd-update's install_verify routine excessive stating
Hackers,
Recently I decided it would be time to upgrade an older laptop that
was still running 6.2-RELEASE to a more recent release 7.1-RELEASE (re-
install not possible, laptop has no cd-rom drive, and does not boot
from a USB one).
Everything went well, including the kernel update. It was not until
after I rebooted and ran:
./freebsd-update.sh -f freebsd-update.conf install
That I started noticing something rather weird. It had been running
for a good 30 minutes before I started wondering what was going on.
top gave me nothing of value, other than that all of the time was
spent running sh, and that it was spending all of its time in system,
not user where I would have expected it. Thinking went wrong I stopped
the process and started it again.
After a ktrace and kdump I found the culprit in install_verify in the
freebsd-update utility, it would read one line, and then run stat on
one of the many files that was listed in /usr/upgrade/files/.
install_verify () {
# Generate a list of hashes
cat $@ |
cut -f 2,7 -d '|' |
grep -E '^f' |
cut -f 2 -d '|' |
sort -u filelist
# Make sure all the hashes exist
while read HASH; do
if ! [ -f files/${HASH}.gz ]; then
echo -n Update files missing --
echo this should never happen.
echo Re-run '$0 fetch'.
return 1
fi
done filelist
# Clean up
rm filelist
}
running find /usr/upgrade/files/ | wc -l showed over 64000 files. So
what was happening here is that freebsd-update.sh is doing due
diligence in checking that all the files exist, however it uses the
built in test utility to do so. While in normal situations this may
not be that big of a deal since the changeset is likely to be small,
running stat on 64000 individual files in a loop is rather slow.
In my case I have just removed the offending code and hoped all went
well, however this is off course not an acceptable solution. I have
not yet come up with an acceptable way to fix the offending code,
hence my post to hackers. I am also not sure as to how I would file a
pr bug report for this situation, any guidance would be greatly
appreciated.
This email has become much longer than I had originally intended. I
just wanted to get this out to see what you all had to say about this.
Cheers,
Bert JW Regeer
Re: Kernel Module - GCC Requires memmove
On 2009-01-22 02:14, Nate Eldredge wrote:
I vaguely recall Linux having a policy that compiling the kernel without
optimization was not supported, possibly because of situations like this.
No, Linux has its own implementations of mem{cmp,cpy,move,set}, both in
fallback C versions, and optimized versions for several arches.
Compiling Linux without optimization will fail at the linking stage, due
to extern inline functions in header files, without implementation in
separate .c files.
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: freebsd-update's install_verify routine excessive stating
Hi,
I've Cc'ed Colin Percival, the author of freebsd-update.
Bert JW Regeer wrote:
Recently I decided it would be time to upgrade an older laptop that
was still running 6.2-RELEASE to a more recent release 7.1-RELEASE
[...]
Everything went well, including the kernel update. It was not until
after I rebooted and ran:
./freebsd-update.sh -f freebsd-update.conf install
That I started noticing something rather weird. It had been running
for a good 30 minutes before I started wondering what was going on.
top gave me nothing of value, other than that all of the time was
spent running sh, and that it was spending all of its time in system,
not user where I would have expected it. Thinking went wrong I stopped
the process and started it again.
After a ktrace and kdump I found the culprit in install_verify in the
freebsd-update utility, it would read one line, and then run stat on
one of the many files that was listed in /usr/upgrade/files/.
install_verify () {
# Generate a list of hashes
cat $@ |
That should be $@ with double quotes. $@ doesn't make
sense without the quotes. Apart from that, it's a typical
case of useless use of cat.
cut -f 2,7 -d '|' |
grep -E '^f' |
cut -f 2 -d '|' |
sort -u filelist
It's unclear why there are two cut commands. The 7th
field isn't used at all. Also, the -E option to grep
is superfluous here. Finally, using awk might be more
efficient than cut + grep.
So I would suggest to replace the whole pipe with this:
awk -F | '$2 ~ /^f/ {print $2}' $@ |
sort -u filelist
# Make sure all the hashes exist
while read HASH; do
if ! [ -f files/${HASH}.gz ]; then
echo -n Update files missing --
echo this should never happen.
echo Re-run '$0 fetch'.
return 1
fi
done filelist
# Clean up
rm filelist
}
running find /usr/upgrade/files/ | wc -l showed over 64000 files. So
what was happening here is that freebsd-update.sh is doing due
diligence in checking that all the files exist, however it uses the
built in test utility to do so. While in normal situations this may
not be that big of a deal since the changeset is likely to be small,
running stat on 64000 individual files in a loop is rather slow.
In my case I have just removed the offending code and hoped all went
well, however this is off course not an acceptable solution. I have
not yet come up with an acceptable way to fix the offending code,
hence my post to hackers. I am also not sure as to how I would file a
pr bug report for this situation, any guidance would be greatly
appreciated.
You are right, that loop doesn't scale very well. I'm not
surprised it is horribly slow with 64000 files on an old
laptop that probably has a disk that isn't too fast.
It would be much better to generate two lists:
- The list of hashes, as already done (filelist)
- A list of gzipped files present, stripped to the hash:
(cd files; echo *.gz) |
tr ' ' '\n' |
sed 's/\.gz$//' filespresent
Note we use echo instead of ls, in order to avoid the
kern.argmax limit. 64000 files would certainly exceed that
limit. Also note that the output is already sorted because
the shell sorts wildcard expansions.
Now that we have those two files, we can use comm(1) to
find out whether there are any hashes in filelist that are
not in filespresent:
if [ -n $(comm -23 filelist filespresent) ]; then
echo -n Update files missing --
...
fi
That solution scales much better because no shell loop is
required at all.
Best regards
Oliver
--
Oliver Fromme, secnetix GmbH Co. KG, Marktplatz 29, 85567 Grafing b. M.
Handelsregister: Registergericht Muenchen, HRA 74606, Geschäftsfuehrung:
secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün-
chen, HRB 125758, Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart
FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd
Life is short (You need Python)
-- Bruce Eckel, ANSI C++ Comitee member, author
of Thinking in C++ and Thinking in Java
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: freebsd-update's install_verify routine excessive stating
Oliver Fromme schrieb:
cut -f 2,7 -d '|' |
grep -E '^f' |
cut -f 2 -d '|' |
sort -u filelist
It's unclear why there are two cut commands. The 7th
field isn't used at all. Also, the -E option to grep
After the first cut the seventh field becomes the second:
echo 'a|b|c|d|e|f|g' | cut -f 2,7 -d '|'
So the second cut selects the original seventh field and fills it into
the file filelist.
So I would suggest to replace the whole pipe with this:
awk -F | '$2 ~ /^f/ {print $2}' $@ |
sort -u filelist
It should print $7, see above.
Christoph
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: device driver: cdesw questions?
on 21/01/2009 16:05 Robert Watson said the following: On Wed, 21 Jan 2009, Andriy Gapon wrote: I also would like the driver to provide a select capability quite similar to that of network (e.g. TCP) sockets using d_poll. I.e. a userland program should be able to query when it can write data to the device without blocking and when it can read data without blocking, plus when an error occurred in the device/driver, so there is no point in further waiting. At this moment I am thoroughly confused by meaning of various event masks described in poll(2). E.g. what is normal priority, non-zero priority and high priority. Which flags should I care about if all data is the same priority for me? Which flag(s) should I set when I'd like to communicate an error condition (e.g. similar to TCP connection reset)? I find that the description of the polling flags is at best confusing in both man pages and specifications. The best bet is to look at the existing TCP semantics, which are basically defined in sopoll_generic(): [snip] A few observations: - Neither POLLHUP nor POLLERR appear to be implemented for sockets (all protocols use sopoll_generic in practice). This is surprising given the wording in the poll(2) man page. - Make sure to distinguish POLLIN and POLLINIGNEOF -- the difference between soreadable() and the test in POLLIGNEOF is that POLLIN also considers SBS_CANTRCVMORE (i.e., at least half-close in the receive direction) but POLLIGNEOF doesn't. I think Kostik's pointer to the pipe_poll() code is a good one, but if you're looking to emulate TCP semantics a bit more exactly, these differences should be taken into account. Robert, Kostik, thank you for the great pointers. From previous network programming I recall that if an error occurs on a TCP socket then select(2) marks the socket as available for reading (and probably for writing too), then a subsequent operation gets actual error code. I think that maybe this was the only way to do it in select-only days. I am a little bit confused about exceptfds parameter of select, the manual page says The only exceptional condition detectable is out-of-band data received on a socket and I am not sure what that would be for TCP and how that could be used in practice. Anyway, I'd probably stick to the same strategy - mark the device as available for reading and writing if an error occurs and then return error code in actual read/write. -- Andriy Gapon ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: usb keyboard vs btx: an SMI theory
on 21/01/2009 21:07 John Baldwin said the following: On Tuesday 16 December 2008 8:16:44 am Andriy Gapon wrote: Again, I am very fuzzy about the exact details, but I think that this is something that could be happening and I think that SMI is of primary interest here. I also think that this might explain to a certain degree the difference in behavior between older btx and newer btx. One thing to keep in mind is that when an SMI# is delivered, the processor enters System Management Mode (SMM). In SMM, the CPU actually uses a different set of memory for its RAM. It also runs in a sort of weird 32-bit real mode. It is not going to call the stock IRQ 1 handler. Instead, it passes data back to normal mode by changing the values restored into registers when exiting SMM. Typically doing an I/O port access to the ports backing the keyboard (0x60 and 0x64) cause an SMI# and the SMM handler emulates the inb/outb request by storing the resulting data for an inb in the %ax register the normal mode sees once it resumes execution after the 'inb' instruction. I've been thinking about that and also decided that my SMI theory is rubbish. On the other hand, I still suspect that there could be some race in protected-real transition, but from looking at the code I can not imagine how it could happen. -- Andriy Gapon ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: freebsd-update's install_verify routine excessive stating
Christoph Mallon wrote:
Oliver Fromme schrieb:
cut -f 2,7 -d '|' |
grep -E '^f' |
cut -f 2 -d '|' |
sort -u filelist
It's unclear why there are two cut commands. The 7th
field isn't used at all. Also, the -E option to grep
After the first cut the seventh field becomes the second:
Ah yes, you're right, of course.
My caffeine level was insufficient, I guess. :-)
So I would suggest to replace the whole pipe with this:
awk -F | '$2 ~ /^f/ {print $2}' $@ |
sort -u filelist
It should print $7, see above.
Right. Thanks for pointing it out.
Best regards
Oliver
--
Oliver Fromme, secnetix GmbH Co. KG, Marktplatz 29, 85567 Grafing b. M.
Handelsregister: Registergericht Muenchen, HRA 74606, Geschäftsfuehrung:
secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün-
chen, HRB 125758, Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart
FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd
Whatever happened to the days when hacking started
at the cerebral cortex, and not at the keyboard?
-- Sid on userfriendly.org by Illiad, 2007-06-20
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: How to detach a foreign driver from a device so my driver can attach?
On Thursday 22 January 2009 2:15:15 am Andre Albsmeier wrote: On Wed, 21-Jan-2009 at 14:08:37 -0500, John Baldwin wrote: hostb as other devices (e.g. agp(4)) need to attach to host-pci bridges as well. Would this work in 6.x as well? You wrote in another mail that in 7.0 agp attaches to hostb. This makes me think that in 6.x things are handled differently. Yes, 6.x does not work this way. -- John Baldwin ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: freebsd-update's install_verify routine excessive stating
Bert JW Regeer xiste...@0x58.com writes: [problems with freebsd-update] You should probably send a copy of this to cperc...@... DES -- Dag-Erling Smørgrav - d...@des.no ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: freebsd-update's install_verify routine excessive stating
Hi.
It's interesting.
On Thu, 22 Jan 2009 13:17:41 +0100 (CET)
Oliver Fromme o...@lurza.secnetix.de wrote:
Hi,
So I would suggest to replace the whole pipe with this:
awk -F | '$2 ~ /^f/ {print $2}' $@ |
sort -u filelist
It would be much better to generate two lists:
- The list of hashes, as already done (filelist)
- A list of gzipped files present, stripped to the hash:
(cd files; echo *.gz) |
tr ' ' '\n' |
sed 's/\.gz$//' filespresent
Note we use echo instead of ls, in order to avoid the
kern.argmax limit. 64000 files would certainly exceed that
limit. Also note that the output is already sorted because
the shell sorts wildcard expansions.
Now that we have those two files, we can use comm(1) to
find out whether there are any hashes in filelist that are
not in filespresent:
if [ -n $(comm -23 filelist filespresent) ]; then
echo -n Update files missing --
...
fi
That solution scales much better because no shell loop is
required at all.
This will probably be the fastest.
awk -F | '
$2 ~ /^f/{required[$7]=$7; count++}
END{FS=[/.];
while(find files -name *.gz | getline0)
if($2 in required)
if(--count=0)
exit(0);
exit(count)}' $@
It verifies entries using hashtable instead of sort.
Therefore, it is O(n+m) instead of O(n*log(n))+O(m*log(m)) in theory.
I am not well aware how good awk's associate array is, though.
Regards,
Hiro
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
