Playing with include-what-you-use shows interesting stuff
Hi, I've been playing with the include-what-you-use[1] llvm tool for some on my personnal projects, as it works very well, I have also played with it on our source tree starting with the bin directory. It shows some interesting results, while the default output is quite aggressive, I just chose to remove the useless headers in each sources. It show some interesting results which seems to come from maybe bad includes in some of our headers. Apparently some of the #include sys/param.h are false positive because others headers shouldn't include it for example (according to des) here is a diff showing what I find that can be removed: http://people.freebsd.org/~bapt/include-what-you-use.diff I think it shouldn't be applied as it but be more analyzed. regards, Bapt pgpVqC7bBkTJA.pgp Description: PGP signature
iso2flash img
I'm doing some work for a client and I need to get a copy of ubuntu to demo for them. I can only use a usb disk (no cdrom), and unfortunately the guys at ubuntu are too lazy to create a flash img themselves and expect you to do it yourself. Of course the tools to do this are only available on Winblows and Mac (hdiutils? I think). They have instructions on the site - which of course are useless to me being a FreeBSD freak ;) I googled a bit and found an old post here from Luigi (http://osdir.com/ml/freebsd-hackers/2008-11/msg00245.html) which had a script to do this, but I'm having trouble with it- is anyone familiar with this? I'm on a bit of a deadline... and yes, I did install syslinux port. Output: ../iso2flash ubuntu-11.10-desktop-i386.iso usage: realpath [-q] path [...] type tree /usr/home/user/Downloads/ubuntu-11.10-desktop-i386.iso image Extract files from /usr/home/user/Downloads/ubuntu-11.10-desktop-i386.iso into /usr/home/user/Downloads/ubuntu-11.10-desktop-i386.iso.tree chmod: /usr/home/user/Downloads/ubuntu-11.10-desktop-i386.iso.tree: No such file or directory total 6 drwxr-xr-x 2 user user 512 Mar 21 18:22 . drwxr-xr-x 6 user user 2560 Mar 21 18:22 .. total 2500 drwxr-xr-x 11 user user 512 Mar 21 18:23 . drwxr-xr-x 6 user user 2560 Mar 21 18:22 .. drwx-- 2 user user 512 Oct 13 01:15 .disk -r 1 user user 225 Oct 13 01:15 README.diskdefines -r 1 user user 143 Oct 13 01:15 autorun.inf drwx-- 3 user user 512 Oct 13 01:15 boot drwx-- 2 user user 512 Oct 13 01:15 casper drwx-- 3 user user 512 Oct 13 01:15 dists drwx-- 2 user user 512 Oct 13 01:15 install drwx-- 2 user user 2560 Oct 13 01:15 isolinux -r 1 user user 4418 Oct 13 01:15 md5sum.txt drwx-- 2 user user 512 Oct 13 01:15 pics drwx-- 4 user user 512 Oct 13 01:15 pool drwx-- 2 user user 512 Oct 13 01:15 preseed -r 1 user user 2491552 Oct 13 01:15 wubi.exe image size is 711676 kb guess type 8+0 records in 8+0 records out 1048576 bytes transferred in 0.005349 secs (196035057 bytes/sec) newfs_msdos: /dev/711676: No such file or directory syslinux: invalid media signature (not a FAT filesystem?) moving boot code mv: rename /usr/home/user/Downloads/ubuntu-11.10-desktop-i386.iso.tree/isolinux.cfg to /usr/home/user/Downloads/ubuntu-11.10-desktop-i386.iso.tree/syslinux.cfg: No such file or directory moving files... init :: non DOS media Cannot initialize '::' Bad target ::/ init :: non DOS media Cannot initialize '::' I'm going to keep on investigating, I tried this on 8.2 and 9.0 with no luck. I intend to attempt a usb install of FreeBSD at some point (on the todo list for some time now), but at this point its still a bit of a mystery; so I think I may have some trouble understanding this atm. A little instruction would go a long way if someone could provide some pointers. Cheers ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: iso2flash img
Da Rock wrote: I googled a bit and found an old post here from Luigi (http://osdir.com/ml/freebsd-hackers/2008-11/msg00245.html) which had a script to do this, but I'm having trouble with it- is anyone familiar with this? I'm on a bit of a deadline... Can't help you with that script (I failed to make it work too), but you might want to try to dd the iso image directly onto USB instead; there where talks that Ubuntu would support this starting at 11.10. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: iso2flash img
Hello, I've forwarded your question to a colleague who is an Ubuntu fan and he replied, that the creation of a booting Ubuntu USB key is possile with Ubuntu tools/methods only, i.e. booting the Ubuntu live CD and following the ways to install it onto the key (and not on disk). He wrote some years ago a small Wiki page about, it is in German but maybe you can clue something from the commands there: http://mikiwiki.org/wiki/Ubuntu_8.10_Intrepid_Ibex/Installation_2009.04.07_usbstick HIH matthias -- Matthias Apitz e g...@unixarea.de - w http://www.unixarea.de/ UNIX since V7 on PDP-11, UNIX on mainframe since ESER 1055 (IBM /370) UNIX on x86 since SVR4.2 UnixWare 2.1.2, FreeBSD since 2.2.5 ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: iso2flash img
On 03/21/12 22:55, Vitaly Magerya wrote: Da Rock wrote: I googled a bit and found an old post here from Luigi (http://osdir.com/ml/freebsd-hackers/2008-11/msg00245.html) which had a script to do this, but I'm having trouble with it- is anyone familiar with this? I'm on a bit of a deadline... Can't help you with that script (I failed to make it work too), but you might want to try to dd the iso image directly onto USB instead; there where talks that Ubuntu would support this starting at 11.10. Interesting. I'll have to look further into how that would work. Also, why wouldn't they just tell Mac users to do that? In the meantime I think I may have stumbled on the solution to the script: In the midst of all the output it mentions usage realpath [-q] path. I wasn't 100% sure exactly what that meant, but I put the full path to the iso and a full path to an img file and I *think* that worked. I've yet to test the result; and I have no idea of the '-q' option ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: iso2flash img
On 03/21/12 23:06, Matthias Apitz wrote: Hello, I've forwarded your question to a colleague who is an Ubuntu fan and he replied, that the creation of a booting Ubuntu USB key is possile with Ubuntu tools/methods only, i.e. booting the Ubuntu live CD and following the ways to install it onto the key (and not on disk). He wrote some years ago a small Wiki page about, it is in German but maybe you can clue something from the commands there: http://mikiwiki.org/wiki/Ubuntu_8.10_Intrepid_Ibex/Installation_2009.04.07_usbstick I'll have to translate that, but I am trying to get a 'live' usb disk to demo on the clients cdrom less unit. I know the cd is live, I assumed I could get a live usb disk from that based on their instructions. For a supposedly user friendly system, obtaining install media is not.. :/ Maybe a little too much debian in the system ;) ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: iso2flash img
On 03/21/12 23:11, Da Rock wrote: On 03/21/12 22:55, Vitaly Magerya wrote: Da Rock wrote: I googled a bit and found an old post here from Luigi (http://osdir.com/ml/freebsd-hackers/2008-11/msg00245.html) which had a script to do this, but I'm having trouble with it- is anyone familiar with this? I'm on a bit of a deadline... Can't help you with that script (I failed to make it work too), but you might want to try to dd the iso image directly onto USB instead; there where talks that Ubuntu would support this starting at 11.10. Interesting. I'll have to look further into how that would work. Also, why wouldn't they just tell Mac users to do that? In the meantime I think I may have stumbled on the solution to the script: In the midst of all the output it mentions usage realpath [-q] path. I wasn't 100% sure exactly what that meant, but I put the full path to the iso and a full path to an img file and I *think* that worked. I've yet to test the result; and I have no idea of the '-q' option Nada. Booted it, and all I got was a blinking cursor, and I couldn't even type hello world! :( Back to square one... I guess I'll try a direct copy. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: iso2flash img
As an alternative I recently purchased a Zalman ZM-VE200 device (there's also a USB3.0 flavor) that lets you copy ISOs to it and it will emulate a CDROM/DVDROM/BDROM for you so you never have to deal with this mess again. It works amazingly well. I was tired of fighting this problem and this is an amazing solution -- I can keep every ISO I ever need on a single drive. http://www.zalman.com/eng/product/Product_Read.asp?idx=431 http://www.zalman.com/eng/product/Product_Read.asp?idx=459 http://www.rmprepusb.com/tutorials/ve200 Hope that helps someone ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Approaching the limit on PV entries
On Tuesday, March 20, 2012 11:37:57 am Sergey Kandaurov wrote: On 22 November 2011 19:29, Mark Saad nones...@longcount.org wrote: Hello All [found this mail in my drafts, not sure if my answer is still useful] I want to get to the bottom of a warning in dmesg. On 7.2-RELEASE and 7.3-RELEASE I have seen the following warning in dmesg. Approaching the limit on PV entries, consider increasing either the vm.pmap.shpgperproc or the vm.pmap.pv_entry_max sysctl. So looking around I see a few posts here and there about how to tune the sysctls to address the warning however I am not 100% sure what each value does. It appears changing vm.pmap.shpgperproc affects the value of vm.pmap.pv_entry_max . Can someone explain the relationship of the two sysctls. Also This is how they are calculated. pv_entry_max = shpgperproc * maxproc + cnt.v_page_count; and, respectively, shpgperproc = (pv_entry_max - cnt.v_page_count) / maxproc; So, changing one sysctl will change another and vice versa. what pitfalls of changing them are. Not known to me (on amd64 platform). I have had vm.pmap.shpgperproc=15000 on 8.1 amd64 with 4G RAM to make some badly written commercial software to work until it was decommissioned to the scrap. FYI, Alan just removed this warning and the associated sysctls from HEAD yesterday because they were made obsolete several years ago. I think they are obsolete even on 7. Certainly on 8. -- John Baldwin ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Approaching the limit on PV entries
On 21 March 2012 19:19, John Baldwin j...@freebsd.org wrote: On Tuesday, March 20, 2012 11:37:57 am Sergey Kandaurov wrote: On 22 November 2011 19:29, Mark Saad nones...@longcount.org wrote: Hello All [found this mail in my drafts, not sure if my answer is still useful] I want to get to the bottom of a warning in dmesg. On 7.2-RELEASE and 7.3-RELEASE I have seen the following warning in dmesg. Approaching the limit on PV entries, consider increasing either the vm.pmap.shpgperproc or the vm.pmap.pv_entry_max sysctl. So looking around I see a few posts here and there about how to tune the sysctls to address the warning however I am not 100% sure what each value does. It appears changing vm.pmap.shpgperproc affects the value of vm.pmap.pv_entry_max . Can someone explain the relationship of the two sysctls. Also This is how they are calculated. pv_entry_max = shpgperproc * maxproc + cnt.v_page_count; and, respectively, shpgperproc = (pv_entry_max - cnt.v_page_count) / maxproc; So, changing one sysctl will change another and vice versa. what pitfalls of changing them are. Not known to me (on amd64 platform). I have had vm.pmap.shpgperproc=15000 on 8.1 amd64 with 4G RAM to make some badly written commercial software to work until it was decommissioned to the scrap. FYI, Alan just removed this warning and the associated sysctls from HEAD yesterday because they were made obsolete several years ago. I think they are obsolete even on 7. Certainly on 8. Yep, and since switching to direct map (somewhere around 7.x on amd64?) made PV entry limit factually obsolete, this is really cool. -- wbr, pluknet ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: iso2flash img
On 21 March 2012 15:52, Mark Felder f...@feld.me wrote: As an alternative I recently purchased a Zalman ZM-VE200 device (there's also a USB3.0 flavor) that lets you copy ISOs to it and it will emulate a CDROM/DVDROM/BDROM for you so you never have to deal with this mess again. It works amazingly well. I was tired of fighting this problem and this is an amazing solution -- I can keep every ISO I ever need on a single drive. http://www.zalman.com/eng/product/Product_Read.asp?idx=431 http://www.zalman.com/eng/product/Product_Read.asp?idx=459 http://www.rmprepusb.com/tutorials/ve200 Hope that helps someone Have to say that I have one too and have had no problems with it yet :) ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Approaching the limit on PV entries
On Wed, Mar 21, 2012 at 12:39 PM, Sergey Kandaurov pluk...@gmail.com wrote: On 21 March 2012 19:19, John Baldwin j...@freebsd.org wrote: On Tuesday, March 20, 2012 11:37:57 am Sergey Kandaurov wrote: On 22 November 2011 19:29, Mark Saad nones...@longcount.org wrote: Hello All [found this mail in my drafts, not sure if my answer is still useful] I want to get to the bottom of a warning in dmesg. On 7.2-RELEASE and 7.3-RELEASE I have seen the following warning in dmesg. Approaching the limit on PV entries, consider increasing either the vm.pmap.shpgperproc or the vm.pmap.pv_entry_max sysctl. So looking around I see a few posts here and there about how to tune the sysctls to address the warning however I am not 100% sure what each value does. It appears changing vm.pmap.shpgperproc affects the value of vm.pmap.pv_entry_max . Can someone explain the relationship of the two sysctls. Also This is how they are calculated. pv_entry_max = shpgperproc * maxproc + cnt.v_page_count; and, respectively, shpgperproc = (pv_entry_max - cnt.v_page_count) / maxproc; So, changing one sysctl will change another and vice versa. what pitfalls of changing them are. Not known to me (on amd64 platform). I have had vm.pmap.shpgperproc=15000 on 8.1 amd64 with 4G RAM to make some badly written commercial software to work until it was decommissioned to the scrap. FYI, Alan just removed this warning and the associated sysctls from HEAD yesterday because they were made obsolete several years ago. I think they are obsolete even on 7. Certainly on 8. Yep, and since switching to direct map (somewhere around 7.x on amd64?) made PV entry limit factually obsolete, this is really cool. -- wbr, pluknet Interesting so this warning is relevant in 7.x ? -- mark saad | nones...@longcount.org ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: [vfs] buf_daemon() slows down write() severely on low-speed CPU
On Thu, Mar 15, 2012 at 08:00:41PM +0100, Svatopluk Kraus wrote:
2012/3/15 Konstantin Belousov kostik...@gmail.com:
On Tue, Mar 13, 2012 at 01:54:38PM +0100, Svatopluk Kraus wrote:
On Mon, Mar 12, 2012 at 7:19 PM, Konstantin Belousov
kostik...@gmail.com wrote:
On Mon, Mar 12, 2012 at 04:00:58PM +0100, Svatopluk Kraus wrote:
Hi,
I have solved a following problem. If a big file (according to
'hidirtybuffers') is being written, the write speed is very poor.
It's observed on system with elan 486 and 32MB RAM (i.e., low speed
CPU and not too much memory) running FreeBSD-9.
Analysis: A file is being written. All or almost all dirty buffers
belong to the file. The file vnode is almost all time locked by
writing process. The buf_daemon() can not flush any dirty buffer as a
chance to acquire the file vnode lock is very low. A number of dirty
buffers grows up very slow and with each new dirty buffer slower,
because buf_daemon() eats more and more CPU time by looping on dirty
buffers queue (with very low or no effect).
This slowing down effect is started by buf_daemon() itself, when
'numdirtybuffers' reaches 'lodirtybuffers' threshold and buf_daemon()
is waked up by own timeout. The timeout fires at 'hz' period, but
starts to fire at 'hz/10' immediately as buf_daemon() fails to reach
'lodirtybuffers' threshold. When 'numdirtybuffers' (now slowly)
reaches ((lodirtybuffers + hidirtybuffers) / 2) threshold, the
buf_daemon() can be waked up within bdwrite() too and it's much worse.
Finally and with very slow speed, the 'hidirtybuffers' or
'dirtybufthresh' is reached, the dirty buffers are flushed, and
everything starts from beginning...
Note that for some time, bufdaemon work is distributed among bufdaemon
thread itself and any thread that fails to allocate a buffer, esp.
a thread that owns vnode lock and covers long queue of dirty buffers.
However, the problem starts when numdirtybuffers reaches
lodirtybuffers count and ends around hidirtybuffers count. There are
still plenty of free buffers in system.
On the system, a buffer size is 512 bytes and the default
thresholds are following:
vfs.hidirtybuffers = 134
vfs.lodirtybuffers = 67
vfs.dirtybufthresh = 120
For example, a 2MB file is copied into flash disk in about 3
minutes and 15 second. If dirtybufthresh is set to 40, the copy time
is about 20 seconds.
My solution is a mix of three things:
1. Suppresion of buf_daemon() wakeup by setting bd_request to 1 in
the main buf_daemon() loop.
I cannot understand this. Please provide a patch that shows what do
you mean there.
curthread-td_pflags |= TDP_NORUNNINGBUF | TDP_BUFNEED;
mtx_lock(bdlock);
for (;;) {
- bd_request = 0;
+ bd_request = 1;
mtx_unlock(bdlock);
Is this a complete patch ? The change just causes lost wakeups for
bufdaemon,
nothing more.
Yes, it's a complete patch. And exactly, it causes lost wakeups which are:
1. !! UNREASONABLE !!, because bufdaemon is not sleeping,
2. not wanted, because it looks that it's correct behaviour for the
sleep with hz/10 period. However, if the sleep with hz/10 period is
expected to be waked up by bd_wakeup(), then bd_request should be set
to 0 just before sleep() call, and then bufdaemon behaviour will be
clear.
No, your description is wrong.
If bufdaemon is unable to flush enough buffers and numdirtybuffers still
greater then lodirtybuffers, then bufdaemon enters qsleep state
without resetting bd_request, with timeouts of one tens of second.
Your patch will cause all wakeups for this case to be lost. This is
exactly the situation when we want bufdaemon to run harder to avoid
possible deadlocks, not to slow down.
All stuff around bd_request and bufdaemon sleep is under bd_lock, so
if bd_request is 0 and bufdaemon is not sleeping, then all wakeups are
unreasonable! The patch is about that mainly.
Wakeups itself are very cheap for the running process. Mostly, it comes
down to locking sleepq and waking all threads that are present in the
sleepq blocked queue. If there is no threads in queue, nothing is done.
I read description of bd_request variable. However, bd_request should
serve as an indicator that buf_daemon() is in sleep. I.e., the
following paradigma should be used:
mtx_lock(bdlock);
bd_request = 0; /* now, it's only time when wakeup() will be meaningful
*/
sleep(bd_request, ..., hz/10);
bd_request = 1; /* in case of timeout, we must set it (bd_wakeup()
already set it) */
mtx_unlock(bdlock);
My patch follows the paradigma. What happens without the patch in
described problem: buf_daemon() fails in its job and goes to sleep
with hz/10 period. It supposes that next early wakeup will do nothing
too. bd_request is untouched but buf_daemon() doesn't know if
malloc pages map to user space
Hi, From within the freeBSD kernel, not all malloc are made equal: * malloc() smaller than KMEM_ZMAX (set to one page size) end up in UMA SLABs, themselves laid out in powers of 2 (from 16 bytes, 32... to 4096 bytes) * bigger malloc() are done through uma_large_malloc() which uses the kmem wired space In my driver, I need to map some malloc-ed memory, obtained from another module, into userspace. The problem: on the smaller mallocs, as well as on some bigeer ones (8k seems fine, 64k fails): vm_map_lookup() fails finding the underlying vm object. Do somebody know how (or better, have a piece of code!) to retrieve the vm_object associated with malloc-ed memory? (small and big ones) As far as I can see in the vm code, there isn't any object associated with the slabs (the smaller mallocs), it seems that a huge chunk of virtual space is used as is, so I presume the virtual addresses where the SLABs are have some remarkable property, with respect to physical addresses, that could allow creating an object from scratch? The usual answer is: use mmap(). It seems mmap() is the solution to everything. But what I dislike with mmap() is the following cost *for each page*: 1/ a page fault 2/ a call to a pager function that will do the on demand mapping. Thank you. Eric ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Improving gcore
Sometimes I have trouble capturing the correct state of a multithreaded process using gcore. That is, it looks like target process might have done some work since the time command was issued and the core file was generated. Looking at the code, gcore calls ptrace(PT_ATTACH...), which internally issues SIGSTOP, and calls waitpid() to wait until the process stops. So, it's quite possible that some threads that are not sleeping interruptibly will continue to run until the process notices the signal. Signals are only checked when a thread that is tagged to handle the signal crosses the user boundary (return from syscall, trap). When the thread finally handles SIGSTOP, it needs to stop all threads, which is done by lighting a flag-bit it each thread. This bit is checked as each thread crosses the user boundary. So, there will always be some state change in the target process from the time SIGSTOP is posted to the time all threads are actually stopped. I was wondering if I could improve this a bit by calling PT_SUSPEND on all threads, instead of posting SIGSTOP and waiting for all threads to stop. Once the core is generated, unsuspend all threads. As with SIGSTOP, individual thread will only notice suspension as they cross user boundary. But there is no overhead of tagging a thread to handle the signal and that thread doing the suspension. The idea is to try and generate the core file which reflects the running state of the process as closely as possible. Does this sound reasonable ? Thanks, Sushanth ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: iso2flash img
On 03/21/12 22:55, Vitaly Magerya wrote: Da Rock wrote: I googled a bit and found an old post here from Luigi (http://osdir.com/ml/freebsd-hackers/2008-11/msg00245.html) which had a script to do this, but I'm having trouble with it- is anyone familiar with this? I'm on a bit of a deadline... Can't help you with that script (I failed to make it work too), but you might want to try to dd the iso image directly onto USB instead; there where talks that Ubuntu would support this starting at 11.10. Nada. Tried that and it didn't work. I'm not sure how that would work given that it uses isolinux to boot- ergo needs a cd to load the kernel. Maybe some way to determine the install media? Bit of an unnecessary complication though if you ask me... just make an iso _and_ an img. I'm sure canonical are not that strapped for resources... ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
