Re: pdflush produces 99% CPU usage with aufs2 on some conditions
Dear J.R. the reason is a patch of the gentoo-patchset. Even the vanilla-sources-2.29.3 work nicely. I created a bugreport at gentoo.org: http://bugs.gentoo.org/show_bug.cgi?id=270805 Thanks again for your time. Cheers Alex -- Alexander Krause Amtmann-Kästner-Platz 11 99091 Erfurt Mail alexander.kra...@ed-solutions.de Tel. +49 (361) 2156472 Mob. +49 (179) 4725826 -- Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT is a gathering of tech-side developers brand creativity professionals. Meet the minds behind Google Creative Lab, Visual Complexity, Processing, iPhoneDevCamp asthey present alongside digital heavyweights like Barbarian Group, R/GA, Big Spaceship. http://www.creativitycat.com
Re: pdflush produces 99% CPU usage with aufs2 on some conditions
Alexander Krause: the reason is a patch of the gentoo-patchset. Even the vanilla-sources-2.29.3 work nicely. I created a bugreport at gentoo.org: http://bugs.gentoo.org/show_bug.cgi?id=3D270805 Thanks again for your time. No problem. Glad to find the cause other than aufs. :-) J. R. Okajima -- Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT is a gathering of tech-side developers brand creativity professionals. Meet the minds behind Google Creative Lab, Visual Complexity, Processing, iPhoneDevCamp asthey present alongside digital heavyweights like Barbarian Group, R/GA, Big Spaceship. http://www.creativitycat.com
Re: pdflush produces 99% CPU usage with aufs2 on some conditions
Dear J.R., Let's focus ext3/4. Bad case is kernel-genkernel-x86-2.6.28-tuxonice-r1. Then what is your good kernel version? And where can I get gentoo kernel source files? i compiled the aufs2-2.6 tree this morning, that is linux-2.6.30-rc5 and tested my problem. Well... and it works ;-) So it seems to be something with the kernel sources. I'll try the latest gentoo-sources and will drop you a message about the state. Sorry for wasting your time. Cheers Alex -- Alexander Krause Amtmann-Kästner-Platz 11 99091 Erfurt Mail alexander.kra...@ed-solutions.de Tel. +49 (361) 2156472 Mob. +49 (179) 4725826 -- Crystal Reports - New Free Runtime and 30 Day Trial Check out the new simplified licensing option that enables unlimited royalty-free distribution of the report engine for externally facing server and web deployment. http://p.sf.net/sfu/businessobjects
Re: pdflush produces 99% CPU usage with aufs2 on some conditions
On Tue May 19 2009, Alexander Krause wrote: Hi, Alexander Krause: What I menat is something like this. # mkdir /mnt/aufs/overlay # mount /dev/sdb2 /mnt/aufs/overlay # while true; do find /mnt/aufs/overlay/ -ls /dev/null; done # seq 100 | while read i; do yes | dd of=3D/mnt/aufs/overlay/tmp/$i count=$i; # while true; do find /mnt/aufs/overlay/ -ls /dev/null; done # while true; do seq 100 | while read i; do yes | dd of=3D/mnt/aufs/overlay/tmp/$i; done I got both lines running for over 5 minutes - no problem. The issue is still there with ext3 but ext2 works fine. Ok, that may be important. It's weird that this only happens with kde4 and only with ext3/4. I'll try to find the program/service which causes the problem and will run an strace or something. Try something with a big executable memory footprint (which kde4 has) - - There is an on-going discussion on LKML about how/why under some conditions executable pages of memory are getting evicted from the page cache while they are still in use (which just causes them to be re-loaded). That problem has the same type of symptom - works fine for awhile (while pages are in the page cache) - and then everything slows way down. The problem reported on LKML can be seen when copying very large files. Mike Cheers Alex -- Crystal Reports - New Free Runtime and 30 Day Trial Check out the new simplified licensing option that enables unlimited royalty-free distribution of the report engine for externally facing server and web deployment. http://p.sf.net/sfu/businessobjects
Re: pdflush produces 99% CPU usage with aufs2 on some conditions
Hi, Try something with a big executable memory footprint (which kde4 has) - - There is an on-going discussion on LKML about how/why under some conditions executable pages of memory are getting evicted from the page cache while they are still in use (which just causes them to be re-loaded). That problem has the same type of symptom - works fine for awhile (while pages are in the page cache) - and then everything slows way down. The problem reported on LKML can be seen when copying very large files. I can not confirm this. I just copied a 1gig file without problems and even running openoffice works. But as soon as i start /usr/kde/4.2/bin/kdeinit4 my load goes to 100% cos of pdflush. Btw, I'm very sory for the wrong info J.R. i testet my old system with aufs1 (aufs 20090126) on 2.6.28 and it does the same stuff with pdflush. Cheers, Alex -- Alexander Krause Amtmann-Kästner-Platz 11 99091 Erfurt Mail alexander.kra...@ed-solutions.de Tel. +49 (361) 2156472 Mob. +49 (179) 4725826 -- Crystal Reports - New Free Runtime and 30 Day Trial Check out the new simplified licensing option that enables unlimited royalty-free distribution of the report engine for externally facing server and web deployment. http://p.sf.net/sfu/businessobjects
Re: pdflush produces 99% CPU usage with aufs2 on some conditions
Alexander Krause: Btw, I'm very sory for the wrong info J.R. i testet my old system with aufs= 1=20 (aufs 20090126) on 2.6.28 and it does the same stuff with pdflush. So your bad case is, - linux-2.6.28 and 29 - last aufs1 and latest aufs2 - kde3.5 and kde4 - ext3 and ext4 Can you clarify the good case? I am afraid I need to check gentoo kernel source. If you fix your good and bad case, then tell me where can I get gentoo kernel source files. J. R. Okajima -- Crystal Reports - New Free Runtime and 30 Day Trial Check out the new simplified licensing option that enables unlimited royalty-free distribution of the report engine for externally facing server and web deployment. http://p.sf.net/sfu/businessobjects
Re: pdflush produces 99% CPU usage with aufs2 on some conditions
Hi, === Bad case === == linux-2.6.28 and 29 == * kernel-genkernel-x86-2.6.29-gentoo-r1 (current one with aufs 2- standalone.tree-29-20090511) but i got this problem since months with the first versions of aufs2 and even 2.6.28 . * kernel-genkernel-x86-2.6.28-tuxonice-r1 (aufs 20090126, tuxonice sources include the gentoo patchset) == aufs == * aufs 20090126 * aufs 2-standalone.tree-29-20090511 == kde == * only kde4, about 1 minute after running /usr/kde/4.2/bin/kdeinit4 == overlay fs == * ext3 or * ext4 === Good case === == linux-2.6.28 and 29 == * i can only test 2.6.29 since this is my current system == aufs == * aufs 2-standalone.tree-29-20090511 == kde == * anything which does not start kdeinit4 == fs == * ext2 or * no additional overlay partition (only squashfs+tmpfs) == additional tests == * other apps are working, that includes: xorg,openoffice,firefox,fluxbox * copying big files is not a problem (i even got a warning that my disk is full, i deleted some files and still got no problems) * your script does not raise the pdflush load I am afraid I need to check gentoo kernel source. If you fix your good and bad case, then tell me where can I get gentoo kernel source files. To be honest, i tried the last weeks before posting to this maillist. It's not really a problem for me but if anyone wants to use kde4 from a live-cd with aufs they'll get this annoying issue. I don't use special CFLAGS or anything, my root was supposed to work everywhere on x86. It consists of grub, my kernel and a 890MB squash file. I can provide you with anything you want, kernel config and so on. I can even upload the files to a server, so you can test it yourself with a usb-memory stick and something which can boot it. Cheers Alex -- Alexander Krause Amtmann-Kästner-Platz 11 99091 Erfurt Mail alexander.kra...@ed-solutions.de Tel. +49 (361) 2156472 Mob. +49 (179) 4725826 -- Crystal Reports - New Free Runtime and 30 Day Trial Check out the new simplified licensing option that enables unlimited royalty-free distribution of the report engine for externally facing server and web deployment. http://p.sf.net/sfu/businessobjects
Re: pdflush produces 99% CPU usage with aufs2 on some conditions
Hello Alexander, Alexander Krause: mount /dev/sda1 /boot #create aufs branch mount -t tmpfs tmpfs /aufs_dynamic mount -t aufs -o br:/aufs_dynamic/=3Drw none /aufs_vertex #add overlay partition mkdir -p /aufs_vertex/mnt/aufs/overlay mount /dev/sda2 /aufs_vertex/mnt/aufs/overlay -o rw mount -o remount,append: /aufs_vertex/mnt/aufs/overlay=3Drw /aufs_vertex #add loop-image mkdir -p /aufs_vertex/mnt/aufs/images/10-desktop.squash/ mount /boot/images/10-desktop.squash /aufs_vertex/mnt/aufs/images/10- desktop.squash/ -o loop mount -o remount,append:/aufs_vertex/mnt/aufs/images/10-desktop.squash/=3Dr= r = /aufs_vertex So you have 3 branches in your /aufs_vertex. - /aufs_dynamic, rw, tmpfs - /aufs_vertex/mnt/aufs/overlay, rw, ?? - /aufs_vertex/mnt/aufs/images/10-desktop.squash, rr, squashfs If you can, please try again without /aufs_vertex/mnt/aufs/overlay, eg. /aufs_vertex = /aufs_dynamic(rw,tmpfs) + /.../10-desktop.squash(rr,squashfs) And run several commands and any version of kde. With these 2 branches, pdflush will not eat your cpu, I hope. And then, - mount /aufs_vertex/mnt/aufs/overlay - write several things to it directly, eg. no aufs - find /aufs_vertex/mnt/aufs/overlay -ls /dev/null repeatedly in background may be good - and see what will happen to pdflush. In /aufs_vertex/mnt/aufs/overlay standalone, pdflush MAY eat your cpu up. I just want you to make sure. But when i start any kde4 application my cpu becomes busy and top shows pdf= lush = with about 99% usage. I've no idea how to debug this. Kde 3.5 works fine an= d = anything else which does not use kde4 works too. So your good case is - linux kernel 2.6.2?? - aufs1, last version - kde3.5 and kde4 and bad case is, - linux-2.6.29 - aufs2 - kde4 Am I right? J. R. Okajima -- Crystal Reports - New Free Runtime and 30 Day Trial Check out the new simplified licensing option that enables unlimited royalty-free distribution of the report engine for externally facing server and web deployment. http://p.sf.net/sfu/businessobjects
Re: pdflush produces 99% CPU usage with aufs2 on some conditions
Hi J.R. thanks for your fast response. So you have 3 branches in your /aufs_vertex. - /aufs_dynamic, rw, tmpfs - /aufs_vertex/mnt/aufs/overlay, rw, ?? It's an EXT4 . - /aufs_vertex/mnt/aufs/images/10-desktop.squash, rr, squashfs If you can, please try again without /aufs_vertex/mnt/aufs/overlay, eg. /aufs_vertex = /aufs_dynamic(rw,tmpfs) + /.../10-desktop.squash(rr,squashfs) Did that and everything works perfect. So the problem comes with the overlay. And run several commands and any version of kde. With these 2 branches, pdflush will not eat your cpu, I hope. Yes, you're right. And then, - mount /aufs_vertex/mnt/aufs/overlay Well, it's kinda difficult to do it that way, cos the creation and mounting of /aufs_vertex is done via /initrc and after the boot It chrootes me into /aufs_vertex . I'll do the mounts etc. from within the aufs_vertex which looks like this: mkdir /mnt/aufs/overlay mount /dev/sdb2 /mnt/aufs/overlay My dmesg sais: EXT4-fs barriers enabled kjournald2 starting: pid 5939, dev sdb2:8, commit interval 5 seconds EXT4 FS on sdb2, internal journal on sdb2:8 EXT4-fs: delayed allocation enabled EXT4-fs: file extents enabled EXT4-fs: mballoc enabled EXT4-fs: mounted filesystem sdb2 with ordered data mode - write several things to it directly, eg. no aufs Created some files without problems. - find /aufs_vertex/mnt/aufs/overlay -ls /dev/null repeatedly in background may be good - and see what will happen to pdflush. while true; do find /mnt/aufs/overlay/ -ls /dev/null; done I got about 50% system usage in top but not pdflush or anything. In /aufs_vertex/mnt/aufs/overlay standalone, pdflush MAY eat your cpu up. I just want you to make sure. Sorry, I don't understand what that should do. # ln /mnt/aufs/overlay standalone ... hard link not allowed for directory.. # ln -fs ,,, works, but no pdflush load. # cd /mnt/aufs/overlay; ln -fs foo /root works, but no pdflush load. So your good case is - linux kernel 2.6.2?? 2.6.28 - aufs1, last version the stable one from sf - kde3.5 and kde4 Works both. and bad case is, - linux-2.6.29 - aufs2 - kde4 Am I right? Correct. FS for /mnt/aufs/overlay was ext4 for both cases. I try an ext3 overlay now, just to make sure. Cheers Alex -- Alexander Krause Amtmann-Kästner-Platz 11 99091 Erfurt Mail alexander.kra...@ed-solutions.de Tel. +49 (361) 2156472 Mob. +49 (179) 4725826 -- Crystal Reports - New Free Runtime and 30 Day Trial Check out the new simplified licensing option that enables unlimited royalty-free distribution of the report engine for externally facing server and web deployment. http://p.sf.net/sfu/businessobjects
Re: pdflush produces 99% CPU usage with aufs2 on some conditions
Correct. FS for /mnt/aufs/overlay was ext4 for both cases. I try an ext3 overlay now, just to make sure. Same problem with ext3. -- Alexander Krause Amtmann-Kästner-Platz 11 99091 Erfurt Mail alexander.kra...@ed-solutions.de Tel. +49 (361) 2156472 Mob. +49 (179) 4725826 -- Crystal Reports - New Free Runtime and 30 Day Trial Check out the new simplified licensing option that enables unlimited royalty-free distribution of the report engine for externally facing server and web deployment. http://p.sf.net/sfu/businessobjects
Re: pdflush produces 99% CPU usage with aufs2 on some conditions
Hi, Alexander Krause: If you can, please try again without /aufs_vertex/mnt/aufs/overlay, eg. /aufs_vertex =3D /aufs_dynamic(rw,tmpfs) + /.../10-desktop.squash(rr,squashfs) Did that and everything works perfect. So the problem comes with the overla= y. And run several commands and any version of kde. With these 2 branches, pdflush will not eat your cpu, I hope. Yes, you're right. OK. Sorry, I don't understand what that should do. What I menat is something like this. # mkdir /mnt/aufs/overlay # mount /dev/sdb2 /mnt/aufs/overlay # while true; do find /mnt/aufs/overlay/ -ls /dev/null; done # seq 100 | while read i; do yes | dd of=/mnt/aufs/overlay/tmp/$i count=$i; done # killall find - aufs1, last version the stable one from sf Just to make sure, please try dmesg | grep aufs. Correct. FS for /mnt/aufs/overlay was ext4 for both cases. I try an ext3=20 overlay now, just to make sure. Please try EXT2. J. R. Okajima -- Crystal Reports - New Free Runtime and 30 Day Trial Check out the new simplified licensing option that enables unlimited royalty-free distribution of the report engine for externally facing server and web deployment. http://p.sf.net/sfu/businessobjects
