Re: [gentoo-user] system lag with gentoo-sources-2.6.35-r2
I am the same problems with 2.6.35, now I downgrade to 2.6.34-r6, it is normal till now 2010/8/23 Alan Warren > Thanks Mark, I'll look into that config option, and try again with top > open. > > In this case I was doing a home backup to a 1TB WD Caviar black formatted > as ext3. > > I also have a raid0 with 2 other non-WD sata drives, and a single WD > velociraptor I can test > with. > > It also doesn't sound too far off that FF could be the culprit (mentioned > above), as I have it > open all the time, and so far it's been the first place I've noticed these > hiccups. That could > be coincidence though, as I've pretty much always got it open and these > hiccups are > system wide. > > > On Mon, Aug 23, 2010 at 10:03 AM, Mark Knecht wrote: > >> On Sun, Aug 22, 2010 at 12:01 PM, Alan Warren >> wrote: >> > Hello, >> > >> > I am having some system performance issues with this kernel release. I >> have >> > a SMP machine (dual xeon nehalem 8 core / 16 threads) with 24gb non-ecc >> > memory. >> > >> > On occasion (seems random so far) my system feels like a Pentium II >> trying >> > to cope with Vista. For example, I was in the middle of tar'ing a >> semi-large >> > file and noticed all of my apps came to a crawl. Scrolling in firefox, >> > typing in the terminal, or trying to navigate in my file manager >> resulted in >> > breif "pauses" that came in waves. On one occasion my system froze >> > completely and I had to manually reset the machine. (that was with >> > 2.6.35-r1) >> > >> > I didn't activate anything "new" in this kernel release that I don't >> > normally activate. ie, no cpuidle driver >> > >> > Is there a proper venu for debugging such matters, or should I just wait >> for >> > this kernel to go prime-time? >> > >> > Thanks for your time, >> > Alan >> > >> >> Hi Alan, >> Sorry for the problems. I've seen them also in the recent past. In >> my case it was on new hardware so I couldn't say it was due to a >> specific kernel release. >> >> 1) What happens when you watch top while doing the tar? Do you by any >> chance see large wait times in top? (Hit '1' to watch all CPUs) If so >> the problem could well be how the kernel is dealing with writing data >> back to the hard drive. I had this problem with the WD Green drives. >> When I changed to WD RAID Edition drives (1/2 the storage for 30% more >> money) the problems disappeared. >> >> 2) If it's not the drive issue then there is a kernel option called (I >> think) RCU_CPU_STALL_DETECTION (or something like that. If you turn it >> on I may generate a trace of what's keeping a core busy to long. >> Mileage will vary. >> >> Good luck, >> Mark >> >> >
Re: [gentoo-user] system lag with gentoo-sources-2.6.35-r2
Thanks Mark, I'll look into that config option, and try again with top open. In this case I was doing a home backup to a 1TB WD Caviar black formatted as ext3. I also have a raid0 with 2 other non-WD sata drives, and a single WD velociraptor I can test with. It also doesn't sound too far off that FF could be the culprit (mentioned above), as I have it open all the time, and so far it's been the first place I've noticed these hiccups. That could be coincidence though, as I've pretty much always got it open and these hiccups are system wide. On Mon, Aug 23, 2010 at 10:03 AM, Mark Knecht wrote: > On Sun, Aug 22, 2010 at 12:01 PM, Alan Warren > wrote: > > Hello, > > > > I am having some system performance issues with this kernel release. I > have > > a SMP machine (dual xeon nehalem 8 core / 16 threads) with 24gb non-ecc > > memory. > > > > On occasion (seems random so far) my system feels like a Pentium II > trying > > to cope with Vista. For example, I was in the middle of tar'ing a > semi-large > > file and noticed all of my apps came to a crawl. Scrolling in firefox, > > typing in the terminal, or trying to navigate in my file manager resulted > in > > breif "pauses" that came in waves. On one occasion my system froze > > completely and I had to manually reset the machine. (that was with > > 2.6.35-r1) > > > > I didn't activate anything "new" in this kernel release that I don't > > normally activate. ie, no cpuidle driver > > > > Is there a proper venu for debugging such matters, or should I just wait > for > > this kernel to go prime-time? > > > > Thanks for your time, > > Alan > > > > Hi Alan, > Sorry for the problems. I've seen them also in the recent past. In > my case it was on new hardware so I couldn't say it was due to a > specific kernel release. > > 1) What happens when you watch top while doing the tar? Do you by any > chance see large wait times in top? (Hit '1' to watch all CPUs) If so > the problem could well be how the kernel is dealing with writing data > back to the hard drive. I had this problem with the WD Green drives. > When I changed to WD RAID Edition drives (1/2 the storage for 30% more > money) the problems disappeared. > > 2) If it's not the drive issue then there is a kernel option called (I > think) RCU_CPU_STALL_DETECTION (or something like that. If you turn it > on I may generate a trace of what's keeping a core busy to long. > Mileage will vary. > > Good luck, > Mark > >
Re: [gentoo-user] system lag with gentoo-sources-2.6.35-r2
On Sun, Aug 22, 2010 at 12:01 PM, Alan Warren wrote: > Hello, > > I am having some system performance issues with this kernel release. I have > a SMP machine (dual xeon nehalem 8 core / 16 threads) with 24gb non-ecc > memory. > > On occasion (seems random so far) my system feels like a Pentium II trying > to cope with Vista. For example, I was in the middle of tar'ing a semi-large > file and noticed all of my apps came to a crawl. Scrolling in firefox, > typing in the terminal, or trying to navigate in my file manager resulted in > breif "pauses" that came in waves. On one occasion my system froze > completely and I had to manually reset the machine. (that was with > 2.6.35-r1) > > I didn't activate anything "new" in this kernel release that I don't > normally activate. ie, no cpuidle driver > > Is there a proper venu for debugging such matters, or should I just wait for > this kernel to go prime-time? > > Thanks for your time, > Alan > Hi Alan, Sorry for the problems. I've seen them also in the recent past. In my case it was on new hardware so I couldn't say it was due to a specific kernel release. 1) What happens when you watch top while doing the tar? Do you by any chance see large wait times in top? (Hit '1' to watch all CPUs) If so the problem could well be how the kernel is dealing with writing data back to the hard drive. I had this problem with the WD Green drives. When I changed to WD RAID Edition drives (1/2 the storage for 30% more money) the problems disappeared. 2) If it's not the drive issue then there is a kernel option called (I think) RCU_CPU_STALL_DETECTION (or something like that. If you turn it on I may generate a trace of what's keeping a core busy to long. Mileage will vary. Good luck, Mark
Re: [gentoo-user] system lag with gentoo-sources-2.6.35-r2
On Sun, Aug 22, 2010 at 2:01 PM, Alan Warren wrote: > Hello, > > I am having some system performance issues with this kernel release. I have > a SMP machine (dual xeon nehalem 8 core / 16 threads) with 24gb non-ecc > memory. > > On occasion (seems random so far) my system feels like a Pentium II trying > to cope with Vista. For example, I was in the middle of tar'ing a semi-large > file and noticed all of my apps came to a crawl. Scrolling in firefox, > typing in the terminal, or trying to navigate in my file manager resulted in > breif "pauses" that came in waves. On one occasion my system froze > completely and I had to manually reset the machine. (that was with > 2.6.35-r1) > > I didn't activate anything "new" in this kernel release that I don't > normally activate. ie, no cpuidle driver Sounds exactly like the same problem I was having. I posted about it recently on this list. Downgrading to the latest 2.6.34 made everything work normally again. > Is there a proper venu for debugging such matters, or should I just wait for > this kernel to go prime-time? Well, 2.6.35 is already prime-time as far as it has been released and is not RC... The proper way to debug would be to do a git bisect of the kernel.org sources until you find the exact patch that broke things. I haven't had time to attempt such a feat yet, so I'm just waiting patiently for someone else to figure it out instead. :)
Re: [gentoo-user] system lag with gentoo-sources-2.6.35-r2
On Sunday 22 August 2010 21:04:56 Robert Bridge wrote: > Well, the fix is in the line for 2.6.36 IIRC, so wouldn't be in an > 2.6.35 kernel. > > That said, the problem supposedly being fixed goes back well before > 2.6.34, so if that kernel works, it suggests that it is a different > issue you are hitting. Alan's problem sounds like the one I've been having through several kernel versions - the way I describe the symptom is that the system "lurches" through its work. It's an i5 box with 4 GB simple RAM (4 cores, 8 threads). I did mention this here a few weeks ago, but when I discovered that killing BOINC, or reducing its allowed CPU load, seemed to cure it I took no further action. > However... If there was a FF update, that could be triggering the bug, > as FF3.5+ are pretty stinky for I/O levels. You may be onto something there. My genlop history of firefox goes back to January, when 3.5.6 was installed together with sys-kernel/gentoo- sources-2.6.32-r1. I don't remember when I first noticed the lurching though. -- Rgds Peter. Linux Counter 5290, 1994-04-23.
Re: [gentoo-user] system lag with gentoo-sources-2.6.35-r2
On Sun, Aug 22, 2010 at 8:35 PM, Alan Warren wrote: > I think you are correct though. It does seem to only happen while the system > is under heavy I/O. > > I've never experienced anything like this in previous versions of the linux > kernel, > and resorting back to gentoo-sources-2.6.34 fixes the issue completely. > > If there are I/O fixes upstream, then I am assuming you are referring to a > cut > that is more recent then gentoo-sources-2.6.35-r2 that the Gentoo devs have > yet to provide their patches to? ( I see vanilla sources has 2.6.35 .3) Well, the fix is in the line for 2.6.36 IIRC, so wouldn't be in an 2.6.35 kernel. That said, the problem supposedly being fixed goes back well before 2.6.34, so if that kernel works, it suggests that it is a different issue you are hitting. However... If there was a FF update, that could be triggering the bug, as FF3.5+ are pretty stinky for I/O levels.
Re: [gentoo-user] system lag with gentoo-sources-2.6.35-r2
Thanks, I'm not very savvy when it comes to working with the kernel beyond using the normal stable cut gentoo provides. I'll research git-bisect and see if I can't figure this out though. I think you are correct though. It does seem to only happen while the system is under heavy I/O. I've never experienced anything like this in previous versions of the linux kernel, and resorting back to gentoo-sources-2.6.34 fixes the issue completely. If there are I/O fixes upstream, then I am assuming you are referring to a cut that is more recent then gentoo-sources-2.6.35-r2 that the Gentoo devs have yet to provide their patches to? ( I see vanilla sources has 2.6.35 .3) Thanks again, Alan On Sun, Aug 22, 2010 at 2:14 PM, Robert Bridge wrote: > On Sun, Aug 22, 2010 at 8:01 PM, Alan Warren > wrote: > > Is there a proper venu for debugging such matters, or should I just wait > for > > this kernel to go prime-time? > > Can you reliably reproduce the problem? If so, and you have a kernel > that works git-bisect should allow you to pinpoint the offending > commit. > > By the sounds of it though, this could be related to the problems > Linux has when under heavy I/O, in which case you best bet would be to > look to the upstream git as there are supposed to be fixes in it. > > Cheers, > RobbieAB. > >
Re: [gentoo-user] system lag with gentoo-sources-2.6.35-r2
On Sun, Aug 22, 2010 at 8:01 PM, Alan Warren wrote: > Is there a proper venu for debugging such matters, or should I just wait for > this kernel to go prime-time? Can you reliably reproduce the problem? If so, and you have a kernel that works git-bisect should allow you to pinpoint the offending commit. By the sounds of it though, this could be related to the problems Linux has when under heavy I/O, in which case you best bet would be to look to the upstream git as there are supposed to be fixes in it. Cheers, RobbieAB.
[gentoo-user] system lag with gentoo-sources-2.6.35-r2
Hello, I am having some system performance issues with this kernel release. I have a SMP machine (dual xeon nehalem 8 core / 16 threads) with 24gb non-ecc memory. On occasion (seems random so far) my system feels like a Pentium II trying to cope with Vista. For example, I was in the middle of tar'ing a semi-large file and noticed all of my apps came to a crawl. Scrolling in firefox, typing in the terminal, or trying to navigate in my file manager resulted in breif "pauses" that came in waves. On one occasion my system froze completely and I had to manually reset the machine. (that was with 2.6.35-r1) I didn't activate anything "new" in this kernel release that I don't normally activate. ie, no cpuidle driver Is there a proper venu for debugging such matters, or should I just wait for this kernel to go prime-time? Thanks for your time, Alan
