Re: [gentoo-user] VPS has issues emerging everything
Hi all, thanks for all your help. I'll try to reply to everything, but first let me annouce the issue is now resolved and gentoo was not at fault. The virtual HD is physically on a raid (unknown config). Mark, the sector size issue you mention, does it have to do with aligning real HD sectors with filesystem sectors (so that stuff like read-ahead will get no-more-no-less than what the kernel wants)? I've read about this kind of setup when I was interested in RAID long ago... Now that I know my hd is actually on a raid, maybe i could benefit some I/O performance improvements by tuning this a bit! Anyway, I was told by the support team that another user on the same physical machine (remember it's a xen VPS) was doing I/O intensive stuff which could have I/O starved my system. I don't understand how starving or even doing some kind of DoS attack could lead to a complete freeze on the console, but eh... They offered to migrate my system to another physical machine, and after that... I was able to perform a complete 'emerge -e system' in one shot without a scratch, I even did it with --jobs=2 and MAKEOPTS=-j4. After that, I started a complete emerge --keep-going --jobs=2 world with MAKEOPTS=-j8... (i got 4 cores: dual xeon 2Ghz) This last emerge is still going on as I write this and is emerging pkg 522 of 620 !! And there were no build errors so far... It's emerging glibc at the moment, so once the big emerge is finished, I'll probably recompile all pkgs that depend on glibc. I believe glibc was actually updated during my very initial update on monday and I haven't come to do that... but I guess everything will go smoothly from here. Thanks again for all your help guys! Simon On Fri, Jan 7, 2011 at 5:02 PM, Jacob Todd jaketodd...@gmail.com wrote: What do you have MAKEOPTS set to?
Re: [gentoo-user] VPS has issues emerging everything
Glad you have a root cause/solution. On Sat, Jan 8, 2011 at 10:49 AM, Simon turne...@gmail.com wrote: SNIP The virtual HD is physically on a raid (unknown config). Mark, the sector size issue you mention, does it have to do with aligning real HD sectors with filesystem sectors (so that stuff like read-ahead will get no-more-no-less than what the kernel wants)? I've read about this kind of setup when I was interested in RAID long ago... Now that I know my hd is actually on a raid, maybe i could benefit some I/O performance improvements by tuning this a bit! As it's RAID underneath it's likely set up correctly. The issue I had in mind was the disk being a 4K/sector disk but the person who built the partition not knowing to align the partition to a 4K boundary. That can cause a _huge_ slowdown. I doubt that's the case here. As this is a hosting service they likely know what they are doing in that area, and if it wasn't done correctly you would have noticed it before I think. Anyway, I was told by the support team that another user on the same physical machine (remember it's a xen VPS) was doing I/O intensive stuff which could have I/O starved my system. I don't understand how starving or even doing some kind of DoS attack could lead to a complete freeze on the console, but eh... Makes sense actually. The other guy took all the disk I/O leaving you with none. If you can't get to the disk then you cannot read ebuilds or write compiled code, or at least not fast. They offered to migrate my system to another physical machine, and after that... I was able to perform a complete 'emerge -e system' in one shot without a scratch, I even did it with --jobs=2 and MAKEOPTS=-j4. After that, I started a complete emerge --keep-going --jobs=2 world with MAKEOPTS=-j8... (i got 4 cores: dual xeon 2Ghz) So now you're in good shape...until some user on the new system starts hogging all the disk I/O and holds you up again. This last emerge is still going on as I write this and is emerging pkg 522 of 620 !! And there were no build errors so far... It's emerging glibc at the moment, so once the big emerge is finished, I'll probably recompile all pkgs that depend on glibc. I believe glibc was actually updated during my very initial update on monday and I haven't come to do that... but I guess everything will go smoothly from here. Thanks again for all your help guys! Simon Good that you got to the root of the problem. Good luck, Mark
Re: [gentoo-user] VPS has issues emerging everything
Just a final word on this... The problem is effectively resolved... i was able to rebuild the system, then world with zero issues. I then ran revdep-rebuild, no issues and no broken links found, I then recompiled pkgs with deps against glibc and ran revdep-rebuild again. The whole thing ran at full capacity and with zero errors. I don't know if I felt as good as this when I found the root cause... I just know that having root again feels great! ;) Okay... and now let's upgrade the kernel... ;P Thanks again, Simon On Sat, Jan 8, 2011 at 3:16 PM, Mark Knecht markkne...@gmail.com wrote: Glad you have a root cause/solution. On Sat, Jan 8, 2011 at 10:49 AM, Simon turne...@gmail.com wrote: SNIP The virtual HD is physically on a raid (unknown config). Mark, the sector size issue you mention, does it have to do with aligning real HD sectors with filesystem sectors (so that stuff like read-ahead will get no-more-no-less than what the kernel wants)? I've read about this kind of setup when I was interested in RAID long ago... Now that I know my hd is actually on a raid, maybe i could benefit some I/O performance improvements by tuning this a bit! As it's RAID underneath it's likely set up correctly. The issue I had in mind was the disk being a 4K/sector disk but the person who built the partition not knowing to align the partition to a 4K boundary. That can cause a _huge_ slowdown. I doubt that's the case here. As this is a hosting service they likely know what they are doing in that area, and if it wasn't done correctly you would have noticed it before I think. Anyway, I was told by the support team that another user on the same physical machine (remember it's a xen VPS) was doing I/O intensive stuff which could have I/O starved my system. I don't understand how starving or even doing some kind of DoS attack could lead to a complete freeze on the console, but eh... Makes sense actually. The other guy took all the disk I/O leaving you with none. If you can't get to the disk then you cannot read ebuilds or write compiled code, or at least not fast. They offered to migrate my system to another physical machine, and after that... I was able to perform a complete 'emerge -e system' in one shot without a scratch, I even did it with --jobs=2 and MAKEOPTS=-j4. After that, I started a complete emerge --keep-going --jobs=2 world with MAKEOPTS=-j8... (i got 4 cores: dual xeon 2Ghz) So now you're in good shape...until some user on the new system starts hogging all the disk I/O and holds you up again. This last emerge is still going on as I write this and is emerging pkg 522 of 620 !! And there were no build errors so far... It's emerging glibc at the moment, so once the big emerge is finished, I'll probably recompile all pkgs that depend on glibc. I believe glibc was actually updated during my very initial update on monday and I haven't come to do that... but I guess everything will go smoothly from here. Thanks again for all your help guys! Simon Good that you got to the root of the problem. Good luck, Mark
[gentoo-user] VPS has issues emerging everything
Hi there, something went wrong during a previous update and now revdep-rebuild will kind-of freeze around 30%. When it freezes like that, the hosting company shows one of it's CPU core is used at 100% non-stop and the rest is idle. I have tried `emerge -e world` and with system, they all fail similarly. I managed to reconfigure it to have a working console and started the `emerge -e system` from the console. After a while it froze, but typed a ctrl-z to suspend the process, then %% to resume it and it unlocked it. It often hung like in between two phases of the emerge process (like after unpacking source, or just before doing the install, etc). I was able to resume the emerge like that a good dozen times until now, and now it's really locked, not responding at all... seems like all 4 cores are used at 100%. It hung while emerging perl and I've included the output on the console below so you can see exactly where it hung. I will now force a reboot on it and retry emerging just perl, I will reply to the list with my results of that. [...] make[1]: Entering directory `/mnt/tmp/portage/dev-lang/perl-5.12.2-r4/work/perl-5.12.2' sh writemain DynaLoader.o perlmain.c rm -f opmini.c cp op.c opmini.c rm -f perlmini.c cp perl.c perlmini.c echo av.c scope.c op.c doop.c doio.c dump.c gv.c hv.c mg.c reentr.c mro.c perl.c perly.c pp.c pp_hot.c pp_ctl.c pp_sys.c regcomp.c regexec.c utf8.c sv.c taint.c toke.c util.c deb.c run.c universal.c pad.c globals.c perlio.c perlapi.c numeric.c mathoms.c locale.c pp_pack.c pp_sort.c miniperlmain.c perlmain.c opmini.c perlmini.c | tr ' ' '\n' .clist make[1]: Leaving directory `/mnt/tmp/portage/dev-lang/perl-5.12.2-r4/work/perl-5.12.2' Finding dependencies for av.o. Finding dependencies for scope.o. Finding dependencies for op.o. Finding dependencies for doop.o. Finding dependencies for doio.o. Finding dependencies for dump.o. Finding dependencies for gv.o. Finding dependencies for hv.o. Finding dependencies for mg.o. Finding dependencies for reentr.o. Finding dependencies for mro.o. Finding dependencies for perl.o. Finding dependencies for perly.o. Finding dependencies for pp.o. Finding dependencies for pp_hot.o. Finding dependencies for pp_ctl.o. Finding dependencies for pp_sys.o. Finding dependencies for regcomp.o. Finding dependencies for regexec.o. Finding dependencies for utf8.o. Finding dependencies for sv.o. Finding dependencies for taint.o. Finding dependencies for toke.o. Finding dependencies for util.o. Finding dependencies for deb.o. Finding dependencies for run.o. Finding dependencies for universal.o. Finding dependencies for pad.o. Finding dependencies for globals.o. Finding dependencies for perlio.o. Finding dependencies for perlapi.o. Finding dependencies for numeric.o. Finding dependencies for mathoms.o. Finding dependencies for locale.o. Finding dependencies for pp_pack.o. Finding dependencies for pp_sort.o. Finding dependencies for miniperlmain.o. Finding dependencies for perlmain.o. Finding dependencies for opmini.o. Finding dependencies for perlmini.o.
Re: [gentoo-user] VPS has issues emerging everything
On Fri, Jan 7, 2011 at 7:56 AM, Simon turne...@gmail.com wrote: Hi there, something went wrong during a previous update and now revdep-rebuild will kind-of freeze around 30%. When it freezes like that, the hosting company shows one of it's CPU core is used at 100% non-stop and the rest is idle. I have tried `emerge -e world` and with system, they all fail similarly. I managed to reconfigure it to have a working console and started the `emerge -e system` from the console. After a while it froze, but typed a ctrl-z to suspend the process, then %% to resume it and it unlocked it. It often hung like in between two phases of the emerge process (like after unpacking source, or just before doing the install, etc). I was able to resume the emerge like that a good dozen times until now, and now it's really locked, not responding at all... seems like all 4 cores are used at 100%. It hung while emerging perl and I've included the output on the console below so you can see exactly where it hung. I will now force a reboot on it and retry emerging just perl, I will reply to the list with my results of that. Uhhgdoesn't sound good at all. If it's dieing on emerge -e system I'd be a little suspicious of a drive or file system failure of some type. Do you have access to smartctl to get to info on the drive? fsck possibly? Good luck, Mark
Re: [gentoo-user] VPS has issues emerging everything
Simon wrote: Hi there, something went wrong during a previous update and now revdep-rebuild will kind-of freeze around 30%. When it freezes like that, the hosting company shows one of it's CPU core is used at 100% non-stop and the rest is idle. I have tried `emerge -e world` and with system, they all fail similarly. I managed to reconfigure it to have a working console and started the `emerge -e system` from the console. After a while it froze, but typed a ctrl-z to suspend the process, then %% to resume it and it unlocked it. It often hung like in between two phases of the emerge process (like after unpacking source, or just before doing the install, etc). I was able to resume the emerge like that a good dozen times until now, and now it's really locked, not responding at all... seems like all 4 cores are used at 100%. It hung while emerging perl and I've included the output on the console below so you can see exactly where it hung. I will now force a reboot on it and retry emerging just perl, I will reply to the list with my results of that. SNIP Have you enabled python3 by any chance? eselect python list should show 2.6 as the active python. Just a thought. Someone else did this the other day and had troubles. Dale :-) :-)
Re: [gentoo-user] VPS has issues emerging everything
Ok, the fsck reported nothing wrong... I still got the same bug again... while doing `emerge -e @system`... It got stuck right after the Installing line below... Dale, I just checked my python version and it was already 2.6, i set it to 2.6 again, just in case, and continued my emerge (the ctrl-z + %% worked at this point). I'll reply here again with the next issue... Thanks, Simon Emerging (35 of 151) sys-devel/binutils-config-1.9-r4 * Package:sys-devel/binutils-config-1.9-r4 * Repository: gentoo * Maintainer: toolch...@gentoo.org * USE: elibc_glibc kernel_linux userland_GNU x86 Unpacking source... Source unpacked in /mnt/tmp/portage/sys-devel/binutils-config-1.9-r4/work Compiling source in /mnt/tmp/portage/sys-devel/binutils-config-1.9-r4/work ... Source compiled. Test phase [not enabled]: sys-devel/binutils-config-1.9-r4 Install binutils-config-1.9-r4 into /mnt/tmp/portage/sys-devel/binutils-config-1.9-r4/image/ category sys-devel Completed installing binutils-config-1.9-r4 into /mnt/tmp/portage/sys-devel/binutils-config-1.9-r4/image/ ecompressdir: bzip2 -9 /usr/share/man Installing (35 of 151) sys-devel/binutils-config-1.9-r4 \ On Fri, Jan 7, 2011 at 1:22 PM, Dale rdalek1...@gmail.com wrote: Simon wrote: Hi there, something went wrong during a previous update and now revdep-rebuild will kind-of freeze around 30%. When it freezes like that, the hosting company shows one of it's CPU core is used at 100% non-stop and the rest is idle. I have tried `emerge -e world` and with system, they all fail similarly. I managed to reconfigure it to have a working console and started the `emerge -e system` from the console. After a while it froze, but typed a ctrl-z to suspend the process, then %% to resume it and it unlocked it. It often hung like in between two phases of the emerge process (like after unpacking source, or just before doing the install, etc). I was able to resume the emerge like that a good dozen times until now, and now it's really locked, not responding at all... seems like all 4 cores are used at 100%. It hung while emerging perl and I've included the output on the console below so you can see exactly where it hung. I will now force a reboot on it and retry emerging just perl, I will reply to the list with my results of that. SNIP Have you enabled python3 by any chance? eselect python list should show 2.6 as the active python. Just a thought. Someone else did this the other day and had troubles. Dale :-) :-)
Re: [gentoo-user] VPS has issues emerging everything
Ok, it actually just froze again after the output below... / Emerging (42 of 151) sys-devel/autoconf-2.65-r1 * autoconf-2.65.tar.bz2 RMD160 SHA1 SHA256 size ;-) ... [ ok ] * Package:sys-devel/autoconf-2.65-r1 * Repository: gentoo * Maintainer: base-sys...@gentoo.org * USE: elibc_glibc kernel_linux userland_GNU x86 Unpacking source... Unpacking autoconf-2.65.tar.bz2 to /mnt/tmp/portage/sys-devel/autoconf-2.65-r1/work * Applying autoconf-2.65-AC_TYPE_INT_T.patch ...[ ok ] Source unpacked in /mnt/tmp/portage/sys-devel/autoconf-2.65-r1/work Compiling source in /mnt/tmp/portage/sys-devel/autoconf-2.65-r1/work/autoconf-2.65 ... * econf: updating autoconf-2.65/build-aux/config.guess with /usr/share/gnuconfig/config.guess * econf: updating autoconf-2.65/build-aux/config.sub with /usr/share/gnuconfig/config.sub ./configure --prefix=/usr --build=i686-pc-linux-gnu --host=i686-pc-linux-gnu --mandir=/usr/share/man --infodir=/usr/share/info --datadir=/usr/share --sysconfdir=/etc --localstatedir=/var/lib --program-suffix=-2.65 checking for a BSD-compatible install... /usr/bin/install -c checking whether build environment is sane... yes checking for a thread-safe mkdir -p... /bin/mkdir -p checking for gawk... gawk checking whether make sets $(MAKE)... yes checking whether /bin/sh -n is known to work... yes checking for characters that cannot appear in file names... none checking whether directories can have trailing spaces... yes checking for expr... /usr/bin/expr checking for GNU M4 that supports accurate traces... /usr/bin/m4 checking whether /usr/bin/m4 accepts --gnu... yes checking how m4 supports trace files... --debugfile checking for perl... /usr/bin/perl checking whether /usr/bin/perl Fcntl::flock is implemented... yes checking for emacs... no checking for emacs... no checking where .elc files should go... ${datadir}/emacs/site-lisp checking for grep that handles long lines and -e... /bin/grep checking for egrep... /bin/grep -E checking for a sed that does not truncate output... /bin/sed checking whether make is case sensitive... yes configure: creating ./config.status config.status: creating tests/Makefile config.status: creating tests/atlocal config.status: creating man/Makefile config.status: creating lib/emacs/Makefile config.status: creating Makefile config.status: creating doc/Makefile config.status: creating lib/Makefile config.status: creating lib/Autom4te/Makefile config.status: creating lib/autoscan/Makefile config.status: creating lib/m4sugar/Makefile config.status: creating lib/autoconf/Makefile config.status: creating lib/autotest/Makefile // On Fri, Jan 7, 2011 at 2:40 PM, Simon turne...@gmail.com wrote: Ok, the fsck reported nothing wrong... I still got the same bug again... while doing `emerge -e @system`... It got stuck right after the Installing line below... Dale, I just checked my python version and it was already 2.6, i set it to 2.6 again, just in case, and continued my emerge (the ctrl-z + %% worked at this point). I'll reply here again with the next issue... Thanks, Simon Emerging (35 of 151) sys-devel/binutils-config-1.9-r4 * Package: sys-devel/binutils-config-1.9-r4 * Repository: gentoo * Maintainer: toolch...@gentoo.org * USE: elibc_glibc kernel_linux userland_GNU x86 Unpacking source... Source unpacked in /mnt/tmp/portage/sys-devel/binutils-config-1.9-r4/work Compiling source in /mnt/tmp/portage/sys-devel/binutils-config-1.9-r4/work ... Source compiled. Test phase [not enabled]: sys-devel/binutils-config-1.9-r4 Install binutils-config-1.9-r4 into /mnt/tmp/portage/sys-devel/binutils-config-1.9-r4/image/ category sys-devel Completed installing binutils-config-1.9-r4 into /mnt/tmp/portage/sys-devel/binutils-config-1.9-r4/image/ ecompressdir: bzip2 -9 /usr/share/man Installing (35 of 151) sys-devel/binutils-config-1.9-r4 \ On Fri, Jan 7, 2011 at 1:22 PM, Dale rdalek1...@gmail.com wrote: Simon wrote: Hi there, something went wrong during a previous update and now revdep-rebuild will kind-of freeze around 30%. When it freezes like that, the hosting company shows one of it's CPU core is used at 100% non-stop and the rest is idle. I have tried `emerge -e world` and with system, they all fail similarly. I managed to reconfigure it to have a working console and started the `emerge -e system` from the console. After a while it froze, but typed a ctrl-z to suspend the process, then %% to resume it and it unlocked it. It often hung like in between two phases of the emerge process (like after unpacking source, or just before doing the install, etc). I was able to resume the emerge like that a good dozen times until now, and now it's really locked, not responding at all... seems
Re: [gentoo-user] VPS has issues emerging everything
Simon wrote: Ok, it actually just froze again after the output below... / Well, we know it is using the right version of python at least. No other ideas at the moment. Dale :-) :-)
Re: [gentoo-user] VPS has issues emerging everything
Ok, was just doing a fsck.ext3 -c -c -C 1 /dev/xvda from within a rescue OS provided by my VPS provider... and it froze at the middle of it!!! So I guess Gentoo is not responsible after all... I've opened a support ticket with them... I'll let you know how it turns out. Simon On Fri, Jan 7, 2011 at 2:42 PM, Simon turne...@gmail.com wrote: Ok, it actually just froze again after the output below... / Emerging (42 of 151) sys-devel/autoconf-2.65-r1 * autoconf-2.65.tar.bz2 RMD160 SHA1 SHA256 size ;-) ... [ ok ] * Package: sys-devel/autoconf-2.65-r1 * Repository: gentoo * Maintainer: base-sys...@gentoo.org * USE: elibc_glibc kernel_linux userland_GNU x86 Unpacking source... Unpacking autoconf-2.65.tar.bz2 to /mnt/tmp/portage/sys-devel/autoconf-2.65-r1/work * Applying autoconf-2.65-AC_TYPE_INT_T.patch ... [ ok ] Source unpacked in /mnt/tmp/portage/sys-devel/autoconf-2.65-r1/work Compiling source in /mnt/tmp/portage/sys-devel/autoconf-2.65-r1/work/autoconf-2.65 ... * econf: updating autoconf-2.65/build-aux/config.guess with /usr/share/gnuconfig/config.guess * econf: updating autoconf-2.65/build-aux/config.sub with /usr/share/gnuconfig/config.sub ./configure --prefix=/usr --build=i686-pc-linux-gnu --host=i686-pc-linux-gnu --mandir=/usr/share/man --infodir=/usr/share/info --datadir=/usr/share --sysconfdir=/etc --localstatedir=/var/lib --program-suffix=-2.65 checking for a BSD-compatible install... /usr/bin/install -c checking whether build environment is sane... yes checking for a thread-safe mkdir -p... /bin/mkdir -p checking for gawk... gawk checking whether make sets $(MAKE)... yes checking whether /bin/sh -n is known to work... yes checking for characters that cannot appear in file names... none checking whether directories can have trailing spaces... yes checking for expr... /usr/bin/expr checking for GNU M4 that supports accurate traces... /usr/bin/m4 checking whether /usr/bin/m4 accepts --gnu... yes checking how m4 supports trace files... --debugfile checking for perl... /usr/bin/perl checking whether /usr/bin/perl Fcntl::flock is implemented... yes checking for emacs... no checking for emacs... no checking where .elc files should go... ${datadir}/emacs/site-lisp checking for grep that handles long lines and -e... /bin/grep checking for egrep... /bin/grep -E checking for a sed that does not truncate output... /bin/sed checking whether make is case sensitive... yes configure: creating ./config.status config.status: creating tests/Makefile config.status: creating tests/atlocal config.status: creating man/Makefile config.status: creating lib/emacs/Makefile config.status: creating Makefile config.status: creating doc/Makefile config.status: creating lib/Makefile config.status: creating lib/Autom4te/Makefile config.status: creating lib/autoscan/Makefile config.status: creating lib/m4sugar/Makefile config.status: creating lib/autoconf/Makefile config.status: creating lib/autotest/Makefile // On Fri, Jan 7, 2011 at 2:40 PM, Simon turne...@gmail.com wrote: Ok, the fsck reported nothing wrong... I still got the same bug again... while doing `emerge -e @system`... It got stuck right after the Installing line below... Dale, I just checked my python version and it was already 2.6, i set it to 2.6 again, just in case, and continued my emerge (the ctrl-z + %% worked at this point). I'll reply here again with the next issue... Thanks, Simon Emerging (35 of 151) sys-devel/binutils-config-1.9-r4 * Package: sys-devel/binutils-config-1.9-r4 * Repository: gentoo * Maintainer: toolch...@gentoo.org * USE: elibc_glibc kernel_linux userland_GNU x86 Unpacking source... Source unpacked in /mnt/tmp/portage/sys-devel/binutils-config-1.9-r4/work Compiling source in /mnt/tmp/portage/sys-devel/binutils-config-1.9-r4/work ... Source compiled. Test phase [not enabled]: sys-devel/binutils-config-1.9-r4 Install binutils-config-1.9-r4 into /mnt/tmp/portage/sys-devel/binutils-config-1.9-r4/image/ category sys-devel Completed installing binutils-config-1.9-r4 into /mnt/tmp/portage/sys-devel/binutils-config-1.9-r4/image/ ecompressdir: bzip2 -9 /usr/share/man Installing (35 of 151) sys-devel/binutils-config-1.9-r4 \ On Fri, Jan 7, 2011 at 1:22 PM, Dale rdalek1...@gmail.com wrote: Simon wrote: Hi there, something went wrong during a previous update and now revdep-rebuild will kind-of freeze around 30%. When it freezes like that, the hosting company shows one of it's CPU core is used at 100% non-stop and the rest is idle. I have tried `emerge -e world` and with system, they all fail similarly. I managed to reconfigure it to have a working console and started the `emerge -e
Re: [gentoo-user] VPS has issues emerging everything
On Fri, Jan 7, 2011 at 12:48 PM, Simon turne...@gmail.com wrote: Ok, was just doing a fsck.ext3 -c -c -C 1 /dev/xvda from within a rescue OS provided by my VPS provider... and it froze at the middle of it!!! So I guess Gentoo is not responsible after all... I've opened a support ticket with them... I'll let you know how it turns out. Simon Good move. Last thing that comes to mind for me. Are you on your own hard drive in this system, or just a partition? Anyway, if this was a 4K sector hard drive, like the newer WD green drives, then partitions that aren't set up right can have long times where the system appears to freeze or simple command can fail due to time-outs. I saw 'freezes' of 10-15 minutes sometimes on the WD10EARS 1TB drives I bought until others helped me figure that one out. Good luck, Mark
Re: [gentoo-user] VPS has issues emerging everything
What do you have MAKEOPTS set to?