Re: [Zope-dev] short svn.zope.org downtime
On 04/08/2010 03:12 PM, Jim Fulton wrote: On Thu, Apr 8, 2010 at 2:59 AM, Christian Theune wrote: On 04/07/2010 05:14 PM, Jens Vagelpohl wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 4/7/10 17:08 , Tres Seaver wrote: I'm seeing frequent periods of 50+% packet loss to the machine, and can't even get top to run on it this morning. Any clues about what is causing the load? I haven't watched it because I don't know when Theuni has set that particular cron job, but I bet the whole repository policy checker is causing a lot of load. The checker runs at 2:15 CE(ST) for about 90 minutes. However, the script only loads recent revisions and does the working copy checkouts step by step instead of a large run, so I don't think it is aggressive enough to cause troubles. (Can't be sure of course.) Is it running against the main repo? or a mirror? Its running against the main repo - we currently don't have a mirror around, but I'll have one again in the next weeks so if it should be an issue I can relocate that in the future. Christian -- Christian Theune · c...@gocept.com gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development smime.p7s Description: S/MIME Cryptographic Signature ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] short svn.zope.org downtime
On Thu, Apr 8, 2010 at 2:59 AM, Christian Theune wrote: > On 04/07/2010 05:14 PM, Jens Vagelpohl wrote: >> -BEGIN PGP SIGNED MESSAGE- >> Hash: SHA1 >> >> On 4/7/10 17:08 , Tres Seaver wrote: >> >>> I'm seeing frequent periods of 50+% packet loss to the machine, and >>> can't even get top to run on it this morning. Any clues about what is >>> causing the load? >> >> I haven't watched it because I don't know when Theuni has set that >> particular cron job, but I bet the whole repository policy checker is >> causing a lot of load. > > The checker runs at 2:15 CE(ST) for about 90 minutes. However, the > script only loads recent revisions and does the working copy checkouts > step by step instead of a large run, so I don't think it is aggressive > enough to cause troubles. (Can't be sure of course.) Is it running against the main repo? or a mirror? Jim -- Jim Fulton ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] short svn.zope.org downtime
On 04/07/2010 05:14 PM, Jens Vagelpohl wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On 4/7/10 17:08 , Tres Seaver wrote: > >> I'm seeing frequent periods of 50+% packet loss to the machine, and >> can't even get top to run on it this morning. Any clues about what is >> causing the load? > > I haven't watched it because I don't know when Theuni has set that > particular cron job, but I bet the whole repository policy checker is > causing a lot of load. The checker runs at 2:15 CE(ST) for about 90 minutes. However, the script only loads recent revisions and does the working copy checkouts step by step instead of a large run, so I don't think it is aggressive enough to cause troubles. (Can't be sure of course.) Christian -- Christian Theune · c...@gocept.com gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] short svn.zope.org downtime
On Wed, Apr 7, 2010 at 11:32 AM, Jens Vagelpohl wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On 4/7/10 17:27 , Jim Fulton wrote: >> We've been having some networking issues over the last week or two that >> we believe to have been addressed. > > I'm still getting alerts from Nagios 2-3 times a day caused by high > packet loss and inability to connect to the services being tested. Hm, OK. We're continuing to work on this and monitor it. Thanks. Jim -- Jim Fulton ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] short svn.zope.org downtime
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 4/7/10 17:27 , Jim Fulton wrote: > We've been having some networking issues over the last week or two that > we believe to have been addressed. I'm still getting alerts from Nagios 2-3 times a day caused by high packet loss and inability to connect to the services being tested. jens -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.8 (Darwin) iEYEARECAAYFAku8pZwACgkQRAx5nvEhZLL6TwCfZ1JDUeVYyriYYbc6xaMRSL1M fvsAn1a4cKBn1IT2+YvgN7bevefhlKyG =wAsX -END PGP SIGNATURE- ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] short svn.zope.org downtime
On Wed, Apr 7, 2010 at 11:08 AM, Tres Seaver wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > Hanno Schlichting wrote: >> Hi, >> >> looks like SVN and viewcvs weren't responding for a couple minutes. I >> could log into the machine via SSH and Apache serving docs.zope.org >> still worked. >> >> I only noticed: >> >> 27059 simon 20 0 98664 6388 4360 R 100 0.2 43327:19 darcs >> >> Why do we have darcs running on that machine and more importantly why >> does it run with 100% cpu load for minutes (or hours?). > > I'm seeing frequent periods of 50+% packet loss to the machine, and > can't even get top to run on it this morning. Any clues about what is > causing the load? We've been having some networking issues over the last week or two that we believe to have been addressed. Jim -- Jim Fulton ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] short svn.zope.org downtime
On 4/7/10 17:17 , Wichert Akkerman wrote: > On 4/7/10 17:14 , Jens Vagelpohl wrote: >> -BEGIN PGP SIGNED MESSAGE- >> Hash: SHA1 >> >> On 4/7/10 17:08 , Tres Seaver wrote: >> >>> I'm seeing frequent periods of 50+% packet loss to the machine, and >>> can't even get top to run on it this morning. Any clues about what is >>> causing the load? >> >> I haven't watched it because I don't know when Theuni has set that >> particular cron job, but I bet the whole repository policy checker is >> causing a lot of load. >> >> There are many other times with high packet loss unrelated to load. Not >> sure how and why that started. It's a very recent issue. > > Did you check using mtr or a similar tool at which hops the packetloss > occur? Currently it looks this this for me: thunderbird mangled that badly. There is it properly: http://pastebin.com/raw.php?i=5yWh9KjN Wichert. ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] short svn.zope.org downtime
On 4/7/10 17:14 , Jens Vagelpohl wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On 4/7/10 17:08 , Tres Seaver wrote: > >> I'm seeing frequent periods of 50+% packet loss to the machine, and >> can't even get top to run on it this morning. Any clues about what is >> causing the load? > > I haven't watched it because I don't know when Theuni has set that > particular cron job, but I bet the whole repository policy checker is > causing a lot of load. > > There are many other times with high packet loss unrelated to load. Not > sure how and why that started. It's a very recent issue. Did you check using mtr or a similar tool at which hops the packetloss occur? Currently it looks this this for me: Host Loss% Snt Last Avg Best Wrst StDev 1. gw255.3d12.high5.net 0.0% 1230.5 1.6 0.5 7.2 1.6 2. vlan159.swcolo1.3d12.xs4all.net 0.0% 1230.8 0.8 0.4 3.0 0.4 3. 0.ge-1-2-0.xr3.d12.xs4all.net 0.0% 1230.4 3.8 0.3 114.7 13.7 4. 0.ge-1-2-0.xr1.sara.xs4all.net 0.0% 1230.9 5.2 0.6 132.8 16.6 5. 10gigabitethernet3-3.core1.ams1.he.net 0.0% 1231.4 2.5 0.6 11.0 2.9 6. 10gigabitethernet4-1.core1.nyc4.he.net 0.0% 123 86.8 89.2 86.0 97.7 3.5 7. 10gigabitethernet1-2.core1.nyc1.he.net 0.0% 123 95.8 88.6 86.1 99.2 3.3 8. nyiix.nyc.cavtel.net 0.0% 122 87.8 87.8 86.8 94.8 1.0 9. lag-3.cr01.cleyohja.cavtel.com 77.9% 122 15741 15370 15157 16128 278.4 10. ??? 11. xe-1-2-2.er01.asbnvaeq.cavtel.com 41.0% 122 103.6 104.2 102.6 122.8 3.1 12. static-98-140-183-6.hicap.cavtel.net 0.0% 122 103.6 108.6 102.6 298.7 26.3 13. v10.rd1.ric.peak10.net 0.0% 122 103.5 103.2 102.4 114.9 1.1 14. cvs.zope.org 0.0% 122 103.5 103.5 102.4 110.3 1.5 which shows two routers at cavtel that are too busy to return a response, but otherwise a normal connection. Wichert. ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] short svn.zope.org downtime
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 4/7/10 17:08 , Tres Seaver wrote: > I'm seeing frequent periods of 50+% packet loss to the machine, and > can't even get top to run on it this morning. Any clues about what is > causing the load? I haven't watched it because I don't know when Theuni has set that particular cron job, but I bet the whole repository policy checker is causing a lot of load. There are many other times with high packet loss unrelated to load. Not sure how and why that started. It's a very recent issue. jens -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.8 (Darwin) iEYEARECAAYFAku8oVYACgkQRAx5nvEhZLJ3nwCZAW0/DWV2OfJZDj1WRCkhFHCd NboAn1uEXW1OhXjHc99deCpZd+KztafT =YJDS -END PGP SIGNATURE- ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] short svn.zope.org downtime
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hanno Schlichting wrote: > Hi, > > looks like SVN and viewcvs weren't responding for a couple minutes. I > could log into the machine via SSH and Apache serving docs.zope.org > still worked. > > I only noticed: > > 27059 simon 20 0 98664 6388 4360 R 100 0.2 43327:19 darcs > > Why do we have darcs running on that machine and more importantly why > does it run with 100% cpu load for minutes (or hours?). I'm seeing frequent periods of 50+% packet loss to the machine, and can't even get top to run on it this morning. Any clues about what is causing the load? Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAku8n+UACgkQ+gerLs4ltQ6m8ACgjCu/lne/2zQ0YvVbCbfVnipz UkYAn29KBmZe6SzKvmNN++wmCjijqB6e =dk3o -END PGP SIGNATURE- ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] short svn.zope.org downtime
On 4/3/10 4:56 PM, Hanno Schlichting wrote: > 27059 simon 20 0 98664 6388 4360 R 100 0.2 43327:19 darcs Ack, my cunning plan to bog down all but the One True VCS is leaking. Must be more *careful*.. But seriously, ha ha ha, no, I have no idea what that process was doing there, sorry all. ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] short svn.zope.org downtime
Hi, looks like SVN and viewcvs weren't responding for a couple minutes. I could log into the machine via SSH and Apache serving docs.zope.org still worked. I only noticed: 27059 simon 20 0 98664 6388 4360 R 100 0.2 43327:19 darcs Why do we have darcs running on that machine and more importantly why does it run with 100% cpu load for minutes (or hours?). Hanno ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
