[ZWeb] IOError: [Errno 28] No space left on device
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi, disk is full again. Last successful transaction was at 2009-08-07 12:50:24 AM systemtime. Regards Michael - -- http://blog.d2m.at http://planetzope.org -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iD8DBQFKe8t7l0uAvQJUKVYRAimfAJ9otDgdqyKkKq5NHynvq8r33+iB4QCguO32 eGQ5Ubc7zc6fLa8NLT6DhzM= =OuuH -END PGP SIGNATURE- ___ Zope-web maillist - Zope-web@zope.org http://mail.zope.org/mailman/listinfo/zope-web
Re: [ZWeb] Error Type, IOError, Error Value, [Errno 28] No space left on device
I just looked and the transaction rates have fallen from about 200-300MB/hr to about 200-300K/hr. :) Jim On Thu, Aug 6, 2009 at 9:19 AM, Michael Haubenwallner wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > Jens Vagelpohl wrote: >> >> On Aug 6, 2009, at 09:43 , Andreas Jung wrote: >> >>> Same issue this morning. This must be fixed soon - otherwise we can >>> not >>> proceed with the releases. >> >> I just looked and for some strange reason the ZODB has grown to over >> 30 GB, which means it more than doubled in size since the last full >> backup on August 2. Looking at the backup files from repozo it appears >> the massive growth happened over the last few days, those daily >> incrementals since August 2 range from 5-6 GB in size each. >> >> I have now freed up 3 GB by deleting the remaining delta files from >> the previous round of backups, but this may not be enough seeing how >> fast the database is growing. Something fishy is going on. >> > > Hi, this was caused by spamming 2 collector issues in member space, each > holding a transcript of about 1 MB and more. > > I removed the issues, blocked the spamming user account and finally > changed default roles for everything collector related to 'Manager'. > > Regards > Michael > > - -- > http://blog.d2m.at > http://planetzope.org > > -BEGIN PGP SIGNATURE- > Version: GnuPG v1.4.6 (GNU/Linux) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ > > iD8DBQFKethml0uAvQJUKVYRAlC0AJ4tkdTGdMrG56mtZM46ASZJFIgNiACfUkjg > 2yX+44OxRtMaZuur2D9sNqQ= > =Wq/v > -END PGP SIGNATURE- > > ___ > Zope-web maillist - zope-...@zope.org > http://mail.zope.org/mailman/listinfo/zope-web > -- Jim Fulton ___ Zope-web maillist - Zope-web@zope.org http://mail.zope.org/mailman/listinfo/zope-web
Re: [ZWeb] Error Type, IOError, Error Value, [Errno 28] No space left on device
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Aug 6, 2009, at 17:44 , Jim Fulton wrote: > Jens, > > We're doing this. (Actually, we're going to copy the backups to > another disk first.) > > We're also going to upgrade the hardware and switch to using ZRS. > > Sorry, I should have acked sooner. Bad me. Bad bad me. No worries, just means less on my plate ;-) jens P.S.: A bigger disk would really be handy as well. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.8 (Darwin) iEYEARECAAYFAkp6+1kACgkQRAx5nvEhZLKZNwCdFiRW2p1ZY6DXMlFPxLTDmFQY eFIAoKxk2VIfGQOs36U4Ie4xPwa/HDMh =zcPd -END PGP SIGNATURE- ___ Zope-web maillist - Zope-web@zope.org http://mail.zope.org/mailman/listinfo/zope-web
Re: [ZWeb] Error Type, IOError, Error Value, [Errno 28] No space left on device
Jens, We're doing this. (Actually, we're going to copy the backups to another disk first.) We're also going to upgrade the hardware and switch to using ZRS. Sorry, I should have acked sooner. Bad me. Bad bad me. Jim On Thu, Aug 6, 2009 at 11:24 AM, Jens Vagelpohl wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > I just want to mention that we will run into space problems again tonight. > > The ZODB is 32 GB in size, and there's 2 GB space left on the drive. Any > repozo run or pack at this point will fail. The only way out I can see > involves the following: > > - delete all repozo backups > > - pack the database manually > > - delete Data.fs.old > > After that repozo will start with a fresh backup. > > Anyone have any better ideas? > > jens > > > -BEGIN PGP SIGNATURE- > Version: GnuPG v1.4.8 (Darwin) > > iEYEARECAAYFAkp69Z4ACgkQRAx5nvEhZLIdzACggnNgk5wp7GIAO4aLX34zJcOD > ULEAnjRxhcr9yb9Upc6952bv7eJPMpJD > =RjRo > -END PGP SIGNATURE- > -- Jim Fulton ___ Zope-web maillist - Zope-web@zope.org http://mail.zope.org/mailman/listinfo/zope-web
Re: [ZWeb] Error Type, IOError, Error Value, [Errno 28] No space left on device
That's pretty much along the lines I was thinking as well when I looked at it real quick this morning. +1 from me, fwiw. --Dave On Aug 6, 2009, at 11:24 AM, Jens Vagelpohl wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > I just want to mention that we will run into space problems again > tonight. > > The ZODB is 32 GB in size, and there's 2 GB space left on the drive. > Any repozo run or pack at this point will fail. The only way out I can > see involves the following: > > - delete all repozo backups > > - pack the database manually > > - delete Data.fs.old > > After that repozo will start with a fresh backup. > > Anyone have any better ideas? > > jens > > > -BEGIN PGP SIGNATURE- > Version: GnuPG v1.4.8 (Darwin) > > iEYEARECAAYFAkp69Z4ACgkQRAx5nvEhZLIdzACggnNgk5wp7GIAO4aLX34zJcOD > ULEAnjRxhcr9yb9Upc6952bv7eJPMpJD > =RjRo > -END PGP SIGNATURE- > ___ > Zope-web maillist - Zope-web@zope.org > http://mail.zope.org/mailman/listinfo/zope-web Systems Administrator Zope Corp. 540-361-1722 da...@zope.com ___ Zope-web maillist - Zope-web@zope.org http://mail.zope.org/mailman/listinfo/zope-web
Re: [ZWeb] Error Type, IOError, Error Value, [Errno 28] No space left on device
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I just want to mention that we will run into space problems again tonight. The ZODB is 32 GB in size, and there's 2 GB space left on the drive. Any repozo run or pack at this point will fail. The only way out I can see involves the following: - delete all repozo backups - pack the database manually - delete Data.fs.old After that repozo will start with a fresh backup. Anyone have any better ideas? jens -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.8 (Darwin) iEYEARECAAYFAkp69Z4ACgkQRAx5nvEhZLIdzACggnNgk5wp7GIAO4aLX34zJcOD ULEAnjRxhcr9yb9Upc6952bv7eJPMpJD =RjRo -END PGP SIGNATURE- ___ Zope-web maillist - Zope-web@zope.org http://mail.zope.org/mailman/listinfo/zope-web
Re: [ZWeb] Error Type, IOError, Error Value, [Errno 28] No space left on device
Thanks! Jim On Thu, Aug 6, 2009 at 9:19 AM, Michael Haubenwallner wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > Jens Vagelpohl wrote: >> >> On Aug 6, 2009, at 09:43 , Andreas Jung wrote: >> >>> Same issue this morning. This must be fixed soon - otherwise we can >>> not >>> proceed with the releases. >> >> I just looked and for some strange reason the ZODB has grown to over >> 30 GB, which means it more than doubled in size since the last full >> backup on August 2. Looking at the backup files from repozo it appears >> the massive growth happened over the last few days, those daily >> incrementals since August 2 range from 5-6 GB in size each. >> >> I have now freed up 3 GB by deleting the remaining delta files from >> the previous round of backups, but this may not be enough seeing how >> fast the database is growing. Something fishy is going on. >> > > Hi, this was caused by spamming 2 collector issues in member space, each > holding a transcript of about 1 MB and more. > > I removed the issues, blocked the spamming user account and finally > changed default roles for everything collector related to 'Manager'. > > Regards > Michael > > - -- > http://blog.d2m.at > http://planetzope.org > > -BEGIN PGP SIGNATURE- > Version: GnuPG v1.4.6 (GNU/Linux) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ > > iD8DBQFKethml0uAvQJUKVYRAlC0AJ4tkdTGdMrG56mtZM46ASZJFIgNiACfUkjg > 2yX+44OxRtMaZuur2D9sNqQ= > =Wq/v > -END PGP SIGNATURE- > > ___ > Zope-web maillist - zope-...@zope.org > http://mail.zope.org/mailman/listinfo/zope-web > -- Jim Fulton ___ Zope-web maillist - Zope-web@zope.org http://mail.zope.org/mailman/listinfo/zope-web
Re: [ZWeb] Error Type, IOError, Error Value, [Errno 28] No space left on device
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jens Vagelpohl wrote: > > On Aug 6, 2009, at 09:43 , Andreas Jung wrote: > >> Same issue this morning. This must be fixed soon - otherwise we can >> not >> proceed with the releases. > > I just looked and for some strange reason the ZODB has grown to over > 30 GB, which means it more than doubled in size since the last full > backup on August 2. Looking at the backup files from repozo it appears > the massive growth happened over the last few days, those daily > incrementals since August 2 range from 5-6 GB in size each. > > I have now freed up 3 GB by deleting the remaining delta files from > the previous round of backups, but this may not be enough seeing how > fast the database is growing. Something fishy is going on. > Hi, this was caused by spamming 2 collector issues in member space, each holding a transcript of about 1 MB and more. I removed the issues, blocked the spamming user account and finally changed default roles for everything collector related to 'Manager'. Regards Michael - -- http://blog.d2m.at http://planetzope.org -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iD8DBQFKethml0uAvQJUKVYRAlC0AJ4tkdTGdMrG56mtZM46ASZJFIgNiACfUkjg 2yX+44OxRtMaZuur2D9sNqQ= =Wq/v -END PGP SIGNATURE- ___ Zope-web maillist - Zope-web@zope.org http://mail.zope.org/mailman/listinfo/zope-web
Re: [ZWeb] Error Type, IOError, Error Value, [Errno 28] No space left on device
Michael, is there any chance you can join us? On Thu, Aug 6, 2009 at 5:04 AM, Jim Fulton wrote: > I'm on #zope-web. > > On Thu, Aug 6, 2009 at 3:43 AM, Andreas Jung wrote: >> Same issue this morning. This must be fixed soon - otherwise we can not >> proceed with the releases. >> >> Andreas >> >> On 03.08.09 19:52, Jim Fulton wrote: >>> We cleaned up some old files and the space problem is addressed, at >>> least for a few months. We'll look into getting some newer hardware >>> for the storage server. >>> >>> We also restarted the app servers, although the app still seems to be >>> quite slow. >>> >>> Jim >>> >>> On Mon, Aug 3, 2009 at 1:36 PM, Jim Fulton wrote: >>> Sorry, I should have responded earlier. We're digging into it. Jim On Mon, Aug 3, 2009 at 1:31 PM, Andreas Jung wrote: > On 03.08.09 19:28, Jens Vagelpohl wrote: > >> On Aug 3, 2009, at 19:25 , Andreas Jung wrote: >> >> >>> On 03.08.09 19:24, Jens Vagelpohl wrote: >>> On Aug 3, 2009, at 19:22 , David Lawson wrote: > Joseph and I both cleaned off some things. I also modified the repozo > backup cleanup script to only keep six days of backups, otherwise we > would have run out of space again either in a week or the next time > the database packed. The major space usage was the Data.fs, the > Data.fs.old, and two full FS backups made by repozo. > Alright, thanks for jumping onto it so quickly! >> >>> Right now zope.org is dead - all requests are hanging. >>> >> It works perfectly fine for me. >> > *shrug* creating a new document took roughly 15 minutes. During > that time all requests were hanging on my side. They terminated > after the documented was created...anyway...old Plone crap. > > Andreas > > ___ > Zope-web maillist - zope-...@zope.org > http://mail.zope.org/mailman/listinfo/zope-web > > > -- Jim Fulton >>> >>> >>> >> >> >> -- >> ZOPYX Ltd. & Co KG \ ZOPYX & Friends >> Charlottenstr. 37/1 \ The experts for your Python, Zope and >> D-72070 Tübingen \ Plone projects >> www.zopyx.com, i...@zopyx.com \ www.zopyx.de/friends, frie...@zopyx.de >> >> E-Publishing, Python, Zope & Plone development, Consulting >> >> >> > > > > -- > Jim Fulton > -- Jim Fulton ___ Zope-web maillist - Zope-web@zope.org http://mail.zope.org/mailman/listinfo/zope-web
Re: [ZWeb] Error Type, IOError, Error Value, [Errno 28] No space left on device
I'm on #zope-web. On Thu, Aug 6, 2009 at 3:43 AM, Andreas Jung wrote: > Same issue this morning. This must be fixed soon - otherwise we can not > proceed with the releases. > > Andreas > > On 03.08.09 19:52, Jim Fulton wrote: >> We cleaned up some old files and the space problem is addressed, at >> least for a few months. We'll look into getting some newer hardware >> for the storage server. >> >> We also restarted the app servers, although the app still seems to be >> quite slow. >> >> Jim >> >> On Mon, Aug 3, 2009 at 1:36 PM, Jim Fulton wrote: >> >>> Sorry, I should have responded earlier. We're digging into it. >>> >>> Jim >>> >>> On Mon, Aug 3, 2009 at 1:31 PM, Andreas Jung wrote: >>> On 03.08.09 19:28, Jens Vagelpohl wrote: > On Aug 3, 2009, at 19:25 , Andreas Jung wrote: > > >> On 03.08.09 19:24, Jens Vagelpohl wrote: >> >>> On Aug 3, 2009, at 19:22 , David Lawson wrote: >>> >>> Joseph and I both cleaned off some things. I also modified the repozo backup cleanup script to only keep six days of backups, otherwise we would have run out of space again either in a week or the next time the database packed. The major space usage was the Data.fs, the Data.fs.old, and two full FS backups made by repozo. >>> Alright, thanks for jumping onto it so quickly! >>> > >> Right now zope.org is dead - all requests are hanging. >> > It works perfectly fine for me. > *shrug* creating a new document took roughly 15 minutes. During that time all requests were hanging on my side. They terminated after the documented was created...anyway...old Plone crap. Andreas ___ Zope-web maillist - zope-...@zope.org http://mail.zope.org/mailman/listinfo/zope-web >>> >>> >>> -- >>> Jim Fulton >>> >>> >> >> >> > > > -- > ZOPYX Ltd. & Co KG \ ZOPYX & Friends > Charlottenstr. 37/1 \ The experts for your Python, Zope and > D-72070 Tübingen \ Plone projects > www.zopyx.com, i...@zopyx.com \ www.zopyx.de/friends, frie...@zopyx.de > > E-Publishing, Python, Zope & Plone development, Consulting > > > -- Jim Fulton ___ Zope-web maillist - Zope-web@zope.org http://mail.zope.org/mailman/listinfo/zope-web
Re: [ZWeb] Error Type, IOError, Error Value, [Errno 28] No space left on device
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Aug 6, 2009, at 09:43 , Andreas Jung wrote: > Same issue this morning. This must be fixed soon - otherwise we can > not > proceed with the releases. I just looked and for some strange reason the ZODB has grown to over 30 GB, which means it more than doubled in size since the last full backup on August 2. Looking at the backup files from repozo it appears the massive growth happened over the last few days, those daily incrementals since August 2 range from 5-6 GB in size each. I have now freed up 3 GB by deleting the remaining delta files from the previous round of backups, but this may not be enough seeing how fast the database is growing. Something fishy is going on. jens -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.8 (Darwin) iEYEARECAAYFAkp6kyMACgkQRAx5nvEhZLKcMACeOu7iaFufinRDHL7QsUkvr9bJ /wIAn3yJqvo1qiA2mE4XlCGodleA93NC =/IOy -END PGP SIGNATURE- ___ Zope-web maillist - Zope-web@zope.org http://mail.zope.org/mailman/listinfo/zope-web
Re: [ZWeb] Error Type, IOError, Error Value, [Errno 28] No space left on device
Same issue this morning. This must be fixed soon - otherwise we can not proceed with the releases. Andreas On 03.08.09 19:52, Jim Fulton wrote: > We cleaned up some old files and the space problem is addressed, at > least for a few months. We'll look into getting some newer hardware > for the storage server. > > We also restarted the app servers, although the app still seems to be > quite slow. > > Jim > > On Mon, Aug 3, 2009 at 1:36 PM, Jim Fulton wrote: > >> Sorry, I should have responded earlier. We're digging into it. >> >> Jim >> >> On Mon, Aug 3, 2009 at 1:31 PM, Andreas Jung wrote: >> >>> On 03.08.09 19:28, Jens Vagelpohl wrote: >>> On Aug 3, 2009, at 19:25 , Andreas Jung wrote: > On 03.08.09 19:24, Jens Vagelpohl wrote: > >> On Aug 3, 2009, at 19:22 , David Lawson wrote: >> >> >>> Joseph and I both cleaned off some things. I also modified the repozo >>> backup cleanup script to only keep six days of backups, otherwise we >>> would have run out of space again either in a week or the next time >>> the database packed. The major space usage was the Data.fs, the >>> Data.fs.old, and two full FS backups made by repozo. >>> >> Alright, thanks for jumping onto it so quickly! >> > Right now zope.org is dead - all requests are hanging. > It works perfectly fine for me. >>> *shrug* creating a new document took roughly 15 minutes. During >>> that time all requests were hanging on my side. They terminated >>> after the documented was created...anyway...old Plone crap. >>> >>> Andreas >>> >>> ___ >>> Zope-web maillist - Zope-web@zope.org >>> http://mail.zope.org/mailman/listinfo/zope-web >>> >>> >>> >> >> >> -- >> Jim Fulton >> >> > > > -- ZOPYX Ltd. & Co KG \ ZOPYX & Friends Charlottenstr. 37/1 \ The experts for your Python, Zope and D-72070 Tübingen \ Plone projects www.zopyx.com, i...@zopyx.com \ www.zopyx.de/friends, frie...@zopyx.de E-Publishing, Python, Zope & Plone development, Consulting begin:vcard fn:Andreas Jung n:Jung;Andreas org:ZOPYX Ltd. & Co. KG adr;quoted-printable:;;Charlottenstr. 37/1;T=C3=BCbingen;;72070;Germany email;internet:i...@zopyx.com title:CEO tel;work:+49-7071-793376 tel;fax:+49-7071-7936840 tel;home:+49-7071-793257 x-mozilla-html:FALSE url:www.zopyx.com version:2.1 end:vcard ___ Zope-web maillist - Zope-web@zope.org http://mail.zope.org/mailman/listinfo/zope-web
