Re: [ZWeb] Error Type, IOError, Error Value, [Errno 28] No space left on device
Same issue this morning. This must be fixed soon - otherwise we can not proceed with the releases. Andreas On 03.08.09 19:52, Jim Fulton wrote: We cleaned up some old files and the space problem is addressed, at least for a few months. We'll look into getting some newer hardware for the storage server. We also restarted the app servers, although the app still seems to be quite slow. Jim On Mon, Aug 3, 2009 at 1:36 PM, Jim Fultonj...@zope.com wrote: Sorry, I should have responded earlier. We're digging into it. Jim On Mon, Aug 3, 2009 at 1:31 PM, Andreas Jungli...@zopyx.com wrote: On 03.08.09 19:28, Jens Vagelpohl wrote: On Aug 3, 2009, at 19:25 , Andreas Jung wrote: On 03.08.09 19:24, Jens Vagelpohl wrote: On Aug 3, 2009, at 19:22 , David Lawson wrote: Joseph and I both cleaned off some things. I also modified the repozo backup cleanup script to only keep six days of backups, otherwise we would have run out of space again either in a week or the next time the database packed. The major space usage was the Data.fs, the Data.fs.old, and two full FS backups made by repozo. Alright, thanks for jumping onto it so quickly! Right now zope.org is dead - all requests are hanging. It works perfectly fine for me. *shrug* creating a new document took roughly 15 minutes. During that time all requests were hanging on my side. They terminated after the documented was created...anyway...old Plone crap. Andreas ___ Zope-web maillist - Zope-web@zope.org http://mail.zope.org/mailman/listinfo/zope-web -- Jim Fulton -- ZOPYX Ltd. Co KG \ ZOPYX Friends Charlottenstr. 37/1 \ The experts for your Python, Zope and D-72070 Tübingen \ Plone projects www.zopyx.com, i...@zopyx.com \ www.zopyx.de/friends, frie...@zopyx.de E-Publishing, Python, Zope Plone development, Consulting begin:vcard fn:Andreas Jung n:Jung;Andreas org:ZOPYX Ltd. Co. KG adr;quoted-printable:;;Charlottenstr. 37/1;T=C3=BCbingen;;72070;Germany email;internet:i...@zopyx.com title:CEO tel;work:+49-7071-793376 tel;fax:+49-7071-7936840 tel;home:+49-7071-793257 x-mozilla-html:FALSE url:www.zopyx.com version:2.1 end:vcard ___ Zope-web maillist - Zope-web@zope.org http://mail.zope.org/mailman/listinfo/zope-web
Re: [ZWeb] Error Type, IOError, Error Value, [Errno 28] No space left on device
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Aug 6, 2009, at 09:43 , Andreas Jung wrote: Same issue this morning. This must be fixed soon - otherwise we can not proceed with the releases. I just looked and for some strange reason the ZODB has grown to over 30 GB, which means it more than doubled in size since the last full backup on August 2. Looking at the backup files from repozo it appears the massive growth happened over the last few days, those daily incrementals since August 2 range from 5-6 GB in size each. I have now freed up 3 GB by deleting the remaining delta files from the previous round of backups, but this may not be enough seeing how fast the database is growing. Something fishy is going on. jens -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.8 (Darwin) iEYEARECAAYFAkp6kyMACgkQRAx5nvEhZLKcMACeOu7iaFufinRDHL7QsUkvr9bJ /wIAn3yJqvo1qiA2mE4XlCGodleA93NC =/IOy -END PGP SIGNATURE- ___ Zope-web maillist - Zope-web@zope.org http://mail.zope.org/mailman/listinfo/zope-web
Re: [ZWeb] Error Type, IOError, Error Value, [Errno 28] No space left on device
I'm on #zope-web. On Thu, Aug 6, 2009 at 3:43 AM, Andreas Jungli...@zopyx.com wrote: Same issue this morning. This must be fixed soon - otherwise we can not proceed with the releases. Andreas On 03.08.09 19:52, Jim Fulton wrote: We cleaned up some old files and the space problem is addressed, at least for a few months. We'll look into getting some newer hardware for the storage server. We also restarted the app servers, although the app still seems to be quite slow. Jim On Mon, Aug 3, 2009 at 1:36 PM, Jim Fultonj...@zope.com wrote: Sorry, I should have responded earlier. We're digging into it. Jim On Mon, Aug 3, 2009 at 1:31 PM, Andreas Jungli...@zopyx.com wrote: On 03.08.09 19:28, Jens Vagelpohl wrote: On Aug 3, 2009, at 19:25 , Andreas Jung wrote: On 03.08.09 19:24, Jens Vagelpohl wrote: On Aug 3, 2009, at 19:22 , David Lawson wrote: Joseph and I both cleaned off some things. I also modified the repozo backup cleanup script to only keep six days of backups, otherwise we would have run out of space again either in a week or the next time the database packed. The major space usage was the Data.fs, the Data.fs.old, and two full FS backups made by repozo. Alright, thanks for jumping onto it so quickly! Right now zope.org is dead - all requests are hanging. It works perfectly fine for me. *shrug* creating a new document took roughly 15 minutes. During that time all requests were hanging on my side. They terminated after the documented was created...anyway...old Plone crap. Andreas ___ Zope-web maillist - zope-...@zope.org http://mail.zope.org/mailman/listinfo/zope-web -- Jim Fulton -- ZOPYX Ltd. Co KG \ ZOPYX Friends Charlottenstr. 37/1 \ The experts for your Python, Zope and D-72070 Tübingen \ Plone projects www.zopyx.com, i...@zopyx.com \ www.zopyx.de/friends, frie...@zopyx.de E-Publishing, Python, Zope Plone development, Consulting -- Jim Fulton ___ Zope-web maillist - Zope-web@zope.org http://mail.zope.org/mailman/listinfo/zope-web
Re: [ZWeb] Error Type, IOError, Error Value, [Errno 28] No space left on device
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jens Vagelpohl wrote: On Aug 6, 2009, at 09:43 , Andreas Jung wrote: Same issue this morning. This must be fixed soon - otherwise we can not proceed with the releases. I just looked and for some strange reason the ZODB has grown to over 30 GB, which means it more than doubled in size since the last full backup on August 2. Looking at the backup files from repozo it appears the massive growth happened over the last few days, those daily incrementals since August 2 range from 5-6 GB in size each. I have now freed up 3 GB by deleting the remaining delta files from the previous round of backups, but this may not be enough seeing how fast the database is growing. Something fishy is going on. Hi, this was caused by spamming 2 collector issues in member space, each holding a transcript of about 1 MB and more. I removed the issues, blocked the spamming user account and finally changed default roles for everything collector related to 'Manager'. Regards Michael - -- http://blog.d2m.at http://planetzope.org -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iD8DBQFKethml0uAvQJUKVYRAlC0AJ4tkdTGdMrG56mtZM46ASZJFIgNiACfUkjg 2yX+44OxRtMaZuur2D9sNqQ= =Wq/v -END PGP SIGNATURE- ___ Zope-web maillist - Zope-web@zope.org http://mail.zope.org/mailman/listinfo/zope-web
Re: [ZWeb] Error Type, IOError, Error Value, [Errno 28] No space left on device
Thanks! Jim On Thu, Aug 6, 2009 at 9:19 AM, Michael Haubenwallnermich...@d2m.at wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jens Vagelpohl wrote: On Aug 6, 2009, at 09:43 , Andreas Jung wrote: Same issue this morning. This must be fixed soon - otherwise we can not proceed with the releases. I just looked and for some strange reason the ZODB has grown to over 30 GB, which means it more than doubled in size since the last full backup on August 2. Looking at the backup files from repozo it appears the massive growth happened over the last few days, those daily incrementals since August 2 range from 5-6 GB in size each. I have now freed up 3 GB by deleting the remaining delta files from the previous round of backups, but this may not be enough seeing how fast the database is growing. Something fishy is going on. Hi, this was caused by spamming 2 collector issues in member space, each holding a transcript of about 1 MB and more. I removed the issues, blocked the spamming user account and finally changed default roles for everything collector related to 'Manager'. Regards Michael - -- http://blog.d2m.at http://planetzope.org -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iD8DBQFKethml0uAvQJUKVYRAlC0AJ4tkdTGdMrG56mtZM46ASZJFIgNiACfUkjg 2yX+44OxRtMaZuur2D9sNqQ= =Wq/v -END PGP SIGNATURE- ___ Zope-web maillist - zope-...@zope.org http://mail.zope.org/mailman/listinfo/zope-web -- Jim Fulton ___ Zope-web maillist - Zope-web@zope.org http://mail.zope.org/mailman/listinfo/zope-web
Re: [ZWeb] Error Type, IOError, Error Value, [Errno 28] No space left on device
That's pretty much along the lines I was thinking as well when I looked at it real quick this morning. +1 from me, fwiw. --Dave On Aug 6, 2009, at 11:24 AM, Jens Vagelpohl wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I just want to mention that we will run into space problems again tonight. The ZODB is 32 GB in size, and there's 2 GB space left on the drive. Any repozo run or pack at this point will fail. The only way out I can see involves the following: - delete all repozo backups - pack the database manually - delete Data.fs.old After that repozo will start with a fresh backup. Anyone have any better ideas? jens -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.8 (Darwin) iEYEARECAAYFAkp69Z4ACgkQRAx5nvEhZLIdzACggnNgk5wp7GIAO4aLX34zJcOD ULEAnjRxhcr9yb9Upc6952bv7eJPMpJD =RjRo -END PGP SIGNATURE- ___ Zope-web maillist - Zope-web@zope.org http://mail.zope.org/mailman/listinfo/zope-web Systems Administrator Zope Corp. 540-361-1722 da...@zope.com ___ Zope-web maillist - Zope-web@zope.org http://mail.zope.org/mailman/listinfo/zope-web
Re: [ZWeb] Error Type, IOError, Error Value, [Errno 28] No space left on device
Jens, We're doing this. (Actually, we're going to copy the backups to another disk first.) We're also going to upgrade the hardware and switch to using ZRS. Sorry, I should have acked sooner. Bad me. Bad bad me. Jim On Thu, Aug 6, 2009 at 11:24 AM, Jens Vagelpohlj...@dataflake.org wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I just want to mention that we will run into space problems again tonight. The ZODB is 32 GB in size, and there's 2 GB space left on the drive. Any repozo run or pack at this point will fail. The only way out I can see involves the following: - delete all repozo backups - pack the database manually - delete Data.fs.old After that repozo will start with a fresh backup. Anyone have any better ideas? jens -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.8 (Darwin) iEYEARECAAYFAkp69Z4ACgkQRAx5nvEhZLIdzACggnNgk5wp7GIAO4aLX34zJcOD ULEAnjRxhcr9yb9Upc6952bv7eJPMpJD =RjRo -END PGP SIGNATURE- -- Jim Fulton ___ Zope-web maillist - Zope-web@zope.org http://mail.zope.org/mailman/listinfo/zope-web
Re: [ZWeb] Error Type, IOError, Error Value, [Errno 28] No space left on device
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Aug 6, 2009, at 17:44 , Jim Fulton wrote: Jens, We're doing this. (Actually, we're going to copy the backups to another disk first.) We're also going to upgrade the hardware and switch to using ZRS. Sorry, I should have acked sooner. Bad me. Bad bad me. No worries, just means less on my plate ;-) jens P.S.: A bigger disk would really be handy as well. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.8 (Darwin) iEYEARECAAYFAkp6+1kACgkQRAx5nvEhZLKZNwCdFiRW2p1ZY6DXMlFPxLTDmFQY eFIAoKxk2VIfGQOs36U4Ie4xPwa/HDMh =zcPd -END PGP SIGNATURE- ___ Zope-web maillist - Zope-web@zope.org http://mail.zope.org/mailman/listinfo/zope-web
Re: [ZWeb] Error Type, IOError, Error Value, [Errno 28] No space left on device
I just looked and the transaction rates have fallen from about 200-300MB/hr to about 200-300K/hr. :) Jim On Thu, Aug 6, 2009 at 9:19 AM, Michael Haubenwallnermich...@d2m.at wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jens Vagelpohl wrote: On Aug 6, 2009, at 09:43 , Andreas Jung wrote: Same issue this morning. This must be fixed soon - otherwise we can not proceed with the releases. I just looked and for some strange reason the ZODB has grown to over 30 GB, which means it more than doubled in size since the last full backup on August 2. Looking at the backup files from repozo it appears the massive growth happened over the last few days, those daily incrementals since August 2 range from 5-6 GB in size each. I have now freed up 3 GB by deleting the remaining delta files from the previous round of backups, but this may not be enough seeing how fast the database is growing. Something fishy is going on. Hi, this was caused by spamming 2 collector issues in member space, each holding a transcript of about 1 MB and more. I removed the issues, blocked the spamming user account and finally changed default roles for everything collector related to 'Manager'. Regards Michael - -- http://blog.d2m.at http://planetzope.org -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iD8DBQFKethml0uAvQJUKVYRAlC0AJ4tkdTGdMrG56mtZM46ASZJFIgNiACfUkjg 2yX+44OxRtMaZuur2D9sNqQ= =Wq/v -END PGP SIGNATURE- ___ Zope-web maillist - zope-...@zope.org http://mail.zope.org/mailman/listinfo/zope-web -- Jim Fulton ___ Zope-web maillist - Zope-web@zope.org http://mail.zope.org/mailman/listinfo/zope-web
[ZWeb] Error Type, IOError, Error Value, [Errno 28] No space left on device
We run out of disk space for the zope.org site. Andreas begin:vcard fn:Andreas Jung n:Jung;Andreas org:ZOPYX Ltd. Co. KG adr;quoted-printable:;;Charlottenstr. 37/1;T=C3=BCbingen;;72070;Germany email;internet:i...@zopyx.com title:CEO tel;work:+49-7071-793376 tel;fax:+49-7071-7936840 tel;home:+49-7071-793257 x-mozilla-html:FALSE url:www.zopyx.com version:2.1 end:vcard ___ Zope-web maillist - Zope-web@zope.org http://mail.zope.org/mailman/listinfo/zope-web
Re: [ZWeb] Error Type, IOError, Error Value, [Errno 28] No space left on device
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Aug 3, 2009, at 18:01 , Andreas Jung wrote: We run out of disk space for the zope.org site. Does this error still occur? I see 12 GB free on that drive, but there's two other people who logged in before me, maybe they deleted something. jens -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.8 (Darwin) iEYEARECAAYFAkp3GmoACgkQRAx5nvEhZLKJ6QCgiRuvijMieeuzE266cY+DoBen VwIAoJSxTmoe5R5+c4czJaf3UcRSvjhn =Lqqi -END PGP SIGNATURE- ___ Zope-web maillist - Zope-web@zope.org http://mail.zope.org/mailman/listinfo/zope-web
Re: [ZWeb] Error Type, IOError, Error Value, [Errno 28] No space left on device
On 03.08.09 19:12, Jens Vagelpohl wrote: On Aug 3, 2009, at 18:01 , Andreas Jung wrote: We run out of disk space for the zope.org site. Does this error still occur? I see 12 GB free on that drive, but there's two other people who logged in before me, maybe they deleted something. Tnx, I can work again. Andreas begin:vcard fn:Andreas Jung n:Jung;Andreas org:ZOPYX Ltd. Co. KG adr;quoted-printable:;;Charlottenstr. 37/1;T=C3=BCbingen;;72070;Germany email;internet:i...@zopyx.com title:CEO tel;work:+49-7071-793376 tel;fax:+49-7071-7936840 tel;home:+49-7071-793257 x-mozilla-html:FALSE url:www.zopyx.com version:2.1 end:vcard ___ Zope-web maillist - Zope-web@zope.org http://mail.zope.org/mailman/listinfo/zope-web
Re: [ZWeb] Error Type, IOError, Error Value, [Errno 28] No space left on device
Joseph and I both cleaned off some things. I also modified the repozo backup cleanup script to only keep six days of backups, otherwise we would have run out of space again either in a week or the next time the database packed. The major space usage was the Data.fs, the Data.fs.old, and two full FS backups made by repozo. --Dave On Aug 3, 2009, at 1:12 PM, Jens Vagelpohl wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Aug 3, 2009, at 18:01 , Andreas Jung wrote: We run out of disk space for the zope.org site. Does this error still occur? I see 12 GB free on that drive, but there's two other people who logged in before me, maybe they deleted something. jens -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.8 (Darwin) iEYEARECAAYFAkp3GmoACgkQRAx5nvEhZLKJ6QCgiRuvijMieeuzE266cY+DoBen VwIAoJSxTmoe5R5+c4czJaf3UcRSvjhn =Lqqi -END PGP SIGNATURE- ___ Zope-web maillist - Zope-web@zope.org http://mail.zope.org/mailman/listinfo/zope-web Systems Administrator Zope Corp. 540-361-1722 da...@zope.com ___ Zope-web maillist - Zope-web@zope.org http://mail.zope.org/mailman/listinfo/zope-web
Re: [ZWeb] Error Type, IOError, Error Value, [Errno 28] No space left on device
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Aug 3, 2009, at 19:22 , David Lawson wrote: Joseph and I both cleaned off some things. I also modified the repozo backup cleanup script to only keep six days of backups, otherwise we would have run out of space again either in a week or the next time the database packed. The major space usage was the Data.fs, the Data.fs.old, and two full FS backups made by repozo. Alright, thanks for jumping onto it so quickly! jens -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.8 (Darwin) iEYEARECAAYFAkp3HUgACgkQRAx5nvEhZLLP6wCdH0owTDMFlWaVtdMFuYfhERYx z7oAn348M9nLuYuLg9NfqC23VBeYSLke =6HrL -END PGP SIGNATURE- ___ Zope-web maillist - Zope-web@zope.org http://mail.zope.org/mailman/listinfo/zope-web
Re: [ZWeb] Error Type, IOError, Error Value, [Errno 28] No space left on device
On 03.08.09 19:24, Jens Vagelpohl wrote: On Aug 3, 2009, at 19:22 , David Lawson wrote: Joseph and I both cleaned off some things. I also modified the repozo backup cleanup script to only keep six days of backups, otherwise we would have run out of space again either in a week or the next time the database packed. The major space usage was the Data.fs, the Data.fs.old, and two full FS backups made by repozo. Alright, thanks for jumping onto it so quickly! Right now zope.org is dead - all requests are hanging. Andreas begin:vcard fn:Andreas Jung n:Jung;Andreas org:ZOPYX Ltd. Co. KG adr;quoted-printable:;;Charlottenstr. 37/1;T=C3=BCbingen;;72070;Germany email;internet:i...@zopyx.com title:CEO tel;work:+49-7071-793376 tel;fax:+49-7071-7936840 tel;home:+49-7071-793257 x-mozilla-html:FALSE url:www.zopyx.com version:2.1 end:vcard ___ Zope-web maillist - Zope-web@zope.org http://mail.zope.org/mailman/listinfo/zope-web
Re: [ZWeb] Error Type, IOError, Error Value, [Errno 28] No space left on device
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Aug 3, 2009, at 19:25 , Andreas Jung wrote: On 03.08.09 19:24, Jens Vagelpohl wrote: On Aug 3, 2009, at 19:22 , David Lawson wrote: Joseph and I both cleaned off some things. I also modified the repozo backup cleanup script to only keep six days of backups, otherwise we would have run out of space again either in a week or the next time the database packed. The major space usage was the Data.fs, the Data.fs.old, and two full FS backups made by repozo. Alright, thanks for jumping onto it so quickly! Right now zope.org is dead - all requests are hanging. It works perfectly fine for me. jens -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.8 (Darwin) iEYEARECAAYFAkp3HkAACgkQRAx5nvEhZLIujwCdEvNmHBw03c8E6fc8Te1YaKdC u7QAnAkIA+njy09Zug9hscZYNzgbUJgR =Z1/f -END PGP SIGNATURE- ___ Zope-web maillist - Zope-web@zope.org http://mail.zope.org/mailman/listinfo/zope-web
Re: [ZWeb] Error Type, IOError, Error Value, [Errno 28] No space left on device
Sorry, I should have responded earlier. We're digging into it. Jim On Mon, Aug 3, 2009 at 1:31 PM, Andreas Jungli...@zopyx.com wrote: On 03.08.09 19:28, Jens Vagelpohl wrote: On Aug 3, 2009, at 19:25 , Andreas Jung wrote: On 03.08.09 19:24, Jens Vagelpohl wrote: On Aug 3, 2009, at 19:22 , David Lawson wrote: Joseph and I both cleaned off some things. I also modified the repozo backup cleanup script to only keep six days of backups, otherwise we would have run out of space again either in a week or the next time the database packed. The major space usage was the Data.fs, the Data.fs.old, and two full FS backups made by repozo. Alright, thanks for jumping onto it so quickly! Right now zope.org is dead - all requests are hanging. It works perfectly fine for me. *shrug* creating a new document took roughly 15 minutes. During that time all requests were hanging on my side. They terminated after the documented was created...anyway...old Plone crap. Andreas ___ Zope-web maillist - zope-...@zope.org http://mail.zope.org/mailman/listinfo/zope-web -- Jim Fulton ___ Zope-web maillist - Zope-web@zope.org http://mail.zope.org/mailman/listinfo/zope-web
