Re: [ZODB-Dev] Problems in ZEO pack in 3.9.x?
On Mon, Apr 26, 2010 at 7:44 PM, Jim Fulton j...@zope.com wrote: Hm. I don't know if it was intentional to ignore POSKeyErrors in the old pack code. It seems like a bad idea to me. Yep, I was wondering if that was a conscious design choice or just accidental behavior. What do folks think about this? Should missing records be ignored? Or should the missing record cause the pack (or maybe just GC) to fail? Mmh, I think having the pack succeed would be nice. It can sometimes take a while until you can fix those PosKeyErrors. Not everyone has the skill to do that. Preventing the ZODB from growing indefinitely during that time would be nice. But doing GC on an inconsistent state is probably a bad idea. Hanno ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] [ZODB 3.8/Blob] Error in rename_or_copy_blob()
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Could this be related to https://bugs.launchpad.net/zodb/+bug/224169 ? Andreas Jim Fulton wrote: On Mon, Apr 26, 2010 at 6:34 AM, Andreas Jung li...@zopyx.com wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi there, a customer site is running Plone 3.3 w/ plone.app.blob. While creating a 'File' instance we get the following error: 2010-04-26T13:17:07 ERROR Zope.SiteErrorLog 1272277027.360.157502129415 http://www.deisa.eu/test-project/conversions/@@generate/generate Traceback (innermost last): Module ZPublisher.Publish, line 125, in publish Module Zope2.App.startup, line 238, in commit Module transaction._manager, line 93, in commit Module transaction._transaction, line 325, in commit Module transaction._transaction, line 432, in _commitResources Module ZODB.Connection, line 762, in tpc_finish Module ZEO.ClientStorage, line 1088, in tpc_finish Module ZEO.ClientStorage, line 1134, in _update_cache Module ZODB.blob, line 772, in rename_or_copy_blob IOError: [Errno 13] Permission denied: '/home/plone/Plone/zeocluster/var/blobstorage/0x00/0x00/0x00/0x00/0x00/0x07/0x4c/0xf0/0x0385b9291ea836aa.blob' The Plone processes is running as user 'plone' (nothing related to setuid/effective-user) and all files and directories are owned by 'plone'. I could even create a new file from the debugger inside rename_or_copy_blob() inside the /home/plone/Plone/zeocluster/var/blobstorage/0x00/0x00/0x00/0x00/0x00/0x07/0x4c/0xf0 directory. The related code in blob.py is: 759 def rename_or_copy_blob(f1, f2, chmod=True): 760 Try to rename f1 to f2, fallback to copy. 761 762 Under certain conditions a rename might not work, e.g. because the target 763 directory is on a different partition. In this case we try to copy the 764 data and remove the old file afterwards. 765 766 767 try: 768 os.rename(f1, f2) 769 except OSError: 770 copied(Copied blob file %r to %r., f1, f2) 771 file1 = open(f1, 'rb') 772 file2 = open(f2, 'wb') ^^^ 773 try: 774 utils.cp(file1, file2) 775 finally: 776 file1.close() 777 file2.close() 778 remove_committed(f1) 779 if chmod: 780 os.chmod(f2, stat.S_IREAD) Is this a known problem No. and how can this be fixed. I have no idea, and have no idea how to reproduce it. Jim - -- ZOPYX Limited | zopyx group Charlottenstr. 37/1 | The full-service network for Zope Plone D-72070 Tübingen| Produce Publish www.zopyx.com | www.produce-and-publish.com - E-Publishing, Python, Zope Plone development, Consulting -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkvWvQYACgkQCJIWIbr9KYzm/ACggwZqZ9A0ZLIp8Bn7KhrHijVw FMsAn1cUdMobgNBVnSMhX05sZC2lIDjn =iKs1 -END PGP SIGNATURE- attachment: lists.vcf___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Problems in ZEO pack in 3.9.x?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 4/27/10 12:29 , Hanno Schlichting wrote: On Mon, Apr 26, 2010 at 7:44 PM, Jim Fulton j...@zope.com wrote: What do folks think about this? Should missing records be ignored? Or should the missing record cause the pack (or maybe just GC) to fail? Mmh, I think having the pack succeed would be nice. It can sometimes take a while until you can fix those PosKeyErrors. Not everyone has the skill to do that. Preventing the ZODB from growing indefinitely during that time would be nice. But doing GC on an inconsistent state is probably a bad idea. IMHO the script should default to failing if the data is inconsistent. Ignoring the error should be a conscious gesture, the user should have to supply some flag like --ignore-errors if inconsistencies should really be ignored. jens -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.8 (Darwin) iEYEARECAAYFAkvWvXkACgkQRAx5nvEhZLJqSQCcCoLaj1VvEiV/1TF2rQQzAQxX iMsAniKcV5iWx3W7ePyJ0JrZBAVIVFx8 =LzxX -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] [ZODB 3.8/Blob] Error in rename_or_copy_blob()
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 For the logs: the target file existed and was not writeable for the 'plone' user (only readable) - not sure if this is intentional an issue with the umask of the system (0002). As workaround I created a money patch changing the permissions of the target file before actually trying to write it back. Andreas Andreas Jung wrote: Could this be related to https://bugs.launchpad.net/zodb/+bug/224169 ? Andreas Jim Fulton wrote: On Mon, Apr 26, 2010 at 6:34 AM, Andreas Jung li...@zopyx.com wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi there, a customer site is running Plone 3.3 w/ plone.app.blob. While creating a 'File' instance we get the following error: 2010-04-26T13:17:07 ERROR Zope.SiteErrorLog 1272277027.360.157502129415 http://www.deisa.eu/test-project/conversions/@@generate/generate Traceback (innermost last): Module ZPublisher.Publish, line 125, in publish Module Zope2.App.startup, line 238, in commit Module transaction._manager, line 93, in commit Module transaction._transaction, line 325, in commit Module transaction._transaction, line 432, in _commitResources Module ZODB.Connection, line 762, in tpc_finish Module ZEO.ClientStorage, line 1088, in tpc_finish Module ZEO.ClientStorage, line 1134, in _update_cache Module ZODB.blob, line 772, in rename_or_copy_blob IOError: [Errno 13] Permission denied: '/home/plone/Plone/zeocluster/var/blobstorage/0x00/0x00/0x00/0x00/0x00/0x07/0x4c/0xf0/0x0385b9291ea836aa.blob' The Plone processes is running as user 'plone' (nothing related to setuid/effective-user) and all files and directories are owned by 'plone'. I could even create a new file from the debugger inside rename_or_copy_blob() inside the /home/plone/Plone/zeocluster/var/blobstorage/0x00/0x00/0x00/0x00/0x00/0x07/0x4c/0xf0 directory. The related code in blob.py is: 759 def rename_or_copy_blob(f1, f2, chmod=True): 760 Try to rename f1 to f2, fallback to copy. 761 762 Under certain conditions a rename might not work, e.g. because the target 763 directory is on a different partition. In this case we try to copy the 764 data and remove the old file afterwards. 765 766 767 try: 768 os.rename(f1, f2) 769 except OSError: 770 copied(Copied blob file %r to %r., f1, f2) 771 file1 = open(f1, 'rb') 772 file2 = open(f2, 'wb') ^^^ 773 try: 774 utils.cp(file1, file2) 775 finally: 776 file1.close() 777 file2.close() 778 remove_committed(f1) 779 if chmod: 780 os.chmod(f2, stat.S_IREAD) Is this a known problem No. and how can this be fixed. I have no idea, and have no idea how to reproduce it. Jim - ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev - -- ZOPYX Limited | zopyx group Charlottenstr. 37/1 | The full-service network for Zope Plone D-72070 Tübingen| Produce Publish www.zopyx.com | www.produce-and-publish.com - E-Publishing, Python, Zope Plone development, Consulting -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkvW72QACgkQCJIWIbr9KYwGtACePGxR3qHejwAHjVUEIZ1W1IHw hU4AoNAiORDVVr9sCCdSuqFrywhnIsS4 =Ei/R -END PGP SIGNATURE- attachment: lists.vcf___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Problems in ZEO pack in 3.9.x?
On Tue, Apr 27, 2010 at 6:29 AM, Hanno Schlichting ha...@hannosch.eu wrote: On Mon, Apr 26, 2010 at 7:44 PM, Jim Fulton j...@zope.com wrote: Hm. I don't know if it was intentional to ignore POSKeyErrors in the old pack code. It seems like a bad idea to me. Yep, I was wondering if that was a conscious design choice or just accidental behavior. What do folks think about this? Should missing records be ignored? Or should the missing record cause the pack (or maybe just GC) to fail? Mmh, I think having the pack succeed would be nice. It can sometimes take a while until you can fix those PosKeyErrors. Not everyone has the skill to do that. Preventing the ZODB from growing indefinitely during that time would be nice. But doing GC on an inconsistent state is probably a bad idea. Then I think the current behavior is correct. You can now disable GC using the pack-gc option: filestorage pack-gc false ... which will allow you to pack away old revisions while you research the POSKeyError issue. Jim -- Jim Fulton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Problems in ZEO pack in 3.9.x?
On Tue, Apr 27, 2010 at 4:08 PM, Jim Fulton j...@zope.com wrote: On Tue, Apr 27, 2010 at 6:29 AM, Hanno Schlichting ha...@hannosch.eu wrote: But doing GC on an inconsistent state is probably a bad idea. Then I think the current behavior is correct. You can now disable GC using the pack-gc option: filestorage pack-gc false ... which will allow you to pack away old revisions while you research the POSKeyError issue. Ok, thanks for the clarification. I'm looking into zc.zodbdgc now :) Hanno ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
