Re: [ZODB-Dev] Relstorage Blob support and Oracle
On Fri, Jun 10, 2011 at 7:17 AM, Shane Hathaway sh...@hathawaymix.org wrote: BTW, I'd like to release a final of RelStorage 1.5, but the #@$%$ Windows tests are failing on Windows XP (and probably other versions of Windows). I think the problem is caused by file descriptors being left open somewhere. If any Windows specialists would like to lend a hand and fix the tests on RelStorage trunk, it would sure be appreciated. Are you getting real test failures or only warnings from the cleanup module during test teardown? I've got myself a checkout of relstorage on Windows and running tests now. So far I only get warning messages like: error: uncaptured python exception, closing channel __main__.ZEOTestServer :28210 at 0x2c9e108 (type 'exceptions.WindowsError':[Error 32] Der Prozess kann nicht auf die Datei zugreifen, da sie von einem anderen Prozess verwendet wird: ' c:\\users\\hannosch\\appdata\\local\\temp\\BlobAdaptedFileStorageTests3nvzbg\\Data.fs' [C:\Python\Python26-Installer\lib\asyncore.py|read|76] [C:\Python\Python26-Installer\lib\asyncore.py|handle_read_event|411] [c:\users\hannosch\.buildout\eggs\zodb3-3.10.3-py2.6-win-amd64.egg\ZEO\tests\zeoserver.py|handle_accept|100][c:\users\hannosch\.buildout\eggs\zodb3-3.10.3-py2.6-win-amd64.egg\ZEO\tests\zeoserver.py|cleanup|34] [c:\users\hannosch\.buildout\eggs\zodb3-3.10.3-py2.6-win-amd64.egg\ZODB\FileStorage\FileStorage.py|cleanup|1253]) where the bit of German text translates to the process cannot access the file, as it is in use by another process. This looks like the typical problem, where some code opens a file without explicitly closing it. But instead relies on garbage collection to do the job during __del__ of the file object. That generally doesn't work well on Windows. Hanno ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Relstorage Blob support and Oracle
On 06/10/2011 02:53 AM, Hanno Schlichting wrote: This looks like the typical problem, where some code opens a file without explicitly closing it. But instead relies on garbage collection to do the job during __del__ of the file object. That generally doesn't work well on Windows. Yes, that's exactly the problem. What to do about it is not obvious. I can't just ignore it. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Relstorage Blob support and Oracle
On 06/10/2011 06:38 AM, Hanno Schlichting wrote: On Fri, Jun 10, 2011 at 1:03 PM, Hanno Schlichtingha...@hannosch.eu wrote: /me is still trying to get the postgres tests to run. I only just now found the relstorage/tests/readme instructions I got the postgres tests running now and get actual test failures. They all look related to blobs. For example: Error in test test_storeBlob_replace (relstorage.tests.test_blobhelper.BlobHelperTest) Traceback (most recent call last): File C:\Python\Python26-Installer\lib\unittest.py, line 289, in run self.tearDown() File c:\relstorage\relstorage\tests\test_blobhelper.py, line 45, in tearDown shutil.rmtree(self.blob_dir) File C:\Python\Python26-Installer\lib\shutil.py, line 216, in rmtree rmtree(fullname, ignore_errors, onerror) File C:\Python\Python26-Installer\lib\shutil.py, line 221, in rmtree onerror(os.remove, fullname, sys.exc_info()) File C:\Python\Python26-Installer\lib\shutil.py, line 219, in rmtree os.remove(fullname) WindowsError: [Error 5] Access denied: 'c:\\users\\hannosch\\appdata\\local\\temp\\tmpcg4taz\\0x01\\0x02hrzkyf.tmp-' There's a couple more happening in the separate blob cleanup threads. Is this what are you seeing as well? Yes. Here is the last part of the output from my buildbot testing ZODB trunk and ZODB 3.9 on Windows: Tests with errors: test_restoreBlob_shared (relstorage.tests.test_blobhelper.BlobHelperTest) test_storeBlob_replace (relstorage.tests.test_blobhelper.BlobHelperTest) test_storeBlob_shared (relstorage.tests.test_blobhelper.BlobHelperTest) test_storeBlob_unshared (relstorage.tests.test_blobhelper.BlobHelperTest) test_vote (relstorage.tests.test_blobhelper.BlobHelperTest) Total: 583 tests, 0 failures, 5 errors in 20 minutes 1.718 seconds. The tests of ZODB 3.8 produce the same errors except test_storeBlob_unshared (since unshared blobs don't work in ZODB 3.8). Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Relstorage Blob support and Oracle
On Fri, Jun 10, 2011 at 07:17, Shane Hathaway sh...@hathawaymix.org wrote: I see your thinking now. RelStorage will download multiple chunks from Oracle but will now only upload one chunk per blob to Oracle. If others want to do the same for PostgreSQL or MySQL, you've set the example of how to do it without disruption. Thank you for not changing the schema. I get a lot of complaints these days anytime the schema changes. I ask just one favor: please run some manual tests with multi-gigabyte blobs that don't fit in RAM. I've committed my refactor now. In testing I ran into some limitations of cx_Oracle; it uses the older (pre 10.1) OCI APIs that only allow offsets and sizes of up to 4GB (e.g. using 32-bit unsigned integers). Moreover, it's naive use of signed integers instead of unsigned ones means that on 32-bit platforms the largest *LOB write and read offsets it can handle is 2GB - 1 byte (sys.maxint). I've seen that the author is already aware that there is a newer API that upgrades the offset and size types to 64-bit unsigned integers, and I've asked him what the status is of cx_Oracle supporting that. In the meantime, to answer your request for a favour: Uploading and downloading blobs within these limits worked great and memory usage wasn't impacted at all. For us these limits are absolutely not a problem. Although our Oracle setup is performing very nicely when it comes to BLOB reading and writing (multiple MBs per second), pulling a BLOB of several GB to a file in your local non-shared blob cache before serving is going to take too long anyway, and you really want to use a shared blob storage for that instead. I've only tested the Oracle blob upload and download methods in the mover, since that's what changed in this commit. I didn't run the full test suite as I lacked the required SYS access on the oracle cluster and didn't have time to set up a local Oracle database. If you could run the test suite against your Oracle setup that'd be great. :-) Next I'll look into adding PostgreSQL support too; it has a nicer API still in that it let's you specify files directly to upload from or download to. -- Martijn Pieters ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] ZODB book now has automated API reference
Hi, work continues on the ZODB book project. We just added API docs (thanks to Dag Odenhall). http://zodb.readthedocs.org/ There is enough material now for people to consider helping us in reviewing or start suggesting more advanced topics (an article about conflict resolution is in the works). If you have tips or simple scripts to work with the ZODB it would be great to have them. Any participation will be greatly appreciated, even if it's just thoughts about the project. Thanks, Carlos de la Guardia ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB book now has automated API reference
One more thing: forgot to say that you can fork the project on Github and I can integrate your pull requests easily! https://github.com/cguardia/ZODB-Documentation Thanks, Carlos de la Guardia On Fri, Jun 10, 2011 at 1:58 PM, Carlos de la Guardia carlos.delaguar...@gmail.com wrote: Hi, work continues on the ZODB book project. We just added API docs (thanks to Dag Odenhall). http://zodb.readthedocs.org/ There is enough material now for people to consider helping us in reviewing or start suggesting more advanced topics (an article about conflict resolution is in the works). If you have tips or simple scripts to work with the ZODB it would be great to have them. Any participation will be greatly appreciated, even if it's just thoughts about the project. Thanks, Carlos de la Guardia ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB book now has automated API reference
On 6/10/11 6:32 PM, Carlos de la Guardia wrote: One more thing: forgot to say that you can fork the project on Github and I can integrate your pull requests easily! https://github.com/cguardia/ZODB-Documentation Thanks, Carlos de la Guardia Do we need to do anything to manage copyright ownership so that we end up with a text that can be printed at some point if we want? David -- David Glick Web Developer davidgl...@groundwire.org 206.286.1235x32 GiveBIG is coming! Give to Groundwire on June 23 and The Seattle Foundation will stretch your money! http://www.groundwire.org/blog/give-big-is-coming ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB book now has automated API reference
Do we need to do anything to manage copyright ownership so that we end up with a text that can be printed at some point if we want? David That is the goal from the beginning. We should talk to Alan Runyan about this, but I guess the idea is to assign copyright to the Zope Foundation. Carlos de la Guardia ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
