Re: [ZODB-Dev] RelStorage: ConflictError causes deadlock without RELSTORAGE_ABORT_EARLY
On 07/19/2014 05:22 PM, Sean Upton wrote: Folks, I have been dealing with locking issues and RelStorage for the past few days, and want to verify what I believe is a bug: without RELSTORAGE_ABORT_EARLY set in environment, tpc_vote() could potentially leave an ILocker adapter setting an RDBMS table lock (originally set in either tpc_begin() or in _prepare_tid()) that does not get removed. Hi Sean, Sorry to take so long to reply. Life is busy! That variable only has an effect on transactions that fail inside tpc_vote(). When tpc_vote() fails, the code expects something to call tpc_abort() later, releasing the commit lock. However, I wonder whether there are cases where the transaction package will only call abort(), not tpc_abort(). If so, RelStorage probably needs to release the lock in abort(). Shane ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage, postgresql 9.2 and zc.async (plone.app.async)
On 03/13/2014 09:06 AM, Simone Deponti wrote: 1. Certain tables remain locked and automatic cleanup functions (e.g. AUTOVACUUM) can't properly run Are you using the poll-interval option? That option tells RelStorage to leave the transaction open. In practice, it's just wrong (although it doesn't corrupt any data.) I think we should remove that option. Shane ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Optimizing RelStorage Packing for large DBs
On 11/15/2013 06:01 PM, Jens W. Klein wrote: The idea is simple: - iterate over all transactions starting with the lowest transaction id (tid) - for each transaction load the object states connected with tid - for each state fetch its outgoing references and fill a table where all incoming references of an object are stored as an array. if an state has no references write it anyway to the table with empty outgoing references I would describe the RelStorage packing algorithm with the same words, but since you reimplemented the algorithm from scratch, you found a more optimal implementation for your database. Good work! Shane ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage 1.5.1 and persistent 4.0.5+ Incompatible (patch)
On 03/07/2013 10:48 AM, jason.mad...@nextthought.com wrote: On Mar 7, 2013, at 11:35, Sean Upton sdup...@gmail.com wrote: On Thu, Mar 7, 2013 at 7:31 AM, jason.mad...@nextthought.com wrote: I only spotted two uses of this assumption in RelStrorage, the above-mentioned `_prepare_tid`, plus `pack`. The following simple patch to change those places to use `raw` makes our own internal tests (python2.7, MySQL) pass. Why not fork https://github.com/zodb/relstorage and submit a pull request? Because I didn't realize that repository existed :) I will do so, thanks. On that note, though, the PyPI page still links to the SVN repository at http://svn.zope.org/relstorage/trunk/ (which is also what comes up in a Google search), and that repository still has all its contents; it's missing the 'MOVED_TO_GITHUB' file that's commonly there when the project has been moved (e.g., [1]). With a bit of searching I found the announcement on this list that development had been moved[2], but to a first glance it looks like SVN is still the place to be. If the move is complete, maybe it would be good to replace the SVN contents with the MOVED_TO_GITHUB pointer? Thanks for the patch and suggestion. I intend to handle RelStorage pull requests during/around PyCon next week. :-) Shane ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Relstorage and over growing database.
On 02/06/2013 04:23 AM, Jürgen Herrmann wrote: I think this is not entirely correct. I ran in to problems serveral times when new_oid was emptied! Maybe Shane can confirm this? (results in read conlfict errors) Ah, that's true. You do need to replicate new_oid. Then I'd like to talk a little about my current relstorage setup here: It's backed by mysql, history-preserving setup. Recently one of our DBs started to grow very quickly and it's object_state.ibd (InnoDB) file is just over 86GB as of today. Packing now fails due to mysql not being able to complete sorts in the object_ref table. object_ref is also very big (36GB MYD file, 25GB MYI file). I took a backup of the DB and let zodbconvert convert it back to a FileStorage, the resulting file is 6GB (!). I will pack it and see how big it is then. I will also investigate how big on disk this DB would be when stored in postgresql. This situation poses another problem for us: using zodbconvert to convert this mess to a Filestorage tages just over an hour when writing to a ramdisk. I suspect converting to postgres will take more than 10 hours, which is unacceptable for us as this is a live database an cannot be offline for more than 2-3 hours in the nicht. So we will have to investigate into a special zodbconvert that uses a two step process: 1. import transactions to new storage from a mysql db backup 2. import rest of transactions that occurred after the backup was made from the live database (which is offline for that time of course) looking at zodbconvert using copyTransactionsFrom() i thnik this should be possible but up to now i did non investigate furhter. maybe shane could confirm this? maybe this could also be transformed into a neat way of getting incremental backups out of zodbs in general? Yes, that could work. As for MySQL growing tables without bounds... well, that wouldn't surprise me very much. Shane ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Relstorage and over growing database.
On 02/07/2013 01:54 PM, Jürgen Herrmann wrote: Am 07.02.2013 21:18, schrieb Jürgen Herrmann: I know that's entirely not your fault but may be worth mentioning in the docs. Relstorage with MySQL works *very* well for DB sizes 5GB or so, above that - not so much :/ Also for the docs: on disk Restorage/MySQL uses 4x the size of a FileStorage with same contents. As packing tables are filled this grows by another factor of ~2. If you don't pack very regularly you might up ending in DBs that donb't permit packing anymore because of the big size very quickly. I suspect there are ways to fix all of that in the MySQL configuration. Like any SQL database, MySQL needs tuning as it grows. Meanwhile, FileStorage doesn't really have any knobs, and it always stores in a fairly optimal way, so it's easier to use. FileStorage has a couple of issues that often drive people to RelStorage: (1) the on-disk format is unique to FileStorage, and there aren't many tools available for analyzing and fixing a broken Data.fs. (2) FileStorage only supports multiple clients through ZEO, which has relatively high latency. If these issues don't impact you, then FileStorage is clearly the better choice for you. Shane ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Relstorage and over growing database.
On 02/01/2013 09:08 PM, Juan A. Diaz wrote: Reading the some comments [0] in the code (relstorage/adapters/schema.py) I could see that the object_ref database is uses during the packing, then the question is, in a history-preserving database there is something that we could do to decrease the size of that table? could be safe truncate that table? We want to move to a storage with history-free, but for now we are looking some options and actions to perform on production without the object_ref is essentially a cache of object_state, and object_refs_added is a list of what's in that cache. Therefore you can freely truncate object_ref as long as you also truncate object_refs_added. Don't truncate them during packing, though. When we realize that the size of the database was over growing we start to make daily packs but now after read this comment I start to think that that could be now part of the problem, could be? Normally in a normal day the DB grows like 2.2GB, but after make a pack the DB size is decrease clouse to 1.5GB or 2GB. If your database grows by 2.2 GB per day, it's not surprising that the database is 15 GB. With drive and RAM sizes today, 15 GB doesn't sound like a problem to me... unless it's on a Raspberry Pi. :-) Shane ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage, PostgreSQL backup method comparison
On 12/26/2012 10:43 AM, Sean Upton wrote: For cron job RelStorage backups (databse not including blobs backed-up seperately, using a PostgreSQL 9.0.x backend), I use both zodbconvert to save FileStorage copies of my database, and pgdump for low-level binary dumps (pg_restore custom format, preserving postgres OIDs). bzip2-compressed, the pgdump backups are always ~2.5 times the size versus the compressed FileStorage -- this puzzles me. I'm using something in my bash backup script that looks like: $PGDUMP -Fc -o -h $SOCKET_HOST -p $SOCKET_PORT $dbname | bzip2 -c - $DESTDIR/pgdump-$dbname-$DATESTAMP.bz2 One database that backs up to 45MB bz2-compressed FileStorage file equates to a 123MB bz2-compressed pgdump custom-format file. I would expect such a ratio in running size, but not in compressed backups of similar data. Generally, I'm wondering, for the same data, what it is that makes my high-level FileStorage dump so much smaller in comparison to the lower-level pgdump alternative? Anyone with hunches or PostgreSQL kung-fu to add insight? My guess is the Postgres blob backup format is inefficient. Also, you are probably backing up the tables used for packing. You might want to use the -T option to exclude the pack tables, but then you'll have to create empty pack tables when restoring. Side-note: the zodbconvert script seems a perfectly viable mechanism for ZODB backup (regardless of whether one uses a RelStorage backend), but I am not sure if anyone else does this. I agree that zodbconvert is a good way to back up the database, although it might not be as fast as pg_dump. Shane ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZEO invalidation message transaction-inbound or outbound
On 12/01/2012 11:22 AM, Andreas Jung wrote: Jim Fulton wrote: On Fri, Nov 30, 2012 at 1:37 AM, Andreas Jung li...@zopyx.com wrote: a customer made the observation that that ZEO clients became inconsistent after some time (large CMF-based application running on Zope 2.12 afaik). Customer made some investigation and noticed that the ZEO invalidations have been queued (in some cases for hours). I can't imagine invalidations being queued for many seconds, much less hours without some serious network pathology. How did they come to this conclusion? I am only asking on customer behalf. This was their observation. Did the customer mean to say that the clients becomes stale (not inconsistent)? I've seen ZEO clients become stale due to network instability. The clients caught up the moment they changed something. This was years ago, though. Shane ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] RelStorage moved to Github
By popular request, I've moved RelStorage development from svn.zope.org to Github [1]. This should make it easier for the ZODB developer community to contribute and submit issues. Happy hacking! Also, I've created a zodb community on github. If you'd like to move or create your ZODB-centric project there, just ask. [1] https://github.com/zodb/relstorage Shane ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Storm/ZEO deadlocks (was Re: [Zope-dev] [announce] NEO 1.0 - scalable and redundant storage for ZODB)
On 08/30/2012 10:14 AM, Marius Gedminas wrote: On Wed, Aug 29, 2012 at 06:30:50AM -0400, Jim Fulton wrote: On Wed, Aug 29, 2012 at 2:29 AM, Marius Gedminas mar...@gedmin.as wrote: On Tue, Aug 28, 2012 at 06:31:05PM +0200, Vincent Pelletier wrote: On Tue, 28 Aug 2012 16:31:20 +0200, Martijn Pieters m...@zopatista.com wrote : Anything else different? Did you make any performance comparisons between RelStorage and NEO? I believe the main difference compared to all other ZODB Storage implementation is the finer-grained locking scheme: in all storage implementations I know, there is a database-level lock during the entire second phase of 2PC, whereas in NEO transactions are serialised only when they alter a common set of objects. This could be a compelling point. I've seen deadlocks in an app that tried to use both ZEO and PostgreSQL via the Storm ORM. (The thread holding the ZEO commit lock was blocked waiting for the PostgreSQL commit to finish, while the PostgreSQL server was waiting for some other transaction to either commit or abort -- and that other transaction couldn't proceed because it was waiting for the ZEO lock.) This sounds like an application/transaction configuration problem. *shrug* Here's the code to reproduce it: http://pastie.org/4617132 To avoid this sort of deadlock, you need to always commit in a a consistent order. You also need to configure ZEO (or NEO) to time-out transactions that take too long to finish the second phase. The deadlock happens in tpc_begin() in both threads, which is the first phase, AFAIU. AFAICS Thread #2 first performs tpc_begin() for ClientStorage and takes the ZEO commit lock. Then it enters tpc_begin() for Storm's StoreDataManager and blocks waiting for a response from PostgreSQL -- which is delayed because the PostgreSQL server is waiting to see if the other thread, Thread #1, will commit or abort _its_ transaction, which is conflicting with the one from Thread #2. Meanwhile Thread #1 is blocked in ZODB's tpc_begin(), trying to acquire the ZEO commit lock held by Thread #2. So thread 1 acquires in this order: 1. PostgreSQL 2. ZEO Thread 2 acquires in this order: 1. ZEO 2. PostgreSQL SQL databases handle deadlocks by detecting and automatically rolling back transactions, while the transaction package expects all data managers to completely avoid deadlocks using the sortKey method. I haven't looked at the code, but I imagine Storm's StoreDataManager implements IDataManager. I wonder if StoreDataManager provides a consistent sortKey. The sortKey method must return a string (not an integer or other object) that is consistent yet different from all other participating data managers. Shane ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Relstorage memcache object lifetimes
On 08/14/2012 09:59 AM, Daniel Garcia wrote: I've noticed that over time the number of objects in memcached grows util memcached begins to evict old objects. I'm trying to figure out how to size the memcached layer. Since there is no lifetime on the objects they remain in the cache until they are evicted by memached. Would there be any drawbacks to adding a lifetime (say 24 hours) to objects? This way the number of objects in the cache would reach some stable number (given enough RAM). I wouldn't expect any stability issues. Shane ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Question about relstorage and pack-gc
On 07/12/2012 01:30 PM, Santi Camps wrote: My specific question is: if I disable pack-gc, can I safety empty object_ref table and free this space? Certainly. However, most of the 23 GB probably consists of blobs; blobs are not shown in the query results you posted. Shane ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Question about relstorage and pack-gc
On 07/13/2012 02:42 PM, Santi Camps wrote: On Fri, Jul 13, 2012 at 7:05 PM, Shane Hathaway sh...@hathawaymix.org mailto:sh...@hathawaymix.org wrote: On 07/12/2012 01:30 PM, Santi Camps wrote: My specific question is: if I disable pack-gc, can I safety empty object_ref table and free this space? Certainly. However, most of the 23 GB probably consists of blobs; blobs are not shown in the query results you posted. Shane Thanks for your answer. I haven't blobs inside ZODB, but truncating object_ref table has shrink the database to 12GB Great improvement, I prefer that config because in this application deletions are rare BTW, to prevent a future possible accident, you should also clear object_refs_added so that RelStorage will know to repopulate object_ref if you ever decide to re-enable pack-gc. Otherwise, re-enabling pack-gc could delete too much. Shane ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage breaks history tab
On 05/23/2012 09:27 AM, Chris Withers wrote: Okay, the issue appears to be that, in some circumstances, RelStorage is leaving the read connection with an open transaction that isn't rolled back. That is what RelStorage is designed to do when you set poll-interval. Does the bug go away when you set poll-interval to 0? Shane ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Relstorage and MSQL
On 05/08/2012 06:34 PM, Dylan Jay wrote: I know it's not all about money but if we were to sponsor development of microsoft sql server support for relstorage, is there someone who knows how and has an estimated cost and available time? I don't have enough cycles now to do it myself, but I will happily assist anyone who wants to take on this project. The code already has 3 mature implementations to serve as a reference. I would plan to spend at least as much time on testing as on development. Like any database, ZODB storage bugs are often expensive to debug, so you really need a reliable storage before you can safely build layers on top of it. Shane ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Unable to acquire commit lock from RelStorage
On 03/04/2012 04:16 PM, Chris Withers wrote: Hi Shane, What does this exception mean: Traceback (innermost last): Module ZPublisher.Publish, line 135, in publish Module Zope2.App.startup, line 291, in commit Module transaction._manager, line 89, in commit Module transaction._transaction, line 329, in commit Module transaction._transaction, line 426, in _commitResources Module ZODB.Connection, line 776, in tpc_vote Module relstorage.storage, line 783, in tpc_vote Module relstorage.storage, line 817, in _vote Module relstorage.storage, line 689, in _prepare_tid Module relstorage.adapters.locker, line 104, in hold_commit_lock StorageError: Unable to acquire commit lock What can cause it? What are the recommended solutions? That means something else is/was holding the commit lock. Find out what is holding it and why. Shane ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] where does 32it vs 64bit matter?
On 01/02/2012 11:54 PM, Chris Withers wrote: Good to know, but my concern is taking a system which is currently running 32-bit clients against RelStorage running on a MySQL on a 64-bit server and replacing the 32-bit clients with 64-bit clients. Is that going to cause any issues? That will not introduce any new data storage issues, no. Shane ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Removing undeleteable objects from ZODB
On 12/19/2011 02:50 AM, Eugene Morozov wrote: Is there some other way to remove them? I would remove all references to the broken objects, then let packing wipe the old objects away. For example: container = app['somefolder'] del container[obj_name] It's possible to replace objects in the manner you posted, but it's usually a bad idea. If you really must do it, you have to set the _p_serial to the _p_serial of the existing object, which you can get using app._p_jar._storage._load(oid). Shane ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Cached object error while upgrading Plone
On 11/21/2011 10:40 AM, Alexandru Plugaru wrote: Hello, I've got this error ( http://pastie.org/2898828 ) while upgrading from plone3 to plone4. The context in which that error happens is this: http://pastie.org/2898959 The only place I could find the error message was in cPersistence.c: http://svn.zope.org/*checkout*/ZODB/branches/3.9/src/persistent/cPersistence.c So I guess my question is how do I know if an object is cached? And maybe a theory of why this is happening. That kind of thing happens when a class changes from non-persistent to persistent or vice-versa. This may be caused by add-ons. When I do a major Plone upgrade, I first disable all add-ons temporarily and that usually does the trick. If that doesn't work, I suggest you get a list of all your add-ons and ask this question on a Plone list. Shane ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Mysql with RelStorage shows very bad results
On 11/17/2011 10:16 AM, Войнаровський Тарас wrote: Convert took around 2 hours and the speed is realy bad. I tried iterating through all objects, by using the findObjectsProviding of zope.app.generations and it took 600 seconds, in contrast to the 20 seconds with FileStorage. The cache works good though (nearly the same results at single object geting multiple times). Maybe mysql set up needs some configuration, that I'm not aware of? Some indexes maybe... I think you misunderstood: FileStorage is known to be faster than RelStorage (except when starting up a large database). However, FileStorage can not be used by multiple instances without ZEO. For a fair comparison, you should set up ZEO on the same hardware and see how it fares. Shane ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage - high ConflictError rate in newly converted DB
On 11/10/2011 05:32 PM, Darryl Dixon - Winterhouse Consulting wrote: * What is the situation with using multiple memcache instances (eg, one per machine - will this lead to cache incoherency - inconsistency between instances, etc? One memcache per client? That's fine as long as you don't set poll-interval. * What is the failure scenario for a memcache instance dying - is it handled gracefully by RelStorage or will there be hanging/blocking/tracebacks until it returns? That is a function of the memcache client module that you choose. The default choice is pylibmc_wrapper, a module in the relstorage package, which handles most memcache errors. I have not seen it hang, but anything that uses TCP introduces the possibility of hanging. You can easily make your own pylibmc wrapper module. * Will this potentially exacerbate the ancient ZEO-Client-Race condition that occurs with load-balancing when data changes? (To be clear, I am referring to when one Zope instance updates/commits some data, and then the client browser makes a new request which is directed to a second Zope instance, which has not yet received an invalidation message for the changed objects, and it returns, eg, a 404 to the client). That can happen if you set poll-interval to a nonzero value and you don't share memcache between instances. It can also happen if you misuse the read-only replica support that's coming in the next version. Otherwise, no. Shane ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Relstorage PostgreSQL and blobs garbage collection
On 11/10/2011 08:56 AM, Александр Неганов wrote: We had large history-free database with lot of deleted Blobs, so we packed and gc-ed it. Many rows from object_state as well as corresponding blob_chunk rows has been deleted, but postgresql large objects stays untouched. It seems that blob_chunk_delete_trigger is not executed in this case, but works fine when i manually delete rows from blob_chunk table. PostgreSQL problem? PostgreSQL server version is 9.0.4, RelStorage version is 1.5.0. Hmm, that trigger is essential for deleting the large objects. There is probably some special case in Postgres that RelStorage is accidentally hitting when it deletes the rows. It has always worked for me. Shane ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage - high ConflictError rate in newly converted DB
On 11/07/2011 03:15 PM, Darryl Dixon - Winterhouse Consulting wrote: Hi All, We have just converted a site with a handful of editors over from ZEO to RelStorage on Oracle and and the rate of ConflictErrors has gone through the roof (5 an hour average continuously for days). This seems very surprising to me. What options/changes should we look at configuring to lower this back down? We used to use ZEO with an invalidation-queue-size of 5000 and a client-cache of 1000. Currently we are using a very basic RelStorage configuration without any additional bits like memcache. Any guidance gratefully appreciated. I have a few thoughts: - You aren't using poll-interval, are you? Make sure that is disabled. - When you switched from ZEO to Oracle without turning on memcache, you lost the ZEO cache and didn't replace it with anything, so now loading objects takes longer, which may lead to more conflicts. I would try turning on memcache. - Do you have multiple storages / databases? Make sure each uses a different Oracle lock ID. Shane ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage with Oracle and separation of rights
On 10/30/2011 07:05 PM, Darryl Dixon - Winterhouse Consulting wrote: Hi All, Part of the setup of our Oracle RelStorage environment involves the DBAs wanting to separate ownership of the schema from the rights to actually use the schema. In other words, user A owns all the tables etc that RelStorage creates, but then when it comes to actually making use of them, the application (Zope) uses user B for the connection. Apparently this is a standard Oracle risk-mitigation strategy and the DBAs are quite firm on this requirement. The way they achieve this is by creating what I would call a 'shadow schema' for user B that consists of a big pile of synonyms and grants pointing back at the 'real' user A schema. This doesn't work with parts of RelStorage. For example, it seems that RelStorage will unilaterally try to 'CREATE TABLE pack_lock' even though that table already exists. When this code runs as user A, Oracle seems to treat this as a no-op and continues on its merry way. When this code runs as user B, Oracle throws a fit and complains that cx_Oracle.DatabaseError: ORA-00955: name is already used by an existing object. Because presumably for user B, 'pack_lock' already exists, but it is a *synonym* not an actual, for-real table. I suspect that other such situations may arise involving, eg Indexes or Sequences. Is there any straightforward way to resolve this? I think you're saying your DBAs want to be in charge of all DDL (Data Definition Language like CREATE TABLE) while apps can only use DML (Data Manipulation Language like SELECT, INSERT, UPDATE, and DELETE). Setting the create-schema option to false should do the trick. I wonder if you'll run into problems with OID management, since some DDL is required there, but other than that and creating the schema, everything RelStorage does with Oracle is (or should be) DML. BTW, I wonder why your DBAs bother with a shadow schema. They can simply revoke the permission to use specific DDL instead. Shane ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Debugging RelStorage hang-ups
On 10/20/2011 05:41 AM, Martijn Pieters wrote: On a test server with a Plone 4.1 upgrade of a client setup, we are experience regular lock-ups of the 2 instances we run. After a restart, after a few hours at least one of the instances will be waiting on Oracle to roll back: File /srv/test-plone4/eggs/RelStorage-1.5.0-py2.6.egg/relstorage/storage.py, line 1228, in poll_invalidations changes, new_polled_tid = self._restart_load_and_poll() File /srv/test-plone4/eggs/RelStorage-1.5.0-py2.6.egg/relstorage/storage.py, line 1202, in _restart_load_and_poll self._adapter.poller.poll_invalidations, prev, ignore_tid) File /srv/test-plone4/eggs/RelStorage-1.5.0-py2.6.egg/relstorage/storage.py, line 254, in _restart_load_and_call self._load_conn, self._load_cursor) File /srv/test-plone4/eggs/RelStorage-1.5.0-py2.6.egg/relstorage/adapters/oracle.py, line 322, in restart_load conn.rollback() I am a bit at a loss at where to start debugging this. Any hints from anyone? Some ideas: - Is Oracle running out of space somewhere, such as the undo/redo logs? - Do rollbacks in Oracle acquire some kind of lock? - Could RAC be the culprit? (Synchronous replication always has weird edge cases.) Shane ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zeo.memcache
On 10/09/2011 08:26 AM, Jim Fulton wrote:
On Sat, Oct 8, 2011 at 4:34 PM, Shane Hathawaysh...@hathawaymix.org wrote:
On 10/05/2011 11:40 AM, Pedro Ferreira wrote:
Hello all,
While doing some googling on ZEO + memcache I came across this:
https://github.com/eleddy/zeo.memcache
Has anybody ever tried it?
Having implemented memcache integration for RelStorage, I now know what
it takes to make a decent connection between memcache and ZODB. The
code at the link above does not look sufficient to me.
I could adapt the cache code in RelStorage for ZEO. I don't think it
would be very difficult. How many people would be interested in such a
thing?
This would be of broad interest!
Can you briefly describe the strategy? How do you arrange that
the client sees a consistent view of the current tid for a given
oid?
(Sorry for not replying sooner--I've been busy.)
As I see it, a cache of this type can take 2 basic approaches: it can
either store {oid: (state, tid)}, or it can store {(oid, tid): (state,
last_tid)}. The former approach is much simpler, but since memcache has
no transaction guarantees whatsoever, it would lead to consistency
errors. The latter approach makes it possible to avoid all consistency
errors even with memcache, but it requires interesting algorithms to
make efficient use of the cache. I chose the latter.
Given the choice to structure the cache as {(oid, tid): (state,
last_tid)}, a simple way to use the cache would be to get the last
committed tid from the database and use that tid for the lookup key.
This would be extremely efficient until the next commit, at which point
the entire cache would become irrelevant and would have to be rebuilt.
Therefore, most of the interesting parts of the cache code in RelStorage
are focused on simply choosing a good tid for the cache lookup operation.
It caches the following things in memcache:
1. A pair of checkpoints.
2. A state and last committed transaction ID for a given transaction ID
and object ID.
3. A commit counter.
The checkpoints are two arbitrary committed transaction IDs. Clients
can use any pair of committed transaction IDs as checkpoints (so it's OK
if the checkpoints disappear from the cache), but the cache is much more
efficient if all clients use the same checkpoints.
Each storage object holds a pair of delta mappings, where each delta
contains {oid: tid}. The deltas contain information about what objects
have changed since the checkpoints: delta0 lists the changes since
checkpoint0 and delta1 lists the changes between checkpoint1 and
checkpoint0. Within each transaction, the delta0 mapping must be updated
before reading from the database.
When retrieving an object, the cache tries to discover the object's
current tid by looking first in delta0. If it's there, then the cache
asks memcache for the object state at that exact tid. If not, the cache
asks memcache for the object state and tid at the current checkpoints.
It is not actually necessary to have 2 checkpoints. It could work with
more checkpoints or only 1 checkpoint, but if there were only 1, each
checkpoint shift would be equivalent to flushing the cache. With more
checkpoints, the cache would often query many keys for each read
operation. 2 checkpoints seems like a good balance.
I wrote more notes about the caching strategy here:
http://svn.zope.org/relstorage/trunk/notes/caching.txt
As I review all of this, I wonder at the moment why I chose to create
delta1. It seems like the system would work without it. I probably
added it because I thought it would improve cache efficiency, but today
I'd rather simplify as much as possible even at the cost of a little
theoretical efficiency.
The commit counter is not very related, but since I brought it up, I'll
explain it briefly: it serves as a way for clients to discover whether
the database has changed without actually reading anything from the
database. It is a counter rather than a transaction ID because that
choice avoids a race condition.
Shane
___
For more information about ZODB, see http://zodb.org/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zeo.memcache
On 10/12/2011 04:53 PM, Shane Hathaway wrote:
Given the choice to structure the cache as {(oid, tid): (state,
last_tid)}, a simple way to use the cache would be to get the last
committed tid from the database and use that tid for the lookup key.
This would be extremely efficient until the next commit, at which point
the entire cache would become irrelevant and would have to be rebuilt.
Therefore, most of the interesting parts of the cache code in RelStorage
are focused on simply choosing a good tid for the cache lookup operation.
Furthermore... anytime the cache chooses a tid other than the most
recently committed tid for the lookup operation, there is a risk that it
will choose a tid that is too old, leading to consistency errors. I have
searched deeply for any such holes and closed some obscure ones, but
it's important to acknowledge the risk.
(BTW, I worked with a client who saw many consistency errors that seemed
to be caused by the cache, but the problem turned out to be a major flaw
in Oracle's documentation of read only mode. The cache operated flawlessly.)
Shane
___
For more information about ZODB, see http://zodb.org/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage zodbconvert, ConflictError on Zope2 startup
On 07/18/2011 02:16 PM, Sean Upton wrote: On Fri, Jul 15, 2011 at 5:35 PM, Shane Hathawaysh...@hathawaymix.org wro= te: I am thinking of changing the memcache code to use a random per-database = key prefix. =A0If I had done that already, you would not have run into this problem, since clearing the database would cause RelStorage to use a new = key prefix. =A0Your old memcache data would have been ignored. +1, and it sounds like by consequence of design, this would be necessarily backward compatible with existing installations of 1.5.0? Yes, it should be fully compatible with existing installations. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage zodbconvert, ConflictError on Zope2 startup
On 07/14/2011 11:21 PM, Sean Upton wrote: On Thu, Jul 14, 2011 at 3:28 PM, Sean Uptonsdup...@gmail.com wrote: Full traceback: http://pastie.org/2214036 I am able to avoid this by commenting out cache-servers and cache-module-name in my zope.conf. Looks like the ConflictError at start-up is self-inflicted: I never stopped memcached to purge it of traces of the old/previous relstorage database/instance I was running in my tests. I was running zodbconvert with --clear, but not specifying use of memcached for zodbconvert destination storage ZConfig values, only for my zope2 instance configurations. Lesson learned: purge memcached of any previous data via restart if you plan to replace an existing RelStorage storage with zodbconvert --clear. This kind of thing has bitten a number of people. When moving databases around, it's easy to forget to clear memcached, and the effects are always weird. I am thinking of changing the memcache code to use a random per-database key prefix. If I had done that already, you would not have run into this problem, since clearing the database would cause RelStorage to use a new key prefix. Your old memcache data would have been ignored. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] POSKeyError
On 06/21/2011 07:18 AM, Erik Dahl wrote: I'm using relstorage 1.4.2 during a batch job I was running last night I got the following errors. 2011-06-21 07:55:02,664 WARNING relstorage: POSKeyError on oid 23916102: no tid found; Current transaction is 256466219826629358; Recent object tids: [] 2011-06-21 07:55:02,665 WARNING relstorage: POSKeyError on oid 23916103: no tid found; Current transaction is 256466219826629358; Recent object tids: [] 2011-06-21 08:00:19,850 WARNING relstorage: POSKeyError on oid 23917425: no tid found; Current transaction is 256466242546653525; Recent object tids: [] 2011-06-21 08:00:19,854 WARNING relstorage: POSKeyError on oid 23917423: no tid found; Current transaction is 256466242546653525; Recent object tids: [] 2011-06-21 08:00:19,865 WARNING relstorage: POSKeyError on oid 23917424: no tid found; Current transaction is 256466242546653525; Recent object tids: [] 2011-06-21 08:01:56,607 WARNING relstorage: POSKeyError on oid 23917839: no tid found; Current transaction is 256466249456206779; Recent object tids: [] 2011-06-21 08:01:56,616 WARNING relstorage: POSKeyError on oid 23917838: no tid found; Current transaction is 256466249456206779; Recent object tids: [] Whenever I see POSKeyError I get really nervous but I see these are marked as warnings. A quick look at the database seemed to show that it was ok. Should I be worried? I would investigate more before saying whether it's OK. Run a pre-pack with GC to populate the object_ref table, then find out what refers to those objects by exploring object_ref. (It would be really nice if someone invented a ZODB browser that's aware of the object_ref table, perhaps through some API that RelStorage has not yet exposed. object_ref makes it easy and fast to discover the lineage of any ZODB object.) Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage and MySQL 5.5
On 06/21/2011 08:22 AM, Philip K. Warren wrote: Does RelStorage work with MySQL 5.5? I saw a note posted back in February on this list that said that 5.5 is not yet supported, however I haven't seen any issues with this configuration. I have been able to successfully run the 1.4.2 and 1.5.0b2 unit tests against a 5.5.13 server on Linux/Mac OS X. That's great! I think RelStorage is ready to become more of a community project, and part of that is changing the meaning of supported. Until now, it has meant Shane is confident that this configuration will not lose your data, because he has tested it repeatedly in a few different ways. In the future, supported should mean several people have tested it, and not necessarily me. I think we need some kind of support matrix that gathers input from the community. I wonder if a wiki would be sufficient. Also, we need a public buildbot. I wonder if there is something like a GitHub for buildbots. I have a private buildbot, but my little server can't easily go public. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Relstorage Blob support and Oracle
On 06/12/2011 04:01 AM, Martijn Pieters wrote: How big a userbase is there for 1.5.0b2 on PostgreSQL? I know schema changes are painful, but in this case we'd only have people on the bleeding edge using a beta version to switch. I think we can come up with a little script that would move these people over to a blob_chunk table with oid columns. That shouldn't be a big problem. In any case, I'll work on a version that, like the Oracle blob story, stores ZODB blobs in chunks up to 2GB in PostgreSQL oid blobs instead. To be clear: I do not intend to touch the OBJECT_STATE table, that table can continue to use bytea as far as I am concerned. Good, because I'm actually pretty certain bytea is the right choice for object_state. I agree that bytea is less than ideal for blob_chunk, so I'm glad you're working to remedy that. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Relstorage Blob support and Oracle
On 06/10/2011 02:53 AM, Hanno Schlichting wrote: This looks like the typical problem, where some code opens a file without explicitly closing it. But instead relies on garbage collection to do the job during __del__ of the file object. That generally doesn't work well on Windows. Yes, that's exactly the problem. What to do about it is not obvious. I can't just ignore it. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Relstorage Blob support and Oracle
On 06/10/2011 06:38 AM, Hanno Schlichting wrote: On Fri, Jun 10, 2011 at 1:03 PM, Hanno Schlichtingha...@hannosch.eu wrote: /me is still trying to get the postgres tests to run. I only just now found the relstorage/tests/readme instructions I got the postgres tests running now and get actual test failures. They all look related to blobs. For example: Error in test test_storeBlob_replace (relstorage.tests.test_blobhelper.BlobHelperTest) Traceback (most recent call last): File C:\Python\Python26-Installer\lib\unittest.py, line 289, in run self.tearDown() File c:\relstorage\relstorage\tests\test_blobhelper.py, line 45, in tearDown shutil.rmtree(self.blob_dir) File C:\Python\Python26-Installer\lib\shutil.py, line 216, in rmtree rmtree(fullname, ignore_errors, onerror) File C:\Python\Python26-Installer\lib\shutil.py, line 221, in rmtree onerror(os.remove, fullname, sys.exc_info()) File C:\Python\Python26-Installer\lib\shutil.py, line 219, in rmtree os.remove(fullname) WindowsError: [Error 5] Access denied: 'c:\\users\\hannosch\\appdata\\local\\temp\\tmpcg4taz\\0x01\\0x02hrzkyf.tmp-' There's a couple more happening in the separate blob cleanup threads. Is this what are you seeing as well? Yes. Here is the last part of the output from my buildbot testing ZODB trunk and ZODB 3.9 on Windows: Tests with errors: test_restoreBlob_shared (relstorage.tests.test_blobhelper.BlobHelperTest) test_storeBlob_replace (relstorage.tests.test_blobhelper.BlobHelperTest) test_storeBlob_shared (relstorage.tests.test_blobhelper.BlobHelperTest) test_storeBlob_unshared (relstorage.tests.test_blobhelper.BlobHelperTest) test_vote (relstorage.tests.test_blobhelper.BlobHelperTest) Total: 583 tests, 0 failures, 5 errors in 20 minutes 1.718 seconds. The tests of ZODB 3.8 produce the same errors except test_storeBlob_unshared (since unshared blobs don't work in ZODB 3.8). Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Relstorage Blob support and Oracle
On 06/09/2011 06:32 AM, Martijn Pieters wrote: We've looked over the RelsStorage ZODB Blob storage implementation and came to the conclusion that the current use of blob chunks is unnecessary in Oracle when using the cx_Oracle database connector. Not splitting ZODB Blobs into chunks may have performance benefits on the Oracle side (especially on 11g) as Oracle can then use read-ahead more efficiently for the larger BLOBs while streaming these back to the client. Well, the ZODB blob interfaces make it impossible to actually stream blobs from the database. The entire blob has to be read into cache before the client can get it, so I'm not sure what you will gain. I'll be refactoring the blob support currently found in RelStorage 1.5b2 to just store one blob in one row, using the cx_Oracle LOB.read() and .write() methods, which let you read and write to blobs in chunks to avoid memory overload, and I'll reuse the blob chunk size to determine how much we read / write per iteration. I am currently leaning towards dropping the chunk column in the Oracle schema altogether; it certainly won't hold any other value than integer 0 after my refactor. Any reason to keep it, other than that others whom already are using 1.5b2 on an Oracle database will now have to drop that column again (or set it to a default value of 0 on insert)? Should the code support reading blob chunks still? No, please don't change the schema. I know all of the supported databases support a form of streaming, but they are all wildly different. I suggest that chunking is much simpler and has little overhead compared with streaming. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage: Pack no longer holding the commit lock for prolonged periods
On 05/23/2011 01:58 PM, Martijn Pieters wrote: I've cleared the last area where RelStorage packing could hold the transaction lock for long periods of time, during empty transaction deletion: http://zope3.pov.lt/trac/changeset/121783/relstorage/trunk During a large pack, this section could potentially hold the transaction lock for a long period of time and thus block an entire RelStorage Zope cluster. It now deletes transactions is batches of 1000 and releases the lock in between. Shane, I have only tested this against PostgreSQL and Oracle, but I am pretty confident the MySQL SQL syntax is correct. This has been run on the live Elkjøp Oracle cluster without problems. It looks very nice, thanks! I'll test it with my Buildbot. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Speeding up ZODB (was redis cache for RelStorage)
On 05/06/2011 06:22 AM, Pedro Ferreira wrote: But isn't RelStorage supposed be slower than FileStorage/ZEO? No, every measurement I've tried suggests RelStorage (with PostgreSQL or MySQL) is faster than ZEO on the same hardware. ZEO has certainly gotten faster lately, but RelStorage still seems to have the advantage AFAICT. OTOH, the speed difference is not dramatic. For many apps it's not even noticeable. But remember that throwing more caches at the problem isn't a solution. It's likely the way you store or query the data from the database that's not optimal. I agree, many things could be improved regarding the data structures we use. However, it is also true that we have a large number of objects that are rarely changed, and that there is no need to fetch from the DB if we can keep them in memory. It sounds like you primarily need a bigger and faster cache. If you want to make minimal changes to your setup, try increasing the size of your ZEO cache and store the ZEO cache on either a RAM disk (try mount -t tmpfs none /some/path) or a solid state disk. Remember that seek time is 5-10 ms with spinning drives, so putting a ZEO cache on a spinning drive can actually kill performance. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Speeding up ZODB (was redis cache for RelStorage)
On 05/06/2011 02:38 PM, Jim Fulton wrote: If there is memory pressure and you take away ram for a ram disk, then you're going to start swapping, which will give you other problems. In my experience, Linux moves pieces of the ZEO cache out of RAM long before it starts swapping much. I tried running ZODB tests off of a RAM disk created that way and got lots of strange failures. :( Hmm, ok. If that's still the case then SSD is a better option. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Speeding up ZODB (was redis cache for RelStorage)
On 05/06/2011 02:14 PM, Shane Hathaway wrote: However, there is a different class of problems that prefetching could help solve. Let's say you have pages with a lot of little pieces on it, such as a comment page with a profile image for every comment. It would be useful to tell ZODB to load all of the objects in a list before accessing them. For example: app._p_jar.load_all(comments) app._p_jar.load_all([comment.profile for comment in comments]) FWIW, a better API name would be activate_all. Code would look like app._p_jar.activate_all(my_objects). Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] redis cache for RelStorage
On 05/02/2011 09:23 PM, 潘俊勇 wrote: It seems that redis is much faster than memcached. Could we use redis as a cache for RelStroage? Are you having speed issues? I suspect either one is so fast that the speed of Redis or Memcached is irrelevant. If you want speed, minimize the latency of the *network*, and that means getting good network hardware. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] weird MySQL gone away error when viewing history tab on Script(Python) with RelStorage 1.4.2
On 05/03/2011 01:39 AM, Chris Withers wrote: On 27/04/2011 18:11, Shane Hathaway wrote: OperationalError: (2006, 'MySQL server has gone away') This is happening across at least two separate instances with separate storages. Any ideas? Maybe the query is killing MySQL, for whatever reason. I don't think so, since I'd then expect all the other storages served by that cluster to complain similarly. Not necessarily. RelStorage reconnects automatically. The reconnection is invisible unless the disconnect happens in the middle of a request, in which case RelStorage raises the error then automatically reconnects on the next request. However, it's literally just the thread handling this particular request that shows the error... Two ideas: - MySQL is dropping the connection after a long period of inactivity, then RelStorage doesn't notice it's dropped until it's too late. Connections that poll take defensive steps to prevent that, but maybe you're using a connection that does not poll. - A single MySQL thread is dying. (Is MySQL multithreaded?) Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] weird MySQL gone away error when viewing history tab on Script(Python) with RelStorage 1.4.2
On 04/27/2011 05:07 AM, Chris Withers wrote:
Hi Shane,
Attempting to view the /manage_change_history_page of a history-keeping
relstorage is giving me:
script statement failed: '\nSELECT 1 FROM current_object WHERE
zoid = %(oid)s\n'; parameters: {'oid': 1163686}
...and then:
Traceback (innermost last):
Module ZPublisher.Publish, line 127, in publish
Module ZPublisher.mapply, line 77, in mapply
Module ZPublisher.Publish, line 47, in call_object
Module Shared.DC.Scripts.Bindings, line 324, in __call__
Module Shared.DC.Scripts.Bindings, line 361, in _bindAndExec
Module App.special_dtml, line 185, in _exec
Module OFS.History, line 120, in manage_change_history
Module relstorage.storage, line 1011, in history
Module relstorage.adapters.dbiter, line 120, in iter_object_history
Module relstorage.adapters.scriptrunner, line 53, in run_script_stmt
Module MySQLdb.cursors, line 174, in execute
Module MySQLdb.connections, line 36, in defaulterrorhandler
OperationalError: (2006, 'MySQL server has gone away')
This is happening across at least two separate instances with separate
storages.
Any ideas?
Maybe the query is killing MySQL, for whatever reason. Are the MySQL
process IDs the same before and after this happens?
Shane
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] python path problem: ImportError when calling zodbconvert (FreeBSD 8.1)
On 04/24/2011 03:36 AM, Nikolaj Groenlund wrote: I guess this is a python path problem (on FreeBSD 8.1). Im trying to convert a Data.fs to Postgresql using zodbconvert. Ive downloaded RelStorage-1.5.0b2 and is running: /usr/local/Plone/Python-2.6/bin/python zodbconvert.py fstodb.conf , to use the version that Plone is running with. The error I get: Traceback (most recent call last): File zodbconvert.py, line 22, inmodule from persistent.TimeStamp import TimeStamp ImportError: No module named persistent.TimeStamp This is not a good way to run it. The error you got is only the first of many. Did you add RelStorage to your buildout.cfg? Once you did that, did you get a zodbconvert command built for you in /usr/local/Plone/zinstance/bin? Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] packing ZODB
On 03/31/2011 04:46 AM, Adam GROSZER wrote: After investigating FileStorage a bit, I found that GC runs on objects, but pack later by transactions. That means that if there's a bigger-ish transaction, we can't get rid of it until all of it's objects are GCed (or superseeded by newer states). No, but we can remove old object states from the big transaction and mark it as partially destroyed (that's what the packed flag is for) so that the transaction is no longer eligible for undo. Both FileStorage and RelStorage do that. Not much space is wasted. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Problem with RelStorage zodbpack
On 03/29/2011 09:16 PM, Erik Dahl wrote: Ok looked a little deeper. I think solution 2 is the way to go (ie clear the object_ref table from references that are in my range of non-packed transactions. Does that sound right? Statement would be: delete from object_ref where tid 255908476364112230; I think once this is done the code will refill the table and go on its way. Thoughts? I haven't read the whole thread, but I will say this: it is quite safe to remove rows from object_ref as long as you *also* remove the corresponding rows from object_refs_added. To be 100% safe, you could just clear both object_refs_added and object_ref before packing. IOW, the object_ref table serves as a cache between pack operations and the object_refs_added table indicates how much of the object_ref cache is filled. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Problem with RelStorage zodbpack
On 03/29/2011 07:39 PM, Erik Dahl wrote: I was running a pack and canceled so that I could reboot my box. After it came back up I tried to restart the pack and got this: [...] File /opt/zenoss/lib/python2.6/site-packages/RelStorage-1.4.2-py2.6.egg/relstorage/adapters/packundo.py, line 397, in _add_refs_for_tid self.runner.run_many(cursor, stmt, add_rows) [...] IntegrityError: (1062, Duplicate entry '255911127406517196-2714-893440' for key 'PRIMARY') Ah, I recognize this now. This is a minor bug that I believe has been fixed for 1.5.0. It only affects packing; it has zero impact on your application. The bug occurs when filling the object_ref table for packing. If you interrupt the pack operation while the object_ref table is being filled, the corresponding entry in object_refs_added will not be created and the next attempt to pack will try again and stumble in exactly this way. The bug has been solved by replacing entries in object_ref rather than just inserting them. The object_ref and object_refs_added tables are only used during packing. Perhaps the quickest thing to do is remove only the offending entries: delete from object_refs_added where tid = 255911127406517196; delete from object_ref where tid = 255911127406517196; A bit of trivia: that error message says that object 2714 in transaction 255911127406517196 has at least one direct reference to object 893440. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Plone 4.0.4 and Postgresql Encoding
On 03/28/2011 04:01 PM, Nikolaj Groenlund wrote: Im using Plone 4.0.4, Postgresql 9.0.3 and RelStorage 1.5.0b2. Currently Im using da_DK.ISO8859-1 encoding in PostgreSQL - would da_DK.UTF-8 be better since Plone is using UTF-8 internally? PS both Encoding, Collation and Ctype are set in Postgresql. Currently, RDBMS-level encoding parameters should not matter. All Unicode strings (such as transaction user name and description) are encoded in bytea fields, so the encoding is opaque to PostgreSQL. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Wrong blob returned in one of the zeo clients
On 03/02/2011 07:38 AM, Jim Fulton wrote: BTW, my sense is that when blobs were first implemented, the emphasis was on shared blob directories and the non-shared blob implementation suffered from neglect, which I at least tried to rectify in ZODB 3.9. That evolutionary path is evident and I think it worked out well enough. Even if it were possible to run a streaming-only blob storage (with no cache nor shared storage), I think it would be a bad idea anyway; it would put extra pressure on the storage. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] blobs missing with relstorage and small blob cache dir
On 02/28/2011 07:37 PM, Shane Hathaway wrote: On 02/28/2011 08:19 AM, Maurits van Rees wrote: I wonder if there is some code that mistakenly throws away the wrong blob; it should throw away the oldest one I'd think, and not the one that it just loaded. The workaround is probably just to not set the blob-cache-size this low. But I tried this low setting to work around a different problem, which I will report in a different post. This reveals how naive the blob cache management is in both RelStorage and ZEO. It is smart enough to use the atime of files (thus you should not disable atime on a filesystem that hosts a blob cache), but the cache management does not guarantee downloaded files are kept long enough to actually read them. Oops, I take it back after reviewing the blob code tonight. ZODB 3.9 and above is not naive about this issue. While ZODB 3.8 allows blob files to disappear at the wrong times, ZODB 3.9 has internal retry mechanisms that ensure blob files will be opened even when the blob cache size is too small. RelStorage can not fix this issue. If you want stable blob support, as Hanno said, you need ZODB 3.9 or above. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Wrong blob returned in one of the zeo clients
On 03/01/2011 08:05 AM, Maurits van Rees wrote: Op 01-03-11 13:54, Maurits van Rees schreef: Op 01-03-11 04:41, Shane Hathaway schreef: On 02/28/2011 09:29 AM, Maurits van Rees wrote: This is now also happening for some images during normal operation, so after any blob migration has been run and existing blob caches have been cleared. What happens is that somehow the file contents for 0xblahblah.blob in the blob caches can differ between two zeo clients, even when the image has not changed at all (only tried with images as it is easier to spot than with other files). Thanks for the report. Looking through the blob code, I noticed that when the system is using a blob cache, it was possible for a thread to get a partial file while another thread is downloading that file. I suspect this is the problem you saw. I have committed a fix. Thanks! I'll test this. No, I am still seeing it. I now see more clearly that the problem is that two files share the same blob name. I completely remove the blob cache and restart the zeo client. I visit image number 1 and get this file in the var/blobcache: 291/0x038c599924e70177.blob I then visit the second image and get served the same file! Now I remove this file from the blob cache, load the second image in a browser and this time I get the correct content for this image, using the same file name in the var/blobcache directory. Now of course loading the first image gives the wrong content. The .layout file is set to 'zeocache'. Any idea where the error might be? Could this be in plone.app.blob? Any chance that this is better in ZODB 3.9+? This appears to be a design bug in the zeocache blob layout: if a single transaction stores multiple blobs, and the OIDs of some of those blobs differ by a multiple of 997, then cache filenames will collide. I think the solution is to incorporate the OID, not just the transaction ID, in the cache filename. If I have diagnosed this correctly, this will have to be solved in ZEO as well. Nice catch! Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Wrong blob returned in one of the zeo clients
On 03/01/2011 01:10 PM, Jim Fulton wrote: On Tue, Mar 1, 2011 at 2:47 PM, Shane Hathawaysh...@hathawaymix.org wrote: Any idea where the error might be? Could this be in plone.app.blob? Any chance that this is better in ZODB 3.9+? This appears to be a design bug in the zeocache blob layout: if a single transaction stores multiple blobs, and the OIDs of some of those blobs differ by a multiple of 997, then cache filenames will collide. I'm at a loss as to why you would think this. The the blob file names use the full oid. Yes I see that now. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Wrong blob returned in one of the zeo clients
On 03/01/2011 08:05 AM, Maurits van Rees wrote: No, I am still seeing it. I now see more clearly that the problem is that two files share the same blob name. I completely remove the blob cache and restart the zeo client. I visit image number 1 and get this file in the var/blobcache: 291/0x038c599924e70177.blob Did you copy that filename exactly? Blob cache filenames should not have an 'x' in them. I would expect to see a '.' where you put the 'x'. I wonder if you somehow got the wrong cache layout, or if something is monkey-patching the cache layout. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Wrong blob returned in one of the zeo clients
On 03/01/2011 02:47 PM, Maurits van Rees wrote: That is pretty weird! I can understand a few duplicates because images are saved in a few sizes, but this is too much. To reiterate some versions: - Plone 3.3.5 - ZODB3 3.8.6-polling - RelStorage 1.5.0b1 - Zope2 2.10.12 - plone.app.blob 1.3 - plone.app.imaging 1.0.1 Blobs are stored in postgres with RelStorage. Layout of the blob cache dir is 'zeocache'. I reproduced your setup and I think I found it: shared-blob-dir false seems to be incompatible with ZODB 3.8, because the blob code in ZODB 3.8 constructs blob filenames in an inflexible way. IOW, BlobCacheLayout was never intended to work with ZODB 3.8. I expect my test runner to confirm this within the next couple of hours, then I'll make it so shared-blob-dir false is not available with ZODB 3.8. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Wrong blob returned in one of the zeo clients
On 03/01/2011 04:32 PM, Maurits van Rees wrote: As a workaround for getting blobstorage to work reliably in a relstorage setup without a shared blob dir: is there any way to store the blobs completely in postgres and have no shared or cached blob dir in the zeo clients? If that would work it should still be easier on memory usage. No. Although some documentation I've seen talks about streaming blobs from the database, neither RelStorage nor ZEO actually streams blobs. Blobs have to be fully downloaded to the filesystem before application code can see them. Without a cache directory, the files would have no reasonable place to go. OTOH, there are a lot of Plone 3 instances out there and converting them to Plone 4 is going to be a lot of work. As an interim step, maybe we should figure out what it would take to Plone 3 work with ZODB 3.9 or 3.10. Perhaps the incompatibility is shallow. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Wrong blob returned in one of the zeo clients
On 02/28/2011 09:29 AM, Maurits van Rees wrote: This is now also happening for some images during normal operation, so after any blob migration has been run and existing blob caches have been cleared. What happens is that somehow the file contents for 0xblahblah.blob in the blob caches can differ between two zeo clients, even when the image has not changed at all (only tried with images as it is easier to spot than with other files). I'll study this. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Pack complete: 80% of a 425GB database packed
On 02/28/2011 07:45 AM, Martijn Pieters wrote: Early this morning, after packing through the weekend, our somewhat overweight Oracle RelStorage ZODB pack was completed. I am still waiting for the final size from the customer DBAs, but before the pack this beast was occupying 425GB. The pack removed 80% of the object_state rows, so hopefully this is now reduced to a more manageable 85GB or so. Is 425GB a new record? ;-) Good work! I think we can remove holding the commit lock during the pack cleanup altogether. For starters, the object_ref and object_refs_added tables are only ever used during packing, so the packing lock is more than enough to protect these. That's correct. And unless I am missing something, we don't need to worry about the transactions table either. The rest of RelStorage ignores transactions where the packed flag has been set to TRUE, so deleting the packed and empty transactions from the table will never lead to deadlocks, right? It might lead to deadlocks, actually. Deadlocks occur when concurrent clients try to acquire locks in different orders. SQL databases handle deadlocks of that type by returning an error to all but one of the deadlocking clients, similar to the way ZODB deals with conflict errors. Also, in SQL, even reading a table usually acquires a lock, so it's really hard to prevent deadlocks without something like a commit lock. Or am I missing something about how RelStorage implements polling and caching? In any case, the history-free version doesn't need to lock at all, because it only ever touches the pack-specific tables in the pack cleanup. True. If the transaction table needs no lock protection either, we can get rid of the lock during cleanup altogether. I'd like to hear confirmation on this though. I wouldn't do that. Too risky. (Many database architects say that deadlocks are normal and should be handled in application code, but in my experience, deadlocks kill applications under load and I prefer to put in the effort to avoid them entirely if possible.) On a side note: I see that the history-preserving object_ref and object_refs_added deletion SQL statements have been optimized for MySQL, but not for the history-free version. Wouldn't those statements not also benefit from using a JOIN? Perhaps, but I prefer not to optimize until I see what happens in practice. Mis-optimized MySQL statements frequently turn out to have something like O(n^3) performance. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] blobs missing with relstorage and small blob cache dir
On 02/28/2011 08:19 AM, Maurits van Rees wrote: I wonder if there is some code that mistakenly throws away the wrong blob; it should throw away the oldest one I'd think, and not the one that it just loaded. The workaround is probably just to not set the blob-cache-size this low. But I tried this low setting to work around a different problem, which I will report in a different post. This reveals how naive the blob cache management is in both RelStorage and ZEO. It is smart enough to use the atime of files (thus you should not disable atime on a filesystem that hosts a blob cache), but the cache management does not guarantee downloaded files are kept long enough to actually read them. The usual Unixy solution is to pass around an open file descriptor instead of a filename, so that even if the blob file is deleted, the thread that asked for a file is sure to be able to read it. However, the ZODB blob APIs demand a filename, not an open file, so that solution is not feasible. We could make this kind of exception retryable, or we could make the blob cache management refuse to delete files within some time limit, but those solutions would be workarounds too. In the end I think it's best to just tell everyone that the minimum reasonable blob cache size is some multiple of your largest blob. I wouldn't set it lower than 1 GB. I would set it much larger for hosting videos or CD/DVD images. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Wrong blob returned in one of the zeo clients
On 02/28/2011 09:29 AM, Maurits van Rees wrote: This is now also happening for some images during normal operation, so after any blob migration has been run and existing blob caches have been cleared. What happens is that somehow the file contents for 0xblahblah.blob in the blob caches can differ between two zeo clients, even when the image has not changed at all (only tried with images as it is easier to spot than with other files). Thanks for the report. Looking through the blob code, I noticed that when the system is using a blob cache, it was possible for a thread to get a partial file while another thread is downloading that file. I suspect this is the problem you saw. I have committed a fix. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Relstorage database permissions
On 02/22/2011 03:10 PM, Maurits van Rees wrote: Hi, Normally RelStorage creates the database tables for you and the user you have specified is the owner of those tables. For security reasons a client does not want this, but wants a different user to own the tables and instead only grant some permissions to the relstorage user. I guess theoretically there could be a bug in the relstorage code that could lead to more problems when the relstorage user has full rights to those tables. I am not losing any sleep over fears like that though. :-) But putting aside a potentially distracting discussion about whether this extra security is needed: which permissions does relstorage really need? Select, update, insert and delete are obvious. I have seen that packing also needs the truncate permission. Everything seems to work with this combination. But for that extra bit peace of mind: am I overlooking a permission? Well, this is why transactions are really nice. If you overlooked anything, it is very likely that some transaction will be aborted normally and you'll get a nice traceback that narrows the problem quickly. So I think you'll be fine. :-) Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage 1.5.0b1 dry-run two phase pack, better pack lock behaviour
On 02/22/2011 01:41 PM, Martijn Pieters wrote: On Tue, Feb 22, 2011 at 19:12, Shane Hathawaysh...@hathawaymix.org wrote: On 02/22/2011 09:25 AM, Martijn Pieters wrote: I haven't yet actually run this code, but the change isn't big. I didn't find any relevant tests to update. Anyone want to venture some feedback? Both ideas are excellent. The new options even open the possibility of running the pre-pack on a copy of the database, then copying the pack tables to the main database for a final run. BTW, should I just commit the patch, or do you want to integrate it yourself? I'll look into working the locking idea into a patch too, but I'll need help with supporting Postgres and MySQL as I don't know their locking semantics. Do your best and I'll take a look after you've committed the patches to the trunk. :-) Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage 1.5.0b1 dry-run two phase pack, better pack lock behaviour
On 02/22/2011 09:25 AM, Martijn Pieters wrote: I haven't yet actually run this code, but the change isn't big. I didn't find any relevant tests to update. Anyone want to venture some feedback? Both ideas are excellent. The new options even open the possibility of running the pre-pack on a copy of the database, then copying the pack tables to the main database for a final run. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Copying zodb's with relstorage
On 02/10/2011 06:30 AM, Santi Camps wrote: I was trying to move a database copy a relstorage zodb and having some issues. The original zodb is mounted using a mount point /original_path If I restore the backup of the database and mount it using exactly the same mount point /original_path in the destination zope, all goes right. But what I want is to replicate the original database N times, so need to have /destination_pathN in the mount point. When I do that, the database seems empty (no object is shown in the mounted point). The normal way to copy objects in ZODB is through ZEXP export and import. Have you tried that? Also, what is your goal? Is there any way to fix this updating registers in SQL ? I know a way to solve it might be to export and import ZEXP, but the database is very big and I'm trying to avoid it I've tried these 2 queries but no effect obtained, these fields seems just informative: update transaction set description=replace(description::text, 'helpdesk_src', 'redesistemas')::bytea; update transaction set username=replace(username::text, 'helpdesk_src', 'redesistemas')::bytea; The username and description in the transaction table affect only what you see in the undo log. I don't think they have any connection with the problem you are trying to solve. Where is stored the information about parent - children objects ? Is the prev_tid field of object_state table ? No. RelStorage stores all such information in pickles, just like FileStorage or any other ZODB storage. If you're using RelStorage because you expect to be able to access and manipulate the object database using SQL, I'm afraid you will be disappointed. All ZODB storages store pickles, and pickles are opaque to SQL. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Copying zodb's with relstorage
On 02/10/2011 07:41 AM, Shane Hathaway wrote: On 02/10/2011 06:30 AM, Santi Camps wrote: I was trying to move a database copy a relstorage zodb and having some issues. The original zodb is mounted using a mount point /original_path If I restore the backup of the database and mount it using exactly the same mount point /original_path in the destination zope, all goes right. But what I want is to replicate the original database N times, so need to have /destination_pathN in the mount point. When I do that, the database seems empty (no object is shown in the mounted point). The normal way to copy objects in ZODB is through ZEXP export and import. Have you tried that? Also, what is your goal? Um, I see you mentioned that you're trying to avoid ZEXP. Ok. ;-) The question remains: What are you trying to accomplish? Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Copying zodb's with relstorage
On 02/10/2011 08:42 AM, Santi Camps wrote: The objective is to duplicate a storage using different mount points. For instance, if we have Database1 - mount_point_1 , create Database2 and Database3 as copies of Database1 (using pg_dump pg_restore), and then mount them as mount_point_2 and mount_point_3 Yes, but why do you want to do that? There might be a better way to accomplish what you're trying to do, or perhaps what you're doing is the right thing to do but there's some bug and you need to describe why that bug is important. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Copying zodb's with relstorage
On 02/10/2011 09:27 AM, Santi Camps wrote: On Thu, Feb 10, 2011 at 5:07 PM, Shane Hathaway sh...@hathawaymix.org mailto:sh...@hathawaymix.org wrote: On 02/10/2011 08:42 AM, Santi Camps wrote: The objective is to duplicate a storage using different mount points. For instance, if we have Database1 - mount_point_1 , create Database2 and Database3 as copies of Database1 (using pg_dump pg_restore), and then mount them as mount_point_2 and mount_point_3 Yes, but why do you want to do that? There might be a better way to accomplish what you're trying to do, or perhaps what you're doing is the right thing to do but there's some bug and you need to describe why that bug is important. Well, the original is a base site I want to replicate to be used for different clients. I often have done that using mount points and zexp's export/import, but was trying to do it copying the database to avoid the long and heavy import process. I see. I don't know why the database seemed empty, but even after you fix that, assuming you're using Plone or similar, you're going to face a deeper problem: the catalog(s) in each site use absolute paths. To fix that, you will have to re-catalog everything in each site, which is probably the most expensive part of the ZEXP import. In other words, I suspect you won't actually save any time with this method of copying. Another possible solution is to shrink your template so that it's not so expensive to copy. If I have understood correctly, then this question is independent of the ZODB storage--you would have exactly the same problem with FileStorage or ZEO. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Transaction Timestamps
FWIW, here is a way to extract timestamps from transaction IDs stored with RelStorage. Kai of HexagonIT suggested it. The timestamps should be in UTC. PostgreSQL: select (tid 32) / 535680 + 1900 as year, 1 + ((tid 32) % 535680) / 44640 as month, 1 + ((tid 32) % 44640) / 1440 as day, ((tid 32) % 1440) / 60 as hour, ((tid 32) % 60) as minute, (tid 4294967295) * 60.0 / 4294967296.0 as second from transaction; MySQL: select (tid 32) div 535680 + 1900 as year, 1 + ((tid 32) % 535680) div 44640 as month, 1 + ((tid 32) % 44640) div 1440 as day, ((tid 32) % 1440) div 60 as hour, ((tid 32) % 60) as minute, (tid 4294967295) * 60.0 / 4294967296.0 as second from transaction; Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage and PosKey errors - is this a risky hotfix?
On 02/02/2011 10:57 AM, Chris Withers wrote: Er, since when? If that were the case, I'm sure Shane would place explicit instructions that it should not be used... Safe is relative. MySQL is a good choice for Facebook, but if I knew my bank was storing my account balance in MySQL, I would close my accounts immediately. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage - what tables and indexes should be present?
On 02/01/2011 10:16 AM, Chris Withers wrote: I notice that one of my history-free storages only has 'new_oid' and 'object_state' tables while all the others have 'new_oid', 'object_ref', 'object_refs_added', 'object_state' and 'pack_object' tables. What's special about this storage? It sounds like RelStorage didn't get a chance to finish creating the schema. In MySQL, DDL statements are not transactional, so errors during schema creation (such as a timeout) leave a partial schema. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage and MySQL wait-timeout
On 02/01/2011 10:01 AM, Chris Withers wrote: OperationalError: (2006, 'MySQL server has gone away') ...which feels a little on the serious side for (what is for MySQL) quite a normal situation to be in. Random disconnects are unacceptable for RelStorage. If MySQL goes away outside transaction boundaries, we *have* to propagate the exception to the application; otherwise consistency is lost. We can only reconnect at the next request or transaction boundary. (Mike Bayer could probably shed some light given the connection pool stuff that SQLAlchemy does to deal with MySQL's behaviour...) Unfortunately, the generic behavior implemented by SQLAlchemy would not work here. It would scramble ZODB. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage - what tables and indexes should be present?
On 02/01/2011 07:35 PM, Chris Withers wrote: On 01/02/2011 17:33, Shane Hathaway wrote: What's special about this storage? It sounds like RelStorage didn't get a chance to finish creating the schema. In MySQL, DDL statements are not transactional, so errors during schema creation (such as a timeout) leave a partial schema. Sounds like something to warn about in the docs for zodbconvert and storage creation... Partial schema creation is specific to MySQL and is not specific to RelStorage. MySQL has many pitfalls and there are far too many of them to list in the RelStorage documentation. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage and MySQL wait-timeout
On 02/01/2011 07:51 PM, Chris Withers wrote: I can understand the problem being fairly terminal if there was a disconnect *during* a timeout, and I'd expect an exception, but not a segfault ;-) I haven't seen segfaults except when the dynamic linker used an incorrect library. Use ldd to check the .so files generated during buildout. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage, history-free, pack causes POSKeyError with BTreeFolder2
On 01/31/2011 06:02 PM, Chris Withers wrote: On 28/01/2011 21:58, Jürgen Herrmann wrote: Afaics you use zodbpack's default of days=0. This is known to produce POSKeyErrors if the database is written to while packing. Where is this documented? It is not known to me. :-) I'll not pack a history-free storage until Shane gives it the okay... Good, because although I fixed a bug your test revealed, your test still fails, and I don't yet know why. :-( Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Error from MySQL when attempting to run zodbconvert
On 01/31/2011 06:30 PM, Chris Withers wrote: Hi Shane, I got the following error when trying to run zodbconvert against a clustered MySQL environment running RHCS: File /var/buildout-eggs/MySQL_python-1.2.3-py2.6-linux-i686.egg/MySQLdb/connections.py, line 36, in defaulterrorhandler raise errorclass, errorvalue _mysql_exceptions.OperationalError: (1598, Binary logging not possible. Message: Transaction level 'READ-COMMITTED' in InnoDB is not safe for binlog mode 'STATEMENT') That's why you have to use row-based replication instead of statement-based replication. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] bug with RelStorage's zodbconvert --clear when clearing a large DB with MySQL
On 02/01/2011 12:19 AM, Chris Withers wrote: Hi Shane, This one's less serious if it is what I think it is: Traceback (most recent call last): File bin/zodbconvert, line 24, inmodule relstorage.zodbconvert.main() File /var/buildout-eggs/RelStorage-1.4.0-py2.6.egg/relstorage/zodbconvert.py, line 89, in main destination.zap_all() File /var/buildout-eggs/RelStorage-1.4.0-py2.6.egg/relstorage/storage.py, line 308, in zap_all self._rollback_load_connection() File /var/buildout-eggs/RelStorage-1.4.0-py2.6.egg/relstorage/storage.py, line 217, in _rollback_load_connection self._load_conn.rollback() _mysql_exceptions.OperationalError: (2006, 'MySQL server has gone away') My guess is that the zap_all took so long that the server had gone away by the time the sql statement had be executed. My guess is MySQL is configured to drop connections when they are idle. That is a bad idea IMHO, so I think raising that exception is the right thing to do, not a bug. I also had a segfault trying to do the same conversion which I'm attributing to the MySQL server being restarted by an overeager DBA mid-converstion but still, that shouldn't cause a segfault, right? I don't know why it would. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage pack with history-free storage results in POSKeyErrors
On 01/28/2011 02:21 AM, Chris Withers wrote: Okay, so next up I restored the backup. This time I set the batch running and *then* started the pack. This eventually caused the POSKeyErrors to begin again. I'm going to have a go at knocking up a small batch script that you can run to reproduce the issue, but there's definitely an issue here when packing while changing lots of data in a ZODB. I look forward to your results! Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage, history-free, pack causes POSKeyError with BTreeFolder2
On 01/28/2011 05:34 AM, Chris Withers wrote: bin/generate --zap --folder_depth 3 Leave it running for a minute or two, and then in another do: bin/zodbpack pack.conf Thanks to this test, I found a serious flaw in the history-free variant of packing. Currently, the history-free and history-preserving variants share the fill_object_refs() method, which assumes that it's possible to enumerate the references from all objects in a specific transaction. However, that assumption is only correct in the history-preserving variant. In history-free storages, object states disappear every time they are replaced by a newer version, causing fill_object_refs() to think the replaced objects have no references. That mistake leads to garbage collection of too many objects. I'm surprised that existing tests did not reveal this. Until I release a fix, I recommend not packing history-free databases. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage and PosKey errors - is this a risky hotfix?
On 01/24/2011 02:02 PM, Anton Stonor wrote: Now, I wonder why these pointers were deleted from the current_object table in the first place. My money is on packing -- and it might fit with the fact that we recently ran a pack that removed an unusual large amount of transactions in a single pack (100.000+ transactions). But I don't know how to investigate the root cause further. Ideas? I have meditated on this for some time now. I mentioned I had an idea about packing, but I studied the design and I don't see any way my idea could work. The design is such that it seems impossible that the pack code could produce an inconsistency between the object_state and current_object tables. I have lots of other ideas now, but I don't know which to pursue. I need a lot more information. It would be helpful if you sent me your database to analyze. Some possible causes: - Have you looked for filesystem-level corruption yet? I asked this before and I am waiting for an answer. - Although there is a pack lock, that lock unfortunately gets released automatically if MySQL disconnects prematurely. Therefore, it is possible to force RelStorage to run multiple pack operations in parallel, which would have unpredictable effects. Is there any possibility that you accidentally ran multiple pack operations in parallel? For example, maybe you have a cron job, or you were setting up a cron job at the time, and you started a pack while the cron job was running. (Normally, any attempt to start parallel pack operations will just generate an error, but if MySQL disconnects in just the right way, you'll get a mess.) - Every SQL database has nasty surprises. Oracle, for example, has a nice read only mode, but it turns out that mode works differently in RAC environments, leading to silent corruption. As a result, we never use that feature of Oracle anymore. Maybe MySQL has some nasty surprises I haven't yet discovered; maybe the MySQL-specific delete using statement doesn't work as expected. - Applications can accidentally cause POSKeyErrors in a variety of ways. For example, persistent objects cached globally can cause POSKeyErrors. Maybe Plone 4 or some add-on uses ZODB incorrectly. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage pack with history-free storage results in POSKeyErrors
On 01/27/2011 03:32 AM, Chris Withers wrote: On 27/01/2011 03:15, Shane Hathaway wrote: Okay, so I'll do: - e2fsck -f Hmm, how do I e2fsck a mounted filesystem? You don't. Don't even try. :-) The MySQL filesystem could be unmounted (after I shut down MySQL!), so I ran e2fsck on it: e2fsck -f /dev/sdb1 e2fsck 1.41.3 (12-Oct-2008) Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information /dev/sdb1: 166/16777216 files (10.2% non-contiguous), 8244979/67107513 blocks - mysqlcheck -c mysqlcheck -c dev_packed -u root -p Enter password: dev_packed.new_oid OK dev_packed.object_ref OK dev_packed.object_refs_added OK dev_packed.object_stateOK dev_packed.pack_object OK Ok, thanks for checking. First up, I get the following failures: IOError: [Errno 2] No such file or directory: '/var/buildout-eggs/RelStorage-1.4.0-py2.6.egg/relstorage/tests/blob/blob_connection.txt' OSError: [Errno 2] No such file or directory: '/var/buildout-eggs/RelStorage-1.4.0-py2.6.egg/relstorage/tests/replicas.conf' My guess is that these files aren't included by setuptools? No, they are included AFAICT. Let's ignore this for now. This is a little weird, as I have max_allowed_packet set to 16M. Increase max_allowed_packet to at least 32M. 16M is just a bit too low. Should these tests fail? No. I think they will all pass once you increase max_allowed_packet. That said, I don't think this has anything to do with the packing bug as The point was to eliminate some of the most likely causes. Now we'll move on. I didn't see any exceptions or, in fact, any logging or output at all from zodbpack, and the only other exceptions seen were the POSKeyErrors... Hmm, you do bring up a good point: zodbpack doesn't configure the logging package. It should. Can you send me your database? Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] question about object invalidation after ReadConflictErrors
On 01/27/2011 03:59 AM, Jürgen Herrmann wrote: hi there! i wrongly posted a bug to the zodb bugtracker on tuesday: https://bugs.launchpad.net/zodb/+bug/707332 as it turns out, that bug report was moot. it didn't fix my problem. out of curiosity, why is the exception raised inside the loop effectifely breaking the loop after the first object invalidated? my debug code showed that sometimes ~20 objects were in _readCurrent. was this a side effect of the relstorage bug involved? or can this happen more frequently? if so, why not invalidate all objects in _readCurrent then before re- raising the ConflictEror? That one invalidation is probably some kind of optimization. I don't think it's necessary. The real invalidation happens later, at a transaction boundary, when _flush_invalidations() is called. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage pack with history-free storage results in POSKeyErrors
On 01/27/2011 03:57 AM, Chris Withers wrote: It would also be *really* handy if zodbpack could run off a normal zope.conf for both logging and storage config (although, in my case, I'd then need to be able to specify which storage to pack, I want to avoid packing one of them!) Please note that the zodbpack utility is simple, short, and has no extra dependencies. I prefer to keep it that way. If you want to do something more interesting, please fork zodbpack. However, more than happy to poke, just tell me where and for what... Is the problem repeatable? If you start with the same database twice and pack, do you end up with POSKeyErrors on the same OIDs? I know that's probably a long test to run, but I'm not yet sure what else to suggest. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ReadConflictErrors with ZODB3.10.1 and Relstorage 1.4.1
On 01/26/2011 05:29 AM, Jürgen Herrmann wrote: _readCurrent is obviously reused and not cleared at transaction boundaries, is that expected?): No! Thanks for the great analysis. This insight is key. RelStorage has a monkey patch of the Connection.sync() method, which has not changed in a long time, so the monkey patch seemed safe enough. Well, sync() changed in ZODB 3.10, but the monkey patch didn't change along with it. Sigh... sorry. I've checked in a fix in Subversion. Please try it out. I need to look at the possible pack issue recently reported before we make a release. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage pack with history-free storage results in POSKeyErrors
On 01/26/2011 11:52 AM, Chris Withers wrote: On 26/01/2011 14:08, Shane Hathaway wrote: I've checked in a fix in Subversion. Please try it out. I need to look at the possible pack issue recently reported before we make a release. Where is this pack issue documented/discussed? See the discussion here with Anton Stonor. We are still only hypothesizing that there's a bug. Also, does RelStorage have a bug tracker anywhere? Not yet. The need for one has not been clear until very recently. RelStorage is turning into a community project and every community project needs a bug tracker. I suggest we use Launchpad. I have a main ZODB storage in a normal RelStorage with several BTreeFolder2's mounted into it that come from a history free storage. I'm using Zope 2.12.7 with ZODB 3.9.6 I attempted to pack the history-free storage with the following config: relstorage destination pack-gc true mysql ... /mysql keep-history false /relstorage The pack was going on while a batch process was pouring data into the ZODB by way for a loop with one transaction per iteration. The pack succeeded fine with no errors but about 10 minutes after the pack started, the batch process started throwing POSKeyErrors of the following sort: File /var/buildout-eggs/ZODB3-3.9.6-py2.6-linux-i686.egg/ZODB/Connection.py, line 897, in _setstate p, serial = self._storage.load(obj._p_oid, '') File /var/buildout-eggs/RelStorage-1.4.0-py2.6.egg/relstorage/storage.py, line 462, in load raise POSKeyError(oid) POSKeyError: 0xf30dbd -- 2011-01-26T10:26:00 WARNING POSKeyError on oid 15925081: no tid found; history-free adapter -- 2011-01-26T10:26:00 ERROR Couldn't load state for 0xf2ff59 Traceback (most recent call last): File /var/buildout-eggs/ZODB3-3.9.6-py2.6-linux-i686.egg/ZODB/Connection.py, line 847, in setstate self._setstate(obj) File /var/buildout-eggs/ZODB3-3.9.6-py2.6-linux-i686.egg/ZODB/Connection.py, line 897, in _setstate p, serial = self._storage.load(obj._p_oid, '') File /var/buildout-eggs/RelStorage-1.4.0-py2.6.egg/relstorage/storage.py, line 462, in load raise POSKeyError(oid) POSKeyError: 0xf2ff59 It looks like these errors are occurring through the web client to this ZODB as well now. Any ideas what's going on here? What more useful information can I provide? That may be related, but first, are you mounting databases? You have to be careful with mounted databases and packing. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage pack with history-free storage results in POSKeyErrors
On 01/26/2011 03:21 PM, Chris Withers wrote: Well, my case matches that case pretty exactly... MySQL 5.1, RelStorage 1.4.0... Anton, were you using a history-free storage? My guess is not. Shane, I went to try your suggestion: If you're sure you have found all of the corruption, you can do this (with all app servers shut down) to re-create the current_object table: delete from current_object; insert into current_object (zoid, tid) select zoid, max(tid) from object_state; ...but of course, history-free storages have no current_object table. Indeed, which is why I think your issue could be quite different from Anton's. The odd thing is I have seen no reports of pack issues for a long time. Why two at once? I still wonder if there might be no bug at all. If you and Anton could check all your logs and database integrity, that would save me a lot of time. FWIW, RelStorage has extensive tests of packing, including stress tests that write to the database while packing. I run those tests automatically using Buildbot on several platforms and with several combinations of ZODB and SQL database versions. None of the pack tests are failing on any platform, with any version of ZODB or any SQL database. If there is a bug, then the next step is to add a test that exercises the bug. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage pack with history-free storage results in POSKeyErrors
On 01/26/2011 04:07 PM, Chris Withers wrote: On 26/01/2011 22:38, Shane Hathaway wrote: The odd thing is I have seen no reports of pack issues for a long time. Why two at once? I still wonder if there might be no bug at all. I'm afraid my wondering is on the other side of that argument ;-) If you and Anton could check all your logs and database integrity, that would save me a lot of time. Okay, so I'll do: - e2fsck -f - mysqlcheck -c Anything else? What logs should I hunt through and what kind of things am I looking for? /var/log/messages and the like. Look for kernel-level errors such as block I/O errors, oops, and system panics. Any such errors have a chance of corrupting files. We need to rule out errors at the kernel or below. FWIW, RelStorage has extensive tests of packing, including stress tests that write to the database while packing. How can I run these? If you don't already have bin/test in your buildout, add a section that creates it. Here is an example based on the buildout.cfg in RelStorage: [buildout] parts = test [test] recipe = zc.recipe.testrunner eggs = relstorage MySQL-python ZODB3 [test] zope.testrunner Next, follow the directions in relstorage/tests/README.txt to create the 4 test databases. Then run bin/test -p -m relstorage. All tests should pass. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage and PosKey errors - is this a risky hotfix?
On 01/24/2011 02:02 PM, Anton Stonor wrote:
Hi there,
We have recently experienced a couple of PosKey errors with a Plone 4
site running RelStorage 1.4.1 and Mysql 5.1.
After digging down we found that the objects that were throwing
PosKeyErrors actually existed in the object_state table with pickles
etc, however not in the current_object table.
After inserting the missing pointers into the current_object table,
everything worked fine:
mysql SELECT zoid, tid FROM object_state WHERE zoid=561701;
+++
| zoid | tid|
+++
| 561701 | 255267099158685832 |
+++
mysql INSERT INTO current_object(zoid, tid) VALUES('561701',
'255267099158685832');
Looks like it works -- but is this a safe way to fix PosKeyErrors?
Now, I wonder why these pointers were deleted from the current_object
table in the first place. My money is on packing -- and it might fit
with the fact that we recently ran a pack that removed an unusual large
amount of transactions in a single pack (100.000+ transactions).
But I don't know how to investigate the root cause further. Ideas?
This suggests MySQL not only lost some data (due to a MySQL bug or a
filesystem-level error), but it failed to enforce a foreign key that is
supposed to ensure this never happens. I think you need to check the
integrity of your filesystem (e2fsck -f) and database (mysqlcheck -c).
You might also reconsider the choice to use MySQL.
If you're sure you have found all of the corruption, you can do this
(with all app servers shut down) to re-create the current_object table:
delete from current_object;
insert into current_object (zoid, tid)
select zoid, max(tid) from object_state;
Shane
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage and PosKey errors - is this a risky hotfix?
On 01/24/2011 06:32 PM, Laurence Rowe wrote: Must this imply a failure to maintain a foreign key constraint? While there are FK constraints on current_object (zoid, tid) - object_state (zoid, tid) there is no foreign key that might prevent a current_object row from being incorrectly deleted. That's true. Thanks for pitching in. 2. Something goes wrong during pack gc (either in the pack logic or on the database). Looking at the code, I think I might see a way that current_object rows could be incorrectly removed by a pack operation, but I'll have to write tests to find out. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage recommended maintenance
On 01/21/2011 10:46 AM, Chris Withers wrote: I'm wondering what the recommended maintenance for these two types of storage are that I use: - keep-history=true, never want to lose any revisions My guess is zodbpack with pack-gc as true, but what do I specify for the number of days in order to keep all history? Is 100 years enough? 365.24 * 100 = 36524 ;-) - keep-history=false, never want any revisions or unreferenced objects My guess is zodbpack with pack-gc as true and days as 0? Sounds right. All storages will be in MySQL, are there any duty-cycle or other parameters I should tweak from their default values? As I recommend to everyone, there should be no need to tweak anything unless you have measurements that tell you the defaults don't fit your site. How often do you recommend running zodbpack? Daily. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage recommended maintenance
On 01/21/2011 03:38 PM, Chris Withers wrote: On 21/01/2011 22:09, Laurence Rowe wrote: On 21 January 2011 20:57, Shane Hathawaysh...@hathawaymix.org wrote: On 01/21/2011 10:46 AM, Chris Withers wrote: I'm wondering what the recommended maintenance for these two types of storage are that I use: - keep-history=true, never want to lose any revisions My guess is zodbpack with pack-gc as true, but what do I specify for the number of days in order to keep all history? Is 100 years enough? 365.24 * 100 = 36524 ;-) Why would you pack a database from which you don't want to lose any revisions. It's a fair point. I can't think of anything, but given that Shane didn't make your point, I wonder if he can? Heh, I am obviously not paying enough attention. Don't pack that database. :-) Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage branch 1.4.0-fastimport
On 01/11/2011 07:06 AM, David Blewett wrote: On Thu, Jan 6, 2011 at 11:51 PM, Shane Hathawaysh...@hathawaymix.org wrote: I am no zodb or relstorage expert, but the changes for the --single-transaction code look good to me. I looked at that branch before, but I felt like the changes were complicated enough to require a comparison with simpler solutions first. In particular, Postgres has an option to disable fsync. Set it in postgresql.conf. Disabling fsync is not normally recommended, but for a large import it's obviously a good idea. Would you compare the speed of --single-transaction with disabled fsync on vanilla RelStorage? Sorry, it's been awhile since I started work on that branch. Disabling fsync is an option for the load, and it should help. However, using a single transaction is much faster in PostgreSQL. Off the top of my head, I can't recall if the branch also uses COPY mode. I recall encountering some problems trying to get COPY to work correctly. Single-transaction mode + COPY, however, is the fastest way to get data into PostgreSQL. Right, but I want zodbconvert to remain a very generic tool. I think of zodbconvert as a reference implementation. A tool that uses PostgreSQL COPY is a good idea but it ought to be a different tool. IMHO, it's actually pretty easy to write optimized ZODB conversion tools. I would suggest writing conversion tools as independent packages, meaning they probably should not import RelStorage, but instead talk to the database directly. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Migrating from filestorage to relstorage with blobs
On 01/10/2011 03:58 PM, Maurits van Rees wrote: Starting with RelStorage 1.5.0a1 (or trunk) there is support for blobs in RelStorage. You have the option to either store the blobs on the file system (var/blobstorage) or store them in the relational database. To clarify, blob support was in version 1.4, but the newest release gives you the option of where to put them. With filestorage combined with blobstorage it seems you only have one option: you get a small Data.fs and a big var/blobstorage directory that contains the files. Question: can the blobs also be stored in the Data.fs (supposing you want that)? Not that I know of. plone.app.blob has migration code to move existing Archetypes files and images to blobstorage: @@blob-file-migration and @@blob-image-migration. I successfully migrated a 100 MB filestorage without blobstorage to a setup of relstorage with blobstorage. Both possible routes (first to relstorage using zodbconvert, then to blobstorage or the other way around) worked. Question: if I start with filestorage plus a var/blobstorage directory, is it possible to migrate this to relstorage with blobs stored in the relational database? It should work, yes. You have to use different blob directories for the source and destination. Posed more general: is there migration code that migrates blobs from a (relational) database to a var/blobstorage directory or the other way around? Yes, if you mean you want to copy the pickles as well, then zodbconvert should handle it. If you don't want to copy the pickles, I'm not sure. I can at least say that in my light testing so far, RelStorage 1.5.0a1 with blobs works fine, so thanks Shane! Thanks for testing. 1.5.0a1 seems to be quite stable, so unless any issue surfaces, it's going to turn into a final release really fast. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage branch 1.4.0-fastimport
On 01/07/2011 10:34 AM, Maurits van Rees wrote: Op 07-01-11 05:51, Shane Hathaway schreef: I looked at that branch before, but I felt like the changes were complicated enough to require a comparison with simpler solutions first. In particular, Postgres has an option to disable fsync. Set it in postgresql.conf. Disabling fsync is not normally recommended, but for a large import it's obviously a good idea. Would you compare the speed of --single-transaction with disabled fsync on vanilla RelStorage? In /etc/postgresql/8.4/main/postgresql.conf I switched these settings off and restarted postgres: fsync = off synchronous_commit = off The zodbconvert of the 150 MB CatalogData.fs went from 3.1 minutes to 2.8. When using the --single-transaction option of the branch it went to 2.2 minutes. So there is still a significant improvement here. Ok, I'll study the branch again. Maybe the changes aren't as deep as I remember. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage branch 1.4.0-fastimport
On 01/06/2011 03:35 PM, Maurits van Rees wrote: davidb started a branch of RelStorage in July 2010 and I wonder if that can be merged. It does two interesting things in the zodbconvert code: - It adds a --single-transaction command line option. From the readme: Import into the destination in a single transaction, instead of one transaction per transaction in the source. This option can significantly speed up conversion times on PostgreSQL. - I adds more logging so you have an indication that yes, something is in fact happening. This is good for your heart when you want to migrate a 16 GB Data.fs. :-) Otherwise you just see 'Storages opened successfully' and then nothing at all for several hours. I am no zodb or relstorage expert, but the changes for the --single-transaction code look good to me. I looked at that branch before, but I felt like the changes were complicated enough to require a comparison with simpler solutions first. In particular, Postgres has an option to disable fsync. Set it in postgresql.conf. Disabling fsync is not normally recommended, but for a large import it's obviously a good idea. Would you compare the speed of --single-transaction with disabled fsync on vanilla RelStorage? I assume the logging additions are less invasive, and if that assumption is correct, there's no problem with merging those. I added some more logging on that branch, mostly because the conversion appeared to be hanging at some unknown point. This was also with the official 1.4.1 release, which was the reason I started experimenting with the fastimport branch to see if that would help. It did not. At least in both test runs mentioned above, the actual time it took was about twenty minutes longer, possibly because the conversion temporarily lost the connection with the postgres server. With the logging I could at least see that it was throwing the old transaction table away; I have seen the same with other tables. Definitely no one else is accessing this database at the same time. So if someone has an idea what could be going on here, that is welcome. Are you sure you were not accidentally running multiple imports in parallel? RelStorage does not throw away tables unless you're rather explicit about it. Anyway, the --single-transaction seems to work and I would say the logging is helpful. So: is there any chance this can be merged to trunk? Not yet. We really need someone to do the fsync test, and if that doesn't do the trick, there has to be a way to accomplish the same thing in a clearer way. I am also interested in the blob support that has been added there. :-) The blob changes are also in 1.5.0a1. That release seems to be a lot more stable than I expected and may soon become 1.5.0 final if no one finds bugs in the new blob option. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Sharing (persisted) strings between threads
On 12/08/2010 02:26 PM, Dylan Jay wrote: I was working with a high volume site recently where 70% of requests called a back end API that could take up to 4sec. For better or worse this was in zope. The best solution to scaling this was to increase the number of threads since this process was now IO bound. You do run out of memory when you do this so this solution would have been helpful. If a shared cache between processes were possible, such as using memcached, that would be even better :) That is already possible: just use RelStorage with memcached and set your ZODB cache size to zero or something small. However, large ZODB applications typically depend on a large number of persistent objects to respond to even the simplest requests, so you would have to optimize the application to load as few objects as possible. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage: Error Installing Plone Site on Oracle RAC
On 12/02/2010 03:25 PM, Darryl Dixon - Winterhouse Consulting wrote: [...snip...] We have added a new schema to our Oracle RAC database and mount point inside Zope, attempted to add a Plone site, and get the following error. If everything else remains the same, but the ZODB Mount in the zope.conf gets pointed back to a regular ZEO/ZODB, it works fine: 2010-12-03T10:31:14 ERROR Zope.SiteErrorLog 1291325474.290.650785606429 http://web-cms-edit-node3.dev.internal:8080/nu-gov-companies-www-preview/manage_addProduct/CMFPlone/addPl oneSite Traceback (innermost last): Module ZPublisher.Publish, line 119, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 42, in call_object Module Products.CMFPlone.factory, line 86, in addPloneSite Module Products.GenericSetup.tool, line 390, in runAllImportStepsFromProfile - __traceback_info__: profile-Products.CMFPlone:plone Module Products.GenericSetup.tool, line 1179, in _runImportStepsFromContext Module Products.GenericSetup.tool, line 1090, in _doRunImportStep - __traceback_info__: content Module Products.CMFCore.exportimport.content, line 37, in importSiteStructure Module Products.CMFCore.exportimport.content, line 166, in import_ Module Products.CMFCore.exportimport.content, line 185, in _makeInstance Module Products.CMFCore.TypesTool, line 716, in constructContent Module Products.CMFCore.TypesTool, line 278, in constructInstance Module Products.CMFCore.TypesTool, line 290, in _finishConstruction Module Products.CMFCore.CMFCatalogAware, line 148, in notifyWorkflowCreated Module Products.CMFCore.WorkflowTool, line 292, in notifyCreated Module Products.CMFCore.WorkflowTool, line 642, in _reindexWorkflowVariables Module Products.Archetypes.CatalogMultiplex, line 66, in reindexObjectSecurity Module Products.ZCatalog.CatalogBrains, line 52, in _unrestrictedGetObject Module OFS.Traversable, line 175, in unrestrictedTraverse - __traceback_info__: (['Members', 'site'], '') IndexError: string index out of range Any thoughts on why RelStorage usage might cause this? So, it seems that this is a Heisenbug - I instrumented OFS.Traversable to catch the error, log some stuff, and re-raise... And the site add then worked without any exception being thrown, no logging appear, etc :-( The bug otherwise was replicated twice prior to the code change. That is somewhat worrisome. Anyone who can shed some light? Make sure the databases are really separated. Use different commit lock IDs, set a different cache prefix for each database (or use different memcached servers), and make sure each database has no data whatsoever before you begin. My guess is some simple exception happened earlier (maybe the database had old objects, or you accidentally used the wrong database at some point, or Plone's setup process stalled too long on some step) and the all-important detailed traceback got swallowed by a greedy except clause in Zope or Plone. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage: Write on master, read from slave(s)?
On 11/29/2010 12:55 PM, Leonardo Santagada wrote: On Mon, Nov 29, 2010 at 5:17 PM, Chris Withersch...@simplistix.co.uk wrote: Wouldn't this require zero replication lag? No. Even if the read replicas had only a second or two delay (or spike delays caused by large writes?), would there not potentially be quite serious problems? No. What I think shane meant was to have clients that only read data (eg anonymous users) so they can read old data without causing problems and clients that edit content use the primary database. Correct, although old means up to a few seconds old. Seems to be a bad way to improve performance, the only advantage I would see from using cache is if you somehow have logged in users that don't edit data and have most/all of the pages dependent on the logged user, for example a forum. This strategy would be bad for a site that expects most users to log in and edit, such as Facebook or Twitter, but it is a proven strategy for forums, wikis, blogs, news sites, stores, and many other kinds of sites. It is more reliable than caching, since failed caches often lead to cascading failures. (See the recent Facebook outage.) Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Relstorage Database Adapter
On 11/24/2010 05:17 AM, Santi Camps wrote: Hi all I'm using relstorage for a long time with very good results. Until know, I also use database adapters like ZPsycopgDA to connect the same SQL database and store information in other application tables. I was thinking to create a Relstorage Database Adapter able to do the work using relstorage configuration and connections. Some preliminary simple tests seems to work: def execute_query(conn, cursor, *args, **kw): query = kw['query'] cursor.execute(query); return cursor query = 'SELECT COUNT(*) FROM kmkey_task' rows = app._p_jar._storage._with_store(execute_query, query=query) print rows.fetchone() The question is, can I go on ? Or use same connections than relstorage could cause problems at ZODB level ? Well, it doesn't look like you're using the two RelStorage connections the way they are intended to be used. For the code snippet you gave, it's likely that you want to use the load connection rather than the store connection. The load connection ensures consistent reads. RelStorage does a number of things to maintain that guarantee. The load connection is never committed, only rolled back. The store connection is intended to be used only during transaction commit. At the beginning of a commit (tpc_begin), RelStorage rolls back the store connection in order to get the most recent updates. The store connection and load connection are often out of sync with each other, so code that uses the store connection should detect and handle conflicting updates. I suspect the load/store connection split is too complicated for most apps that just want to interact with the database, so I haven't exposed any documented API. I considered making an API when I worked on repoze.pgtextindex, but I concluded that pgtextindex can be a bit lazy about consistency and therefore doesn't need all the extra complexity that reusing RelStorage connections would bring. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage support to Microsoft SQLServer
On 11/19/2010 10:39 AM, Sachin Tekale wrote: On 2010-11-17 18:12, Shane Hathaway wrote: No, RelStorage does not require two phase commit support from the database. It turns out that a simple commit lock, which is sufficient for preventing deadlocks and unreported conflicts Are there any performance implications of not using two phase commit feature? Database-level two-phase commit is, in general, much slower than ordinary commit. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] 32-bit vs 64-bit - RelStorage on MySQL
On 11/18/2010 09:54 AM, Chris Withers wrote: On 18/11/2010 16:48, Hanno Schlichting wrote: On Thu, Nov 18, 2010 at 5:19 PM, Leonardo Santagadasantag...@gmail.com wrote: On Thu, Nov 18, 2010 at 1:47 PM, Chris Withersch...@simplistix.co.uk wrote: On 18/11/2010 15:39, Marius Gedminas wrote: About the only noticeable difference -- other than the obvious memory growth What obvious memory growth? The one from pointers and anything related to memory going from 32bits to 64bits in size. Py_objects get fatter because of that. For Zope based applications I've generally seen 50% to 100% memory growth when moving to 64bit. That's why we stick to 32bit for a number of memory hungry applications (*wink* Plone). Right, but, even if the MySQL server is 64-bit, if the clients are 32-bit, they won't have this problem, will they? Ok, you have a 32 bit Python client connecting to a 64 bit MySQL server, right? I don't expect any problems. All the pickling and unpickling will be done on 32-bit clients. MySQL sees the pickles as opaque binary streams. The only remaining question is whether the MySQL wire protocol can handle your setup. I suspect it can. If it can't, then you'll be filing MySQL bugs, since RelStorage, Python, and even the MySQL-Python DB-API adapter are all ignorant of the MySQL wire protocol. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage support to Microsoft SQLServer
On 11/17/2010 08:29 AM, Sachin Tekale wrote: Hi all, I have read that there is a problem to implement MS-SQL adapter for Relstorage because the “Two phase commit” feature is not exposed by MS-SQL server . No, RelStorage does not require two phase commit support from the database. It turns out that a simple commit lock, which is sufficient for preventing deadlocks and unreported conflicts. Is there solution to overcome this problem, Without introducing too many layers? Can we use PyMSSQL and ADODB Python extension to implement the relstorage Adapter for MS-SQL. Has any one tried this before? Are there any other major concerns you would have for developing MS-SQL adapter component? Please point us in right direction on the subject. Using mxODBC is a good start, but you still need SQL designed for MS-SQL, since SQL is only loosely standardized. I could help you out. Email me privately if you are interested. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
