Re: [ZODB-Dev] RelStorage: ConflictError causes deadlock without RELSTORAGE_ABORT_EARLY

2014-09-05 Thread Shane Hathaway
On 07/19/2014 05:22 PM, Sean Upton wrote: Folks, I have been dealing with locking issues and RelStorage for the past few days, and want to verify what I believe is a bug: without RELSTORAGE_ABORT_EARLY set in environment, tpc_vote() could potentially leave an ILocker adapter setting an RDBMS

Re: [ZODB-Dev] RelStorage, postgresql 9.2 and zc.async (plone.app.async)

2014-03-14 Thread Shane Hathaway
On 03/13/2014 09:06 AM, Simone Deponti wrote: 1. Certain tables remain locked and automatic cleanup functions (e.g. AUTOVACUUM) can't properly run Are you using the poll-interval option? That option tells RelStorage to leave the transaction open. In practice, it's just wrong (although it

Re: [ZODB-Dev] Optimizing RelStorage Packing for large DBs

2013-11-18 Thread Shane Hathaway
On 11/15/2013 06:01 PM, Jens W. Klein wrote: The idea is simple: - iterate over all transactions starting with the lowest transaction id (tid) - for each transaction load the object states connected with tid - for each state fetch its outgoing references and fill a table where all

Re: [ZODB-Dev] RelStorage 1.5.1 and persistent 4.0.5+ Incompatible (patch)

2013-03-08 Thread Shane Hathaway
On 03/07/2013 10:48 AM, jason.mad...@nextthought.com wrote: On Mar 7, 2013, at 11:35, Sean Upton sdup...@gmail.com wrote: On Thu, Mar 7, 2013 at 7:31 AM, jason.mad...@nextthought.com wrote: I only spotted two uses of this assumption in RelStrorage, the above-mentioned `_prepare_tid`, plus

Re: [ZODB-Dev] Relstorage and over growing database.

2013-02-07 Thread Shane Hathaway
On 02/06/2013 04:23 AM, Jürgen Herrmann wrote: I think this is not entirely correct. I ran in to problems serveral times when new_oid was emptied! Maybe Shane can confirm this? (results in read conlfict errors) Ah, that's true. You do need to replicate new_oid. Then I'd like to talk a little

Re: [ZODB-Dev] Relstorage and over growing database.

2013-02-07 Thread Shane Hathaway
On 02/07/2013 01:54 PM, Jürgen Herrmann wrote: Am 07.02.2013 21:18, schrieb Jürgen Herrmann: I know that's entirely not your fault but may be worth mentioning in the docs. Relstorage with MySQL works *very* well for DB sizes 5GB or so, above that - not so much :/ Also for the docs: on disk

Re: [ZODB-Dev] Relstorage and over growing database.

2013-02-02 Thread Shane Hathaway
On 02/01/2013 09:08 PM, Juan A. Diaz wrote: Reading the some comments [0] in the code (relstorage/adapters/schema.py) I could see that the object_ref database is uses during the packing, then the question is, in a history-preserving database there is something that we could do to decrease the

Re: [ZODB-Dev] RelStorage, PostgreSQL backup method comparison

2012-12-27 Thread Shane Hathaway
On 12/26/2012 10:43 AM, Sean Upton wrote: For cron job RelStorage backups (databse not including blobs backed-up seperately, using a PostgreSQL 9.0.x backend), I use both zodbconvert to save FileStorage copies of my database, and pgdump for low-level binary dumps (pg_restore custom format,

Re: [ZODB-Dev] ZEO invalidation message transaction-inbound or outbound

2012-12-03 Thread Shane Hathaway
On 12/01/2012 11:22 AM, Andreas Jung wrote: Jim Fulton wrote: On Fri, Nov 30, 2012 at 1:37 AM, Andreas Jung li...@zopyx.com wrote: a customer made the observation that that ZEO clients became inconsistent after some time (large CMF-based application running on Zope 2.12 afaik). Customer made

[ZODB-Dev] RelStorage moved to Github

2012-09-01 Thread Shane Hathaway
By popular request, I've moved RelStorage development from svn.zope.org to Github [1]. This should make it easier for the ZODB developer community to contribute and submit issues. Happy hacking! Also, I've created a zodb community on github. If you'd like to move or create your

Re: [ZODB-Dev] Storm/ZEO deadlocks (was Re: [Zope-dev] [announce] NEO 1.0 - scalable and redundant storage for ZODB)

2012-08-30 Thread Shane Hathaway
On 08/30/2012 10:14 AM, Marius Gedminas wrote: On Wed, Aug 29, 2012 at 06:30:50AM -0400, Jim Fulton wrote: On Wed, Aug 29, 2012 at 2:29 AM, Marius Gedminas mar...@gedmin.as wrote: On Tue, Aug 28, 2012 at 06:31:05PM +0200, Vincent Pelletier wrote: On Tue, 28 Aug 2012 16:31:20 +0200, Martijn

Re: [ZODB-Dev] Relstorage memcache object lifetimes

2012-08-16 Thread Shane Hathaway
On 08/14/2012 09:59 AM, Daniel Garcia wrote: I've noticed that over time the number of objects in memcached grows util memcached begins to evict old objects. I'm trying to figure out how to size the memcached layer. Since there is no lifetime on the objects they remain in the cache until they

Re: [ZODB-Dev] Question about relstorage and pack-gc

2012-07-13 Thread Shane Hathaway
On 07/12/2012 01:30 PM, Santi Camps wrote: My specific question is: if I disable pack-gc, can I safety empty object_ref table and free this space? Certainly. However, most of the 23 GB probably consists of blobs; blobs are not shown in the query results you posted. Shane

Re: [ZODB-Dev] Question about relstorage and pack-gc

2012-07-13 Thread Shane Hathaway
On 07/13/2012 02:42 PM, Santi Camps wrote: On Fri, Jul 13, 2012 at 7:05 PM, Shane Hathaway sh...@hathawaymix.org mailto:sh...@hathawaymix.org wrote: On 07/12/2012 01:30 PM, Santi Camps wrote: My specific question is: if I disable pack-gc, can I safety empty object_ref table

Re: [ZODB-Dev] RelStorage breaks history tab

2012-05-23 Thread Shane Hathaway
On 05/23/2012 09:27 AM, Chris Withers wrote: Okay, the issue appears to be that, in some circumstances, RelStorage is leaving the read connection with an open transaction that isn't rolled back. That is what RelStorage is designed to do when you set poll-interval. Does the bug go away when

Re: [ZODB-Dev] Relstorage and MSQL

2012-05-08 Thread Shane Hathaway
On 05/08/2012 06:34 PM, Dylan Jay wrote: I know it's not all about money but if we were to sponsor development of microsoft sql server support for relstorage, is there someone who knows how and has an estimated cost and available time? I don't have enough cycles now to do it myself, but I will

Re: [ZODB-Dev] Unable to acquire commit lock from RelStorage

2012-03-07 Thread Shane Hathaway
On 03/04/2012 04:16 PM, Chris Withers wrote: Hi Shane, What does this exception mean: Traceback (innermost last): Module ZPublisher.Publish, line 135, in publish Module Zope2.App.startup, line 291, in commit Module transaction._manager, line 89, in commit Module transaction._transaction, line

Re: [ZODB-Dev] where does 32it vs 64bit matter?

2012-01-04 Thread Shane Hathaway
On 01/02/2012 11:54 PM, Chris Withers wrote: Good to know, but my concern is taking a system which is currently running 32-bit clients against RelStorage running on a MySQL on a 64-bit server and replacing the 32-bit clients with 64-bit clients. Is that going to cause any issues? That will

Re: [ZODB-Dev] Removing undeleteable objects from ZODB

2011-12-19 Thread Shane Hathaway
On 12/19/2011 02:50 AM, Eugene Morozov wrote: Is there some other way to remove them? I would remove all references to the broken objects, then let packing wipe the old objects away. For example: container = app['somefolder'] del container[obj_name] It's possible to replace objects in the

Re: [ZODB-Dev] Cached object error while upgrading Plone

2011-11-21 Thread Shane Hathaway
On 11/21/2011 10:40 AM, Alexandru Plugaru wrote: Hello, I've got this error ( http://pastie.org/2898828 ) while upgrading from plone3 to plone4. The context in which that error happens is this: http://pastie.org/2898959 The only place I could find the error message was in cPersistence.c:

Re: [ZODB-Dev] Mysql with RelStorage shows very bad results

2011-11-17 Thread Shane Hathaway
On 11/17/2011 10:16 AM, Войнаровський Тарас wrote: Convert took around 2 hours and the speed is realy bad. I tried iterating through all objects, by using the findObjectsProviding of zope.app.generations and it took 600 seconds, in contrast to the 20 seconds with FileStorage. The cache works

Re: [ZODB-Dev] RelStorage - high ConflictError rate in newly converted DB

2011-11-11 Thread Shane Hathaway
On 11/10/2011 05:32 PM, Darryl Dixon - Winterhouse Consulting wrote: * What is the situation with using multiple memcache instances (eg, one per machine - will this lead to cache incoherency - inconsistency between instances, etc? One memcache per client? That's fine as long as you don't set

Re: [ZODB-Dev] Relstorage PostgreSQL and blobs garbage collection

2011-11-10 Thread Shane Hathaway
On 11/10/2011 08:56 AM, Александр Неганов wrote: We had large history-free database with lot of deleted Blobs, so we packed and gc-ed it. Many rows from object_state as well as corresponding blob_chunk rows has been deleted, but postgresql large objects stays untouched. It seems that

Re: [ZODB-Dev] RelStorage - high ConflictError rate in newly converted DB

2011-11-08 Thread Shane Hathaway
On 11/07/2011 03:15 PM, Darryl Dixon - Winterhouse Consulting wrote: Hi All, We have just converted a site with a handful of editors over from ZEO to RelStorage on Oracle and and the rate of ConflictErrors has gone through the roof (5 an hour average continuously for days). This seems very

Re: [ZODB-Dev] RelStorage with Oracle and separation of rights

2011-10-31 Thread Shane Hathaway
On 10/30/2011 07:05 PM, Darryl Dixon - Winterhouse Consulting wrote: Hi All, Part of the setup of our Oracle RelStorage environment involves the DBAs wanting to separate ownership of the schema from the rights to actually use the schema. In other words, user A owns all the tables etc that

Re: [ZODB-Dev] Debugging RelStorage hang-ups

2011-10-20 Thread Shane Hathaway
On 10/20/2011 05:41 AM, Martijn Pieters wrote: On a test server with a Plone 4.1 upgrade of a client setup, we are experience regular lock-ups of the 2 instances we run. After a restart, after a few hours at least one of the instances will be waiting on Oracle to roll back: File

Re: [ZODB-Dev] zeo.memcache

2011-10-12 Thread Shane Hathaway
On 10/09/2011 08:26 AM, Jim Fulton wrote: On Sat, Oct 8, 2011 at 4:34 PM, Shane Hathawaysh...@hathawaymix.org wrote: On 10/05/2011 11:40 AM, Pedro Ferreira wrote: Hello all, While doing some googling on ZEO + memcache I came across this: https://github.com/eleddy/zeo.memcache Has anybody

Re: [ZODB-Dev] zeo.memcache

2011-10-12 Thread Shane Hathaway
On 10/12/2011 04:53 PM, Shane Hathaway wrote: Given the choice to structure the cache as {(oid, tid): (state, last_tid)}, a simple way to use the cache would be to get the last committed tid from the database and use that tid for the lookup key. This would be extremely efficient until the next

Re: [ZODB-Dev] RelStorage zodbconvert, ConflictError on Zope2 startup

2011-07-19 Thread Shane Hathaway
On 07/18/2011 02:16 PM, Sean Upton wrote: On Fri, Jul 15, 2011 at 5:35 PM, Shane Hathawaysh...@hathawaymix.org wro= te: I am thinking of changing the memcache code to use a random per-database = key prefix. =A0If I had done that already, you would not have run into this problem, since

Re: [ZODB-Dev] RelStorage zodbconvert, ConflictError on Zope2 startup

2011-07-15 Thread Shane Hathaway
On 07/14/2011 11:21 PM, Sean Upton wrote: On Thu, Jul 14, 2011 at 3:28 PM, Sean Uptonsdup...@gmail.com wrote: Full traceback: http://pastie.org/2214036 I am able to avoid this by commenting out cache-servers and cache-module-name in my zope.conf. Looks like the ConflictError at start-up is

Re: [ZODB-Dev] POSKeyError

2011-06-22 Thread Shane Hathaway
On 06/21/2011 07:18 AM, Erik Dahl wrote: I'm using relstorage 1.4.2 during a batch job I was running last night I got the following errors. 2011-06-21 07:55:02,664 WARNING relstorage: POSKeyError on oid 23916102: no tid found; Current transaction is 256466219826629358; Recent object tids:

Re: [ZODB-Dev] RelStorage and MySQL 5.5

2011-06-22 Thread Shane Hathaway
On 06/21/2011 08:22 AM, Philip K. Warren wrote: Does RelStorage work with MySQL 5.5? I saw a note posted back in February on this list that said that 5.5 is not yet supported, however I haven't seen any issues with this configuration. I have been able to successfully run the 1.4.2 and 1.5.0b2

Re: [ZODB-Dev] Relstorage Blob support and Oracle

2011-06-13 Thread Shane Hathaway
On 06/12/2011 04:01 AM, Martijn Pieters wrote: How big a userbase is there for 1.5.0b2 on PostgreSQL? I know schema changes are painful, but in this case we'd only have people on the bleeding edge using a beta version to switch. I think we can come up with a little script that would move these

Re: [ZODB-Dev] Relstorage Blob support and Oracle

2011-06-10 Thread Shane Hathaway
On 06/10/2011 02:53 AM, Hanno Schlichting wrote: This looks like the typical problem, where some code opens a file without explicitly closing it. But instead relies on garbage collection to do the job during __del__ of the file object. That generally doesn't work well on Windows. Yes, that's

Re: [ZODB-Dev] Relstorage Blob support and Oracle

2011-06-10 Thread Shane Hathaway
On 06/10/2011 06:38 AM, Hanno Schlichting wrote: On Fri, Jun 10, 2011 at 1:03 PM, Hanno Schlichtingha...@hannosch.eu wrote: /me is still trying to get the postgres tests to run. I only just now found the relstorage/tests/readme instructions I got the postgres tests running now and get actual

Re: [ZODB-Dev] Relstorage Blob support and Oracle

2011-06-09 Thread Shane Hathaway
On 06/09/2011 06:32 AM, Martijn Pieters wrote: We've looked over the RelsStorage ZODB Blob storage implementation and came to the conclusion that the current use of blob chunks is unnecessary in Oracle when using the cx_Oracle database connector. Not splitting ZODB Blobs into chunks may have

Re: [ZODB-Dev] RelStorage: Pack no longer holding the commit lock for prolonged periods

2011-05-24 Thread Shane Hathaway
On 05/23/2011 01:58 PM, Martijn Pieters wrote: I've cleared the last area where RelStorage packing could hold the transaction lock for long periods of time, during empty transaction deletion: http://zope3.pov.lt/trac/changeset/121783/relstorage/trunk During a large pack, this section

Re: [ZODB-Dev] Speeding up ZODB (was redis cache for RelStorage)

2011-05-06 Thread Shane Hathaway
On 05/06/2011 06:22 AM, Pedro Ferreira wrote: But isn't RelStorage supposed be slower than FileStorage/ZEO? No, every measurement I've tried suggests RelStorage (with PostgreSQL or MySQL) is faster than ZEO on the same hardware. ZEO has certainly gotten faster lately, but RelStorage still

Re: [ZODB-Dev] Speeding up ZODB (was redis cache for RelStorage)

2011-05-06 Thread Shane Hathaway
On 05/06/2011 02:38 PM, Jim Fulton wrote: If there is memory pressure and you take away ram for a ram disk, then you're going to start swapping, which will give you other problems. In my experience, Linux moves pieces of the ZEO cache out of RAM long before it starts swapping much. I tried

Re: [ZODB-Dev] Speeding up ZODB (was redis cache for RelStorage)

2011-05-06 Thread Shane Hathaway
On 05/06/2011 02:14 PM, Shane Hathaway wrote: However, there is a different class of problems that prefetching could help solve. Let's say you have pages with a lot of little pieces on it, such as a comment page with a profile image for every comment. It would be useful to tell ZODB to load

Re: [ZODB-Dev] redis cache for RelStorage

2011-05-03 Thread Shane Hathaway
On 05/02/2011 09:23 PM, 潘俊勇 wrote: It seems that redis is much faster than memcached. Could we use redis as a cache for RelStroage? Are you having speed issues? I suspect either one is so fast that the speed of Redis or Memcached is irrelevant. If you want speed, minimize the latency of the

Re: [ZODB-Dev] weird MySQL gone away error when viewing history tab on Script(Python) with RelStorage 1.4.2

2011-05-03 Thread Shane Hathaway
On 05/03/2011 01:39 AM, Chris Withers wrote: On 27/04/2011 18:11, Shane Hathaway wrote: OperationalError: (2006, 'MySQL server has gone away') This is happening across at least two separate instances with separate storages. Any ideas? Maybe the query is killing MySQL, for whatever reason

Re: [ZODB-Dev] weird MySQL gone away error when viewing history tab on Script(Python) with RelStorage 1.4.2

2011-04-27 Thread Shane Hathaway
On 04/27/2011 05:07 AM, Chris Withers wrote: Hi Shane, Attempting to view the /manage_change_history_page of a history-keeping relstorage is giving me: script statement failed: '\nSELECT 1 FROM current_object WHERE zoid = %(oid)s\n'; parameters: {'oid': 1163686} ...and

Re: [ZODB-Dev] python path problem: ImportError when calling zodbconvert (FreeBSD 8.1)

2011-04-25 Thread Shane Hathaway
On 04/24/2011 03:36 AM, Nikolaj Groenlund wrote: I guess this is a python path problem (on FreeBSD 8.1). Im trying to convert a Data.fs to Postgresql using zodbconvert. Ive downloaded RelStorage-1.5.0b2 and is running: /usr/local/Plone/Python-2.6/bin/python zodbconvert.py fstodb.conf , to

Re: [ZODB-Dev] packing ZODB

2011-03-31 Thread Shane Hathaway
On 03/31/2011 04:46 AM, Adam GROSZER wrote: After investigating FileStorage a bit, I found that GC runs on objects, but pack later by transactions. That means that if there's a bigger-ish transaction, we can't get rid of it until all of it's objects are GCed (or superseeded by newer states).

Re: [ZODB-Dev] Problem with RelStorage zodbpack

2011-03-30 Thread Shane Hathaway
On 03/29/2011 09:16 PM, Erik Dahl wrote: Ok looked a little deeper. I think solution 2 is the way to go (ie clear the object_ref table from references that are in my range of non-packed transactions. Does that sound right? Statement would be: delete from object_ref where tid

Re: [ZODB-Dev] Problem with RelStorage zodbpack

2011-03-30 Thread Shane Hathaway
On 03/29/2011 07:39 PM, Erik Dahl wrote: I was running a pack and canceled so that I could reboot my box. After it came back up I tried to restart the pack and got this: [...] File /opt/zenoss/lib/python2.6/site-packages/RelStorage-1.4.2-py2.6.egg/relstorage/adapters/packundo.py, line

Re: [ZODB-Dev] Plone 4.0.4 and Postgresql Encoding

2011-03-28 Thread Shane Hathaway
On 03/28/2011 04:01 PM, Nikolaj Groenlund wrote: Im using Plone 4.0.4, Postgresql 9.0.3 and RelStorage 1.5.0b2. Currently Im using da_DK.ISO8859-1 encoding in PostgreSQL - would da_DK.UTF-8 be better since Plone is using UTF-8 internally? PS both Encoding, Collation and Ctype are set in

Re: [ZODB-Dev] Wrong blob returned in one of the zeo clients

2011-03-02 Thread Shane Hathaway
On 03/02/2011 07:38 AM, Jim Fulton wrote: BTW, my sense is that when blobs were first implemented, the emphasis was on shared blob directories and the non-shared blob implementation suffered from neglect, which I at least tried to rectify in ZODB 3.9. That evolutionary path is evident and I

Re: [ZODB-Dev] blobs missing with relstorage and small blob cache dir

2011-03-01 Thread Shane Hathaway
On 02/28/2011 07:37 PM, Shane Hathaway wrote: On 02/28/2011 08:19 AM, Maurits van Rees wrote: I wonder if there is some code that mistakenly throws away the wrong blob; it should throw away the oldest one I'd think, and not the one that it just loaded. The workaround is probably just

Re: [ZODB-Dev] Wrong blob returned in one of the zeo clients

2011-03-01 Thread Shane Hathaway
On 03/01/2011 08:05 AM, Maurits van Rees wrote: Op 01-03-11 13:54, Maurits van Rees schreef: Op 01-03-11 04:41, Shane Hathaway schreef: On 02/28/2011 09:29 AM, Maurits van Rees wrote: This is now also happening for some images during normal operation, so after any blob migration has been run

Re: [ZODB-Dev] Wrong blob returned in one of the zeo clients

2011-03-01 Thread Shane Hathaway
On 03/01/2011 01:10 PM, Jim Fulton wrote: On Tue, Mar 1, 2011 at 2:47 PM, Shane Hathawaysh...@hathawaymix.org wrote: Any idea where the error might be? Could this be in plone.app.blob? Any chance that this is better in ZODB 3.9+? This appears to be a design bug in the zeocache blob layout:

Re: [ZODB-Dev] Wrong blob returned in one of the zeo clients

2011-03-01 Thread Shane Hathaway
On 03/01/2011 08:05 AM, Maurits van Rees wrote: No, I am still seeing it. I now see more clearly that the problem is that two files share the same blob name. I completely remove the blob cache and restart the zeo client. I visit image number 1 and get this file in the var/blobcache:

Re: [ZODB-Dev] Wrong blob returned in one of the zeo clients

2011-03-01 Thread Shane Hathaway
On 03/01/2011 02:47 PM, Maurits van Rees wrote: That is pretty weird! I can understand a few duplicates because images are saved in a few sizes, but this is too much. To reiterate some versions: - Plone 3.3.5 - ZODB3 3.8.6-polling - RelStorage 1.5.0b1 - Zope2 2.10.12 - plone.app.blob

Re: [ZODB-Dev] Wrong blob returned in one of the zeo clients

2011-03-01 Thread Shane Hathaway
On 03/01/2011 04:32 PM, Maurits van Rees wrote: As a workaround for getting blobstorage to work reliably in a relstorage setup without a shared blob dir: is there any way to store the blobs completely in postgres and have no shared or cached blob dir in the zeo clients? If that would work it

Re: [ZODB-Dev] Wrong blob returned in one of the zeo clients

2011-02-28 Thread Shane Hathaway
On 02/28/2011 09:29 AM, Maurits van Rees wrote: This is now also happening for some images during normal operation, so after any blob migration has been run and existing blob caches have been cleared. What happens is that somehow the file contents for 0xblahblah.blob in the blob caches can

Re: [ZODB-Dev] Pack complete: 80% of a 425GB database packed

2011-02-28 Thread Shane Hathaway
On 02/28/2011 07:45 AM, Martijn Pieters wrote: Early this morning, after packing through the weekend, our somewhat overweight Oracle RelStorage ZODB pack was completed. I am still waiting for the final size from the customer DBAs, but before the pack this beast was occupying 425GB. The pack

Re: [ZODB-Dev] blobs missing with relstorage and small blob cache dir

2011-02-28 Thread Shane Hathaway
On 02/28/2011 08:19 AM, Maurits van Rees wrote: I wonder if there is some code that mistakenly throws away the wrong blob; it should throw away the oldest one I'd think, and not the one that it just loaded. The workaround is probably just to not set the blob-cache-size this low. But I

Re: [ZODB-Dev] Wrong blob returned in one of the zeo clients

2011-02-28 Thread Shane Hathaway
On 02/28/2011 09:29 AM, Maurits van Rees wrote: This is now also happening for some images during normal operation, so after any blob migration has been run and existing blob caches have been cleared. What happens is that somehow the file contents for 0xblahblah.blob in the blob caches can

Re: [ZODB-Dev] Relstorage database permissions

2011-02-25 Thread Shane Hathaway
On 02/22/2011 03:10 PM, Maurits van Rees wrote: Hi, Normally RelStorage creates the database tables for you and the user you have specified is the owner of those tables. For security reasons a client does not want this, but wants a different user to own the tables and instead only grant

Re: [ZODB-Dev] RelStorage 1.5.0b1 dry-run two phase pack, better pack lock behaviour

2011-02-25 Thread Shane Hathaway
On 02/22/2011 01:41 PM, Martijn Pieters wrote: On Tue, Feb 22, 2011 at 19:12, Shane Hathawaysh...@hathawaymix.org wrote: On 02/22/2011 09:25 AM, Martijn Pieters wrote: I haven't yet actually run this code, but the change isn't big. I didn't find any relevant tests to update. Anyone want to

Re: [ZODB-Dev] RelStorage 1.5.0b1 dry-run two phase pack, better pack lock behaviour

2011-02-22 Thread Shane Hathaway
On 02/22/2011 09:25 AM, Martijn Pieters wrote: I haven't yet actually run this code, but the change isn't big. I didn't find any relevant tests to update. Anyone want to venture some feedback? Both ideas are excellent. The new options even open the possibility of running the pre-pack on a

Re: [ZODB-Dev] Copying zodb's with relstorage

2011-02-10 Thread Shane Hathaway
On 02/10/2011 06:30 AM, Santi Camps wrote: I was trying to move a database copy a relstorage zodb and having some issues. The original zodb is mounted using a mount point /original_path If I restore the backup of the database and mount it using exactly the same mount point /original_path

Re: [ZODB-Dev] Copying zodb's with relstorage

2011-02-10 Thread Shane Hathaway
On 02/10/2011 07:41 AM, Shane Hathaway wrote: On 02/10/2011 06:30 AM, Santi Camps wrote: I was trying to move a database copy a relstorage zodb and having some issues. The original zodb is mounted using a mount point /original_path If I restore the backup of the database and mount

Re: [ZODB-Dev] Copying zodb's with relstorage

2011-02-10 Thread Shane Hathaway
On 02/10/2011 08:42 AM, Santi Camps wrote: The objective is to duplicate a storage using different mount points. For instance, if we have Database1 - mount_point_1 , create Database2 and Database3 as copies of Database1 (using pg_dump pg_restore), and then mount them as mount_point_2 and

Re: [ZODB-Dev] Copying zodb's with relstorage

2011-02-10 Thread Shane Hathaway
On 02/10/2011 09:27 AM, Santi Camps wrote: On Thu, Feb 10, 2011 at 5:07 PM, Shane Hathaway sh...@hathawaymix.org mailto:sh...@hathawaymix.org wrote: On 02/10/2011 08:42 AM, Santi Camps wrote: The objective is to duplicate a storage using different mount points

[ZODB-Dev] Transaction Timestamps

2011-02-03 Thread Shane Hathaway
FWIW, here is a way to extract timestamps from transaction IDs stored with RelStorage. Kai of HexagonIT suggested it. The timestamps should be in UTC. PostgreSQL: select (tid 32) / 535680 + 1900 as year, 1 + ((tid 32) % 535680) / 44640 as month, 1 + ((tid 32) % 44640) / 1440 as

Re: [ZODB-Dev] RelStorage and PosKey errors - is this a risky hotfix?

2011-02-02 Thread Shane Hathaway
On 02/02/2011 10:57 AM, Chris Withers wrote: Er, since when? If that were the case, I'm sure Shane would place explicit instructions that it should not be used... Safe is relative. MySQL is a good choice for Facebook, but if I knew my bank was storing my account balance in MySQL, I would

Re: [ZODB-Dev] RelStorage - what tables and indexes should be present?

2011-02-01 Thread Shane Hathaway
On 02/01/2011 10:16 AM, Chris Withers wrote: I notice that one of my history-free storages only has 'new_oid' and 'object_state' tables while all the others have 'new_oid', 'object_ref', 'object_refs_added', 'object_state' and 'pack_object' tables. What's special about this storage? It

Re: [ZODB-Dev] RelStorage and MySQL wait-timeout

2011-02-01 Thread Shane Hathaway
On 02/01/2011 10:01 AM, Chris Withers wrote: OperationalError: (2006, 'MySQL server has gone away') ...which feels a little on the serious side for (what is for MySQL) quite a normal situation to be in. Random disconnects are unacceptable for RelStorage. If MySQL goes away outside

Re: [ZODB-Dev] RelStorage - what tables and indexes should be present?

2011-02-01 Thread Shane Hathaway
On 02/01/2011 07:35 PM, Chris Withers wrote: On 01/02/2011 17:33, Shane Hathaway wrote: What's special about this storage? It sounds like RelStorage didn't get a chance to finish creating the schema. In MySQL, DDL statements are not transactional, so errors during schema creation

Re: [ZODB-Dev] RelStorage and MySQL wait-timeout

2011-02-01 Thread Shane Hathaway
On 02/01/2011 07:51 PM, Chris Withers wrote: I can understand the problem being fairly terminal if there was a disconnect *during* a timeout, and I'd expect an exception, but not a segfault ;-) I haven't seen segfaults except when the dynamic linker used an incorrect library. Use ldd to

Re: [ZODB-Dev] RelStorage, history-free, pack causes POSKeyError with BTreeFolder2

2011-01-31 Thread Shane Hathaway
On 01/31/2011 06:02 PM, Chris Withers wrote: On 28/01/2011 21:58, Jürgen Herrmann wrote: Afaics you use zodbpack's default of days=0. This is known to produce POSKeyErrors if the database is written to while packing. Where is this documented? It is not known to me. :-) I'll not pack

Re: [ZODB-Dev] Error from MySQL when attempting to run zodbconvert

2011-01-31 Thread Shane Hathaway
On 01/31/2011 06:30 PM, Chris Withers wrote: Hi Shane, I got the following error when trying to run zodbconvert against a clustered MySQL environment running RHCS: File /var/buildout-eggs/MySQL_python-1.2.3-py2.6-linux-i686.egg/MySQLdb/connections.py, line 36, in defaulterrorhandler

Re: [ZODB-Dev] bug with RelStorage's zodbconvert --clear when clearing a large DB with MySQL

2011-01-31 Thread Shane Hathaway
On 02/01/2011 12:19 AM, Chris Withers wrote: Hi Shane, This one's less serious if it is what I think it is: Traceback (most recent call last): File bin/zodbconvert, line 24, inmodule relstorage.zodbconvert.main() File

Re: [ZODB-Dev] RelStorage pack with history-free storage results in POSKeyErrors

2011-01-28 Thread Shane Hathaway
On 01/28/2011 02:21 AM, Chris Withers wrote: Okay, so next up I restored the backup. This time I set the batch running and *then* started the pack. This eventually caused the POSKeyErrors to begin again. I'm going to have a go at knocking up a small batch script that you can run to reproduce

Re: [ZODB-Dev] RelStorage, history-free, pack causes POSKeyError with BTreeFolder2

2011-01-28 Thread Shane Hathaway
On 01/28/2011 05:34 AM, Chris Withers wrote: bin/generate --zap --folder_depth 3 Leave it running for a minute or two, and then in another do: bin/zodbpack pack.conf Thanks to this test, I found a serious flaw in the history-free variant of packing. Currently, the history-free and

Re: [ZODB-Dev] RelStorage and PosKey errors - is this a risky hotfix?

2011-01-27 Thread Shane Hathaway
On 01/24/2011 02:02 PM, Anton Stonor wrote: Now, I wonder why these pointers were deleted from the current_object table in the first place. My money is on packing -- and it might fit with the fact that we recently ran a pack that removed an unusual large amount of transactions in a single pack

Re: [ZODB-Dev] RelStorage pack with history-free storage results in POSKeyErrors

2011-01-27 Thread Shane Hathaway
On 01/27/2011 03:32 AM, Chris Withers wrote: On 27/01/2011 03:15, Shane Hathaway wrote: Okay, so I'll do: - e2fsck -f Hmm, how do I e2fsck a mounted filesystem? You don't. Don't even try. :-) The MySQL filesystem could be unmounted (after I shut down MySQL!), so I ran e2fsck

Re: [ZODB-Dev] question about object invalidation after ReadConflictErrors

2011-01-27 Thread Shane Hathaway
On 01/27/2011 03:59 AM, Jürgen Herrmann wrote: hi there! i wrongly posted a bug to the zodb bugtracker on tuesday: https://bugs.launchpad.net/zodb/+bug/707332 as it turns out, that bug report was moot. it didn't fix my problem. out of curiosity, why is the exception raised inside

Re: [ZODB-Dev] RelStorage pack with history-free storage results in POSKeyErrors

2011-01-27 Thread Shane Hathaway
On 01/27/2011 03:57 AM, Chris Withers wrote: It would also be *really* handy if zodbpack could run off a normal zope.conf for both logging and storage config (although, in my case, I'd then need to be able to specify which storage to pack, I want to avoid packing one of them!) Please note

Re: [ZODB-Dev] ReadConflictErrors with ZODB3.10.1 and Relstorage 1.4.1

2011-01-26 Thread Shane Hathaway
On 01/26/2011 05:29 AM, Jürgen Herrmann wrote: _readCurrent is obviously reused and not cleared at transaction boundaries, is that expected?): No! Thanks for the great analysis. This insight is key. RelStorage has a monkey patch of the Connection.sync() method, which has not changed

Re: [ZODB-Dev] RelStorage pack with history-free storage results in POSKeyErrors

2011-01-26 Thread Shane Hathaway
On 01/26/2011 11:52 AM, Chris Withers wrote: On 26/01/2011 14:08, Shane Hathaway wrote: I've checked in a fix in Subversion. Please try it out. I need to look at the possible pack issue recently reported before we make a release. Where is this pack issue documented/discussed? See

Re: [ZODB-Dev] RelStorage pack with history-free storage results in POSKeyErrors

2011-01-26 Thread Shane Hathaway
On 01/26/2011 03:21 PM, Chris Withers wrote: Well, my case matches that case pretty exactly... MySQL 5.1, RelStorage 1.4.0... Anton, were you using a history-free storage? My guess is not. Shane, I went to try your suggestion: If you're sure you have found all of the corruption, you can

Re: [ZODB-Dev] RelStorage pack with history-free storage results in POSKeyErrors

2011-01-26 Thread Shane Hathaway
On 01/26/2011 04:07 PM, Chris Withers wrote: On 26/01/2011 22:38, Shane Hathaway wrote: The odd thing is I have seen no reports of pack issues for a long time. Why two at once? I still wonder if there might be no bug at all. I'm afraid my wondering is on the other side of that argument

Re: [ZODB-Dev] RelStorage and PosKey errors - is this a risky hotfix?

2011-01-24 Thread Shane Hathaway
On 01/24/2011 02:02 PM, Anton Stonor wrote: Hi there, We have recently experienced a couple of PosKey errors with a Plone 4 site running RelStorage 1.4.1 and Mysql 5.1. After digging down we found that the objects that were throwing PosKeyErrors actually existed in the object_state table

Re: [ZODB-Dev] RelStorage and PosKey errors - is this a risky hotfix?

2011-01-24 Thread Shane Hathaway
On 01/24/2011 06:32 PM, Laurence Rowe wrote: Must this imply a failure to maintain a foreign key constraint? While there are FK constraints on current_object (zoid, tid) - object_state (zoid, tid) there is no foreign key that might prevent a current_object row from being incorrectly deleted.

Re: [ZODB-Dev] RelStorage recommended maintenance

2011-01-21 Thread Shane Hathaway
On 01/21/2011 10:46 AM, Chris Withers wrote: I'm wondering what the recommended maintenance for these two types of storage are that I use: - keep-history=true, never want to lose any revisions My guess is zodbpack with pack-gc as true, but what do I specify for the number of days in order

Re: [ZODB-Dev] RelStorage recommended maintenance

2011-01-21 Thread Shane Hathaway
On 01/21/2011 03:38 PM, Chris Withers wrote: On 21/01/2011 22:09, Laurence Rowe wrote: On 21 January 2011 20:57, Shane Hathawaysh...@hathawaymix.org wrote: On 01/21/2011 10:46 AM, Chris Withers wrote: I'm wondering what the recommended maintenance for these two types of storage are that I

Re: [ZODB-Dev] RelStorage branch 1.4.0-fastimport

2011-01-11 Thread Shane Hathaway
On 01/11/2011 07:06 AM, David Blewett wrote: On Thu, Jan 6, 2011 at 11:51 PM, Shane Hathawaysh...@hathawaymix.org wrote: I am no zodb or relstorage expert, but the changes for the --single-transaction code look good to me. I looked at that branch before, but I felt like the changes were

Re: [ZODB-Dev] Migrating from filestorage to relstorage with blobs

2011-01-10 Thread Shane Hathaway
On 01/10/2011 03:58 PM, Maurits van Rees wrote: Starting with RelStorage 1.5.0a1 (or trunk) there is support for blobs in RelStorage. You have the option to either store the blobs on the file system (var/blobstorage) or store them in the relational database. To clarify, blob support was in

Re: [ZODB-Dev] RelStorage branch 1.4.0-fastimport

2011-01-07 Thread Shane Hathaway
On 01/07/2011 10:34 AM, Maurits van Rees wrote: Op 07-01-11 05:51, Shane Hathaway schreef: I looked at that branch before, but I felt like the changes were complicated enough to require a comparison with simpler solutions first. In particular, Postgres has an option to disable fsync. Set

Re: [ZODB-Dev] RelStorage branch 1.4.0-fastimport

2011-01-06 Thread Shane Hathaway
On 01/06/2011 03:35 PM, Maurits van Rees wrote: davidb started a branch of RelStorage in July 2010 and I wonder if that can be merged. It does two interesting things in the zodbconvert code: - It adds a --single-transaction command line option. From the readme: Import into the

Re: [ZODB-Dev] Sharing (persisted) strings between threads

2010-12-08 Thread Shane Hathaway
On 12/08/2010 02:26 PM, Dylan Jay wrote: I was working with a high volume site recently where 70% of requests called a back end API that could take up to 4sec. For better or worse this was in zope. The best solution to scaling this was to increase the number of threads since this process was

Re: [ZODB-Dev] RelStorage: Error Installing Plone Site on Oracle RAC

2010-12-02 Thread Shane Hathaway
On 12/02/2010 03:25 PM, Darryl Dixon - Winterhouse Consulting wrote: [...snip...] We have added a new schema to our Oracle RAC database and mount point inside Zope, attempted to add a Plone site, and get the following error. If everything else remains the same, but the ZODB Mount in the

Re: [ZODB-Dev] RelStorage: Write on master, read from slave(s)?

2010-11-29 Thread Shane Hathaway
On 11/29/2010 12:55 PM, Leonardo Santagada wrote: On Mon, Nov 29, 2010 at 5:17 PM, Chris Withersch...@simplistix.co.uk wrote: Wouldn't this require zero replication lag? No. Even if the read replicas had only a second or two delay (or spike delays caused by large writes?), would there not

Re: [ZODB-Dev] Relstorage Database Adapter

2010-11-24 Thread Shane Hathaway
On 11/24/2010 05:17 AM, Santi Camps wrote: Hi all I'm using relstorage for a long time with very good results. Until know, I also use database adapters like ZPsycopgDA to connect the same SQL database and store information in other application tables. I was thinking to create a Relstorage

Re: [ZODB-Dev] RelStorage support to Microsoft SQLServer

2010-11-19 Thread Shane Hathaway
On 11/19/2010 10:39 AM, Sachin Tekale wrote: On 2010-11-17 18:12, Shane Hathaway wrote: No, RelStorage does not require two phase commit support from the database. It turns out that a simple commit lock, which is sufficient for preventing deadlocks and unreported conflicts Are there any

Re: [ZODB-Dev] 32-bit vs 64-bit - RelStorage on MySQL

2010-11-18 Thread Shane Hathaway
On 11/18/2010 09:54 AM, Chris Withers wrote: On 18/11/2010 16:48, Hanno Schlichting wrote: On Thu, Nov 18, 2010 at 5:19 PM, Leonardo Santagadasantag...@gmail.com wrote: On Thu, Nov 18, 2010 at 1:47 PM, Chris Withersch...@simplistix.co.uk wrote: On 18/11/2010 15:39, Marius Gedminas

Re: [ZODB-Dev] RelStorage support to Microsoft SQLServer

2010-11-17 Thread Shane Hathaway
On 11/17/2010 08:29 AM, Sachin Tekale wrote: Hi all, I have read that there is a problem to implement MS-SQL adapter for Relstorage because the “Two phase commit” feature is not exposed by MS-SQL server . No, RelStorage does not require two phase commit support from the database. It turns

  1   2   3   >