Re: [ZODB-Dev] Invalidations

2005-04-27 Thread Shane Hathaway
Tim Peters wrote: Jim Fulton] We should probably think harder about the semantics of sync. But it implied a transaction boundary -- specifically, an abort. You wouldn't want this to happen automatically. I assume Rajeev doesn't really want to call sync() automatically, because that's never

Re: [ZODB-Dev] [Proposal] Size controlled ZODB cache

2005-06-02 Thread Shane Hathaway
Dieter Maurer wrote: Currently, the ZODB cache can only be controlled via the maximal number of objects. This makes configuration complex as the actual limiting factor is the amount of available RAM and it is very difficult to estimate the size of the objects in the cache. I therefore

[ZODB-Dev] Re: [Proposal] Drop Mount.py from ZODB 3.5

2005-06-04 Thread Shane Hathaway
Tim Peters wrote: http://www.zope.org/Collectors/Zope/1800 describes some of the code problems with Zope's current way of mounting databases. ZODB 3.4 (still) has a Mount.py module, unused and untested by ZODB. Jim and I were both surprised today to discover that Zope (2.8) still

Re: [ZODB-Dev] Reloading product in 2.8

2005-07-14 Thread Shane Hathaway
Jim Fulton wrote: It also doesn't handle global data properly. It tries to do something that Python modules were never designed to support, which is to load them more than once. However, given the existence of the reload() builtin, someone apparently believed Python modules *were* designed

Re: [ZODB-Dev] Reloading product in 2.8

2005-07-15 Thread Shane Hathaway
Tim Peters wrote: [Jim Fulton] I agree. The right way to refresh is to detect code changes (ideally using Linux's brand new inotify mechanism, or something similar when inotify is not available), display a please wait message to the user, and restart the process. If you're changing more

Re: [ZODB-Dev] ZEO ClientStorage's use of the asyncore main loop (was [Fwd: Re: [Zope3-dev] Twisted Publisher and Zope 2])

2005-12-09 Thread Shane Hathaway
Jim Fulton wrote: Tim and I have discussed this for some time. We think an asynchronous I/O approach is still appropriate, to handle asynchronous messages from servers to clients, but we need to get away from expecting a server to provide the asyncore main loop needed by ZEO. Rather, ZEO should

Re: [ZODB-Dev] OID length

2005-12-09 Thread Shane Hathaway
Tim Peters wrote: Some auxiliary code would definitely break if they weren't strings. For example, ZEO cache tracing has its own trace-file format, which originally blew up left and right when Shane tried to use it with APE. That got generalized (and, alas, the trace files got bigger) to live

Re: [ZODB-Dev] OID length

2005-12-09 Thread Shane Hathaway
Tim Peters wrote: [Shane Hathaway] It can do this because it has to provide its own DB and Connection objects anyway. I trust that's for more reasons than _just_ because it doesn't want to use '\0'*8 as the root-object oid. It's for uniformity. In Ape, the strategy for choosing OIDs

Re: [ZODB-Dev] PGStorage

2008-01-24 Thread Shane Hathaway
Andreas Jung wrote: Excellent. I re-tried my benchmarks of course. CopyPaste of a complete new Plone site is now nearly as fast as with Filestorage (just by looking on my watch). The tests for commiting 100 objects are also blazing fast: roughly 1.2 - 1.5 seconds for committing. That's roughly

[ZODB-Dev] RelStorage now in Subversion

2008-01-26 Thread Shane Hathaway
Hi all, RelStorage now exists in the Zope subversion repository here: http://svn.zope.org/relstorage/trunk/ I have also created a wiki page: http://wiki.zope.org/ZODB/RelStorage Testing help and database-specific advice would be appreciated! With both PostgreSQL and Oracle, all

Re: [ZODB-Dev] RelStorage now in Subversion

2008-01-27 Thread Shane Hathaway
Andreas Jung wrote: Thanks a lot for the great work! One question: what is the ZODB patch actually doing? Is it for optimization purposes? Is it specfic to RelStorage or does it provide some general ZODB improvement(s)? The patch is intended to be a general ZODB improvement that enables

[ZODB-Dev] Speedy RelStorage/PostgreSQL

2008-01-30 Thread Shane Hathaway
Hi guys, I just committed some performance enhancements to RelStorage, then I performed two simple speed comparisons with FileStorage using a stopwatch. I was surprised at the results. Test #1: I measured the time to create a new database from scratch and start handling requests. I did

Re: [ZODB-Dev] Speedy RelStorage/PostgreSQL

2008-01-30 Thread Shane Hathaway
Andreas Jung wrote: You're system is appearently very slow. Tested the same myself on an Opteron system (2.6 GHz) against a remote Postgres 8.1 DB...takes roughly 5-6 seconds and is basically as fast a FileStorage...great! Oops, you're right... I was running in a power saving mode. With the

Re: [ZODB-Dev] Speedy RelStorage/PostgreSQL

2008-01-30 Thread Shane Hathaway
David Pratt wrote: Hi Shane. Congratulations. Can you advise of compatibility with ZODB 3.8. Many thanks. Yes, it's fully compatible now. A few tests didn't pass at first, but they only failed due to the lack of some stub methods. After I added those methods and created a patch that

Re: [ZODB-Dev] Speedy RelStorage/PostgreSQL

2008-01-30 Thread Shane Hathaway
Flavio Coelho wrote: I think it would be a good idea to update the wiki with instructions of how to enable RelStorage for use with Standalone ZODB. I am particularly interested in doing it. I am anxious to benchmark it Well, let me think through what the wiki might say: 1. Download ZODB

Re: [ZODB-Dev] RelStorage now in Subversion

2008-01-31 Thread Shane Hathaway
Jim Fulton wrote: On Jan 27, 2008, at 3:11 AM, Shane Hathaway wrote: I hope the patch, or a modified version of the patch, will be accepted for inclusion in ZODB 3.9. A monkey patch version is possible, but I'm trying to avoid that. I'm tentatively planning on including it in 3.9. I need

[ZODB-Dev] Re: RelStorage now in Subversion

2008-01-31 Thread Shane Hathaway
Tres Seaver wrote: I've got a question for you on the removal of the LISTEN / NOTIFY stuff in the Postgresql adapter: it looks to me as though your code was not doing the async check-for-notifies as shown in the 'notify.py' in the psycopg2 examples:

Re: [ZODB-Dev] RelStorage now in Subversion

2008-01-31 Thread Shane Hathaway
Dieter Maurer wrote: I am surprised that you think to be able to play with the polling frequency. Postgres will deliver objects as they have been when the transaction started. Therefore, when you start a postgres transaction you must invalidate any object in your cache that has

Re: [ZODB-Dev] RelStorage now in Subversion

2008-02-02 Thread Shane Hathaway
Dieter Maurer wrote: Unless, you begin a new transaction on your load connection after the write connection was committed, your load connection will not see the data written over your write connection. Good point. After a commit, we *must* poll. This implies, the read connection must start

Re: [ZODB-Dev] RelStorage now in Subversion

2008-02-03 Thread Shane Hathaway
Dieter Maurer wrote: Your conflict resolution requires that it sees a transaction as soon as it is commited. The supported relational databases may have this property -- but I expect we do not have a written garantee that this will definitely be the case. I plan to make a test which

Re: [ZODB-Dev] RelStorage now in Subversion

2008-02-03 Thread Shane Hathaway
Dieter Maurer wrote: For read committed this means: it garantees that I will only see committed transactions but not necessarily that I will see the effect of a transaction as soon as it is committed. Your conflict resolution requires that it sees a transaction as soon as it is

Re: [ZODB-Dev] Fixing POSKeyErrors :-)

2008-02-13 Thread Shane Hathaway
Chris Withers wrote: I did try this: data._p_jar = app._p_jar data._p_oid = oid app.x = data import transaction transaction.get().note('Fix POSKeyError') transaction.commit() ...but it didn't work. I don't know if the differences are significant. app.x = data won't work because

Re: [ZODB-Dev] Fixing POSKeyErrors :-)

2008-02-13 Thread Shane Hathaway
Jim Fulton wrote: Also, in the medium term, you can use zc.FileStorage to disable GC during packing. My plan is that this will become a built-in (and cleaner) FileStorage feature in ZODB 3.9. By disabling GC, you mean that packing retains at least one version of every object, correct?

Re: [ZODB-Dev] Fixing POSKeyErrors :-)

2008-02-13 Thread Shane Hathaway
Paul Winkler wrote: We've been discussing doing this for openplans.org... one question that came up that none of us knew the answer to: If I undo some changes to content, what happens to the index values for that content? Does undo fire an ObjectModifiedEvent? Last time I looked it didn't.

Re: [ZODB-Dev] ZODB and Parallel-Python

2008-02-19 Thread Shane Hathaway
Flavio Coelho wrote: The solution I am trying is to offload the db handling code to a separate module that can be added to the import list of a PP job: dbserve.py Code: import ZODB import ZEO.ClientStorage class Data: def __init__(self): #setting up ZEO server

[ZODB-Dev] RelStorage 1.0 beta

2008-02-22 Thread Shane Hathaway
I am pleased to announce the beta release of RelStorage 1.0! RelStorage is a storage implementation for ZODB that stores pickles in a relational database. Features in this release: * Full support for PostgreSQL 8.1+, Oracle 10g, and MySQL 5.0+. * Supports whole-database migration to and

[ZODB-Dev] RelStorage 1.0c1

2008-02-27 Thread Shane Hathaway
RelStorage 1.0c1 is now available. It has two changes: - It is now installable using easy_install and an egg is available. - Transaction metadata is now stored as raw bytes, following the transaction documentation. This change requires a schema migration for all supported databases.

Re: [ZODB-Dev] Re: ZODB Benchmarks

2008-03-04 Thread Shane Hathaway
Roché Compaan wrote: On Mon, 2008-02-25 at 07:36 +0200, Roché Compaan wrote: I'll update my blog post with the final stats and let you know when it is ready. I'll have to keep running these tests because the more I run them the faster the ZODB becomes ;-) Would you have guessed that the ZODB

Re: [ZODB-Dev] Re: ZODB Benchmarks

2008-03-04 Thread Shane Hathaway
Lennart Regebro wrote: On Tue, Mar 4, 2008 at 9:27 PM, Shane Hathaway [EMAIL PROTECTED] wrote: - Did you use optimal methods of retrieval in Postgres? It is frequently not necessary to pull the data into the application. Copying to another table could be faster than fetching rows

Re: [ZODB-Dev] Re: ZODB Benchmarks

2008-03-05 Thread Shane Hathaway
Roché Compaan wrote: On Tue, 2008-03-04 at 13:27 -0700, Shane Hathaway wrote: On a related topic, you might be interested in the RelStorage performance charts I just posted. Don't take them too seriously, but I think the charts are useful. http://shane.willowrise.com/archives/relstorage-10

Re: [ZODB-Dev] Zeo Server as a single point of failure

2008-03-10 Thread Shane Hathaway
Kenneth Miller wrote: I'm currently developing my application ontop of zodb/zeo, and the idea of having the Zeo server the only single point of failure still bothers me. My question is, has anyone thought of any solutions for this. My idea was to have a few or more background

Re: [ZODB-Dev] Zeo Server as a single point of failure

2008-03-10 Thread Shane Hathaway
Christian Theune wrote: Hi, On Mon, Mar 10, 2008 at 05:34:59PM -0500, Kenneth Miller wrote: Hello all, I'm currently developing my application ontop of zodb/zeo, and the idea of having the Zeo server the only single point of failure still bothers me. My question is, has anyone thought

Re: [ZODB-Dev] Re: Zeo Server as a single point of failure

2008-03-12 Thread Shane Hathaway
Tres Seaver wrote: What does Shane have to do with DirectoryStorage? For the record: I am not at all associated with DirectoryStorage, other than I think it's a good project. Shane ___ For more information about ZODB, see the ZODB Wiki:

Re: [ZODB-Dev] anecdotal speed comparison with RelStorage

2008-03-25 Thread Shane Hathaway
Chris Withers wrote: FileStorage-over-ZEO managed the test in 3 mins 20 seconds Relatorage-to-Oracle managed the test in 3 mins 18 seconds Cool, although it's not clear the storage speed had a major impact. If it did have a major impact, then the same test using plain FileStorage or

Re: [ZODB-Dev] pack script for RelStorage

2008-03-27 Thread Shane Hathaway
Chris Withers wrote: Shane Hathaway wrote: Can't you use the standard manage_pack() method? Yes, pack it about the same way you would pack a ZEO storage. OK, but I do that using fspack.py ;-) Running fspack doesn't work with the storage online, does it? Maybe you're talking about

[ZODB-Dev] Re: RelStorage Oracle Barf...

2008-03-29 Thread Shane Hathaway
Chris Withers wrote: Hi Shane, First real world error from the Oracle adapter: Module ZPublisher.Publish, line 125, in publish Module Zope2.App.startup, line 238, in commit Module transaction._manager, line 96, in commit Module transaction._transaction, line 395, in commit

Re: [ZODB-Dev] pack script for RelStorage

2008-03-29 Thread Shane Hathaway
Eric Brun wrote: Hi all, I had convert a FileStorage (Data.fs) in to a RelStorage : Postgresql 8.1, psycopg2, Zope 2.9.8 , Python 2.4.5, Plone 2.1 + FileSystemStorage Zope 2.9.8 have ZODB 3.6.3 version but patch for 3.7.1 run ... I have poor performance like FileStorage perhaps more bad My

Re: [ZODB-Dev] pack script for RelStorage

2008-04-01 Thread Shane Hathaway
Eric Brun wrote: My pack take 2 days ! And my db have the same size as before start packing :-( That is to be expected; with PostgreSQL, you need to vacuum after packing, and even then, it prefers not to actually give back the space unless you vacuum full. You should enable the autovacuum

Re: [ZODB-Dev] Re: Relstorage

2008-04-23 Thread Shane Hathaway
Alan Runyan wrote: I have a feeling its CMFEditions because he is trying to do fancy stuff with the transaction note(s) in the ZODB. At a gut level: I feel 100x more comfortable with Shane's code (ZODB/RelStoage) than CMFEditions. Which is notoriously insane. Thanks. :-) However, early

Re: [ZODB-Dev] Completing ZEO authentication support

2008-04-23 Thread Shane Hathaway
Wichert Akkerman wrote: Shane Hathaway wrote: Wichert Akkerman wrote: I would like to get that fix in for Zope 2.11, which will use ZODB 3.8. Are there any objections to changing this there as well? This is a bit of a grey area between a bugfix and completing an unfinished feature, and I'm

Re: [ZODB-Dev] Completing ZEO authentication support

2008-04-23 Thread Shane Hathaway
Andreas Jung wrote: --On 23. April 2008 09:20:00 -0600 Shane Hathaway [EMAIL PROTECTED] wrote: AFAIK, Jim is working on major ZODB changes for ZODB 3.9. Could we move those major changes into ZODB 3.10 and create ZODB 3.9 with only modest feature enhancements, then use ZODB 3.9 with Zope

Re: [ZODB-Dev] Completing ZEO authentication support

2008-04-23 Thread Shane Hathaway
Jim Fulton wrote: On Apr 23, 2008, at 11:39 AM, Shane Hathaway wrote: Andreas Jung wrote: --On 23. April 2008 09:20:00 -0600 Shane Hathaway [EMAIL PROTECTED] wrote: AFAIK, Jim is working on major ZODB changes for ZODB 3.9. Could we move those major changes into ZODB 3.10 and create ZODB

Re: [ZODB-Dev] Completing ZEO authentication support

2008-04-23 Thread Shane Hathaway
Andreas Jung wrote: Zope 2.11b1 was released in December 2007 :-) Somebody forgot to upload it to zope.org, then. I see only the alpha 1 release: http://www.zope.org/Products/ I'll assume you're correct about the beta being released, so never mind on the ZODB 3.9 idea. Shane

Re: [ZODB-Dev] RelStorage in ZODB 3.8? (was Completing ZEO authentication support)

2008-04-23 Thread Shane Hathaway
Andreas Jung wrote: --On 23. April 2008 09:39:57 -0600 Shane Hathaway [EMAIL PROTECTED] wrote: Andreas Jung wrote: --On 23. April 2008 09:20:00 -0600 Shane Hathaway [EMAIL PROTECTED] wrote: AFAIK, Jim is working on major ZODB changes for ZODB 3.9. Could we move those major changes

Re: [ZODB-Dev] Completing ZEO authentication support

2008-04-24 Thread Shane Hathaway
Wichert Akkerman wrote: I think the onle proposed changes that would be in this new 3.9 are your Scratch the proposal--I suggested it when I didn't realize Zope 2.11b1 was already out. invalidation patch and my authentication patch, is that correct? Considering the interest there appears

[ZODB-Dev] Re: Suggestion: buildout friendly relstorage

2008-04-29 Thread Shane Hathaway
Anton Stonor wrote: I'm about to give RelStorage a spin - it is looking promising. However, it would be great if the zodbconvert.py could be buildout friendly. What is needed is just to move zodbconvert.py inside the relstorage package and define an entry point for it in the setup.py. A few

Re: [ZODB-Dev] Re: Suggestion: buildout friendly relstorage

2008-04-29 Thread Shane Hathaway
Shane Hathaway wrote: I've attached my buildout.cfg. Everything works except for the creation of the zodbconvert script. Any idea what I'm doing wrong? Ahem, here's the attachment. Shane [buildout] parts = zodb3-fakeegg relstorage-patch plone zope2 productdistros

Re: [ZODB-Dev] Re: Suggestion: buildout friendly relstorage

2008-04-29 Thread Shane Hathaway
Andreas Jung wrote: --On 29. April 2008 01:11:43 -0600 Shane Hathaway [EMAIL PROTECTED] wrote: I've attached my buildout.cfg. Everything works except for the creation of the zodbconvert script. Any idea what I'm doing wrong? You need something like that (scripts sections) parts

Re: [ZODB-Dev] Re: Suggestion: buildout friendly relstorage

2008-04-29 Thread Shane Hathaway
Andreas Jung wrote: --On 29. April 2008 01:11:43 -0600 Shane Hathaway [EMAIL PROTECTED] wrote: I've attached my buildout.cfg. Everything works except for the creation of the zodbconvert script. Any idea what I'm doing wrong? You need something like that (scripts sections) parts

Re: [ZODB-Dev] Re: Suggestion: buildout friendly relstorage

2008-04-29 Thread Shane Hathaway
Andreas Jung wrote: --On 29. April 2008 01:11:43 -0600 Shane Hathaway [EMAIL PROTECTED] wrote: I've attached my buildout.cfg. Everything works except for the creation of the zodbconvert script. Any idea what I'm doing wrong? You need something like that (scripts sections) parts

Re: [ZODB-Dev] Re: Relstorage

2008-05-02 Thread Shane Hathaway
Anton Stonor wrote: I just had a look at this and am capable of reproducing it (on Plone 3.1.1) -- but also with filestorage! So it doesn't really look like a RelStorage issue to me. Thanks for working on this, Anton. I think I can call this issue closed WRT RelStorage. Shane

Re: [ZODB-Dev] RelStorage and BLOBs

2008-05-06 Thread Shane Hathaway
David Durham, Jr. wrote: My questions is whether or not RelStorage supports BLOBs. My situation is that I have a Plone site with user uploaded images and other files, that should presumably be stored as BLOBs with RelStorage. I'm open to placing BLOBs outside of RelStorage if that's reasonably

Re: [ZODB-Dev] Note for the docs: RelStorage requires at least MySQLdb 1.2.2 to work with Mysql

2008-05-07 Thread Shane Hathaway
Anton Stonor wrote: Just wanted to share that RelStorage does not work with the MySQLdb 1.2.1 adapter. 1.2.2 seems to work fine. [...] Maybe this requirement could go into the docs? Well, it's already in the docs. Look for 1.2.1 in README.txt.

Re: [ZODB-Dev] RelStorage and BLOBs

2008-05-07 Thread Shane Hathaway
Wichert Akkerman wrote: For some of us a the main reason for using RelStorage is to be able to satisfy the 'all data must be stored in our oracle cluster' requirement from enterprise customers. That means we can not use NFS mounts to store blobs. You have a point, and I know how stubborn

Re: [ZODB-Dev] RelStorage: Unable to acquire commit lock

2008-05-10 Thread Shane Hathaway
On 9. Mai 2008, at 18:31, Shane Hathaway wrote: Perhaps the scope of the commit lock in MySQL is the entire database server rather than just the database, which would explain this. We should make sure each database has its own commit lock and pack lock. Stefan H. Holek wrote: yeas and yes

Re: [ZODB-Dev] RelStorage: Which tables to replicate?

2008-05-22 Thread Shane Hathaway
Stefan H. Holek wrote: We have put up two boxes in a MySQL master-master replication setup [1]. As long as we only write to one of the masters all is fine. Writing to both masters (and expecting MySQL to sort it out) is giving us trouble. We see things like: Last_Errno: 1062 Last_Error:

Re: [ZODB-Dev] Relstorage: Slow packing with Mysql when GC is turned on

2008-05-26 Thread Shane Hathaway
Anton Stonor wrote: While running the pack, the expensive query is this one according to show processlist: UPDATE pack_object SET keep = TRUE WHERE keep = FALSE AND zoid IN ( SELECT DISTINCT to_zoid FROM object_ref JOIN temp_pack_visit USING (zoid) ) show processlist says: Copying to

Re: [ZODB-Dev] Advice on ZODB with large datasets

2008-06-18 Thread Shane Hathaway
[EMAIL PROTECTED] wrote: We have a large dataset of 650,000+ records that I'd like to examine easily in Python. I have figured out how to put this into a ZODB file that totals 4 GB in size. But I'm new to ZODB and very large databases, and have a few questions. 1. The data is in a IOBTree

Re: [ZODB-Dev] RelStorage: Clearing temp_store in replication-friendly way

2008-07-24 Thread Shane Hathaway
Stefan H. Holek wrote: I have made two checkins to relstorage 1.1 branch: It's really cool to have another contributor. Thanks! 1.1c1 has already been tagged, so I fixed the change log. [1] http://svn.zope.org/relstorage/branches/1.1/?rev=88789view=rev [2]

[ZODB-Dev] RelStorage 1.1c1 Released

2008-07-24 Thread Shane Hathaway
RelStorage 1.1c1 is now available on PyPI [1]! This is the time to try the new features in RelStorage 1.1. This release has been mainly focused on performance improvements: - Optimized packing on all supported databases and minimized the impact of packing on concurrent commits. This should

Re: [ZODB-Dev] RelStorage: Clearing temp_store in replication-friendly way

2008-07-24 Thread Shane Hathaway
Stephan Richter wrote: We should get Brian Aker into this discussion, since is one of the main architects of MySQL and one of the founders of memcached. As Shane knows, I have some access to Brian these days, so we could have an online meeting talking about it. Cool. Memcache could do a

Re: [ZODB-Dev] RelStorage: SVN trunk not up-to-date

2008-07-27 Thread Shane Hathaway
Stefan H. Holek wrote: Most projects on svn.zope.org have the policy to perform new work on the trunk, cutting maintenance branches from it when appropriate. RelStorage appears to forge ahead on its 1.1 branch, allowing the trunk to go stale. What's the policy here? If I wanted to create

Re: [ZODB-Dev] RelStorage: Clearing temp_store in replication-friendly way

2008-07-27 Thread Shane Hathaway
Stefan H. Holek wrote: In MySQL, temporary tables live in RAM. So when a slave goes down, its copy of the table vanishes. When the slave comes back up the log may still contain commands using the temporary table however, causing execution to barf (and replication to stop). To fix this

Re: [ZODB-Dev] Translation of Relstorage documentation to Brazilian Portuguese

2008-09-08 Thread Shane Hathaway
Rogerio Ferreira wrote: I translated the Relstorage documentation (README.txt) to Brazilian Portuguese. I would like to available to svn from project... Cool. Since it's on svn.zope.org, you'll need to sign a standard Zope Subversion contributor agreement.

Re: [ZODB-Dev] Backing up Data.fs and blob directory

2008-09-08 Thread Shane Hathaway
Chris Withers wrote: Sidnei da Silva wrote: Keep in mind rsync is not erm, trivial to get going on Windows. Really? I've never had problems with cygwin... +1, Cygwin with rsync, ssh, and a short script is reliable and fairly easy to set up, in my experience. Shane

Re: [ZODB-Dev] Experiences with Relstorage performance for setups with heavy writes

2008-09-17 Thread Shane Hathaway
Dieter Maurer wrote: Andreas Jung wrote at 2008-9-12 10:31 +0200: anyone having experiences with the performance of Relstorage on Zope installations which heavy parallel writes (which is often a bottleneck). Does Relstorage provide any significant advantages over ZEO. As Relstorage

Re: [ZODB-Dev] Broken instances after refactoring in ZODB

2008-10-03 Thread Shane Hathaway
Marius Gedminas wrote: 1. create modules in their old locations that import all the classes from the new locations (and do nothing else). 2. write a script that opens the DB, iterates over all persistent objects in it, touches every single one of them (obj._p_activate()

Re: [ZODB-Dev] Broken instances after refactoring in ZODB

2008-10-06 Thread Shane Hathaway
Leonardo Santagada wrote: Why doesn't zodb has a table of some form for this info? I heard that sometimes for very small objects the string containing this information can use up to 30% of the whole space of the file (using FileStorage). How does RelStorage store this? RelStorage does

Re: [ZODB-Dev] Amazon SimpleDB Adapter

2008-10-11 Thread Shane Hathaway
Benjamin Liles wrote: Currently at the Plone conference it seems that a large number of people are beginning to host their Plone sites on the Amazon EC2 service. A simpleDB adapter might be a good way to provide persistent storage for an EC2 base Zope instance. Has there been any interest in

Re: [ZODB-Dev] Amazon SimpleDB Adapter

2008-10-11 Thread Shane Hathaway
Laurence Rowe wrote: I'm not sure RelStorage is the best place for it - SimpleDB is very different to relational databases. RelStorage doesn't use much of a relational database either (except during packing). A couple of years ago I experimented with s3storage [1]. This turned out to be

Re: [ZODB-Dev] Amazon SimpleDB Adapter

2008-10-13 Thread Shane Hathaway
David Pratt wrote: There is no lack of storage options out there these days. My opinion is that if you want the true characteristics of the storage, you won't make it behave like a zodb. Lovely systems came to a similar conclusion after trying relstorage and then moving to use storm in a

Re: [ZODB-Dev] Open Connections with RelStorage

2008-10-23 Thread Shane Hathaway
Santi Camps wrote: RelStorage is really fantastic, I'm starting to use it in some production environments with postgresql 8.1 and works really fine. But I've observed the number of open connections is always increasing, but never decreases. How many connections are we talking about? It

Re: [ZODB-Dev] Open Connections with RelStorage

2008-10-23 Thread Shane Hathaway
Santi Camps wrote: I've 8 zope's with 4 threads each one, and I've seen more than 50 open connections used by relstorage. I was hopping one connection per thread, like postgressql adapters does. The number of connections of relstorage doesn't depend of number of zope threads ?. RelStorage

Re: [ZODB-Dev] Open Connections with RelStorage

2008-10-23 Thread Shane Hathaway
Stephan Richter wrote: Shane, is there some wisdom here of a preferred setup in terms of how many threads per Zope and how many connections max on a server? A good setup probably isn't very different from a ZEO setup, with one exception: RelStorage shares a lot less state between threads than

Re: [ZODB-Dev] Open Connections with RelStorage

2008-10-23 Thread Shane Hathaway
Shane Hathaway wrote: Santi Camps wrote: I've 8 zope's with 4 threads each one, and I've seen more than 50 open connections used by relstorage. I was hopping one connection per thread, like postgressql adapters does. The number of connections of relstorage doesn't depend of number of zope

Re: [ZODB-Dev] RelStorage support in 3.9

2008-10-30 Thread Shane Hathaway
Jim Fulton wrote: I'd like to include RelStorage support in ZODB 3.9. The current patch is insufficient because it doesn't include tests or interfaces that describe the interaction between RelStorage and ZODB. I'll need these in the next few weeks, if support is to be included in 3.9.

Re: [ZODB-Dev] Relstorage pack problems

2009-01-19 Thread Shane Hathaway
Santi Camps wrote: We've been using RelStorage 1.1.c1 in production environments over postgresql 8.1 for some time. It has been working really fine, but yesterday we have a big problem packing a relstorage zodb mounted as a zope mount point. I'm guessing that your mounted object is not

Re: [ZODB-Dev] Relstorage pack problems

2009-01-20 Thread Shane Hathaway
Santi Camps wrote: Sorry, I miss to say we are using Zope 2.9.4, and that mounted object is already reachable from the mounted database root (adding a ZODB mount point to the root thought ZMI). Ok. Although, the resultant database, after pack, has a big size. So, not all was deleted, but

Re: [ZODB-Dev] Relstorage pack problems

2009-01-22 Thread Shane Hathaway
Santi Camps wrote: I already know what happens: either object_ref and object_refs_added tables are completely empty in my database. I can't understand why, Those tables only get populated by packing. If you're saying that they are still empty even after a successful pre_pack run with gc

Re: [ZODB-Dev] Relstorage pack problems

2009-01-22 Thread Shane Hathaway
Santi, I hope you don't mind me discussing your database in public. I'm not going to talk about anything that looks like it could be private. Other RelStorage users might benefit from the analysis. Looking at your database, I see that something bad happened just before transaction

Re: [ZODB-Dev] Relstorage pack problems

2009-01-22 Thread Shane Hathaway
Laurence Rowe wrote: Shane Hathaway wrote: I should note that this KeyError occurs while trying to report on a KeyError. I need to fix that. Fortunately, the same error pops out anyway. There's a fix for this in the Jarn branch. Note that to collect more interesting data it rolls back

Re: [ZODB-Dev] How to turn off 'GC' when packing on ZODB3.6.2

2009-01-22 Thread Shane Hathaway
eastxing wrote: For choice 3, 'RelStorage' provide a 'ZODBConverter', but I do not to know if it can used with ZODB3.6.2? I think RelStorage could support ZODB 3.6 without much effort, but I haven't tested it. I'd go with Jim's suggestion of using a new ZEO server. Shane

Re: [ZODB-Dev] Relstorage pack problems

2009-01-23 Thread Shane Hathaway
Santi Camps wrote: I attach the script we use to do the conversion. Be free to include in Relstorage if you think it's useful and it is well done (as I said, I really don't know much about ZODB, I just mix zodbconvert.py with some DirectoryStorage code) Ah-ha, you removed from

Re: [ZODB-Dev] Relstorage pack problems

2009-01-23 Thread Shane Hathaway
Alan Runyan wrote: On Fri, Jan 23, 2009 at 1:45 PM, Shane Hathaway sh...@hathawaymix.org mailto:sh...@hathawaymix.org wrote: = select zoid from current_object where tid != (select max(tid) from object_state where object_state.zoid = current_object.zoid); zoid

Re: [ZODB-Dev] Connection lost using RelStorage on Oracle 10g

2009-01-31 Thread Shane Hathaway
Riccardo Minet wrote: Module relstorage.relstorage, line 983, in poll_invalidations Module relstorage.relstorage, line 154, in _restart_load Module relstorage.adapters.oracle, line 351, in restart_load DatabaseError: ORA-03135: connection lost contact It's happening right at a point where

Re: [ZODB-Dev] Connection lost using RelStorage on Oracle 10g

2009-02-02 Thread Shane Hathaway
Riccardo Minet wrote: thanks a lot for your answer. Your diagnosis seems to be correct but maybe the same fix should be applied to the restart_store method too. See the log below: Good catch! Module relstorage.relstorage, line 505, in tpc_begin Module relstorage.relstorage, line 186, in

Re: [ZODB-Dev] Connection lost using RelStorage on Oracle 10g

2009-02-06 Thread Shane Hathaway
Riccardo Minet wrote: 2009-02-06T12:57:50 ERROR Zope.SiteErrorLog http://laerte.te.actalis.it:18080/Control_Panel/Database/main/manage_pack Traceback (innermost last): Module ZPublisher.Publish, line 119, in publish Module ZPublisher.mapply, line 88, in mapply Module

Re: [ZODB-Dev] Can I use RelStorage with ZODB3.9.0

2009-03-05 Thread Shane Hathaway
eastxing wrote: About one month ago, I asked a question about 'ZODB pack' and got suggestions to update to new ZODB version. It took me one month to update my site from Plone2.5.5(with Zope2.9.6-final,ZODB3.6.2) to Plone3.1.7(with Zope2.10.7,ZODB3.7.3). Then I update zasync(an schedule

Re: [ZODB-Dev] Can I use RelStorage with ZODB3.9.0

2009-03-06 Thread Shane Hathaway
eastxing wrote: Hi Shane, Please. thanks! eastxing Done. See: http://packages.willowrise.org/ The page now includes a patched version of ZODB 3.9.0a12. Shane ___ For more information about ZODB, see the ZODB Wiki:

Re: [ZODB-Dev] Relstorage and ZEO

2009-03-31 Thread Shane Hathaway
Anthony Gerrard wrote: OK thanks. To confirm you can have a cluster of Zope instances all using relstorage to connect to the same database. This negates the need for ZEO as the database handles the multi-threaded data access. Yes. Nice ASCII art. :-) Shane

Re: [ZODB-Dev] Version of mysql to use with Relstorage

2009-03-31 Thread Shane Hathaway
Anthony Gerrard wrote: The Relstorage pypi site recommends against using version 5.1.23 of MySQL because of major bugs. As it doesn't state explicitly what these bugs are I can't tell whether they're fixed in the latest version (5.1.32). Does anyone know whether it's safe to use this latest

Re: [ZODB-Dev] Estrange behaviour with BTreeFolder2

2009-04-01 Thread Shane Hathaway
Santi Camps wrote: I'm using RelStorage, but this shouldn't affect. I will try the same tests in a FileStorage to be sure It seems to be a Relstorage specific issue. The same tests using FileStorage doesn't produce memory garbage ... It could be something related to poll-interval

Re: [ZODB-Dev] Estrange behaviour with BTreeFolder2

2009-04-01 Thread Shane Hathaway
Santi Camps wrote: I'm not saying that pickle cache was affected, I'm saying that loading a BTreeFolder2 seems very affected. In a FileStorage system, objects contained in the BTreeFolder are not loaded when the BTreeFolder itself is loaded, but in RelStorage, a lot of contained objects are

Re: [ZODB-Dev] merging databases containing multidatabase references

2009-04-24 Thread Shane Hathaway
Christian Theune wrote: On Thu, 2009-04-23 at 15:02 -0700, David Glick wrote: Having now realized the folly of trying to split things into multiple databases, I would really like to merge these 3 databases back into 1. Any ideas for how I could go about this? Based on

Re: [ZODB-Dev] merging databases containing multidatabase references

2009-04-27 Thread Shane Hathaway
David Glick wrote: Taking the existing ExportImport.py code as a starting point, it looks like I would need to: 1. modify the persistent_load method to handle rewriting extended references of types 'n' and 'm' 2. provide some way to process several export files at once, and input a

[ZODB-Dev] Proposal: cross database reference seat belt

2009-04-28 Thread Shane Hathaway
Problem --- In multi-database configurations, ZODB applications can unintentionally create cross-database references. The causes include moving (rather than copying) an object between containers, storing an object in a session database and later moving it to the main database, and using a

Re: [ZODB-Dev] Proposal: cross database reference seat belt

2009-04-28 Thread Shane Hathaway
Jim Fulton wrote: Thanks for writing this up! Do we really need per-object control? It would be simpler to simply allow or disallow (new) cross-data references at the database level. I think the most common use case for cross-database connection limitation is support for mounts.

[ZODB-Dev] Proposal (version 2): cross database reference seat belt

2009-04-28 Thread Shane Hathaway
Problem --- In multi-database configurations, ZODB applications can unintentionally create cross-database references. The causes include moving (rather than copying) an object between containers, storing an object in a session database and later moving it to the main database, and using a

Re: [ZODB-Dev] ZODB 3.9.0b1 released

2009-05-17 Thread Shane Hathaway
Tim Cook wrote: Tests with failures: /home/tim/ZODB3/ZODB3-3.9.0b1/src/ZEO/scripts/zeopack.test check7ZODBThreads (ZODB.tests.testMVCCMappingStorage.MVCCMappingStorageTests) Total: 3691 tests, 2 failures, 0 errors in 17 minutes 13.990 seconds. FWIW, I just fixed both of those failures

Re: [ZODB-Dev] URGENT: ZODB down - Important Software Application at CERN

2009-05-26 Thread Shane Hathaway
Jim Fulton wrote: FileStorage indexes can't be saved after they reach a certain size, where size roughly based on the number of objects. I need to find a way to fix this. It might be interesting to use SQLite for FileStorage indexes. With SQLite, we wouldn't have to store the whole index

  1   2   3   >