Re: [ZODB-Dev] Problems packing Zope 2.9.8 storage
Shane Hathaway wrote: Look at this chart in particular, which suggests that RelStorage with PostgreSQL or MySQL is several times faster than ZEO for heavy read loads: Cool, now all I need is something like zeoraid for postgres (or, at a push, mysql)... Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Problems packing Zope 2.9.8 storage
Chris Withers wrote: Shane Hathaway wrote: Look at this chart in particular, which suggests that RelStorage with PostgreSQL or MySQL is several times faster than ZEO for heavy read loads: Cool, now all I need is something like zeoraid for postgres (or, at a push, mysql)... There are existing replication solutions for PostgreSQL and I am fairly confident at least one of them will work with RelStorage. It's a simple schema, after all. I know RelStorage replication works with MySQL. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Problems packing Zope 2.9.8 storage
Chris Withers wrote: I've got a ZODB that's ballooned and I'm now having problems packing with zeopack. I tried with zeopack, which appeared to just hang (still in top, but no cpu usage for an hour or two), so I ctrl-c'ed it. Now when I try, I just get the text Already packing back. The ZEO server does all the packing work. The zeopack utility only tells the ZEO server to get started, then waits for it to finish. It probably wasn't hung. There is a .old file there, but it was last modified over a week ago. The analysis phase of packing can take a long time. A week is too much, though. :-) If so, what should I look to do now? If I were solving it, I would see what happens if I copy the storage to RelStorage and then pack it. Maybe that's just me though. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Problems packing Zope 2.9.8 storage
Shane Hathaway wrote: There is a .old file there, but it was last modified over a week ago. The analysis phase of packing can take a long time. A week is too much, though. :-) Yeah, this is off the back of an automated pack/backup/unpack/verify to keep two storages in sync on two database servers, one live, one hot standby. The process went wrong because one of the storages has ballooned in size... If so, what should I look to do now? If I were solving it, I would see what happens if I copy the storage to RelStorage and then pack it. Maybe that's just me though. I thought RelStorage was slow at packing? cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Problems packing Zope 2.9.8 storage
Shane Hathaway wrote: Chris Withers wrote: I thought RelStorage was slow at packing? Not anymore. I optimized packing last December/January and it has been plenty fast ever since, AFAIK. Interesting. Got any metrics on its speed compared to FileStorage, including packing? cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Problems packing Zope 2.9.8 storage
Chris Withers wrote: Shane Hathaway wrote: Chris Withers wrote: I thought RelStorage was slow at packing? Not anymore. I optimized packing last December/January and it has been plenty fast ever since, AFAIK. Interesting. Got any metrics on its speed compared to FileStorage, including packing? I don't have numbers on pack speed, but I've eliminated all the speed issues people have brought up. People have been looking at these performance charts for quite a while: http://shane.willowrise.com/archives/how-to-fix-the-mysql-write-speed/ http://shane.willowrise.com/archives/relstorage-111-coming-soon/ http://shane.willowrise.com/archives/relstorage-10-and-measurements/ Look at this chart in particular, which suggests that RelStorage with PostgreSQL or MySQL is several times faster than ZEO for heavy read loads: http://shane.willowrise.com/wp-content/uploads/2009/03/relstorage-113-speed-10k.png I am waiting for someone to dispute these results. I *want* ZEO to be faster, since its code paths are simpler than any relational database, but the performance measurements suggest that under pressure, RelStorage with MySQL or PostgreSQL can read 3 objects in the time it takes ZEO with FileStorage to read 1. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Problems packing Zope 2.9.8 storage
On Mon, Sep 14, 2009 at 2:44 PM, Shane Hathaway sh...@hathawaymix.org wrote: People have been looking at these performance charts for quite a while: Are the benchmarks publicly available? I looked through the blog posts and svn.zope.org but couldn't find them. -- Benji York Senior Software Engineer Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Problems packing Zope 2.9.8 storage
Benji York wrote: On Mon, Sep 14, 2009 at 2:44 PM, Shane Hathaway sh...@hathawaymix.org wrote: People have been looking at these performance charts for quite a while: Are the benchmarks publicly available? I looked through the blog posts and svn.zope.org but couldn't find them. http://svn.zope.org/relstorage/trunk/relstorage/tests/speedtest.py?view=markup That script generates CSV, which I paste into this spreadsheet to produce the graphs: http://svn.zope.org/relstorage/trunk/relstorage/tests/comparison.ods Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Problems packing Zope 2.9.8 storage
On Mon, Sep 14, 2009 at 3:52 PM, Shane Hathaway sh...@hathawaymix.org wrote: Benji York wrote: On Mon, Sep 14, 2009 at 2:44 PM, Shane Hathaway sh...@hathawaymix.org wrote: People have been looking at these performance charts for quite a while: Are the benchmarks publicly available? I looked through the blog posts and svn.zope.org but couldn't find them. [snip answer] Thanks! -- Benji York Senior Software Engineer Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
