Re: [ZODB-Dev] PGStorage
--On 24. Januar 2008 15:42:01 +0100 Andreas Jung [EMAIL PROTECTED] wrote: --On 23. Januar 2008 18:17:18 +0100 Andreas Jung [EMAIL PROTECTED] wrote: --On 22. Januar 2008 21:17:45 -0500 Stephan Richter [EMAIL PROTECTED] wrote: On Tuesday 22 January 2008, Dieter Maurer wrote: OracleStorage was abandoned because it was almost an order or magnitude slower than FileStorage. Actually, Lovely Systems uses PGStorage because it is faster for them. It would be interesting where PGS is faster than filestorage. Ok, I just tried made a simple benchmark for testing write performance. I created a Plone site and then created a copy using copy/paste within the ZMI. (AMD Dualcore 2.6 GHz, Postgres 7.4.7 running on dedicated DB server): CopyPaste using Filestorage: 3-4 seconds CopyPaste using PGStorage: 30-40 seconds Alan asked me to so some further testing. I wrote a small script creating 100 Documents within one transaction (both in CMF and Plone) and I calculated the transaction time (not included the overhead for creating the instances)...the numbers are pretty much the self-speaking: CMFPlone PGStorage10-12 secs 15-18 secs Filestorage 0.3-0.4 secs 0.4-0.6 secs I must mention that all tests were done using PGStorage 0.1 - the only version available to me. Rumors say that Shane might release improved versions. So the numbers must be taken with care as they reflect the state of PGStorage as of 2006. Andreas pgpGhRO9tzjfl.pgp Description: PGP signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] PGStorage
--On 24. Januar 2008 11:21:28 -0400 David Pratt [EMAIL PROTECTED] wrote: Hi Andreas. No need to rely on rumors. The current source is available here: http://pgstorage.cvs.sourceforge.net/pgstorage/ Excellent. I re-tried my benchmarks of course. CopyPaste of a complete new Plone site is now nearly as fast as with Filestorage (just by looking on my watch). The tests for commiting 100 objects are also blazing fast: roughly 1.2 - 1.5 seconds for committing. That's roughly 10x faster than with PGStorage 0.1 and about 2-3 slower than Filestorage...not that bad. So PGStorage appears as an interesting alternative. Andreas pgpTPVH7FHEOg.pgp Description: PGP signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] PGStorage
Andreas Jung wrote: Excellent. I re-tried my benchmarks of course. CopyPaste of a complete new Plone site is now nearly as fast as with Filestorage (just by looking on my watch). The tests for commiting 100 objects are also blazing fast: roughly 1.2 - 1.5 seconds for committing. That's roughly 10x faster than with PGStorage 0.1 and about 2-3 slower than Filestorage...not that bad. So PGStorage appears as an interesting alternative. Hi y'all, I just noticed this thread. Interesting timing! :-) Jarn (formerly Plone Solutions) is now funding development of PGStorage into a new storage that interfaces with either Oracle or PostgreSQL. The new storage is called RelStorage and they have agreed to make it open source! I've been plugging away at it for a while and just last night I got Oracle 10g XE to pass the ZODB tests. It was no small feat (it involved rewriting the packing algorithm) and to achieve it I had to temporarily break the PostgreSQL support. As soon as I get PostgreSQL operation back in order, I intend to post the code on svn.zope.org. As for benchmarks, I've decided it's not really fair to compare FileStorage with RelStorage/PGStorage, since RelStorage provides functionality comparable with ZEO and ZRS. For small, single-server sites, FileStorage is going to continue to be the performance leader, because it doesn't have to use the network stack and simply has less work to do. Instead, I've been comparing ZEO+FileStorage with RelStorage+PostgreSQL. In that light, my most recent graphs suggest that RelStorage wins especially as you scale up. YMMV of course! Even if RelStorage wins, I'm sure Jim won't stay still. I bet a lot more performance could be squeezed out of ZEO and ZRS. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] PGStorage
Zvezdan Petkovic wrote at 2008-1-23 17:15 -0500: On Jan 23, 2008, at 4:05 PM, Flavio Coelho wrote: sorry, I never meant to email you personally I have been wrong: Flavio has not forgotten the list, I had not looked carefully enough. Sorry! -- Dieter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] PGStorage
Thats a good thing to know, I will do some testing of my own... Thanks Flávio On Jan 23, 2008 2:36 AM, David Pratt [EMAIL PROTECTED] wrote: Yes, Shane had done some benchmarking about a year or so ago. PGStorage was actually faster with small writes but slower for larger ones. As far as packing, as a zodb implementation, packing is still required to reduce the size of data in Postgres. BTW Stephan, where is Lovely using it - a site example? I had read some time ago that they were exploring it but not that it was being used. Regards, David Stephan Richter wrote: On Tuesday 22 January 2008, Dieter Maurer wrote: OracleStorage was abandoned because it was almost an order or magnitude slower than FileStorage. Actually, Lovely Systems uses PGStorage because it is faster for them. Regards, Stephan ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev -- Flávio Codeço Coelho My grandfather once told me that there were two kinds of people: those who do the work and those who take the credit. He told me to try to be in the first group; there was much less competition. Indira Gandhi registered Linux user # 386432 get counted at http://counter.li.org ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] PGStorage
On Jan 22, 2008, at 8:44 PM, Marius Gedminas wrote: On Tue, Jan 22, 2008 at 05:43:42PM -0200, Flavio Coelho wrote: Actually what I am trying to run away from is the packing monster ;-) I want to be able to use an OO database without the inconvenience of having it growing out of control and then having to spend hours packing the database every once in a while. (I do a lot of writes in my DBs). Do this Holy grail of databases exist? :-) I've learned to love this aspect of FileStorage. Sure, the Data.fs is measured in gigabytes, but when the users run to you crying help! help! why is this bit of data not what I expected it to be? you can dig into object history from a debugzope console and figure what changed it and when. Or restore deleted data, for that matter. We can have our cake and eat it too. We can keep multiple revisions and pack them incrementally. Keeping multiple revisions is almost necessary, because without doing so, MVCC isn't effective. The old Berkeley DB storage did this. It automatically packed records older than a configurable time. Berkeley DB storage didnt' work out the first time around. My experience optimizing packing for FileStorage reminded me how much I want an alternative to FileStorage for large active databases. I intend to revisit Berkeley DB storage one of these days. Jim -- Jim Fulton Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] PGStorage
Jim, What would you consider a large active database? curiously, alan ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] PGStorage
Flavio Coelho wrote: Actually what I am trying to run away from is the packing monster ;-) Jim has done a great deal of work on packing (that will go into 3.9 I presume) that should make your pack 3 to 6 times faster (depending on if you do garbage collection at pack time or not). I want to be able to use an OO database without the inconvenience of having it growing out of control and then having to spend hours packing the database every once in a while. (I do a lot of writes in my DBs). Do this Holy grail of databases exist? :-) Why not put the pack in cron? -- Benji York Senior Software Engineer Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] PGStorage
--On 22. Januar 2008 21:17:45 -0500 Stephan Richter [EMAIL PROTECTED] wrote: On Tuesday 22 January 2008, Dieter Maurer wrote: OracleStorage was abandoned because it was almost an order or magnitude slower than FileStorage. Actually, Lovely Systems uses PGStorage because it is faster for them. It would be interesting where PGS is faster than filestorage. Ok, I just tried made a simple benchmark for testing write performance. I created a Plone site and then created a copy using copy/paste within the ZMI. (AMD Dualcore 2.6 GHz, Postgres 7.4.7 running on dedicated DB server): CopyPaste using Filestorage: 3-4 seconds CopyPaste using PGStorage: 30-40 seconds I doubt that one could optimize the write performance using PGStorage significantly. DCOracleStorage had a similar bad performance compared to Filestorage (10 times slower as far as I can remember). Andreas pgpeWpFXRU5v5.pgp Description: PGP signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] PGStorage
Andreas, Could you try to mount the catalog separately? My understanding is the catalog is what makes storages a misery. CMF/portal_catalog mounted as Filestorage and CMF could be mounted as PGStorage. I presume you would see much more reasonable performance? cheers -- Alan Runyan Enfold Systems, Inc. http://www.enfoldsystems.com/ phone: +1.713.942.2377x111 fax: +1.832.201.8856 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] PGStorage
Likely but I don#t know how to setup my instance in order to make the copied instance making use of mounted catalog. What about creating a Plone site with a FileStorage. Then mount a subfolder into PGStorage say /Plone/some_folder then you could see create af older /Plone/new_folder and dump buncha content and paste it into /Plone/foo_folder and paste it into /Plone/some_folder and see the difference? If there is a benefit..what would be arguments for running a mixed setup? My understanding is that each object in a catalog (Plone has 3 cataogs) has numerous other objects associated with it. Something like: each record in a catalog may have an object for each index/metadata attribute you are capturing. and possibly a few others. Each catalog entries contain ts of object per object being indexed.That is my understanding. -- Alan Runyan Enfold Systems, Inc. http://www.enfoldsystems.com/ phone: +1.713.942.2377x111 fax: +1.832.201.8856 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] PGStorage
--On 23. Januar 2008 13:32:29 -0600 Alan Runyan [EMAIL PROTECTED] wrote: Likely but I don#t know how to setup my instance in order to make the copied instance making use of mounted catalog. What about creating a Plone site with a FileStorage. Then mount a subfolder into PGStorage say /Plone/some_folder then you could see create af older /Plone/new_folder and dump buncha content and paste it into /Plone/foo_folder and paste it into /Plone/some_folder and see the difference? If there is a benefit..what would be arguments for running a mixed setup? My understanding is that each object in a catalog (Plone has 3 cataogs) has numerous other objects associated with it. Something like: each record in a catalog may have an object for each index/metadata attribute you are capturing. and possibly a few others. Each catalog entries contain ts of object per object being indexed.That is my understanding. That would be an artificial test. You basically want to keep your data in one storage type - in this case either Filestorage or PGStorage. Why no mix? For an application like Plone the catalog is as important as the data. You don#t want to lose data during production. Reindexing can be very expensive and possibly causes a longer down time. On the other hand you don't want run two different storage types at the same time - one point of failure more. Copy and paste of a whole is likely not the common usecase but burst writes as in this particular case appear to be very slow with PGStorage. Andreas pgpsGjKg6sttD.pgp Description: PGP signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] PGStorage
Flavio Coelho wrote at 2008-1-22 17:43 -0200: ... Actually what I am trying to run away from is the packing monster ;-) Jim has optimized pack consideraly (-- zc.FileStorage). I, too, have worked on pack optimization the last few days (we cannot yet use Jims work because we are using ZODB 3.4 while Jims optimization is for ZODB 3.8) and obtained speedups of more then 80 persent. I want to be able to use an OO database without the inconvenience of having it growing out of control and then having to spend hours packing the database every once in a while. (I do a lot of writes in my DBs). Do this Holy grail of databases exist? :-) The pack equivalent of Postgres is called vacuum full. It is more disruptive than packing Maybe, you have a look at the old bsddbstorage. It could be configured to not use historical data. Support was discontinued due to lack of interest -- but I report this for the second time within a week or so. This may indicate a renewed interest. BTW: stay on the list. I do not like personal emails. -- Dieter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] PGStorage
On Jan 23, 2008 6:45 PM, Dieter Maurer [EMAIL PROTECTED] wrote: Flavio Coelho wrote at 2008-1-22 17:43 -0200: ... Actually what I am trying to run away from is the packing monster ;-) Jim has optimized pack consideraly (-- zc.FileStorage). I, too, have worked on pack optimization the last few days (we cannot yet use Jims work because we are using ZODB 3.4 while Jims optimization is for ZODB 3.8) and obtained speedups of more then 80 persent. I want to be able to use an OO database without the inconvenience of having it growing out of control and then having to spend hours packing the database every once in a while. (I do a lot of writes in my DBs). Do this Holy grail of databases exist? :-) The pack equivalent of Postgres is called vacuum full. It is more disruptive than packing Maybe, you have a look at the old bsddbstorage. It could be configured to not use historical data. Support was discontinued due to lack of interest -- but I report this for the second time within a week or so. This may indicate a renewed interest. Thanks I will look at it. BTW: stay on the list. I do not like personal emails. sorry, I never meant to email you personally, this due to the default configuration of this list, that sets the reply-to to the poster instead of to the list...So I may have been too quick to the send button on a reply. -- Dieter -- Flávio Codeço Coelho My grandfather once told me that there were two kinds of people: those who do the work and those who take the credit. He told me to try to be in the first group; there was much less competition. Indira Gandhi registered Linux user # 386432 get counted at http://counter.li.org ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] PGStorage
Thanks Dieter, Actually what I am trying to run away from is the packing monster ;-) I want to be able to use an OO database without the inconvenience of having it growing out of control and then having to spend hours packing the database every once in a while. (I do a lot of writes in my DBs). Do this Holy grail of databases exist? :-) Flávio On Jan 22, 2008 5:35 PM, Dieter Maurer [EMAIL PROTECTED] wrote: Flavio Coelho wrote at 2008-1-22 10:57 -0200: ... Can anyone tell me if PGstorage is stable enough for production use? I expect that it will behave similar to OracleStorage. OracleStorage was abandoned because it was almost an order or magnitude slower than FileStorage. Carefully think whether you really need pickle data in a relational database (rather than in the file system). -- Dieter -- Flávio Codeço Coelho My grandfather once told me that there were two kinds of people: those who do the work and those who take the credit. He told me to try to be in the first group; there was much less competition. Indira Gandhi registered Linux user # 386432 get counted at http://counter.li.org ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] PGStorage
On Tue, Jan 22, 2008 at 05:43:42PM -0200, Flavio Coelho wrote: Actually what I am trying to run away from is the packing monster ;-) I want to be able to use an OO database without the inconvenience of having it growing out of control and then having to spend hours packing the database every once in a while. (I do a lot of writes in my DBs). Do this Holy grail of databases exist? :-) I've learned to love this aspect of FileStorage. Sure, the Data.fs is measured in gigabytes, but when the users run to you crying help! help! why is this bit of data not what I expected it to be? you can dig into object history from a debugzope console and figure what changed it and when. Or restore deleted data, for that matter. Marius Gedminas -- MSDOS didn't get as bad as it is overnight -- it took over ten years of careful development. -- [EMAIL PROTECTED] signature.asc Description: Digital signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] PGStorage
On Tuesday 22 January 2008, Dieter Maurer wrote: OracleStorage was abandoned because it was almost an order or magnitude slower than FileStorage. Actually, Lovely Systems uses PGStorage because it is faster for them. Regards, Stephan -- Stephan Richter Web Software Design, Development and Training Google me. Zope Stephan Richter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] PGStorage
On Tuesday 22 January 2008, David Pratt wrote: BTW Stephan, where is Lovely using it - a site example? I had read some time ago that they were exploring it but not that it was being used. I think they are using it for video.vol.at, but you should contact them making sure they actually use it now. Regards, Stephan -- Stephan Richter Web Software Design, Development and Training Google me. Zope Stephan Richter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
