Re: [ZODB-Dev] Re: ZODB Benchmarks

2008-03-25 Thread Christian Theune
Hi, On Fri, Mar 21, 2008 at 09:08:28PM +0100, Dieter Maurer wrote: Chris Withers wrote at 2008-3-20 22:22 +: Roché Compaan wrote: Not yet, they are very time consuming. I plan to do the same tests over ZEO next to determine what overhead ZEO introduces. Remember to try introducing

Re: [ZODB-Dev] Re: ZODB Benchmarks

2008-03-25 Thread Benji York
Christian Theune wrote: I talked to Brian Aker (MySQL guy) two weeks ago and he proposed that we should look into a technique called `group commit` to get rid of the commit contention. Does anybody know this technique already and maybe has a pointer for me? I'd never heard the phrase until

Re: [ZODB-Dev] Re: ZODB Benchmarks

2008-03-25 Thread Dieter Maurer
Benji York wrote at 2008-3-25 09:40 -0400: Christian Theune wrote: I talked to Brian Aker (MySQL guy) two weeks ago and he proposed that we should look into a technique called `group commit` to get rid of the commit contention. ... Summary: fsync is slow (and the cornerstone of most commit

Re: [ZODB-Dev] Re: ZODB Benchmarks

2008-03-25 Thread Benji York
Dieter Maurer wrote: We do not yet precisely the cause of our commit contentions. Hard to tell what'll make them better then. ;) Almost surely there are several causes that all can lead to contention. We already found: * client side causes (while the client helds to commit lock)

Re: [ZODB-Dev] Re: ZODB Benchmarks

2008-03-25 Thread Dieter Maurer
Benji York wrote at 2008-3-25 14:24 -0400: ... commit contentions ... Almost surely there are several causes that all can lead to contention. We already found: * client side causes (while the client helds to commit lock) - garbage collections (which can block a client in the

Re: [ZODB-Dev] Re: ZODB Benchmarks

2008-03-21 Thread Dieter Maurer
Chris Withers wrote at 2008-3-20 22:22 +: Roché Compaan wrote: Not yet, they are very time consuming. I plan to do the same tests over ZEO next to determine what overhead ZEO introduces. Remember to try introducing more app servers and see where the bottleneck comes ;-) We have seen

Re: [ZODB-Dev] Re: ZODB Benchmarks

2008-03-20 Thread Chris Withers
Roché Compaan wrote: Not yet, they are very time consuming. I plan to do the same tests over ZEO next to determine what overhead ZEO introduces. Remember to try introducing more app servers and see where the bottleneck comes ;-) Am I right in thinking the storage servers is still

Re: [ZODB-Dev] Re: ZODB Benchmarks

2008-03-19 Thread Manuel Vazquez Acosta
Roché Compaan wrote: On Mon, 2008-02-25 at 07:36 +0200, Roché Compaan wrote: I'll update my blog post with the final stats and let you know when it is ready. I'll have to keep running these tests because the more I run them the faster the ZODB becomes ;-) Would you have guessed that the

Re: [ZODB-Dev] Re: ZODB Benchmarks

2008-03-05 Thread Roché Compaan
On Tue, 2008-03-04 at 13:27 -0700, Shane Hathaway wrote: Roché Compaan wrote: On Mon, 2008-02-25 at 07:36 +0200, Roché Compaan wrote: I'll update my blog post with the final stats and let you know when it is ready. I'll have to keep running these tests because the more I run them the

Re: [ZODB-Dev] Re: ZODB Benchmarks

2008-03-05 Thread Roché Compaan
On Tue, 2008-03-04 at 23:00 +0100, Bernd Dorn wrote: On 04.03.2008, at 22:16, Shane Hathaway wrote: Lennart Regebro wrote: On Tue, Mar 4, 2008 at 9:27 PM, Shane Hathaway [EMAIL PROTECTED] wrote: - Did you use optimal methods of retrieval in Postgres? It is frequently not necessary

Re: [ZODB-Dev] Re: ZODB Benchmarks

2008-03-05 Thread David Pratt
Hi Roche. I figured this out once and it was included in PGStorage so it should be in relstorage also. Take a look at get_db_size method in postgres adapter. relstorage is in the zope repository. Regards, David Roché Compaan wrote: - How much disk space does each database consume when

Re: [ZODB-Dev] Re: ZODB Benchmarks

2008-03-05 Thread Lennart Regebro
On Tue, Mar 4, 2008 at 10:16 PM, Shane Hathaway [EMAIL PROTECTED] wrote: Not if you're only retrieving intermediate information. Sure. And my point is that in a typical websetting, you don't. You either retrieve data to be displayed, or you insert data from a HTTP post. Massaging data from one

Re: [ZODB-Dev] Re: ZODB Benchmarks

2008-03-05 Thread Roché Compaan
On Tue, 2008-03-04 at 13:27 -0700, Shane Hathaway wrote: On a related topic, you might be interested in the RelStorage performance charts I just posted. Don't take them too seriously, but I think the charts are useful. http://shane.willowrise.com/archives/relstorage-10-and-measurements/

Re: [ZODB-Dev] Re: ZODB Benchmarks

2008-03-05 Thread Shane Hathaway
Roché Compaan wrote: On Tue, 2008-03-04 at 13:27 -0700, Shane Hathaway wrote: On a related topic, you might be interested in the RelStorage performance charts I just posted. Don't take them too seriously, but I think the charts are useful.

Re: [ZODB-Dev] Re: ZODB Benchmarks

2008-03-05 Thread Benji York
Roché Compaan wrote: On Tue, 2008-03-04 at 13:27 -0700, Shane Hathaway wrote: Maybe if you set up a ZODB cache that allows just over 10 million objects, the lookup time will drop to microseconds. You might need a lot of RAM to do that, though. Maybe, but somehow I think that disk IO will

Re: [ZODB-Dev] Re: ZODB Benchmarks

2008-03-05 Thread David Pratt
Hi Benji. Have you any settings to recommend or use a default. Many thanks. Regards, David Benji York wrote: Roché Compaan wrote: On Tue, 2008-03-04 at 13:27 -0700, Shane Hathaway wrote: Maybe if you set up a ZODB cache that allows just over 10 million objects, the lookup time will drop to

Re: [ZODB-Dev] Re: ZODB Benchmarks

2008-03-05 Thread Benji York
David Pratt wrote: Hi Benji. Have you any settings to recommend or use a default. Many thanks. For benchmarking? No. Too high and you'll spend a bunch of time swapping to free up space for disk cache; too low and you may not have a large enough disk cache to be effective. -- Benji York

Re: [ZODB-Dev] Re: ZODB Benchmarks

2008-03-05 Thread Izak Burger
Benji York wrote: If you're on Linux, you can tweak swappiness (/proc/sys/vm/swappiness; http://lwn.net/Articles/83588/) to affect how much RAM is used for the page cache and how much for your process. While we're on that subject. We recently had a box that would take strain for almost no

Re: [ZODB-Dev] Re: ZODB Benchmarks

2008-03-04 Thread Shane Hathaway
Roché Compaan wrote: On Mon, 2008-02-25 at 07:36 +0200, Roché Compaan wrote: I'll update my blog post with the final stats and let you know when it is ready. I'll have to keep running these tests because the more I run them the faster the ZODB becomes ;-) Would you have guessed that the ZODB

Re: [ZODB-Dev] Re: ZODB Benchmarks

2008-03-04 Thread Lennart Regebro
On Tue, Mar 4, 2008 at 9:27 PM, Shane Hathaway [EMAIL PROTECTED] wrote: - Did you use optimal methods of insertion in Postgres, such as COPY? Also note that a standard way to insert a lot of data into a relational database is to temporarily drop indexes and re-create them after insertion.

Re: [ZODB-Dev] Re: ZODB Benchmarks

2008-03-04 Thread Shane Hathaway
Lennart Regebro wrote: On Tue, Mar 4, 2008 at 9:27 PM, Shane Hathaway [EMAIL PROTECTED] wrote: - Did you use optimal methods of retrieval in Postgres? It is frequently not necessary to pull the data into the application. Copying to another table could be faster than fetching rows. But is

Re: [ZODB-Dev] Re: ZODB Benchmarks

2008-03-04 Thread Bernd Dorn
On 04.03.2008, at 22:16, Shane Hathaway wrote: Lennart Regebro wrote: On Tue, Mar 4, 2008 at 9:27 PM, Shane Hathaway [EMAIL PROTECTED] wrote: - Did you use optimal methods of retrieval in Postgres? It is frequently not necessary to pull the data into the application. Copying to another

Re: [ZODB-Dev] Re: ZODB Benchmarks

2008-02-24 Thread Roché Compaan
I made a lovely mistake in my first round of benchmarks. Lovely, in that it puts the ZODB in a much better light. When I first ran the Postgres test, I neglected to put an index on the key field of the table. I only added the index before I timed lookups on Postgres but forgot to retest insertion.

Re: [ZODB-Dev] Re: ZODB Benchmarks

2008-02-08 Thread Dieter Maurer
Roché Compaan wrote at 2008-2-7 21:44 +0200: ... There are use cases where having a container in the ZODB that can handle large volumes and maintain a high insertion rate would be very convenient. An example of such a use case would be a site with millions of members where each member has their

Re: [ZODB-Dev] Re: ZODB Benchmarks

2008-02-07 Thread Roché Compaan
On Thu, 2008-02-07 at 20:26 +0100, Dieter Maurer wrote: Roché Compaan wrote at 2008-2-7 21:21 +0200: ... So if I asked you to build a data structure for the ZODB that can do insertions at a rate comparable to Postgres on high volumes, do you think that it can be done? If you need a high

Re: [ZODB-Dev] Re: ZODB Benchmarks

2008-02-07 Thread Dieter Maurer
Roché Compaan wrote at 2008-2-7 21:21 +0200: ... So if I asked you to build a data structure for the ZODB that can do insertions at a rate comparable to Postgres on high volumes, do you think that it can be done? If you need a high write rate, the ZODB is probably not optimal. Ask yourself

Re: [ZODB-Dev] Re: ZODB Benchmarks

2008-02-07 Thread Roché Compaan
On Thu, 2008-02-07 at 00:39 +0100, Dieter Maurer wrote: If I understand correctly, for each insertion 3 calls are made to persistent_id? This is still very far from the 66 I mentioned above? You did not understand correctly. You insert an entry. The insertion modifies (at least) one

Re: [ZODB-Dev] Re: ZODB Benchmarks

2008-02-06 Thread Roché Compaan
On Tue, 2008-02-05 at 19:17 +0100, Dieter Maurer wrote: Roché Compaan wrote at 2008-2-4 20:54 +0200: ... I don't follow? There are 2 insertions and there are 1338046 calls to persistent_id. Doesn't this suggest that there are 66 objects persisted per insertion? This seems way to high?

Re: RE : [ZODB-Dev] Re: ZODB Benchmarks

2008-02-06 Thread Dieter Maurer
Mignon, Laurent wrote at 2008-2-6 08:06 +0100: After a lot of tests and benchmark, my feeling is that the ZODB does not seem suitable for systems managing many data stored in a plane hierarchy. The application that we currently develop is a business process management system in opposition to a

Re: [ZODB-Dev] Re: ZODB Benchmarks

2008-02-06 Thread Dieter Maurer
Roché Compaan wrote at 2008-2-6 20:18 +0200: On Tue, 2008-02-05 at 19:17 +0100, Dieter Maurer wrote: Roché Compaan wrote at 2008-2-4 20:54 +0200: ... I don't follow? There are 2 insertions and there are 1338046 calls to persistent_id. Doesn't this suggest that there are 66 objects

Re: [ZODB-Dev] Re: ZODB Benchmarks

2008-02-05 Thread Dieter Maurer
Roché Compaan wrote at 2008-2-4 20:54 +0200: ... I don't follow? There are 2 insertions and there are 1338046 calls to persistent_id. Doesn't this suggest that there are 66 objects persisted per insertion? This seems way to high? Jim told you that persistent_id is called for each object and

Re: [ZODB-Dev] Re: ZODB Benchmarks

2008-02-04 Thread David Binger
On Feb 4, 2008, at 1:54 PM, Roché Compaan wrote: I don't follow? There are 2 insertions and there are 1338046 calls to persistent_id. Doesn't this suggest that there are 66 objects persisted per insertion? This seems way to high? It seems like there is some confusion about the

Re: [ZODB-Dev] Re: ZODB Benchmarks

2008-02-04 Thread Roché Compaan
On Sun, 2008-02-03 at 22:05 +0100, Dieter Maurer wrote: The number of persitent_id calls suggests that a written persistent object has a mean value of 65 subobjects -- which fits well will OOBuckets. However, when the profile is for commits with 100 insertions each, then the number of

Re: [ZODB-Dev] Re: ZODB Benchmarks

2008-02-03 Thread Dieter Maurer
Roché Compaan wrote at 2008-2-3 09:15 +0200: ... I have tried different commit intervals. The published results are for a commit interval of 100, iow 100 inserts per commit. Your profile looks very surprising: I would expect that for a single insertion, typically one persistent object

Re: [ZODB-Dev] Re: ZODB Benchmarks

2008-02-02 Thread Dieter Maurer
Roché Compaan wrote at 2008-2-1 21:17 +0200: I have completed my first round of benchmarks on the ZODB and welcome any criticism and advise. I summarised our earlier discussion and additional findings in this blog entry:

Re: [ZODB-Dev] Re: ZODB Benchmarks

2008-02-02 Thread Roché Compaan
On Sat, 2008-02-02 at 22:10 +0100, Dieter Maurer wrote: Roché Compaan wrote at 2008-2-1 21:17 +0200: I have completed my first round of benchmarks on the ZODB and welcome any criticism and advise. I summarised our earlier discussion and additional findings in this blog entry:

Re: [ZODB-Dev] Re: ZODB Benchmarks

2008-02-01 Thread Roché Compaan
I have completed my first round of benchmarks on the ZODB and welcome any criticism and advise. I summarised our earlier discussion and additional findings in this blog entry: http://www.upfrontsystems.co.za/Members/roche/where-im-calling-from/zodb-benchmarks -- Roché Compaan Upfront Systems

Re: [ZODB-Dev] Re: ZODB Benchmarks

2007-12-06 Thread Jim Fulton
On Nov 6, 2007, at 3:17 PM, Roché Compaan wrote: On Tue, 2007-11-06 at 14:51 -0500, Jim Fulton wrote: On Nov 6, 2007, at 2:40 PM, Sidnei da Silva wrote: Despite this change there are still a huge amount of unexplained calls to the 'persistent_id' method of the ObjectWriter in serialize.py.

Re: [ZODB-Dev] Re: ZODB Benchmarks

2007-12-06 Thread Roché Compaan
On Thu, 2007-12-06 at 15:05 -0500, Jim Fulton wrote: On Dec 6, 2007, at 2:40 PM, Godefroid Chapelle wrote: Jim Fulton wrote: On Nov 6, 2007, at 2:40 PM, Sidnei da Silva wrote: Despite this change there are still a huge amount of unexplained calls to the 'persistent_id' method of the

Re: [ZODB-Dev] Re: ZODB Benchmarks

2007-11-06 Thread Sidnei da Silva
Despite this change there are still a huge amount of unexplained calls to the 'persistent_id' method of the ObjectWriter in serialize.py. Why 'unexplained'? 'persistent_id' is called from the Pickler instance being used in ObjectWriter._dump(). It is called for each and every single object

Re: [ZODB-Dev] Re: ZODB Benchmarks

2007-11-06 Thread Roché Compaan
On Tue, 2007-11-06 at 17:40 -0200, Sidnei da Silva wrote: Despite this change there are still a huge amount of unexplained calls to the 'persistent_id' method of the ObjectWriter in serialize.py. Why 'unexplained'? 'persistent_id' is called from the Pickler instance being used in

Re: [ZODB-Dev] Re: ZODB Benchmarks

2007-11-06 Thread Jim Fulton
On Nov 6, 2007, at 3:01 PM, Roché Compaan wrote: On Tue, 2007-11-06 at 17:40 -0200, Sidnei da Silva wrote: Despite this change there are still a huge amount of unexplained calls to the 'persistent_id' method of the ObjectWriter in serialize.py. Why 'unexplained'? 'persistent_id' is

Re: [ZODB-Dev] Re: ZODB Benchmarks

2007-11-06 Thread Roché Compaan
On Tue, 2007-11-06 at 14:51 -0500, Jim Fulton wrote: On Nov 6, 2007, at 2:40 PM, Sidnei da Silva wrote: Despite this change there are still a huge amount of unexplained calls to the 'persistent_id' method of the ObjectWriter in serialize.py. Why 'unexplained'? 'persistent_id' is

Re: [ZODB-Dev] Re: ZODB Benchmarks

2007-11-02 Thread Christian Theune
Hi, Am Freitag, den 02.11.2007, 09:56 -0400 schrieb David Binger: On Nov 2, 2007, at 8:39 AM, Lennart Regebro wrote: On 11/2/07, Matt Hamilton [EMAIL PROTECTED] wrote: That may just end up causing delays periodically in transactions... ie delays that the user sees, as opposed to

Re: [ZODB-Dev] Re: ZODB Benchmarks

2007-11-02 Thread David Binger
On Nov 2, 2007, at 8:39 AM, Lennart Regebro wrote: On 11/2/07, Matt Hamilton [EMAIL PROTECTED] wrote: That may just end up causing delays periodically in transactions... ie delays that the user sees, as opposed to doing it via another thread or something. But then as only one thread would

Re: [ZODB-Dev] Re: ZODB Benchmarks

2007-11-02 Thread Roché Compaan
On Fri, 2007-11-02 at 16:00 +, Laurence Rowe wrote: Matt Hamilton wrote: David Binger dbinger at mems-exchange.org writes: On Nov 2, 2007, at 6:20 AM, Lennart Regebro wrote: Lots of people don't do nightly packs, I'm pretty sure such a process needs to be completely automatic.

Re: [ZODB-Dev] Re: ZODB Benchmarks

2007-11-02 Thread David Binger
On Nov 2, 2007, at 6:20 AM, Lennart Regebro wrote: Lots of people don't do nightly packs, I'm pretty sure such a process needs to be completely automatic. The question is weather doing it in a separate process in the background, or ever X transactions, or every X seconds, or something. Okay,

Re: [ZODB-Dev] Re: ZODB Benchmarks

2007-11-02 Thread David Binger
On Nov 2, 2007, at 5:48 AM, Lennart Regebro wrote: On 11/1/07, Matt Hamilton [EMAIL PROTECTED] wrote: An interesting idea. Surely we need the opposite though, and that is an additional BTree with a very large bucket size, as we want to minimize the chance of a bucket split when inserting?

Re: [ZODB-Dev] Re: ZODB Benchmarks

2007-11-02 Thread Lennart Regebro
On 11/1/07, Matt Hamilton [EMAIL PROTECTED] wrote: An interesting idea. Surely we need the opposite though, and that is an additional BTree with a very large bucket size, as we want to minimize the chance of a bucket split when inserting? Then we occasionally consolidate and move the items

Re: [ZODB-Dev] Re: ZODB Benchmarks

2007-11-02 Thread Russ Ferriday
This is the 'batch' or 'distribute' pattern that crops up in many fields. The best path is normally to understand what the conflicts are, and where the time is spent. If in, this case, much time is spent in the preamble, and the actual inserts are quick, then diving down one time through

Re: [ZODB-Dev] Re: ZODB Benchmarks

2007-11-01 Thread Roché Compaan
On Wed, 2007-10-31 at 10:47 -0400, David Binger wrote: On Oct 31, 2007, at 7:35 AM, Roché Compaan wrote: Thanks for the explanation. The actual insertion is very fast. Your benchmark is dominated by the time to serialize the changes due to an insertion. You should usually have just 2

Re: [ZODB-Dev] Re: ZODB Benchmarks

2007-11-01 Thread Gary Poster
On Nov 1, 2007, at 4:25 PM, Matt Hamilton wrote: David Binger dbinger at mems-exchange.org writes: On Nov 1, 2007, at 7:05 AM, Matt Hamilton wrote: Ie we perhaps look at a catalog data structure in which writes are initially done to some kind of queue then moved to the BTrees at a later

Re: [ZODB-Dev] Re: ZODB Benchmarks

2007-11-01 Thread David Binger
On Nov 1, 2007, at 4:25 PM, Matt Hamilton wrote: David Binger dbinger at mems-exchange.org writes: On Nov 1, 2007, at 7:05 AM, Matt Hamilton wrote: Ie we perhaps look at a catalog data structure in which writes are initially done to some kind of queue then moved to the BTrees at a later

Re: [ZODB-Dev] Re: ZODB Benchmarks

2007-11-01 Thread Russ Ferriday
Quick note... Smaller buckets, fewer conflicts, more overhead on reading and writing. Larger buckets, more conflicts, less overhead on reading and writing. One bucket ... constant conflicts. I'd bet that the additional tree with tiny buckets would be best. Transfer them into the normal tree

Re: [ZODB-Dev] Re: ZODB Benchmarks

2007-10-31 Thread Sidnei da Silva
I think someone proposed to have something just like a WAL in ZODB. That could be an interesting optimization. -- Sidnei da Silva Enfold Systemshttp://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214 ___ For more

Re: [ZODB-Dev] Re: ZODB Benchmarks

2007-10-31 Thread Roché Compaan
On Wed, 2007-10-31 at 10:00 +, Laurence Rowe wrote: It looks like ZODB performance in your test has the same O(log n) performance as PostgreSQL checkpoints (the periodic drops in your graph). This should come as no surprise. B-Trees have a theoretical Search/Insert/Delete time