Hi,
On Fri, Mar 21, 2008 at 09:08:28PM +0100, Dieter Maurer wrote:
Chris Withers wrote at 2008-3-20 22:22 +:
Roché Compaan wrote:
Not yet, they are very time consuming. I plan to do the same tests over
ZEO next to determine what overhead ZEO introduces.
Remember to try introducing
Christian Theune wrote:
I talked to Brian Aker (MySQL guy) two weeks ago and he proposed that we
should look into a technique called `group commit` to get rid of the commit
contention.
Does anybody know this technique already and maybe has a pointer for me?
I'd never heard the phrase until
Benji York wrote at 2008-3-25 09:40 -0400:
Christian Theune wrote:
I talked to Brian Aker (MySQL guy) two weeks ago and he proposed that we
should look into a technique called `group commit` to get rid of the commit
contention.
...
Summary: fsync is slow (and the cornerstone of most commit
Dieter Maurer wrote:
We do not yet precisely the cause of our commit contentions.
Hard to tell what'll make them better then. ;)
Almost surely there are several causes that all can lead to contention.
We already found:
* client side causes (while the client helds to commit lock)
Benji York wrote at 2008-3-25 14:24 -0400:
... commit contentions ...
Almost surely there are several causes that all can lead to contention.
We already found:
* client side causes (while the client helds to commit lock)
- garbage collections (which can block a client in the
Chris Withers wrote at 2008-3-20 22:22 +:
Roché Compaan wrote:
Not yet, they are very time consuming. I plan to do the same tests over
ZEO next to determine what overhead ZEO introduces.
Remember to try introducing more app servers and see where the
bottleneck comes ;-)
We have seen
Roché Compaan wrote:
Not yet, they are very time consuming. I plan to do the same tests over
ZEO next to determine what overhead ZEO introduces.
Remember to try introducing more app servers and see where the
bottleneck comes ;-)
Am I right in thinking the storage servers is still
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Chris Withers wrote:
Roché Compaan wrote:
Not yet, they are very time consuming. I plan to do the same tests over
ZEO next to determine what overhead ZEO introduces.
Remember to try introducing more app servers and see where the
bottleneck
Roché Compaan wrote:
On Mon, 2008-02-25 at 07:36 +0200, Roché Compaan wrote:
I'll update my blog post with the final stats and let you know when it
is ready.
I'll have to keep running these tests because the more I run them the
faster the ZODB becomes ;-) Would you have guessed that the
On Tue, 2008-03-04 at 13:27 -0700, Shane Hathaway wrote:
Roché Compaan wrote:
On Mon, 2008-02-25 at 07:36 +0200, Roché Compaan wrote:
I'll update my blog post with the final stats and let you know when it
is ready.
I'll have to keep running these tests because the more I run them the
On Tue, 2008-03-04 at 23:00 +0100, Bernd Dorn wrote:
On 04.03.2008, at 22:16, Shane Hathaway wrote:
Lennart Regebro wrote:
On Tue, Mar 4, 2008 at 9:27 PM, Shane Hathaway
[EMAIL PROTECTED] wrote:
- Did you use optimal methods of retrieval in Postgres? It is
frequently not necessary
Hi Roche. I figured this out once and it was included in PGStorage so it
should be in relstorage also. Take a look at get_db_size method in
postgres adapter. relstorage is in the zope repository.
Regards,
David
Roché Compaan wrote:
- How much disk space does each database consume when
On Tue, Mar 4, 2008 at 10:16 PM, Shane Hathaway [EMAIL PROTECTED] wrote:
Not if you're only retrieving intermediate information.
Sure. And my point is that in a typical websetting, you don't. You
either retrieve data to be displayed, or you insert data from a HTTP
post. Massaging data from one
On Tue, 2008-03-04 at 13:27 -0700, Shane Hathaway wrote:
On a related topic, you might be interested in the RelStorage
performance charts I just posted. Don't take them too seriously, but
I
think the charts are useful.
http://shane.willowrise.com/archives/relstorage-10-and-measurements/
Roché Compaan wrote:
On Tue, 2008-03-04 at 13:27 -0700, Shane Hathaway wrote:
On a related topic, you might be interested in the RelStorage
performance charts I just posted. Don't take them too seriously, but
I
think the charts are useful.
Roché Compaan wrote:
On Tue, 2008-03-04 at 13:27 -0700, Shane Hathaway wrote:
Maybe if you set up a ZODB cache that allows just over 10 million
objects, the lookup time will drop to microseconds. You might need a
lot of RAM to do that, though.
Maybe, but somehow I think that disk IO will
Benji York wrote:
If you're on Linux, you can tweak swappiness (/proc/sys/vm/swappiness;
http://lwn.net/Articles/83588/) to affect how much RAM is used for the
page cache and how much for your process.
While we're on that subject. We recently had a box that would take
strain for almost no
Roché Compaan wrote:
On Mon, 2008-02-25 at 07:36 +0200, Roché Compaan wrote:
I'll update my blog post with the final stats and let you know when it
is ready.
I'll have to keep running these tests because the more I run them the
faster the ZODB becomes ;-) Would you have guessed that the ZODB
On Tue, Mar 4, 2008 at 9:27 PM, Shane Hathaway [EMAIL PROTECTED] wrote:
- Did you use optimal methods of insertion in Postgres, such as COPY?
Also note that a standard way to insert a lot of data into a relational
database is to temporarily drop indexes and re-create them after
insertion.
Lennart Regebro wrote:
On Tue, Mar 4, 2008 at 9:27 PM, Shane Hathaway [EMAIL PROTECTED] wrote:
- Did you use optimal methods of retrieval in Postgres? It is
frequently not necessary to pull the data into the application. Copying
to another table could be faster than fetching rows.
But is
On 04.03.2008, at 22:16, Shane Hathaway wrote:
Lennart Regebro wrote:
On Tue, Mar 4, 2008 at 9:27 PM, Shane Hathaway
[EMAIL PROTECTED] wrote:
- Did you use optimal methods of retrieval in Postgres? It is
frequently not necessary to pull the data into the application.
Copying
to another
I made a lovely mistake in my first round of benchmarks. Lovely, in
that it puts the ZODB in a much better light. When I first ran the
Postgres test, I neglected to put an index on the key field of the
table. I only added the index before I timed lookups on Postgres but
forgot to retest insertion.
On Thu, 2008-02-07 at 20:26 +0100, Dieter Maurer wrote:
Roché Compaan wrote at 2008-2-7 21:21 +0200:
...
So if I asked you to build a data structure for the ZODB that can do
insertions at a rate comparable to Postgres on high volumes, do you
think that it can be done?
If you need a high
Roché Compaan wrote at 2008-2-7 21:21 +0200:
...
So if I asked you to build a data structure for the ZODB that can do
insertions at a rate comparable to Postgres on high volumes, do you
think that it can be done?
If you need a high write rate, the ZODB is probably not optimal.
Ask yourself
On Thu, 2008-02-07 at 00:39 +0100, Dieter Maurer wrote:
If I understand correctly, for each insertion 3 calls are made to
persistent_id? This is still very far from the 66 I mentioned above?
You did not understand correctly.
You insert an entry. The insertion modifies (at least) one
On Tue, 2008-02-05 at 19:17 +0100, Dieter Maurer wrote:
Roché Compaan wrote at 2008-2-4 20:54 +0200:
...
I don't follow? There are 2 insertions and there are 1338046 calls
to persistent_id. Doesn't this suggest that there are 66 objects
persisted per insertion? This seems way to high?
Mignon, Laurent wrote at 2008-2-6 08:06 +0100:
After a lot of tests and benchmark, my feeling is that the ZODB does not seem
suitable for systems managing many data stored in a plane hierarchy.
The application that we currently develop is a business process management
system in opposition to a
Roché Compaan wrote at 2008-2-6 20:18 +0200:
On Tue, 2008-02-05 at 19:17 +0100, Dieter Maurer wrote:
Roché Compaan wrote at 2008-2-4 20:54 +0200:
...
I don't follow? There are 2 insertions and there are 1338046 calls
to persistent_id. Doesn't this suggest that there are 66 objects
Roché Compaan wrote at 2008-2-4 20:54 +0200:
...
I don't follow? There are 2 insertions and there are 1338046 calls
to persistent_id. Doesn't this suggest that there are 66 objects
persisted per insertion? This seems way to high?
Jim told you that persistent_id is called for each object and
On Feb 4, 2008, at 1:54 PM, Roché Compaan wrote:
I don't follow? There are 2 insertions and there are 1338046 calls
to persistent_id. Doesn't this suggest that there are 66 objects
persisted per insertion? This seems way to high?
It seems like there is some confusion about the
On Sun, 2008-02-03 at 22:05 +0100, Dieter Maurer wrote:
The number of persitent_id calls suggests that a written
persistent object has a mean value of 65 subobjects -- which
fits well will OOBuckets.
However, when the profile is for commits with 100 insertions each,
then the number of
Roché Compaan wrote at 2008-2-3 09:15 +0200:
...
I have tried different commit intervals. The published results are for a
commit interval of 100, iow 100 inserts per commit.
Your profile looks very surprising:
I would expect that for a single insertion, typically
one persistent object
Roché Compaan wrote at 2008-2-1 21:17 +0200:
I have completed my first round of benchmarks on the ZODB and welcome
any criticism and advise. I summarised our earlier discussion and
additional findings in this blog entry:
On Sat, 2008-02-02 at 22:10 +0100, Dieter Maurer wrote:
Roché Compaan wrote at 2008-2-1 21:17 +0200:
I have completed my first round of benchmarks on the ZODB and welcome
any criticism and advise. I summarised our earlier discussion and
additional findings in this blog entry:
I have completed my first round of benchmarks on the ZODB and welcome
any criticism and advise. I summarised our earlier discussion and
additional findings in this blog entry:
http://www.upfrontsystems.co.za/Members/roche/where-im-calling-from/zodb-benchmarks
--
Roché Compaan
Upfront Systems
On Dec 7, 2007, at 10:55 AM, Godefroid Chapelle wrote:
Jim Fulton wrote:
It sounds like I should write some pickle and cPickle tests and we
should update the ZODB trunk to take advantage of this. (/me fears
gettimg mired in Python 3.)
Jim
Would you do that on Python 2.4, 2.5 or ... ?
Godefroid Chapelle wrote:
Jim Fulton wrote:
On Dec 6, 2007, at 2:40 PM, Godefroid Chapelle wrote:
Jim Fulton wrote:
On Nov 6, 2007, at 2:40 PM, Sidnei da Silva wrote:
Despite this change there are still a huge amount
of unexplained calls to the 'persistent_id' method of the
ObjectWriter
Jim Fulton wrote:
On Dec 7, 2007, at 10:55 AM, Godefroid Chapelle wrote:
Jim Fulton wrote:
It sounds like I should write some pickle and cPickle tests and we
should update the ZODB trunk to take advantage of this. (/me fears
gettimg mired in Python 3.)
Jim
Would you do that on Python
Jim Fulton wrote:
On Nov 6, 2007, at 2:40 PM, Sidnei da Silva wrote:
Despite this change there are still a huge amount
of unexplained calls to the 'persistent_id' method of the ObjectWriter
in serialize.py.
Why 'unexplained'? 'persistent_id' is called from the Pickler instance
being used in
On Nov 6, 2007, at 3:17 PM, Roché Compaan wrote:
On Tue, 2007-11-06 at 14:51 -0500, Jim Fulton wrote:
On Nov 6, 2007, at 2:40 PM, Sidnei da Silva wrote:
Despite this change there are still a huge amount
of unexplained calls to the 'persistent_id' method of the
ObjectWriter
in serialize.py.
On Dec 6, 2007, at 2:40 PM, Godefroid Chapelle wrote:
Jim Fulton wrote:
On Nov 6, 2007, at 2:40 PM, Sidnei da Silva wrote:
Despite this change there are still a huge amount
of unexplained calls to the 'persistent_id' method of the
ObjectWriter
in serialize.py.
Why 'unexplained'?
On Thu, 2007-12-06 at 15:05 -0500, Jim Fulton wrote:
On Dec 6, 2007, at 2:40 PM, Godefroid Chapelle wrote:
Jim Fulton wrote:
On Nov 6, 2007, at 2:40 PM, Sidnei da Silva wrote:
Despite this change there are still a huge amount
of unexplained calls to the 'persistent_id' method of the
Despite this change there are still a huge amount
of unexplained calls to the 'persistent_id' method of the ObjectWriter
in serialize.py.
Why 'unexplained'? 'persistent_id' is called from the Pickler instance
being used in ObjectWriter._dump(). It is called for each and every
single object
On Tue, 2007-11-06 at 17:40 -0200, Sidnei da Silva wrote:
Despite this change there are still a huge amount
of unexplained calls to the 'persistent_id' method of the ObjectWriter
in serialize.py.
Why 'unexplained'? 'persistent_id' is called from the Pickler instance
being used in
On Nov 6, 2007, at 3:01 PM, Roché Compaan wrote:
On Tue, 2007-11-06 at 17:40 -0200, Sidnei da Silva wrote:
Despite this change there are still a huge amount
of unexplained calls to the 'persistent_id' method of the
ObjectWriter
in serialize.py.
Why 'unexplained'? 'persistent_id' is
On Tue, 2007-11-06 at 14:51 -0500, Jim Fulton wrote:
On Nov 6, 2007, at 2:40 PM, Sidnei da Silva wrote:
Despite this change there are still a huge amount
of unexplained calls to the 'persistent_id' method of the
ObjectWriter
in serialize.py.
Why 'unexplained'? 'persistent_id' is
Hi,
Am Freitag, den 02.11.2007, 09:56 -0400 schrieb David Binger:
On Nov 2, 2007, at 8:39 AM, Lennart Regebro wrote:
On 11/2/07, Matt Hamilton [EMAIL PROTECTED] wrote:
That may just end up causing delays periodically in
transactions... ie delays
that the user sees, as opposed to
On Nov 2, 2007, at 8:39 AM, Lennart Regebro wrote:
On 11/2/07, Matt Hamilton [EMAIL PROTECTED] wrote:
That may just end up causing delays periodically in
transactions... ie delays
that the user sees, as opposed to doing it via another thread or
something. But
then as only one thread would
Matt Hamilton wrote:
David Binger dbinger at mems-exchange.org writes:
On Nov 2, 2007, at 6:20 AM, Lennart Regebro wrote:
Lots of people don't do nightly packs, I'm pretty sure such a process
needs to be completely automatic. The question is weather doing it in
a separate process in the
On Fri, 2007-11-02 at 16:00 +, Laurence Rowe wrote:
Matt Hamilton wrote:
David Binger dbinger at mems-exchange.org writes:
On Nov 2, 2007, at 6:20 AM, Lennart Regebro wrote:
Lots of people don't do nightly packs, I'm pretty sure such a process
needs to be completely automatic.
On Nov 2, 2007, at 6:20 AM, Lennart Regebro wrote:
Lots of people don't do nightly packs, I'm pretty sure such a process
needs to be completely automatic. The question is weather doing it in
a separate process in the background, or ever X transactions, or every
X seconds, or something.
Okay,
On Nov 2, 2007, at 5:48 AM, Lennart Regebro wrote:
On 11/1/07, Matt Hamilton [EMAIL PROTECTED] wrote:
An interesting idea. Surely we need the opposite though, and that
is an
additional BTree with a very large bucket size, as we want to
minimize the
chance of a bucket split when inserting?
On 11/1/07, Matt Hamilton [EMAIL PROTECTED] wrote:
An interesting idea. Surely we need the opposite though, and that is an
additional BTree with a very large bucket size, as we want to minimize the
chance of a bucket split when inserting? Then we occasionally consolidate and
move the items
David Binger dbinger at mems-exchange.org writes:
On Nov 2, 2007, at 6:20 AM, Lennart Regebro wrote:
Lots of people don't do nightly packs, I'm pretty sure such a process
needs to be completely automatic. The question is weather doing it in
a separate process in the background, or
This is the 'batch' or 'distribute' pattern that crops up in many
fields.
The best path is normally to understand what the conflicts are, and
where the time is spent.
If in, this case, much time is spent in the preamble, and the actual
inserts are quick, then diving down one time through
Laurence Rowe l at lrowe.co.uk writes:
So why is PosgreSQL so much faster? It's using a Write-Ahead-Log for
inserts. Instead of inserting into the (B-Tree based) data files at
every transaction commit it writes a record to the WAL. This does not
require traversal of the B-Tree and has O(1)
On Wed, 2007-10-31 at 10:47 -0400, David Binger wrote:
On Oct 31, 2007, at 7:35 AM, Roché Compaan wrote:
Thanks for the explanation.
The actual insertion is very fast. Your benchmark is dominated by
the time to serialize the changes due to an insertion.
You should usually have just 2
On Nov 1, 2007, at 4:25 PM, Matt Hamilton wrote:
David Binger dbinger at mems-exchange.org writes:
On Nov 1, 2007, at 7:05 AM, Matt Hamilton wrote:
Ie we perhaps look at a catalog data structure
in which writes are initially done to some kind of queue then moved
to the
BTrees at a later
On Nov 1, 2007, at 4:25 PM, Matt Hamilton wrote:
David Binger dbinger at mems-exchange.org writes:
On Nov 1, 2007, at 7:05 AM, Matt Hamilton wrote:
Ie we perhaps look at a catalog data structure
in which writes are initially done to some kind of queue then moved
to the
BTrees at a later
Quick note...
Smaller buckets, fewer conflicts, more overhead on reading and writing.
Larger buckets, more conflicts, less overhead on reading and writing.
One bucket ... constant conflicts.
I'd bet that the additional tree with tiny buckets would be best.
Transfer them into the normal tree
It looks like ZODB performance in your test has the same O(log n)
performance as PostgreSQL checkpoints (the periodic drops in your
graph). This should come as no surprise. B-Trees have a theoretical
Search/Insert/Delete time complexity equal to the height of the tree,
which is (up to) log(n).
I think someone proposed to have something just like a WAL in ZODB.
That could be an interesting optimization.
--
Sidnei da Silva
Enfold Systemshttp://enfoldsystems.com
Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214
___
For more
On Wed, 2007-10-31 at 10:00 +, Laurence Rowe wrote:
It looks like ZODB performance in your test has the same O(log n)
performance as PostgreSQL checkpoints (the periodic drops in your
graph). This should come as no surprise. B-Trees have a theoretical
Search/Insert/Delete time
63 matches
Mail list logo
