Sean Allen wrote at 2008-3-25 15:23 -0400:
> ...
>
>On Mar 25, 2008, at 2:54 PM, Dieter Maurer wrote:
>> Benji York wrote at 2008-3-25 14:24 -0400:
>>> ... commit contentions ...
Almost surely there are several causes that all can lead to
contention.
We already found:
>>>
On Mar 25, 2008, at 2:54 PM, Dieter Maurer wrote:
Benji York wrote at 2008-3-25 14:24 -0400:
... commit contentions ...
Almost surely there are several causes that all can lead to
contention.
We already found:
* client side causes (while the client helds to commit lock)
- garbage col
Benji York wrote at 2008-3-25 14:24 -0400:
> ... commit contentions ...
>> Almost surely there are several causes that all can lead to contention.
>>
>> We already found:
>>
>> * client side causes (while the client helds to commit lock)
>>
>> - garbage collections (which can block a cl
Dieter Maurer wrote:
We do not yet precisely the cause of our commit contentions.
Hard to tell what'll make them better then. ;)
Almost surely there are several causes that all can lead to contention.
We already found:
* client side causes (while the client helds to commit lock)
-
Benji York wrote at 2008-3-25 09:40 -0400:
>Christian Theune wrote:
>> I talked to Brian Aker (MySQL guy) two weeks ago and he proposed that we
>> should look into a technique called `group commit` to get rid of the "commit
>> contention".
> ...
>Summary: fsync is slow (and the cornerstone of most
Christian Theune wrote:
I talked to Brian Aker (MySQL guy) two weeks ago and he proposed that we
should look into a technique called `group commit` to get rid of the "commit
contention".
Does anybody know this technique already and maybe has a pointer for me?
I'd never heard the phrase until r
Hi,
On Fri, Mar 21, 2008 at 09:08:28PM +0100, Dieter Maurer wrote:
> Chris Withers wrote at 2008-3-20 22:22 +:
> >Roché Compaan wrote:
> >> Not yet, they are very time consuming. I plan to do the same tests over
> >> ZEO next to determine what overhead ZEO introduces.
> >
> >Remember to try in
Chris Withers wrote at 2008-3-20 22:22 +:
>Roché Compaan wrote:
>> Not yet, they are very time consuming. I plan to do the same tests over
>> ZEO next to determine what overhead ZEO introduces.
>
>Remember to try introducing more app servers and see where the
>bottleneck comes ;-)
We have see
Roché Compaan wrote:
Not yet, they are very time consuming. I plan to do the same tests over
ZEO next to determine what overhead ZEO introduces.
Remember to try introducing more app servers and see where the
bottleneck comes ;-)
Am I right in thinking the storage servers is still essentially
On Thu, 2008-03-20 at 00:00 -0400, Manuel Vazquez Acosta wrote:
> Roché Compaan wrote:
> > On Mon, 2008-02-25 at 07:36 +0200, Roché Compaan wrote:
> >> I'll update my blog post with the final stats and let you know when it
> >> is ready.
> >>
> >
> > I'll have to keep running these tests because t
Roché Compaan wrote:
> On Mon, 2008-02-25 at 07:36 +0200, Roché Compaan wrote:
>> I'll update my blog post with the final stats and let you know when it
>> is ready.
>>
>
> I'll have to keep running these tests because the more I run them the
> faster the ZODB becomes ;-) Would you have guessed th
Benji York wrote:
If you're on Linux, you can tweak swappiness (/proc/sys/vm/swappiness;
http://lwn.net/Articles/83588/) to affect how much RAM is used for the
page cache and how much for your process.
While we're on that subject. We recently had a box that would take
strain for almost no rea
David Pratt wrote:
Hi Benji. Have you any settings to recommend or use a default. Many thanks.
For benchmarking? No.
Too high and you'll spend a bunch of time swapping to free up space for
disk cache; too low and you may not have a large enough disk cache to be
effective.
--
Benji York
Sen
Hi Benji. Have you any settings to recommend or use a default. Many thanks.
Regards,
David
Benji York wrote:
Roché Compaan wrote:
On Tue, 2008-03-04 at 13:27 -0700, Shane Hathaway wrote:
Maybe if you set up a ZODB cache that allows just over 10 million
objects, the lookup time will drop to mi
Roché Compaan wrote:
On Tue, 2008-03-04 at 13:27 -0700, Shane Hathaway wrote:
Maybe if you set up a ZODB cache that allows just over 10 million
objects, the lookup time will drop to microseconds. You might need a
lot of RAM to do that, though.
Maybe, but somehow I think that disk IO will pre
Roché Compaan wrote:
On Tue, 2008-03-04 at 13:27 -0700, Shane Hathaway wrote:
On a related topic, you might be interested in the RelStorage
performance charts I just posted. Don't take them too seriously, but
I
think the charts are useful.
http://shane.willowrise.com/archives/relstorage-10-a
On Tue, 2008-03-04 at 13:27 -0700, Shane Hathaway wrote:
> On a related topic, you might be interested in the RelStorage
> performance charts I just posted. Don't take them too seriously, but
> I
> think the charts are useful.
>
> http://shane.willowrise.com/archives/relstorage-10-and-measureme
On Tue, Mar 4, 2008 at 10:16 PM, Shane Hathaway <[EMAIL PROTECTED]> wrote:
> Not if you're only retrieving intermediate information.
Sure. And my point is that in a typical websetting, you don't. You
either retrieve data to be displayed, or you insert data from a HTTP
post. Massaging data from on
Hi Roche. I figured this out once and it was included in PGStorage so it
should be in relstorage also. Take a look at get_db_size method in
postgres adapter. relstorage is in the zope repository.
Regards,
David
Roché Compaan wrote:
- How much disk space does each database consume when there
On Tue, 2008-03-04 at 23:00 +0100, Bernd Dorn wrote:
> On 04.03.2008, at 22:16, Shane Hathaway wrote:
>
> > Lennart Regebro wrote:
> >> On Tue, Mar 4, 2008 at 9:27 PM, Shane Hathaway
> >> <[EMAIL PROTECTED]> wrote:
> >>> - Did you use optimal methods of retrieval in Postgres? It is
> >>> freque
On Tue, 2008-03-04 at 13:27 -0700, Shane Hathaway wrote:
> Roché Compaan wrote:
> > On Mon, 2008-02-25 at 07:36 +0200, Roché Compaan wrote:
> >> I'll update my blog post with the final stats and let you know when it
> >> is ready.
> >>
> >
> > I'll have to keep running these tests because the more
On 04.03.2008, at 22:16, Shane Hathaway wrote:
Lennart Regebro wrote:
On Tue, Mar 4, 2008 at 9:27 PM, Shane Hathaway
<[EMAIL PROTECTED]> wrote:
- Did you use optimal methods of retrieval in Postgres? It is
frequently not necessary to pull the data into the application.
Copying
to anothe
Lennart Regebro wrote:
On Tue, Mar 4, 2008 at 9:27 PM, Shane Hathaway <[EMAIL PROTECTED]> wrote:
- Did you use optimal methods of retrieval in Postgres? It is
frequently not necessary to pull the data into the application. Copying
to another table could be faster than fetching rows.
But i
On Tue, Mar 4, 2008 at 9:27 PM, Shane Hathaway <[EMAIL PROTECTED]> wrote:
> - Did you use optimal methods of insertion in Postgres, such as COPY?
> Also note that a standard way to insert a lot of data into a relational
> database is to temporarily drop indexes and re-create them after
> insert
Roché Compaan wrote:
On Mon, 2008-02-25 at 07:36 +0200, Roché Compaan wrote:
I'll update my blog post with the final stats and let you know when it
is ready.
I'll have to keep running these tests because the more I run them the
faster the ZODB becomes ;-) Would you have guessed that the ZODB
On Mon, 2008-02-25 at 07:36 +0200, Roché Compaan wrote:
> I'll update my blog post with the final stats and let you know when it
> is ready.
>
I'll have to keep running these tests because the more I run them the
faster the ZODB becomes ;-) Would you have guessed that the ZODB is
faster at both i
I made a "lovely" mistake in my first round of benchmarks. Lovely, in
that it puts the ZODB in a much better light. When I first ran the
Postgres test, I neglected to put an index on the key field of the
table. I only added the index before I timed lookups on Postgres but
forgot to retest insertion
Roché Compaan wrote at 2008-2-7 21:44 +0200:
> ...
>There are use cases where having a container in the ZODB that can handle
>large volumes and maintain a high insertion rate would be very
>convenient. An example of such a use case would be a site with millions
>of members where each member has the
On Thu, 2008-02-07 at 20:26 +0100, Dieter Maurer wrote:
> Roché Compaan wrote at 2008-2-7 21:21 +0200:
> > ...
> >So if I asked you to build a data structure for the ZODB that can do
> >insertions at a rate comparable to Postgres on high volumes, do you
> >think that it can be done?
>
> If you nee
Roché Compaan wrote at 2008-2-7 21:21 +0200:
> ...
>So if I asked you to build a data structure for the ZODB that can do
>insertions at a rate comparable to Postgres on high volumes, do you
>think that it can be done?
If you need a high write rate, the ZODB is probably not optimal.
Ask yourself wh
On Thu, 2008-02-07 at 00:39 +0100, Dieter Maurer wrote:
> >If I understand correctly, for each insertion 3 calls are made to
> >"persistent_id"? This is still very far from the 66 I mentioned above?
>
> You did not understand correctly.
>
> You insert an entry. The insertion modifies (at least) o
Roché Compaan wrote at 2008-2-6 20:18 +0200:
>On Tue, 2008-02-05 at 19:17 +0100, Dieter Maurer wrote:
>> Roché Compaan wrote at 2008-2-4 20:54 +0200:
>> > ...
>> >I don't follow? There are 2 insertions and there are 1338046 calls
>> >to persistent_id. Doesn't this suggest that there are 66 obje
Mignon, Laurent wrote at 2008-2-6 08:06 +0100:
>After a lot of tests and benchmark, my feeling is that the ZODB does not seem
>suitable for systems managing many data stored in a plane hierarchy.
>The application that we currently develop is a business process management
>system in opposition to
On Tue, 2008-02-05 at 19:17 +0100, Dieter Maurer wrote:
> Roché Compaan wrote at 2008-2-4 20:54 +0200:
> > ...
> >I don't follow? There are 2 insertions and there are 1338046 calls
> >to persistent_id. Doesn't this suggest that there are 66 objects
> >persisted per insertion? This seems way to
Roché Compaan wrote at 2008-2-4 20:54 +0200:
> ...
>I don't follow? There are 2 insertions and there are 1338046 calls
>to persistent_id. Doesn't this suggest that there are 66 objects
>persisted per insertion? This seems way to high?
Jim told you that "persistent_id" is called for each object
On Feb 4, 2008, at 1:54 PM, Roché Compaan wrote:
I don't follow? There are 2 insertions and there are 1338046 calls
to persistent_id. Doesn't this suggest that there are 66 objects
persisted per insertion? This seems way to high?
It seems like there is some confusion about the corresponde
On Sun, 2008-02-03 at 22:05 +0100, Dieter Maurer wrote:
> The number of "persitent_id" calls suggests that a written
> persistent object has a mean value of 65 subobjects -- which
> fits well will OOBuckets.
>
> However, when the profile is for commits with 100 insertions each,
> then the number o
Roché Compaan wrote at 2008-2-3 09:15 +0200:
> ...
>I have tried different commit intervals. The published results are for a
>commit interval of 100, iow 100 inserts per commit.
>
>> Your profile looks very surprising:
>>
>> I would expect that for a single insertion, typically
>> one persiste
On Sat, 2008-02-02 at 22:10 +0100, Dieter Maurer wrote:
> Roché Compaan wrote at 2008-2-1 21:17 +0200:
> >I have completed my first round of benchmarks on the ZODB and welcome
> >any criticism and advise. I summarised our earlier discussion and
> >additional findings in this blog entry:
> >http://w
Roché Compaan wrote at 2008-2-1 21:17 +0200:
>I have completed my first round of benchmarks on the ZODB and welcome
>any criticism and advise. I summarised our earlier discussion and
>additional findings in this blog entry:
>http://www.upfrontsystems.co.za/Members/roche/where-im-calling-from/zodb-b
I have completed my first round of benchmarks on the ZODB and welcome
any criticism and advise. I summarised our earlier discussion and
additional findings in this blog entry:
http://www.upfrontsystems.co.za/Members/roche/where-im-calling-from/zodb-benchmarks
--
Roché Compaan
Upfront Systems
[I'm resending this mail since it seems it never reached the mailing list?]
On Thu, 2007-12-06 at 15:05 -0500, Jim Fulton wrote:
> On Dec 6, 2007, at 2:40 PM, Godefroid Chapelle wrote:
>
> > Jim Fulton wrote:
> >> On Nov 6, 2007, at 2:40 PM, Sidnei da Silva wrote:
> Despite this change there
On Thu, 2007-12-06 at 15:05 -0500, Jim Fulton wrote:
> On Dec 6, 2007, at 2:40 PM, Godefroid Chapelle wrote:
>
> > Jim Fulton wrote:
> >> On Nov 6, 2007, at 2:40 PM, Sidnei da Silva wrote:
> Despite this change there are still a huge amount
> of unexplained calls to the 'persistent_id' m
On Nov 6, 2007, at 3:17 PM, Roché Compaan wrote:
On Tue, 2007-11-06 at 14:51 -0500, Jim Fulton wrote:
On Nov 6, 2007, at 2:40 PM, Sidnei da Silva wrote:
Despite this change there are still a huge amount
of unexplained calls to the 'persistent_id' method of the
ObjectWriter
in serialize.py.
On Nov 6, 2007, at 2:30 PM, Roché Compaan wrote:
If you can cut the b-node branching factor in half, I bet your
benchmark will run almost twice as fast.
I increased the 'DEFAULT_MAX_BUCKET_SIZE' from 30 to 3 and
DEFAULT_MAX_BTREE_SIZE from 250 to 2500 and it didn't make any
noticeable dif
On Tue, Nov 06, 2007 at 10:01:24PM +0200, Roché Compaan wrote:
> On Tue, 2007-11-06 at 17:40 -0200, Sidnei da Silva wrote:
> > > Despite this change there are still a huge amount
> > > of unexplained calls to the 'persistent_id' method of the ObjectWriter
> > > in serialize.py.
> >
> > Why 'unexpl
On Tue, 2007-11-06 at 14:51 -0500, Jim Fulton wrote:
> On Nov 6, 2007, at 2:40 PM, Sidnei da Silva wrote:
>
> >> Despite this change there are still a huge amount
> >> of unexplained calls to the 'persistent_id' method of the
> >> ObjectWriter
> >> in serialize.py.
> >
> > Why 'unexplained'? 'pe
On Tue, 2007-11-06 at 15:08 -0500, Jim Fulton wrote:
> On Nov 6, 2007, at 3:01 PM, Roché Compaan wrote:
>
> > On Tue, 2007-11-06 at 17:40 -0200, Sidnei da Silva wrote:
> >>> Despite this change there are still a huge amount
> >>> of unexplained calls to the 'persistent_id' method of the
> >>> Ob
On Nov 6, 2007, at 3:01 PM, Roché Compaan wrote:
On Tue, 2007-11-06 at 17:40 -0200, Sidnei da Silva wrote:
Despite this change there are still a huge amount
of unexplained calls to the 'persistent_id' method of the
ObjectWriter
in serialize.py.
Why 'unexplained'? 'persistent_id' is called
On Tue, 2007-11-06 at 17:40 -0200, Sidnei da Silva wrote:
> > Despite this change there are still a huge amount
> > of unexplained calls to the 'persistent_id' method of the ObjectWriter
> > in serialize.py.
>
> Why 'unexplained'? 'persistent_id' is called from the Pickler instance
> being used in
On Nov 6, 2007, at 2:40 PM, Sidnei da Silva wrote:
Despite this change there are still a huge amount
of unexplained calls to the 'persistent_id' method of the
ObjectWriter
in serialize.py.
Why 'unexplained'? 'persistent_id' is called from the Pickler instance
being used in ObjectWriter._du
> Despite this change there are still a huge amount
> of unexplained calls to the 'persistent_id' method of the ObjectWriter
> in serialize.py.
Why 'unexplained'? 'persistent_id' is called from the Pickler instance
being used in ObjectWriter._dump(). It is called for each and every
single object r
On Wed, 2007-10-31 at 10:47 -0400, David Binger wrote:
> On Oct 31, 2007, at 7:35 AM, Roché Compaan wrote:
>
> > Thanks for the explanation.
>
> The actual insertion is very fast. Your benchmark is dominated by
> the time to serialize the changes due to an insertion.
>
> You should usually have
Laurence Rowe wrote:
Essentially you end up with a solution very similar to QueueCatalog but
with the queue being searchable.
The pain is then in modifying all of the indexes to search the queue in
addition to their standard data structures.
In many applications it is acceptable to have a ca
On Fri, 2007-11-02 at 16:00 +, Laurence Rowe wrote:
> Matt Hamilton wrote:
> > David Binger mems-exchange.org> writes:
> >
> >>
> >> On Nov 2, 2007, at 6:20 AM, Lennart Regebro wrote:
> >>
> >>> Lots of people don't do nightly packs, I'm pretty sure such a process
> >>> needs to be completely
This is the 'batch' or 'distribute' pattern that crops up in many
fields.
The best path is normally to understand what the conflicts are, and
where the time is spent.
If in, this case, much time is spent in the preamble, and the actual
inserts are quick, then diving down one time through th
On Nov 2, 2007, at 10:58 AM, Lennart Regebro wrote:
It seems to me having one thread doing a background consolidation one
transaction at a time seems a better way to go,
Maybe, but maybe that just causes big buckets to get invalidated
in all of the clients over and over again, when we could a
On 11/2/07, David Binger <[EMAIL PROTECTED]> wrote:
> > But wouldn't then all other threads get a conflict?
>
> If they are trying to do insertions at the same time as the
> consolidation, yes.
> This data structure won't stop insertion conflicts, the intent is to
> make them
> less frequent.
But
On Nov 2, 2007, at 10:18 AM, Christian Theune wrote:
Wouldn't a queue be a good data structure to do that? IIRC ZC already
wrote a queue that doesn't conflict:
http://svn.zope.de/zope.org/zc.queue/trunk/src/zc/queue/queue.txt
If you store key/value pairs in the queue, you can do a step-by-ste
Hi,
Am Freitag, den 02.11.2007, 09:56 -0400 schrieb David Binger:
> On Nov 2, 2007, at 8:39 AM, Lennart Regebro wrote:
>
> > On 11/2/07, Matt Hamilton <[EMAIL PROTECTED]> wrote:
> >> That may just end up causing delays periodically in
> >> transactions... ie delays
> >> that the user sees, as o
On Nov 2, 2007, at 8:39 AM, Lennart Regebro wrote:
On 11/2/07, Matt Hamilton <[EMAIL PROTECTED]> wrote:
That may just end up causing delays periodically in
transactions... ie delays
that the user sees, as opposed to doing it via another thread or
something. But
then as only one thread woul
On 11/2/07, Matt Hamilton <[EMAIL PROTECTED]> wrote:
> That may just end up causing delays periodically in transactions... ie delays
> that the user sees, as opposed to doing it via another thread or something.
> But
> then as only one thread would be doing this at a time it might not be too bad.
On Nov 2, 2007, at 6:20 AM, Lennart Regebro wrote:
Lots of people don't do nightly packs, I'm pretty sure such a process
needs to be completely automatic. The question is weather doing it in
a separate process in the background, or ever X transactions, or every
X seconds, or something.
Okay,
On 11/2/07, David Binger <[EMAIL PROTECTED]> wrote:
> I think that option would work. I think it would suffice to do a
> "Big.update(Small); Small.clear()" operation before a nightly pack.
Lots of people don't do nightly packs, I'm pretty sure such a process
needs to be completely automatic. The
On Nov 2, 2007, at 5:48 AM, Lennart Regebro wrote:
On 11/1/07, Matt Hamilton <[EMAIL PROTECTED]> wrote:
An interesting idea. Surely we need the opposite though, and that
is an
additional BTree with a very large bucket size, as we want to
minimize the
chance of a bucket split when inserting
On 11/1/07, Matt Hamilton <[EMAIL PROTECTED]> wrote:
> An interesting idea. Surely we need the opposite though, and that is an
> additional BTree with a very large bucket size, as we want to minimize the
> chance of a bucket split when inserting? Then we occasionally consolidate and
> move the it
Quick note...
Smaller buckets, fewer conflicts, more overhead on reading and writing.
Larger buckets, more conflicts, less overhead on reading and writing.
One bucket ... constant conflicts.
I'd bet that the additional tree with tiny buckets would be best.
Transfer them into the normal tree on
On Nov 1, 2007, at 4:25 PM, Matt Hamilton wrote:
David Binger mems-exchange.org> writes:
On Nov 1, 2007, at 7:05 AM, Matt Hamilton wrote:
Ie we perhaps look at a catalog data structure
in which writes are initially done to some kind of queue then moved
to the
BTrees at a later point.
A
On Nov 1, 2007, at 4:25 PM, Matt Hamilton wrote:
David Binger mems-exchange.org> writes:
On Nov 1, 2007, at 7:05 AM, Matt Hamilton wrote:
Ie we perhaps look at a catalog data structure
in which writes are initially done to some kind of queue then moved
to the
BTrees at a later point.
A s
On Wed, 2007-10-31 at 10:47 -0400, David Binger wrote:
> On Oct 31, 2007, at 7:35 AM, Roché Compaan wrote:
>
> > Thanks for the explanation.
>
> The actual insertion is very fast. Your benchmark is dominated by
> the time to serialize the changes due to an insertion.
>
> You should usually have
On Nov 1, 2007, at 7:05 AM, Matt Hamilton wrote:
Ie we perhaps look at a catalog data structure
in which writes are initially done to some kind of queue then moved
to the
BTrees at a later point.
A suggestion: use a pair of BTrees, one with a high branching factor
(bucket size)
and one w
On Oct 31, 2007, at 7:35 AM, Roché Compaan wrote:
Thanks for the explanation.
The actual insertion is very fast. Your benchmark is dominated by
the time to serialize the changes due to an insertion.
You should usually have just 2 instances to serialize per insertion of
a new instance: the
On Wed, 2007-10-31 at 10:00 +, Laurence Rowe wrote:
> It looks like ZODB performance in your test has the same O(log n)
> performance as PostgreSQL checkpoints (the periodic drops in your
> graph). This should come as no surprise. B-Trees have a theoretical
> Search/Insert/Delete time comple
I think someone proposed to have something just like a WAL in ZODB.
That could be an interesting optimization.
--
Sidnei da Silva
Enfold Systemshttp://enfoldsystems.com
Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214
___
For more
74 matches
Mail list logo
