Re: [Zope3-Users] Zope 3 Capacities
David Johnson wrote: Today, alone, we've processed 15M TX in about 2 hours (2080tps), using a single MySQL server (not TTW), and MySQL just yawns. In our experience, only MySQL and Oracle are capable of doing this (and Oracle only in versions 9 and above). This is also why I ask so many RDBMS questions on this list. FWIW, PostgreSQL can do this too if you keep the schema simple and pay attention to fsync behavior. (Note that MySQL doesn't generally fsync by default, while PostgreSQL does.) You need to fsync to be sure you don't lose transactions in the event of a power failure or system crash. To fsync more than about 300 times per second, you need NVRAM (aka "write cache") in the disk controller. This usually means a SAN. I do think we will need multiple instances. I would like to use Zope and perhaps replace Gadfly with MySQL. Has anyone done this in Zope 3? If so, how? I would estimate based upon my preliminary tests that Zope with ZODB can handle about 300/tps on our hardware. You're working well beyond the speed that ZODB is currently optimized for, so ZODB doesn't seem like a good match. However, it might be a good experiment to disable the fsync in ZODB; this would let you know how fast ZODB could perform on theoretical hardware that takes zero time to fsync. The versioning feature of Zope alone would blow our clients away. Can you mimic versioning in the RDBMS? Shane ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] Zope 3 Capacities
On 1/20/06, David Johnson <[EMAIL PROTECTED]> wrote: > Today, alone, we've processed 15M TX in about 2 hours (2080tps), using a > single MySQL server (not TTW), and MySQL just yawns. *Writing* transactions? > how? I would estimate based upon my preliminary tests that Zope with ZODB > can handle about 300/tps on our hardware. If that is writing, then the hardware is better than mine, that's for sure. ;) > The versioning feature of Zope alone would blow our clients away. What exactly do you mean with "Versioning feature" in this context? -- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.cps-project.org/ ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] Zope 3 Capacities
On Friday 20 January 2006 08:23, David Johnson wrote: > I do think we will need multiple instances. I would like to use Zope and > perhaps replace Gadfly with MySQL. Has anyone done this in Zope 3? If so, > how? I would estimate based upon my preliminary tests that Zope with ZODB > can handle about 300/tps on our hardware. Note that Gadfly is only a very, very simple default RDB that we provide to test the RDB capabilities in Zope. Of course you can hook up any RDB you want to including MySQL for which there are RDB adapters. You can also look into sqlos, which provides mapping RDB rows to objects inside Zope. However with those solutions you are loosing some features, like versioning, unless you put some effort into the development to map the versioning APIs to sqlos. Regards, Stephan -- Stephan Richter CBU Physics & Chemistry (B.S.) / Tufts Physics (Ph.D. student) Web2k - Web Software Design, Development and Training ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
RE: [Zope3-Users] Zope 3 Capacities
I wish I could say what we're doing, but I know potential competitors are listening and these lists are public. I can discuss privately if you would like to know more. You're close with the NSA stuff, although on a much smaller and public scale. Today, alone, we've processed 15M TX in about 2 hours (2080tps), using a single MySQL server (not TTW), and MySQL just yawns. In our experience, only MySQL and Oracle are capable of doing this (and Oracle only in versions 9 and above). This is also why I ask so many RDBMS questions on this list. I do think we will need multiple instances. I would like to use Zope and perhaps replace Gadfly with MySQL. Has anyone done this in Zope 3? If so, how? I would estimate based upon my preliminary tests that Zope with ZODB can handle about 300/tps on our hardware. The versioning feature of Zope alone would blow our clients away. > -Original Message- > From: Jim Fulton [mailto:[EMAIL PROTECTED] > Sent: Friday, January 20, 2006 6:23 AM > To: David Johnson > Cc: 'Andreas Jung'; zope3-users@zope.org > Subject: Re: [Zope3-Users] Zope 3 Capacities > > David Johnson wrote: > > We're looking at 10-100 billion tx per year stored and performed. > > > > Partly I'm trying to gauge where the dividing line is between using the > ZODB > > and not, and also estimate how many server instances should be running. > > I'm not sure any transactional database will handle that sort of rate with > a single database. We did some tests 2 years ago and with commodity > hardware, we were able to commit around 50 simple transactions per second > (tps) > to a file storage over ZEO. This is about 60 times slower than you need, > assuming > 100 billion tx per year or about 3000 tps. I imagine you could do > somewhat better > than that if you got beefier hardware. > > A *quick* google on transaction rates yielded a fairly old article: > >http://www.wintercorp.com/rwintercolumns/ie_9903.html > > At that time, most of the databases tested on Unix did less that 100 tps. > Many of them much less. Of course that was a long time ago. > Does anyone know of more recent data? > > Of course, if you can segregate your data, you can get higher > transaction rates by employing multiple database servers, ZODB > or otherwise. I've faily confident that this is what you'll > need to do. > > Jim > > -- > Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! > CTO (540) 361-1714http://www.python.org > Zope Corporation http://www.zope.com http://www.zope.org ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] Zope 3 Capacities
David Johnson wrote: We're looking at 10-100 billion tx per year stored and performed. Partly I'm trying to gauge where the dividing line is between using the ZODB and not, and also estimate how many server instances should be running. I'm not sure any transactional database will handle that sort of rate with a single database. We did some tests 2 years ago and with commodity hardware, we were able to commit around 50 simple transactions per second (tps) to a file storage over ZEO. This is about 60 times slower than you need, assuming 100 billion tx per year or about 3000 tps. I imagine you could do somewhat better than that if you got beefier hardware. A *quick* google on transaction rates yielded a fairly old article: http://www.wintercorp.com/rwintercolumns/ie_9903.html At that time, most of the databases tested on Unix did less that 100 tps. Many of them much less. Of course that was a long time ago. Does anyone know of more recent data? Of course, if you can segregate your data, you can get higher transaction rates by employing multiple database servers, ZODB or otherwise. I've faily confident that this is what you'll need to do. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] Zope 3 Capacities
--On 20. Januar 2006 10:52:05 +0100 Lennart Regebro <[EMAIL PROTECTED]> wrote: On 1/20/06, Andreas Jung <[EMAIL PROTECTED]> wrote: It's even worser (due to a calculation error): 10 billion would mean 310 tx/second but 100 billion would mean 3100 tx/second...you should look for something _bigger_. 3100 WRITE transactions per second? 3100 changed or added objects per second? Yes, I want to know what kind of web app creates that amount of data too. ;) Perhaps the new NSA-we-collect-all-your-data database? -aj pgpncvsSeLr0e.pgp Description: PGP signature ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] Zope 3 Capacities
On 1/20/06, Andreas Jung <[EMAIL PROTECTED]> wrote: > It's even worser (due to a calculation error): 10 billion would mean 310 > tx/second but 100 billion would mean 3100 tx/second...you should look for > something _bigger_. 3100 WRITE transactions per second? 3100 changed or added objects per second? Yes, I want to know what kind of web app creates that amount of data too. ;) -- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.cps-project.org/ ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] Zope 3 Capacities
Andreas Jung wrote: This would be 31-310 tx/second...forget the ZODB :-) It's even worser (due to a calculation error): 10 billion would mean 310 tx/second but 100 billion would mean 3100 tx/second...you should look for something _bigger_. I'm curious to know what the app is ;-) I don't know of many applications which can actually generate that level of transactions... Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
RE: [Zope3-Users] Zope 3 Capacities
--On 19. Januar 2006 21:00:15 +0100 Andreas Jung <[EMAIL PROTECTED]> wrote: --On 19. Januar 2006 13:54:55 -0600 David Johnson <[EMAIL PROTECTED]> wrote: We're looking at 10-100 billion tx per year stored and performed. This would be 31-310 tx/second...forget the ZODB :-) It's even worser (due to a calculation error): 10 billion would mean 310 tx/second but 100 billion would mean 3100 tx/second...you should look for something _bigger_. -aj pgpg5pCXBkT2Y.pgp Description: PGP signature ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] Zope 3 Capacities
Andreas Jung wrote: --On 19. Januar 2006 13:54:55 -0600 David Johnson <[EMAIL PROTECTED]> wrote: We're looking at 10-100 billion tx per year stored and performed. This would be 31-310 tx/second...forget the ZODB :-) Well, if those transactions just add to a log, ZODB could handle it on suitable hardware. ZODB could also handle it if there's no concurrency. But you probably have concurrency and you're probably building indexes at the same time. An RDBMS is just easier for that kind of task. Shane ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
RE: [Zope3-Users] Zope 3 Capacities
--On 19. Januar 2006 13:54:55 -0600 David Johnson <[EMAIL PROTECTED]> wrote: We're looking at 10-100 billion tx per year stored and performed. This would be 31-310 tx/second...forget the ZODB :-) -aj pgpz69nycXjX6.pgp Description: PGP signature ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
RE: [Zope3-Users] Zope 3 Capacities
We're looking at 10-100 billion tx per year stored and performed. Partly I'm trying to gauge where the dividing line is between using the ZODB and not, and also estimate how many server instances should be running. > -Original Message- > From: Andreas Jung [mailto:[EMAIL PROTECTED] > Sent: Thursday, January 19, 2006 1:29 PM > To: David Johnson; zope3-users@zope.org > Subject: Re: [Zope3-Users] Zope 3 Capacities > > > > --On 19. Januar 2006 12:26:42 -0600 David Johnson <[EMAIL PROTECTED]> > wrote: > > > I'm trying to scope out Zope 3's capacities. Does anyone have a large > > ZODB? How big is it (in terms of objects)? Any performance issues? > > Does anyone have a very high volume site? What kind of volume with what > > kind of degree of application logic? > > > > > > > > What architectural things have affected performance? I general we're > > looking at using an RDBMS, but we're trying to get a better feeling of > > when we can use the ZODB and when we should stick to the RDBMS. I like > > the ZODB because of it's versioning information in particular. > > > > > > > > We have various datasets: > > > > 1. Transactions: 10-100 billion objects > > per day, month, yr? > > > 2. Users: 10-100 million objects > > up to 1-2 million objects seem to be common for large sites nowadays > > > 3. Contacts: 100,000 objects > > 4. Customers: 100-1000 objects > > Possibly you might consider a hybrid solution using the ZODB and a RDBMS. > > -aj ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] Zope 3 Capacities
--On 19. Januar 2006 12:26:42 -0600 David Johnson <[EMAIL PROTECTED]> wrote: I'm trying to scope out Zope 3's capacities. Does anyone have a large ZODB? How big is it (in terms of objects)? Any performance issues? Does anyone have a very high volume site? What kind of volume with what kind of degree of application logic? What architectural things have affected performance? I general we're looking at using an RDBMS, but we're trying to get a better feeling of when we can use the ZODB and when we should stick to the RDBMS. I like the ZODB because of it's versioning information in particular. We have various datasets: 1. Transactions: 10-100 billion objects per day, month, yr? 2. Users: 10-100 million objects up to 1-2 million objects seem to be common for large sites nowadays 3. Contacts: 100,000 objects 4. Customers: 100-1000 objects Possibly you might consider a hybrid solution using the ZODB and a RDBMS. -aj pgpdR1yldcisJ.pgp Description: PGP signature ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] Zope 3 Capacities
On 1/19/06, David Johnson <[EMAIL PROTECTED]> wrote: > I'm trying to scope out Zope 3's capacities. A quick note: In the things you mention, Zope 2 and Zope 3 are similar, as they both use the same ZODB. So answers to your question could equally be done with Zope 2 example. And I personally do not have a high volume site, but such sites has been installed. The people who did that gets to answer on how big they were. :) The bottleneck with ZODB tends to be writing transactions, while just reading transactions is handled with much greater ease, so that's the most critical data in tis type of evaluations, I would guess. -- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.cps-project.org/ ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
