[Zope] Re: Zope scalabilty and problems
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Dieter Maurer wrote: > Florent Guillaume wrote at 2005-9-5 16:49 +0200: > >>... >> >>>It is just that the CMF is unable to fix a year long bug >> >>It's not "CMF"'s fault, it's only the responsibility of Windows developers >>running CMF. I for one couldn't care less. > > > Really? > >That the CMF makes an hierarchical filesystem scan for >each access to a filesystem directory view >should be the responsibility of the Windows developers running CMF? > >There are no bugs: if you meat one, it proves that you >use the wrong system... > > >>... >> >>> then each access to a "filesystem directory view" triggers >>> a hierarchical file system scan. >>> >>> The result is a slowdown by several orders of magnitude. >> >>I believe Tres has now fixed this. > > It was not yet fixed when I made an svn checkout of the CMF 1.5 branch > about 2 weeks ago. The patch I merged for the CMF 1.5.2 release (back in late July), did not completely remove the 'os.pathwalk': it merely made the exclusion of the .svn directory work cleanly. The better fix would be to use the Win32 APIs for file monitoring; Mark Hammond volunteered to look into that. Tres. - -- === Tres Seaver +1 202-558-7113 [EMAIL PROTECTED] Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFDHXCv+gerLs4ltQ4RAgShAKCuvxwdqHpYGUQMLIxTUT1YGXbbqwCguFIH KANLJ4XcxzwPMnhcYI6raE8= =94l4 -END PGP SIGNATURE- ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Re: Zope scalabilty and problems
Florent Guillaume wrote at 2005-9-5 16:49 +0200: > ... >> It is just that the CMF is unable to fix a year long bug > >It's not "CMF"'s fault, it's only the responsibility of Windows developers >running CMF. I for one couldn't care less. Really? That the CMF makes an hierarchical filesystem scan for each access to a filesystem directory view should be the responsibility of the Windows developers running CMF? There are no bugs: if you meat one, it proves that you use the wrong system... > ... >> then each access to a "filesystem directory view" triggers >> a hierarchical file system scan. >> >> The result is a slowdown by several orders of magnitude. > >I believe Tres has now fixed this. It was not yet fixed when I made an svn checkout of the CMF 1.5 branch about 2 weeks ago. -- Dieter ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Re: Zope scalabilty and problems
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Florent Guillaume wrote: > Dieter Maurer wrote: > >>> I am setting up sites using Plone and am concerned to read some of the >>> comments on it. Is it really that bad performance wise? >> >> No, it is not. >> >> It is just that the CMF is unable to fix a year long bug > > > It's not "CMF"'s fault, it's only the responsibility of Windows > developers running CMF. I for one couldn't care less. > >> which affects Windows only when Zope is run in debug mode: >> >> then each access to a "filesystem directory view" triggers >> a hierarchical file system scan. >> >> The result is a slowdown by several orders of magnitude. It got *much* worse when CMF shifted to use subversion, which has many more bookkeeping files in the .svn directories. Plone developers who work on Windows sent up an anguished cry just after we switched over, while they were trying to prepare their 2.1 release candidate. > I believe Tres has now fixed this. I applied a tourniquet, which was the major reason for pushing out the 1.5.3 release. Mark Hammond volunteered (at the Plone Symposium in New Orleans) to look at the feasibility of a better solution which would use the native Win32 file monitoring APIs: http://www.zope.org/Collectors/CMF/367 As Chris noted, running with 'debug-mode off' also avoids this issue, if you can't upgrade CMF for some reason (and you *should* be running this way if performance matters, anyway). Tres - -- === Tres Seaver +1 202-558-7113 [EMAIL PROTECTED] Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFDHHG6+gerLs4ltQ4RAl49AJ9+R/aUhA/2UvEWEsRZKIqMnf7fRwCglVT9 4iuXchoDl+Bx6rLMrvd91JI= =Gx7s -END PGP SIGNATURE- ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Re: Zope scalabilty and problems
Dieter Maurer wrote: I am setting up sites using Plone and am concerned to read some of the comments on it. Is it really that bad performance wise? No, it is not. It is just that the CMF is unable to fix a year long bug It's not "CMF"'s fault, it's only the responsibility of Windows developers running CMF. I for one couldn't care less. which affects Windows only when Zope is run in debug mode: then each access to a "filesystem directory view" triggers a hierarchical file system scan. The result is a slowdown by several orders of magnitude. I believe Tres has now fixed this. Florent -- Florent Guillaume, Nuxeo (Paris, France) CTO, Director of R&D +33 1 40 33 71 59 http://nuxeo.com [EMAIL PROTECTED] ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Re: Zope scalabilty and problems
michael nt milne wrote: I am setting up sites using Plone and am concerned to read some of the comments on it. Is it really that bad performance wise? "it depends", unless you have a lot of experience with it, Zope and CMF, you might well find yourself with a slow site on your hands and not many apparent options ;-) I'm using CMFMember also-why is it 'apalling'? It's an incredibly heavyweight and inefficient solution to a problem that requires an extremely fast and efficient solution: user object are used a LOT in most Zope apps, CMFMember makes them slow, in a numbner of ways. Generating writes which result in AT-based member objects being reindexed and often resulting in conflicterrors that dramatically slow down a site is not a good thing... cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Re: Zope scalabilty and problems
Hi Dieter, Dieter Maurer wrote: I am setting up sites using Plone and am concerned to read some of the comments on it. Is it really that bad performance wise? No, it is not. ...I agree, if it is correctly set up and optimised, which isn't the case if someone is here complaining about performance, which sadly, new users blown away by Plone's functionality often are... It is just that the CMF is unable to fix a year long bug which affects Windows only when Zope is run in debug mode: then each access to a "filesystem directory view" triggers a hierarchical file system scan. The result is a slowdown by several orders of magnitude. FWIW, my experiences with Plone are predominantly on Linux, and that's what my comments are based on... The problem disappears when you either: * turn off "debug mode" Yes, this is a biggie. Thankfully, I believe newer versions of Zope ship with this turned off by default? cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Re: Zope scalabilty and problems
Chris has opinions on performance based on his experiences. Others have different viewpoints. Here is an more empirical treatment of what you might expect: http://www.ifpeople.net/resources/downloads/PlonePerformance.pdf --Paul michael nt milne wrote: Hi I am setting up sites using Plone and am concerned to read some of the comments on it. Is it really that bad performance wise? I'm using CMFMember also-why is it 'apalling'? On 9/2/05, N.Davis <[EMAIL PROTECTED]> wrote: Chris Withers wrote: Andrew Sawyers wrote: Did Matt indicate if he was running multiple zeo app servers? It might help to be spreading the load. 1, writes a day is not outrageous Yes, but he's using Plone, which implies not only all of the CMF reindexing overhead, but also all the AT and Plone layers on top. That could get even worse if he's gone to town with a workflow tool or is using any of the truly appalling Plone addons, such as CMFMember, CMFForum, etc... cheers, Chris This sounds like the classic abstraction vs performance debate. In the late 80s some people criticised C++ and OOP versus C for the same reasons. Might it not be better, rather than telling people not to use Plone + add-ons for this reason, to just push for across the board performance improvements, which means removing bottlenecks in Plone, Archetypes, specific Products, as well as the underlying Zope/CMF ? Also some of the products you are talking about may be somewhat immature and you know what they say, don't do premature optimisation, do logic first, performance tune later. What you describe as "appalling" might perhaps be able to be performance tuned? BTW CPSSkins is another performance hit. Would you care to list the worst culprits? ;-) Regards Nick ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev ) ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev ) ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Re: Zope scalabilty and problems
michael nt milne wrote at 2005-9-2 09:13 -0400: >I am setting up sites using Plone and am concerned to read some of the >comments on it. Is it really that bad performance wise? No, it is not. It is just that the CMF is unable to fix a year long bug which affects Windows only when Zope is run in debug mode: then each access to a "filesystem directory view" triggers a hierarchical file system scan. The result is a slowdown by several orders of magnitude. The problem disappears when you either: * do not run on Windows * turn off "debug mode" * fix the silly bug Of course, a Plone site is not as fast as a static one. But, I hope, you expect this... -- Dieter ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Re: Zope scalabilty and problems
Hi I am setting up sites using Plone and am concerned to read some of the comments on it. Is it really that bad performance wise? I'm using CMFMember also-why is it 'apalling'? On 9/2/05, N.Davis <[EMAIL PROTECTED]> wrote: > Chris Withers wrote: > > Andrew Sawyers wrote: > > > >> Did Matt indicate if he was running multiple zeo app servers? It > >> might help > >> to be spreading the load. 1, writes a day is not outrageous > > > > > > Yes, but he's using Plone, which implies not only all of the CMF > > reindexing overhead, but also all the AT and Plone layers on top. That > > could get even worse if he's gone to town with a workflow tool or is > > using any of the truly appalling Plone addons, such as CMFMember, > > CMFForum, etc... > > > > cheers, > > > > Chris > > > > This sounds like the classic abstraction vs performance debate. In the > late 80s some people criticised C++ and OOP versus C for the same reasons. > > Might it not be better, rather than telling people not to use Plone + > add-ons for this reason, to just push for across the board performance > improvements, which means removing bottlenecks in Plone, Archetypes, > specific Products, as well as the underlying Zope/CMF ? Also some of the > products you are talking about may be somewhat immature and you know > what they say, don't do premature optimisation, do logic first, > performance tune later. > > What you describe as "appalling" might perhaps be able to be performance > tuned? > > BTW CPSSkins is another performance hit. Would you care to list the > worst culprits? ;-) > > Regards > Nick > > > ___ > Zope maillist - Zope@zope.org > http://mail.zope.org/mailman/listinfo/zope > ** No cross posts or HTML encoding! ** > (Related lists - > http://mail.zope.org/mailman/listinfo/zope-announce > http://mail.zope.org/mailman/listinfo/zope-dev ) > ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Re: Zope scalabilty and problems
Chris Withers wrote: Andrew Sawyers wrote: Did Matt indicate if he was running multiple zeo app servers? It might help to be spreading the load. 1, writes a day is not outrageous Yes, but he's using Plone, which implies not only all of the CMF reindexing overhead, but also all the AT and Plone layers on top. That could get even worse if he's gone to town with a workflow tool or is using any of the truly appalling Plone addons, such as CMFMember, CMFForum, etc... cheers, Chris This sounds like the classic abstraction vs performance debate. In the late 80s some people criticised C++ and OOP versus C for the same reasons. Might it not be better, rather than telling people not to use Plone + add-ons for this reason, to just push for across the board performance improvements, which means removing bottlenecks in Plone, Archetypes, specific Products, as well as the underlying Zope/CMF ? Also some of the products you are talking about may be somewhat immature and you know what they say, don't do premature optimisation, do logic first, performance tune later. What you describe as "appalling" might perhaps be able to be performance tuned? BTW CPSSkins is another performance hit. Would you care to list the worst culprits? ;-) Regards Nick ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Re: Zope scalabilty and problems
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Kennamore, Matthew G [NTK] wrote: > Here is our situation, we originally launched zope with a small user > base in mind and like most IT implementations it got away from us > quickly and we have over 1000 users now. We are experiencing huge > slowdowns and lots of complaints of performance. > > Let me outline our system: > > I have a load balancer that round robins incoming request to 3 zeo > servers > > I have 1 zope server on the back en that severs the data, this is a > large single processor machine with 4 gigs of ram > > Each zeo server is 2 gigs of ram, single processor 3.2 gighz > > The throughput doesn't appear to be the problem as ram/processors > are. > > Short term I plan to change all zeo's to 4 gigs ram and add 3 > additional servers. > > > The Technologies we are using are Zope, Zeo, plone and so forth. Why > are we having these large issues? I see sites with hundreds of > thousands of users and they don't seem to have these issues? We plan > to recode our site to get away from the Data.fs as soon as we can but > that is a ways off yet. What am I doing wrong can anyone help? What versions of Python, Zope, etc. are you running, and on what OS? Your latest posting (the "blocked transaction" entries in your storage server logs) sounds like you might be running Zope 2.6.x. Tres. - -- === Tres Seaver +1 202-558-7113 [EMAIL PROTECTED] Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFDFygV+gerLs4ltQ4RArPjAJ4jda2vJJtQVZHbCGvt4gH7oRa3agCeMbMF +0jnwU7OXY+kge35A9gny4s= =3G6a -END PGP SIGNATURE- ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
