Re: [Zope-dev] Apache Mod_rewrite question
Hello Ganesha,
Wednesday, July 14, 2004, 1:40:14 AM, you wrote:
I'm using mod_rewrite with zope but without maps.
Can your describe in details your task?
GS Hi,
GS I came across a strange behavior of mod_rewrite while using with zope.
GS Although it is not a zope question, I was wondering if any of you have
GS run into this. We are using mod_rewrite to redirect the users to one of
GS the zope sites of a Zeo cluster.
GS The directive RewriteMap has unexpected behavior when it is used to
GS define more than one map with same map name in diffrent places. Normally
GS all the directives within a tag applies only to that tag.
GS For example:
GS VirtualHost *
GS ServerName www.foo.com
GS DocumentRoot /var/www/foo
GS RewriteEngine on
GS RewriteLog/var/log/httpd/foo.log
GS RewriteLogLevel 2
GS RewriteMap map rnd:/etc/httpd/conf/foomap.txt
GS RewriteRule ^/Foo(.*)
GS http://127.0.0.1${map:IP}/VirtualHostBase/http/foo/$1 [L,P]
GS /VirtualHost
GS VirtualHost *
GS ServerName www.bar.com
GS DocumentRoot /var/www/bar
GS RewriteEngine on
GS RewriteLog/var/log/httpd/bar.log
GS RewriteLogLevel 2
GS RewriteMap map rnd:/etc/httpd/conf/barmap.txt
GS RewriteRule ^/Bar(.*)
GS http://127.0.0.1${map:IP}/VirtualHostBase/http/bar/$1 [L,P]
GS /VirtualHost
GS The above is expected to work. However ModRewrite, does not read the map
GS file foomap.txt or barmap.txt everytime it needs it and it caches the
GS results after the first time it reads for faster operation.
GS The problem is the cache treats both the file as the same since they
GS have the same name! Hence if you visit foo.com first, the data stored in
GS foomap.txt is returned from the cache next time you visit foo.com or
GS bar.com. This causes apache to redirect into the wrong zope site.
GS Has anyone else experienced similar problems ?
GS thanks,
GS ganesh
GS Zeomega LLC
GS Open minds' Open Solutions
GS Plano, Texas, USA
GS Bangalore, India
GS phone: 972-731-6750
GS http://www.zeomega.com
GS ___
GS Zope-Dev maillist - [EMAIL PROTECTED]
GS http://mail.zope.org/mailman/listinfo/zope-dev
GS ** No cross posts or HTML encoding! **
GS (Related lists -
GS http://mail.zope.org/mailman/listinfo/zope-announce
GS http://mail.zope.org/mailman/listinfo/zope )
--
Best regards,
Eugenemailto:[EMAIL PROTECTED]
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] any limits on object number?
We are builiding a large portal using Zope. We need to create a large number of objects. The data component of the objects is small, but each object carries lots of metadata. My question is: Is there any limit on the number of objects in a given folder? I am not planning to use any external RDBMS. Are there any known performace isssues when the numbeer of objects increase, particularly when we store them in the same folder? thanks in advance. Nagarjuna ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] any limits on object number?
On Wed, 2004-07-14 at 14:20, Andreas Jung wrote: --On Mittwoch, 14. Juli 2004 13:46 Uhr +0530 Nagarjuna G. [EMAIL PROTECTED] wrote: We are builiding a large portal using Zope. We need to create a large number of objects. How large? in one case, we want the metadata of all the wikipedia contents, that will be around 300,000 objects. The data component of the objects is small, but each object carries lots of metadata. My question is: Is there any limit on the number of objects in a given folder? No, but you should use BTreeFolder2 in any case and don't use the standard Zope folder! thanks, this is certainly going to help, I will try inheriting this class. I am not planning to use any external RDBMS. Are there any known performace isssues when the numbeer of objects increase, particularly when we store them in the same folder? Maybe tell us some details. The project is GNOWSYS, an application that builds semantic web portals on top of Zope. When we tried to upload objects from wikipedia's sqldump of after about 40,000 objects in a folder the script went for a toss. roughly each time we ran the script the same problem. even to visit the folder of even 1000 objects through ZMI is a pain. I think that is where BTreeFolder2 will be very useful. I am also thinking to use a squid cache style of storing objects in zodb, where the objects go into many subfolders (i dont know what is this method called technically) with several sub folders with names alphabetized/numbered. this way i can store large number within zodb though not in the same folder/BTreefolder2. Should I use both methods? Not related to Zope, but what about filesystems? there must be a limit on the number of files within one folder depending on which filesystem one uses. Thanks Nagarjuna ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: any limits on object number?
On Wed, 14 Jul 2004 13:46:57 +0530 Nagarjuna G. [EMAIL PROTECTED] wrote: We are builiding a large portal using Zope. We need to create a large number of objects. The data component of the objects is small, but each object carries lots of metadata. My question is: Is there any limit on the number of objects in a given folder? I am not planning to use any external RDBMS. Are there any known performace isssues when the numbeer of objects increase, particularly when we store them in the same folder? Normal zope folders should probably not be used to hold more than a few dozen items. They store a list of their children in a single ZODB record, and as their numbers increase so do the size of the transactions that change the folder. Also normal Zope folders do not handle concurrent updates and will thus not perform well when multiple users are adding items to the folder. Shane Hathaway's BTreeFolder2 product solves these problems. It is the thing to use when you want to store large numbers of objects in a single folder. It also handles concurrency much better. hth, -Casey ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] any limits on object number?
When we tried to upload objects from wikipedia's sqldump of after about 40,000 objects in a folder the script went for a toss. roughly each time we ran the script the same problem. even to visit the folder of even 1000 objects through ZMI is a pain. I think that is where BTreeFolder2 will be very useful. Your observations make sense in the context of the standard Zope folder. BTreeFolder2 will help. I am assuming whatever script you use to populate the folder(s) commits the transaction ever N objects, right? I am also thinking to use a squid cache style of storing objects in zodb, where the objects go into many subfolders (i dont know what is this method called technically) with several sub folders with names alphabetized/numbered. this way i can store large number within zodb though not in the same folder/BTreefolder2. Should I use both methods? I would definitely use the subfolder method. Not related to Zope, but what about filesystems? there must be a limit on the number of files within one folder depending on which filesystem one uses. The ZODB, in case you use the standard FileStorage, is one monolithic file. So there is no large number of files on the server file system regardless of how many objects you store. For files on the file systems your choice of file system depends not just on the number of objects but also its size. To re-use your earlier example of Squid cache files, which are numerous and mostly small, ReiserFS is normally recommended. It deals much better with large quantities of files in one folder. jens ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] any limits on object number?
[resend] We are builiding a large portal using Zope. We need to create a large number of objects. How large? in one case, we want the metadata of all the wikipedia contents, that will be around 300,000 objects. The data component of the objects is small, but each object carries lots of metadata. My question is: Is there any limit on the number of objects in a given folder? No, but you should use BTreeFolder2 in any case and don't use the standard Zope folder! 'Folders' provide an easy gui for managing sub-items one at a time, and a single interface for accessing sub-items that enables things like the tree view (in the left pane of the management interface) to work with different folder implementations. Im not sure any of that is a benefit to you, and Folders arent the only kind of ZODB container. I suspect you would be better off directly using a BTree of some kind. -- Toby Dickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] TAL Hotfix 2004-07-14 for Zope 2.7.0, 2.7.1
This hotfix product fixes a security bug in Page Templates. This fix ensures that values substituted in named slots in translated elements are properly encoded. If encoding is not desired and the source of the replacement text is trusted, the structure modifier can be used with the tal:content or tal:replace attribute to explicitly disable encoding. Affected Versions This fix applies to Zope 2.7.0 and 2.7.1. Zope versions 2.7.2 and newer already contain this fix, and do not require this hotfix. This fix also obsoletes 'Hotfix_20040713', so that should be uninstalled when this hotfix is installed. See the README.txt file provided with 'Hotfix_20040713' for instructions on removing that hotfix. Getting the Hotfix You can download the hotfix at: http://zope.org/Products/Zope/Hotfix_2004-07-14/Zope%202.7.0%20-%202.7.1/ The product contains a README.txt file with installation instructions. -Fred -- Fred L. Drake, Jr. fred at zope.com Zope Corporation ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Apache Mod_rewrite question
Hi,
We are using mod_rewrite to redirect the incoming requests to one of the
zope instances running on the same machine. It is an SMP machine and has
4 zope instances. The map basically contains
IP port1|port2|port3
Where port1 and port2 etc are the ports where the zope is running. The
problem is mod_rewrite caches the map file based on map name instead of
file as I have mentioned below. But we have solved the problem by using
different map names in each of the VirtualHost sections.
thanks,
ganesh
Zeomega LLC
Open minds' Open Solutions
Plano, Texas, USA
Bangalore, India
phone: 972-731-6750
http://www.zeomega.com
On Wed, 2004-07-14 at 02:23, Eugene wrote:
Hello Ganesha,
Wednesday, July 14, 2004, 1:40:14 AM, you wrote:
I'm using mod_rewrite with zope but without maps.
Can your describe in details your task?
GS Hi,
GS I came across a strange behavior of mod_rewrite while using with zope.
GS Although it is not a zope question, I was wondering if any of you have
GS run into this. We are using mod_rewrite to redirect the users to one of
GS the zope sites of a Zeo cluster.
GS The directive RewriteMap has unexpected behavior when it is used to
GS define more than one map with same map name in diffrent places. Normally
GS all the directives within a tag applies only to that tag.
GS For example:
GS VirtualHost *
GS ServerName www.foo.com
GS DocumentRoot /var/www/foo
GS RewriteEngine on
GS RewriteLog/var/log/httpd/foo.log
GS RewriteLogLevel 2
GS RewriteMap map rnd:/etc/httpd/conf/foomap.txt
GS RewriteRule ^/Foo(.*)
GS http://127.0.0.1${map:IP}/VirtualHostBase/http/foo/$1 [L,P]
GS /VirtualHost
GS VirtualHost *
GS ServerName www.bar.com
GS DocumentRoot /var/www/bar
GS RewriteEngine on
GS RewriteLog/var/log/httpd/bar.log
GS RewriteLogLevel 2
GS RewriteMap map rnd:/etc/httpd/conf/barmap.txt
GS RewriteRule ^/Bar(.*)
GS http://127.0.0.1${map:IP}/VirtualHostBase/http/bar/$1 [L,P]
GS /VirtualHost
GS The above is expected to work. However ModRewrite, does not read the map
GS file foomap.txt or barmap.txt everytime it needs it and it caches the
GS results after the first time it reads for faster operation.
GS The problem is the cache treats both the file as the same since they
GS have the same name! Hence if you visit foo.com first, the data stored in
GS foomap.txt is returned from the cache next time you visit foo.com or
GS bar.com. This causes apache to redirect into the wrong zope site.
GS Has anyone else experienced similar problems ?
GS thanks,
GS ganesh
GS Zeomega LLC
GS Open minds' Open Solutions
GS Plano, Texas, USA
GS Bangalore, India
GS phone: 972-731-6750
GS http://www.zeomega.com
GS ___
GS Zope-Dev maillist - [EMAIL PROTECTED]
GS http://mail.zope.org/mailman/listinfo/zope-dev
GS ** No cross posts or HTML encoding! **
GS (Related lists -
GS http://mail.zope.org/mailman/listinfo/zope-announce
GS http://mail.zope.org/mailman/listinfo/zope )
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
RewriteMaps inefficient (was: [Zope-dev] Apache Mod_rewrite question)
Ganesha Shanmuganathan wrote at 2004-7-13 16:40 -0500: ... The directive RewriteMap has unexpected behavior when it is used to define more than one map with same map name in diffrent places. Normally all the directives within a tag applies only to that tag. There are more strange things with Apache's RewriteMaps. Be warned: they are very inefficiently implemented for large maps! We used such maps to let Apache serve static CMF content as folkore reports that Apache is an order of magnitude faster than Zope in serving static content. It turned out that the solution using Apache RewriteMaps was slower than directly serving this content from Zope -- by a factor of 5! The explanation: Apache caches only single successful lookups in the map file. Each access to a key not in the cache results in parsing the map file until the key is found -- the whole map file when it does not contain the key. For larger map files, this has desastrous effects. Now, we use RewriteMaps only when the map files are very small (not more than some dozen entries). ... maps with the same name ... The problem is the cache treats both the file as the same since they have the same name! Hence if you visit foo.com first, the data stored in foomap.txt is returned from the cache next time you visit foo.com or bar.com. This causes apache to redirect into the wrong zope site. We did not observe this behaviour... -- Dieter ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
