Re: [Zope-dev] Re: More resilient indexes
On Mon, 2007-09-17 at 19:57 -0400, Tres Seaver wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Roché Compaan wrote: Hi there Are the current indexing solutions that we have in Zope 2 and 3 the best that we can have? Is there any way to improve the amount of concurrent indexing a field or text index can handle. The Zope 3 implementation doesn't look significantly different to the Zope 2 one in that it still uses a BTree for forward and reverse index. I have a basic stress test where 10 concurrent threads index 10 random words in a FieldIndex and get too many conflict errors to even consider this a usable scalable indexing solution. I don't really want to index objects in another backend. I really would like to make the ZODB work for me here. Are there solutions here? Can one employ some of the QueueCatalog conflict resolution strategies to make indexes more resilient? Or should one use some locking strategy instead? Check out QueueCatalog, which batches up indexing changes for processing within a single thread. svn://svn.zope.org/repos/main/Products.QueueCatalog I use QueueCatalog often and I know how it works. But if an application requires immediate indexing then QueueCatalog is not a solution. Sorry if I was unclear but what I'm really asking is if it is possible to improve the conflict handling of the current indexes that we have in Zope. I am also asking if concurrent indexing in the ZODB is a realistic goal. The reason I mentioned QueueCatalog is not because of its batch indexing in one thread, but because it has a lot of conflict handling on the queue itself. Sessions might be another good example of a product that tries hard to handle conflicts. Do these products have strategies that can be made to work for indexes? -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Zope Tests: 5 OK
Summary of messages to the zope-tests list. Period Mon Sep 17 12:00:00 2007 UTC to Tue Sep 18 12:00:00 2007 UTC. There were 5 messages: 5 from Zope Unit Tests. Tests passed OK --- Subject: OK : Zope-2.7 Python-2.3.6 : Linux From: Zope Unit Tests Date: Mon Sep 17 20:53:49 EDT 2007 URL: http://mail.zope.org/pipermail/zope-tests/2007-September/008356.html Subject: OK : Zope-2.8 Python-2.3.6 : Linux From: Zope Unit Tests Date: Mon Sep 17 20:55:39 EDT 2007 URL: http://mail.zope.org/pipermail/zope-tests/2007-September/008357.html Subject: OK : Zope-2.9 Python-2.4.4 : Linux From: Zope Unit Tests Date: Mon Sep 17 20:57:09 EDT 2007 URL: http://mail.zope.org/pipermail/zope-tests/2007-September/008358.html Subject: OK : Zope-2.10 Python-2.4.4 : Linux From: Zope Unit Tests Date: Mon Sep 17 20:58:43 EDT 2007 URL: http://mail.zope.org/pipermail/zope-tests/2007-September/008359.html Subject: OK : Zope-trunk Python-2.4.4 : Linux From: Zope Unit Tests Date: Mon Sep 17 21:00:18 EDT 2007 URL: http://mail.zope.org/pipermail/zope-tests/2007-September/008360.html ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: Fwd: [Zope-dev] Options replacing DateTime with datetime!?
Hey again Lennart,
OK, I'm just going after my old notes here, and they could be wrong.
It could be that these types of timezones doesn't work in a datetime string?
Maybe. If there's a problem with parsing, I'll be happy to fix that.
The important part is that there is some sort of way to tell the
module what the local timezone is, so that you can test conversions.
Just use the POSIX-defined TZ variable, and the gettz() method
to retrieve the timezone.
from dateutil.tz import gettz
os.environ[TZ] = Brazil/East
gettz()
tzfile('/usr/share/zoneinfo/Brazil/East')
os.environ[TZ] = US/Eastern
gettz()
tzfile('/usr/share/zoneinfo/US/Eastern')
(...)
But it makes it impossible without actually modifying the modules code
somehow, and in my book, it is reasonable to call that impossible.
Just use gettz(), as explained above. You'll get a richer source
of information for free.
Of course nothing is impossible in computing.
(...)
If I monkey-patch the module for testing, then am I really testing
what goes on in production?
These are interesting statements. I won't move into discussing them
because it won't benefit the main point. If you'd enjoy some general
conceptual discussion we can do so privately.
--
Gustavo Niemeyer
http://niemeyer.net
___
Zope-Dev maillist - Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: More resilient indexes
Roché Compaan wrote at 2007-9-18 08:55 +0200: ... Sorry if I was unclear but what I'm really asking is if it is possible to improve the conflict handling of the current indexes that we have in Zope. I am also asking if concurrent indexing in the ZODB is a realistic goal. I have implemented Conflict Reduced Indexes. They essentially work as follows: Standard indexes use for efficiency reasons a complex dance with a quite high conflict potential: The document list for a term can have 3 implementation missing, represented as an integer, represented as an IITreeSet. Whenever the implentation type changes, a conflict will occur when a concurrent request accesses the same document list. The conflict reduced indexes use only 2 implementation types: missing and IITreeSet and once, the list used an IITreeSet, it remains this way. This can leverage the conflict resolution build in OOBTree and IITreeSet quite well. Nevertheless, it turned out that these separate indexes were not worth the efford (meanwhile, they have been replaced by ManagableIndex). Keep in mind, that the conflict behaviour improves when your have lots of indexed data because the modifications then spread over a large tree, significantly reducing the conflict probability. -- Dieter ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Mixing recipes (zc.recipe.cmmi reuse)
Hi there, Not sure this is the right list, but let's give it a try. I would like to use the 'patches' functionality from zc.recipe.cmmi together with other recipes. I believe this is useful functionality and is interesting to all sorts of recipes, not only to cmmi-based ones. So the question is, does the zc.buildout architecture support reusing options from a recipe on other recipes? Or would it require the recipe writer to explicitly 'subclass' (?) cmmi to get patch functionality? Or even, would a 'post-fetch'/'pre-build' generalization be desired, 'patch' being one such application? -- Sidnei da Silva Enfold Systemshttp://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214 ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: Mixing recipes (zc.recipe.cmmi reuse)
Sidnei da Silva wrote: Hi there, Not sure this is the right list, but let's give it a try. I would like to use the 'patches' functionality from zc.recipe.cmmi together with other recipes. I believe this is useful functionality and is interesting to all sorts of recipes, not only to cmmi-based ones. So the question is, does the zc.buildout architecture support reusing options from a recipe on other recipes? Or would it require the recipe writer to explicitly 'subclass' (?) cmmi to get patch functionality? Or even, would a 'post-fetch'/'pre-build' generalization be desired, 'patch' being one such application? I've seen subclassing, or explicit instantiation, e.g. instantiate the dependent recipe class as an instance variable and just call its methods. I don't think you really need any magic from buildout here. Recipes are pretty simple Python classes. Just make sure you pass the options etc from __init__.py and you should be fine. I think plone.recipe.plone uses plone.recipe.distro and plone.recipe.egg in various ways. You may want to look at that. Martin -- Acquisition is a jealous mistress ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope] Two ZEO instances running on same server?
I've got one ZEO instance of Plone 2.5.0 and one ZEO instance of Plone 2.5.3running on one UNIX server. These were both installed using the unified installer for UNIX. I've modified the ZEO ports and also all the ZOPE ports so that there a no conflicts between the two instances. When I access the ZMI in both instances I am seeing Plone sites mirrored between the two instances. Even though the data.fs file for both sit in different areas opt/Plone-2.5/zeocluster/ opt/Plone-2.5.3/zeocluster. What I get is site names etc displayed which haven't been created in that instance and vice versa. When clicking on the site, no actual data is displayed and also skins aren't correct. It also looks like the /Products directories are mirrored in ZOPE. However if I go into site set-up in Plone and add/remove products the correct products are listed for each instance. -- michael ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Two ZEO instances running on same server?
On 9/18/07, michael nt milne [EMAIL PROTECTED] wrote: What I get is site names etc displayed which haven't been created in that instance and vice versa. When clicking on the site, no actual data is displayed and also skins aren't correct. It also looks like the /Products directories are mirrored in ZOPE. However if I go into site set-up in Plone and add/remove products the correct products are listed for each instance. Sounds like you'll need to double-check your ports, you have got some wires crossed somewhere. You can doublecheck this by looking at the Data.fs sizes and modification times in both ZEO server instances, I suspect that you have both your Zope clusters write to one Data.fs only. -- Martijn Pieters ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Two ZEO instances running on same server?
- Original Message - From: michael nt milne [EMAIL PROTECTED] Subject: Re: [Zope] Two ZEO instances running on same server? The data.fs directories are both different though. It's only things like site names and product listings that are being mirrored. Not actual site content... As Michael mentioned, this sounds like you have some ports (or ZODB mount points) set incorrectly. To confirm: 1- examine both Data.fs files and note the sizes and last modification times 2- make a change (that will affect the ZODB) in only one of the Zope instances 3- examine both Data.fs files. Did the file size/modification time change in the expected Data.fs file and not change in the 'other' Data.fs file? Jonathan On 9/18/07, Martijn Pieters [EMAIL PROTECTED] wrote: On 9/18/07, michael nt milne [EMAIL PROTECTED] wrote: What I get is site names etc displayed which haven't been created in that instance and vice versa. When clicking on the site, no actual data is displayed and also skins aren't correct. It also looks like the /Products directories are mirrored in ZOPE. However if I go into site set-up in Plone and add/remove products the correct products are listed for each instance. Sounds like you'll need to double-check your ports, you have got some wires crossed somewhere. You can doublecheck this by looking at the Data.fs sizes and modification times in both ZEO server instances, I suspect that you have both your Zope clusters write to one Data.fs only. -- Martijn Pieters -- michael ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev ) No virus found in this incoming message. Checked by AVG Free Edition. Version: 7.5.487 / Virus Database: 269.13.21/1012 - Release Date: 9/16/2007 6:32 PM ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Two ZEO instances running on same server?
Hi The data.fs directories are both different though. It's only things like site names and product listings that are being mirrored. Not actual site content... On 9/18/07, Martijn Pieters [EMAIL PROTECTED] wrote: On 9/18/07, michael nt milne [EMAIL PROTECTED] wrote: What I get is site names etc displayed which haven't been created in that instance and vice versa. When clicking on the site, no actual data is displayed and also skins aren't correct. It also looks like the /Products directories are mirrored in ZOPE. However if I go into site set-up in Plone and add/remove products the correct products are listed for each instance. Sounds like you'll need to double-check your ports, you have got some wires crossed somewhere. You can doublecheck this by looking at the Data.fs sizes and modification times in both ZEO server instances, I suspect that you have both your Zope clusters write to one Data.fs only. -- Martijn Pieters -- michael ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Two ZEO instances running on same server?
thanks for the help. Will carry out what you are saying like you have some ports (or ZODB mount points) set incorrectly. ports all not conflicting will check mountpoints..is this specified in the zope.conf? On 9/18/07, Jonathan [EMAIL PROTECTED] wrote: - Original Message - From: michael nt milne [EMAIL PROTECTED] Subject: Re: [Zope] Two ZEO instances running on same server? The data.fs directories are both different though. It's only things like site names and product listings that are being mirrored. Not actual site content... As Michael mentioned, this sounds like you have some ports (or ZODB mount points) set incorrectly. To confirm: 1- examine both Data.fs files and note the sizes and last modification times 2- make a change (that will affect the ZODB) in only one of the Zope instances 3- examine both Data.fs files. Did the file size/modification time change in the expected Data.fs file and not change in the 'other' Data.fs file? Jonathan On 9/18/07, Martijn Pieters [EMAIL PROTECTED] wrote: On 9/18/07, michael nt milne [EMAIL PROTECTED] wrote: What I get is site names etc displayed which haven't been created in that instance and vice versa. When clicking on the site, no actual data is displayed and also skins aren't correct. It also looks like the /Products directories are mirrored in ZOPE. However if I go into site set-up in Plone and add/remove products the correct products are listed for each instance. Sounds like you'll need to double-check your ports, you have got some wires crossed somewhere. You can doublecheck this by looking at the Data.fs sizes and modification times in both ZEO server instances, I suspect that you have both your Zope clusters write to one Data.fs only. -- Martijn Pieters -- michael ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev ) No virus found in this incoming message. Checked by AVG Free Edition. Version: 7.5.487 / Virus Database: 269.13.21/1012 - Release Date: 9/16/2007 6:32 PM ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev ) -- michael ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Two ZEO instances running on same server?
- Original Message - From: michael nt milne [EMAIL PROTECTED] To: zope list user zope@zope.org Sent: Tuesday, September 18, 2007 9:25 AM Subject: Re: [Zope] Two ZEO instances running on same server? thanks for the help. Will carry out what you are saying like you have some ports (or ZODB mount points) set incorrectly. ports all not conflicting will check mountpoints..is this specified in the zope.conf? Yes. Also viewable via the ZMI (in the 'Add' drop-down list select 'ZODB Mount Point'), but less info there. Jonathan ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
