Re: [Zope-dev] http://cvs.zope.org/Products/Scheduler vs Xron for cron-like functionality. Advice?
At 20:40 2002-10-05 +0200, Joachim Werner said: I've noticed that Xcron uses ZPublishers client to trigger events. This seems a bit out-dated to me, wouldn't it be better to use restrictedTraverse? After looking at Xron in more detail and comparing it to the Scheduler stuff in the Zope CVS, I guess that going the Scheduler way is probably more promising in the long run. Xron seems to have quite a few major design flaws. One I haven't mentioned yet is that you can only trigger DTML methods. In a modern Zope installation, you'd probably want to call Scripts (Python) more often, though of course you can call everything FROM DTML .. Yes. I also see at least one pre-built events like a DBPackEvent and possibly more. Events may need to be restricted to only be created in the Scehduler. Not sure that we need this one. Just an idea that pop by :-) Regards, Johan Carlsson -- Torped Strategi och Kommunikation AB Johan Carlsson [EMAIL PROTECTED] Mail: Birkagatan 9 SE-113 36 Stockholm Sweden Visit: Västmannagatan 67, Stockholm, Sweden Phone +46-(0)8-32 31 23 Fax +46-(0)8-32 31 83 Mobil +46-(0)70-558 25 24 http://www.torped.se http://www.easypublisher.com ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] http://cvs.zope.org/Products/Scheduler vs Xron for cron-like functionality. Advice?
The Xron product release is almost a year old, but Scheduler seems to be only available via CVS. Can anyone comment as to the maturity/stability of these products? Any advice would be much appreciated. I can't vouch for either of the products you mentioned, but I've had good luck with cron on linux teamed with wget :-) There doesn't seem to be much support in Zope for use cases where an action is not event driven, ie not a request/response pair. *** Vaporware Follows*** What *I* want, (but haven't written yet) is a product that can do continuous low level scraping of legacy data sources and feed this data into an RDBMS or the ZODB. A kind of helper daemon. How each instance is scheduled would be one of many adjustable per instance config options :-) However, assuming that I do get around to building this product, I don't know where on the scale between ugly hack and elegant, reusable solution it will fall. This will also determine whether you ever see it on zope.org :-) Adam Hi All, I'm interested in starting to maintain Xron, if it has potential to be a stable products. From the code it doesn't seem to do any strange things, but I would like to know if anybody has experience of using it in a production environment, or any other experiences and that would recommend not using it in a production environment. The central feature I'm interested in is the ability to setup packing policies inside Zope. I know many people uses Cron or Scheduler but I like to have one platform independent solution and preferably inside the Zope process. Regards, Johan Carlsson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] http://cvs.zope.org/Products/Scheduler vs Xron for cron-like functionality. Advice?
At 11:32 2002-10-05 +0200, Joachim Werner said: Hi All, I'm interested in starting to maintain Xron, if it has potential to be a stable products. From the code it doesn't seem to do any strange things, but I would like to know if anybody has experience of using it in a production environment, or any other experiences and that would recommend not using it in a production environment. We are using it, but there seem to be some problems. One of them is that I frequently have cases where Xron doesn't reschedule properly (I am using the improved user interface stuff for Xron, so the bug could also be in there). What happens is that an event that is scheduled for daily execution at midnight will not be rescheduled for the next day after it was executed, but reset to the year 1970, which actually means it is switched off. The other problem is more obvious, but still it is a major issue: Xron seems to be incompatible with ZEO at the moment. The reason why is that the Xron process starts on every ZEO client machine, so everything is executed more than once. I could think of two policies to get around that, both of which should be selectable as an option on a per-event base: - Xron just runs on one server, e.g. the one that is on the fastest machine; this is useful for cases where you need the events to be executed on the same machine all the time, e.g. if you want to write stuff to the server's local file system Seems like the most sensible to me. This should be possible to do by splitting the product in two? The Dispatcher and the Scheduler. The Scheduler gets installed on both server and clients but the Dispatcher only gets installed on the server that should run the triggers. I also would like to have a Dispatch monitor ttw where the thread can be start and stopped and the log could be read. Possibly having multiple Dispatcher (for Virtual Hosting situations), which creates another problem because the Dispatcher is started during product instanciation there must be a registry of Dispatchers or someway to start the distributed Dispatcher. For example if the server is restarted. One potential problem would also be notification of failure. I belive I've seen a zLOG email notification product what might be useful. - Xron runs on all servers (i.e. ZEO clients), but on a first come first serve base, i.e. the server that executes the event first blocks the others from doing so, too. This would need some kind of inter-process locking, which I think should be provided by ZEO. I'm presently not aware of any such services in ZEO? Another issue is that Xron will use the current virtual host settings when it executes and reschedules an event. That means that the entries in the Xron Schedule ZCatalog will have different URLs. In some cases the URL that is used to execute an event could be important. E.g., we use Apache with some tricky rewrite rules in front of Zope, and to get around the Apache server the Zope server has to be called from a different URL. I've noticed that Xcron uses ZPublishers client to trigger events. This seems a bit out-dated to me, wouldn't it be better to use restrictedTraverse? Also all events must be given by it's physicalPath, something we learned the hard way. VirtualPath may be translated to PhysicalPaths when needed, something I think we added to our version of VHM, which we would be happy to share. Who should the events be run as? To day the are run as Anonymous even though there are arguments for passing user name and password I can find any in the UI. I think Event should be run as Owner or ProxyRole. But I'm not sure? I am willing to help with maintaining Xron, especially the part concerning ZEO, because we really need it ... Joachim Excellent! :-) Best Regards, Johan Carlsson -- Torped Strategi och Kommunikation AB Johan Carlsson [EMAIL PROTECTED] Mail: Birkagatan 9 SE-113 36 Stockholm Sweden Visit: Västmannagatan 67, Stockholm, Sweden Phone +46-(0)8-32 31 23 Fax +46-(0)8-32 31 83 Mobil +46-(0)70-558 25 24 http://www.torped.se http://www.easypublisher.com ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] http://cvs.zope.org/Products/Scheduler vs Xron for cron-like functionality. Advice?
At 10:30 2002-10-05 +0200, Jean Jordaan wrote: From the code it doesn't seem to do any strange things, but I would like to know if anybody has experience of using it in a production environment, We have seen the following happen a couple of times: we have Reminders, subclassed from XronDTMLMethod. When a Reminder triggers, it sends email (using MailHost). When this fails, Xron goes into a spin, the Reminder retriggering forever, bloating Data.fs with a Gb or more overnight. We've started looking at the code a couple of times, but everything looked fine .. Can't see the reason, yet anyway. But Client.call of the RPC will always return something correct, even if the method that is being called fails? Using Traversal instead, if the call would the Dispatcher would know about it. A when I should be able to stop the event from triggering, and potentially send an e-mail to the admin (which must not fail ;-). I'm trying replacing RPC with restrictedTraverse right now. I'm not sure if I have a security manage in the context of Dispatcher. But it works. Some thoughts: Now it seems like XcronMethods rely on REQUEST to change timer, because it needs it to be able to override for ZClasses. I don't have a REQUEST object, because I'm calling from the Dispatcher thread. This is how I think I'm gone solve it. Split trigger in two separate methods: trigger and reschedule. This also means if trigger raises an exception, reschedule will not be called preventing trigger from being called and fail again. It will still be allowed to override trigger, but reschedule will not be allowed. I think there should be a very narrow bit of code that actually can change the settings of an event. I got a felling there maybe potential security and stability problems here. And I'm very concerned about the ZCatalog being safe here. Is it possible to spoof a catalogObject call from un-secure ttw-code? Also I'm think the trigger time should be changed to a start time and a recurrent setting. The next event should always be calculated from these setting, never be written. Which mean an event could be a read only operation, avoiding unnecessary ZODB bloating. The recurrent setting should be iCal compatible to be able to use Xron for managing triggers in a future Calender server, but this is current not a prioritized feature for me. I think it will happen though in the future. Also, a bit of a contradiction to the read-only events above, I think there is allot of unnecessary blather to the zLOG. I would rather let the Event keep an history over not critical events. It could be optional, I still want the Event to keep a volatile history log to prevent it from being called several times for the same occurrence. This wouldn't be perfect because volatile attributes are thread specific but it could prevent the object from going stall. Best Regards, Johan Carlsson -- Torped Strategi och Kommunikation AB Johan Carlsson [EMAIL PROTECTED] Mail: Birkagatan 9 SE-113 36 Stockholm Sweden Visit: Västmannagatan 67, Stockholm, Sweden Phone +46-(0)8-32 31 23 Fax +46-(0)8-32 31 83 Mobil +46-(0)70-558 25 24 http://www.torped.se http://www.easypublisher.com ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] __getattr__ and Acquisition
At 16:29 2002-09-23 +, Florent Guillaume said: You didn't look a the code I pointed to you. Look at CMFCore/Skinnable to be precise (not everything is to be used of course). * In __of__ you store the parent in a volatile attribute, * in __getattr__ you lookup the volatile attribute This works as long as your object is not used in several different acquisition contexts. Hi Florent, Would you care to elaborate on that last sentence? :-) Could you give an example when it wouldn't work? Best Regards, Johan Carlsson -- Torped Strategi och Kommunikation AB Johan Carlsson [EMAIL PROTECTED] Mail: Birkagatan 9 SE-113 36 Stockholm Sweden Visit: Västmannagatan 67, Stockholm, Sweden Phone +46-(0)8-32 31 23 Fax +46-(0)8-32 31 83 Mobil +46-(0)70-558 25 24 http://www.torped.se http://www.easypublisher.com ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] WebDAV Redirects or overriding IDs duing creation
Hi gurus, I'll try this question on this list as well (a part form [EMAIL PROTECTED]). Is it possible to make a redirect during a WebDAV request. For instance I have a PROPFIND one an object that has been renamed and I whan the PROPFIND to access the object with the new ID. I have tried to return a prop result with the information from the renamed object, but the client seems to ignore that (no kidding its MS Web Folder, so what did I expect ;-) Is there a correct way to do this. What I want to do is for the server to override the requested ID on any object or collection that is being created with something more appropriate like the created objects/collections meta_type. MS WebFolders defaults to New Folder for collections. Best Regards, Johan Carlsson -- Torped Strategi och Kommunikation AB Johan Carlsson [EMAIL PROTECTED] Mail: Birkagatan 9 SE-113 36 Stockholm Sweden Visit: Västmannagatan 67, Stockholm, Sweden Phone +46-(0)8-32 31 23 Fax +46-(0)8-32 31 83 Mobil +46-(0)70-558 25 24 http://www.torped.se http://www.easypublisher.com ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Proxy Object / __getattr__ / Acquisition
At 14:28 2002-08-29 -0700, [EMAIL PROTECTED] said:
I am trying to implement a proxy class (specifically for the purposes of
multi-versioned document objects (folderish proxies that contain the object
Hi Sean,
We've been stealing some code from CMFCore.Skinnable to
do similar things (multi-language objects):
Skinnable gets the skins when the objects is being wrapped (e.g.
in the objects __of__ method) and overrides __getattr__ which
uses the skins or falls back to an unbound superGetAttr method
that points to the inherited __getattr__.
(The superGetAttr have pussled us because it seams like superGetAttr is None,
but everything still work as expected.)
We replace the skin with a VeryTinyDataWrapper which (of course) implements
Acquisition.Implicit.
Would this work better that the way your doing it?
We currently implementing this so I don't know for sure that it
is faster, but it's from CMF (a key part to) so it shouldn't be to slow or?
def setupCurrentLanguageData(self):
#the request part is a rest from Skinnable, I don't think it can
be removed.
request=self.request
#replace this with anything that returns
lang_code=self.EasyLanguageService.getCurrentLanguage()
ob = self._current_language.get(lang_code, VerySmallData())
self._v_c_language = (request, ob, {})
def __getattr__(self, name, marker=None):
# OK, see if we can find the language service:
if not name.startswith('_') and not name.startswith('aq_'):
cl = self._v_c_language
if cl is not None:
request, ob, ignore = cl
if not ignore.has_key(name):
subob = getattr(ob, name, _marker)
if subob is not _marker:
# Return it in context of self, forgetting
# its location and acting as if it were located
# in self.
return aq_base(subob)
else:
ignore[name] = 1
if superGetAttr is None:
raise AttributeError, name
return superGetAttr(self, name)
def __of__(self, parent):
'''
Sneakily sets up the current language then returns the wrapper
that Acquisition.Implicit.__of__() would return.
'''
w_self = ImplicitAcquisitionWrapper(aq_base(self), parent)
try:
w_self.setupCurrentLanguageData()
except:
# This shouldn't happen, even if the requested current language
# does not exist.
import sys
from zLOG import LOG, ERROR
LOG('CMFCore', ERROR, 'Unable to setupCurrentLanguageData()',
error=sys.exc_info())
return w_self
Regards,
Johan Carlsson
--
Torped Strategi och Kommunikation AB
Johan Carlsson
[EMAIL PROTECTED]
Mail:
Birkagatan 9
SE-113 36 Stockholm
Sweden
Visit:
Västmannagatan 67, Stockholm, Sweden
Phone +46-(0)8-32 31 23
Fax +46-(0)8-32 31 83
Mobil +46-(0)70-558 25 24
http://www.torped.se
http://www.easypublisher.com
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] to __of__ or not to __of__ ?
At 23:38 2002-08-14 -0400, Casey Duncan said:
The Item class mixes in Traversable, which is probably responsible for
making
this work TTW. It also gives you a bunch of other stuff that many Zope
classes need, like DAV support, copy support, ZMI tab support and security.
It doen't mix-in acquisition tho.
If you need support for acquistion machinery (including __of__ which creates
wrappers), then you need to mix-in Acquisition.Implicit. The SimpleItem
class
mixes this and Persistence in for you (along with Item), so many Zope
classes
derive form SimpleItem. You need Persistence to store instances of your
class
in the ZODB, and implicit acquisition is the standard for Zope objects.
In short I would recommend mixing in SimpleItem or you may find that your
objects are not stored properly in the ZODB.
-Casey
You also might want to wrap the object before returning them,
because the are stored in _item and not in the AixtraTableProduct.
def getItem(self,key): #{{{
returns item
data = self._items
if data.has_key(key):
item = self._items[key]
if not hasattr(item,'key'):
setattr(item,'key',key)
return item.__of__(self) --- wrap the item before returned
Regards,
Johan Carlsson
--
Torped Strategi och Kommunikation AB
Johan Carlsson
[EMAIL PROTECTED]
Mail:
Birkagatan 9
SE-113 36 Stockholm
Sweden
Visit:
Västmannagatan 67, Stockholm, Sweden
Phone +46-(0)8-32 31 23
Fax +46-(0)8-32 31 83
Mobil +46-(0)70-558 25 24
http://www.torped.se
http://www.easypublisher.com
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] PythonWin/Textpad, external import of Zope module and Persistence
At 10:54 2002-08-15 +0100, Chris Withers wrote: Johan Carlsson [Torped] wrote: I'm trying to setup to run with either TextPad, which can run external program like python scripts and then parse the result and making Traceback lines linked to the line/file where the error occurred. No sure exactly what you're trying to do here, can you privde more detail? Well, I solved the problem another way, when I realized that I could parse the Stupid Log File and make traceback's clickable in that file: I added a small python script that only gives me the latest stuff in the SLF, and it works quite nice. Now when I get an error I just press CTRL-1 inside Textpad and I get the traceback, click on the error rows and it will automatically open the correct file and scroll to the row where the error occured. For anyone using TextPad on Windows this should quite a time saver. (I know that I forgot to implement a check for if the SLF has been reseted, something my Zope start-up script does everytime I restart the server. http://www.zope.org/Members/johanc/Tips_and_HowTos/textpad_tooltips_howto It's almost like having a RAD tool for Zope. Well not really, but it's an improvement. It should also be possible to make a small script that takes a input file with REQUEST information and then make the HTTP request to the development server and return any traceback messages from the reply to TextPad. (SLF only shows error messages when a product is refreshed, this way it should be possible to retrieve traceback from the running code. (Interactive debugging has to wait, but in Windows it would be terrific to use PythonWin for the task. But starting Zope from within PythonWin usually has the effect of PythonWin hangling. I might be because I have a really slow laptop :-) Or try using Python which has an build in debugger which I like to be able to use. The first problem I encounter in both scenarios is that when a file import Globals which in its turn from Persistence import Persistent, this fails. Mostly because Persistence is hot patched by ZODB. Is there any smart way to setup a virtual Zope environment without importing Zope (which mounts the Data.fs) or by importing Zope by in a light way without mounting a storage or just mounting a lights storage. 'import Zope' is the only sane thing I can think you'd want to do. Zope has lots of magic that happens when you do that, and unless you do that, things won't behave as you expect. You will need to mount the ZODB, but if you use ZEO then this isn't a problem as your webserver can mount the same storage. I think I have a look at how unit test does this, when I have the time. It would probably solve my problems. Running UnitTest from within TextPad or PythonWin would be a nice way to develop thing in. (Unfortunately I'm not up in speed with writing UnitTests) Thanks. Johan cheers, Chris -- Torped Strategi och Kommunikation AB Johan Carlsson [EMAIL PROTECTED] Mail: Birkagatan 9 SE-113 36 Stockholm Sweden Visit: Västmannagatan 67, Stockholm, Sweden Phone +46-(0)8-32 31 23 Fax +46-(0)8-32 31 83 Mobil +46-(0)70-558 25 24 http://www.torped.se http://www.easypublisher.com ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] PythonWin/Textpad, external import of Zope module andPersistence
Hi, I'm trying to setup to run with either TextPad, which can run external program like python scripts and then parse the result and making Traceback lines linked to the line/file where the error occurred. Or try using Python which has an build in debugger which I like to be able to use. The first problem I encounter in both scenarios is that when a file import Globals which in its turn from Persistence import Persistent, this fails. Mostly because Persistence is hot patched by ZODB. Is there any smart way to setup a virtual Zope environment without importing Zope (which mounts the Data.fs) or by importing Zope by in a light way without mounting a storage or just mounting a lights storage. The only thing I want to do in this first phase is to be able to make syntax/sanity checks. Regards, Johan Carlsson -- Torped Strategi och Kommunikation AB Johan Carlsson [EMAIL PROTECTED] Mail: Birkagatan 9 SE-113 36 Stockholm Sweden Visit: Västmannagatan 67, Stockholm, Sweden Phone +46-(0)8-32 31 23 Fax +46-(0)8-32 31 83 Mobil +46-(0)70-558 25 24 http://www.torped.se http://www.easypublisher.com ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] PythonWin/Textpad, external import of Zope module and Persistence
The only thing I want to do in this first phase is to be able to make syntax/sanity checks. I use something like this for writing and testing external methods, having TextPad run the same python zope uses: Thanks for you tips. Here's another one I just published on the Zope.org. Textpad tool for parsing Stupid Log files to make traceback print outs clickable Additional details: TextPad from Helios Software Solutions (http://textpad.com) is an excellent tool for working with Python and Zope-files. Besides support for text coloring of common Python and Zope syntaxes it is possible to setup Tools that run a program or a script. The output from such a execution can be conneced to a RegExp that makes lines hyperlinks to Files and Lines according to the output. http://www.zope.org/Memebers http://www.zope.org/Members/johanc/Tips_and_HowTos/textpad_tooltips_howto Regards, Johan Carlsson -- Torped Strategi och Kommunikation AB Johan Carlsson [EMAIL PROTECTED] Mail: Birkagatan 9 SE-113 36 Stockholm Sweden Visit: Västmannagatan 67, Stockholm, Sweden Phone +46-(0)8-32 31 23 Fax +46-(0)8-32 31 83 Mobil +46-(0)70-558 25 24 http://www.torped.se http://www.easypublisher.com ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] __record_schema__ of Brains (Was: Record.pyd)
At 21:28 2002-08-10 -0400, Casey Duncan said:
On Saturday 10 August 2002 11:25 am, Johan Carlsson [Torped] wrote:
Now that I understand how the data tuples are copied to the brain
I'm not at all sure adding a filter when copying the tuple will optimize
thing, because of the overhead in the filter process.
This occurs lazily so the savings would be heavily dependant on the
application. For most web apps presenting small batches of records, the
savings in limiting columns returned would be pretty minimal.
But there must be some though implementing Record.pyd i C, but off course
I suppose Record.pyd was first used for ZSQL?
An easy filter would be to let __record_schema__ control which columns to
save, as it works to day __record_schema__ must point on a sequence starting
with 0, so I can't specify indexes into the tuple like this:
__record_schema__= {'hey':12, 'dude': 22}
Maybe this is easy to change in the record.pyd, or I just implement it
in a special brain base class?
After revisited Record.c I realized that the tuple from the catalogs self.data
is stored as a tuple (or as a C-array I suppose?) in a Record or as attributes
depending on what you provide to the constructor.
I suppose coping data to a C-array is much faster than creating
attributes on each brain, but if the array is large and the number attributes
needed to be set is small it might be the other way around.
I have no idea where they would break even.
Maybe I just will settle with having two different brain base classes and use
one that suits the current need.
The general usage is to put a minimal set of columns in metadata, only enough
to create a results page and load the objects in cases where either large,
dynamic or otherwise arbitrary data elements are needed.
Yes, and that is somewhat restricting.
My current applications use several different catalogs to get the
width of the meta_data down. The downside of this approach is
that I end up with allot of catalogs and that it's a multitude time more
things to do for management, e.g. I must reindex all catalogs instead
of just one.
My primary goals are:
1. Get a general ZCatalog that can be used for all ZCatalog requirement
(not only site searches),
2. Implement feature that removes the need for external RDBS (for instance
report generation is hard with ZCatalogs because of the lack of
grouping/statistics).
3. Make ZCatalogs easier to manage, for instance the need of updating
indexes and meta_data
definitions every time you change your applications data structure is
annoying, especially at
development time. Objects could tell the ZCatalog which meta_data and
indexes it wants removing
the need to manually add them. Off course you will need to clean up the
ZCatalog from time to time.
(The way that I solved the group/calc part of my project, I don't think
it will lead to memory bloat. I'm going to implement a LacyGroupMap
which take an extra parameter (a list of IISet). Each brain created
in the LacyMap will have methods for calculations directly on the self.data
in the Catalog. The data it self will not be stored.
There will most probably be a pre calculate method that calculate all
variables that are applicable and caches the result.)
Sounds like a pretty good solution. However, I would be hesitant in creating
direct dependancies on the internal Catalog data structures if you can help
it (sometimes you can't though).
I could soften the dependency by providing the catalog with an interface for
calculations and give the brain an reference to the catalog it self and
use the interface on that reference.
One way to reduce memory consumption in wide Catalogs would be
to have LacyBrains (vertical lacyness, there might be reasons
why that would be a bad idea, which I'm not aware of)
That would pretty much require a rewrite of the Catalog as the data
structures
would need to be completely different. It would introduce significant
database overhead since each metadata field would need to be loaded
individually. I think that would negate whatever performance benefit metadata
might have over simply loading the objects.
I'm not sure that it would be necessary to change the data structure, the
brain could
use the same method as the LacyMap uses to load the data.
But LacyBrain would need to save all applicable data at once to be efficient.
The different would be that the brain will not fetch any data before the first
attribute has been called. When the first is called all applicable data will
be copied to the attribute according to __record_schema__.
This would probably not be more efficient for regular use of brains, but for
calculated group brains they wouldn't need to store the data at all if
they only used calculated fields.
Another way would be to have multiple data attributes in the Catalog, like
tables, and to join the tuples from them with a from table1, table2
statement.
In this way it would be possible to control the width
Re: [Zope-dev] __record_schema__ of Brains (Was: Record.pyd)
At 08:59 2002-08-09 -0400, Casey Duncan said: __record_schema__ is simply a dictionary which maps field names to column positions (ints) so that the record knows the index of each field in the record tuples. See line 154 of Catalog.py to see how it is initialized to the Metadata schema plus a few extra columns for catalog rid and scores. Hi Casey (and zope-dev), Thanks! After some experimenting I realized that :-) One of the reasons I was because I am thinking about how to implement a SELECT col1 as 'name', ... type of feature for ZCatalogs. I'm not entirely sure it's an good idea to start with, but I'm thinking in the line of large ZCatalogs (by large I mean allot of columns in the self.data structure). If all columns are copied the brains would grow larger as well and by selecting explicitly which columns should be copied to the brain they would be lighter. Now that I understand how the data tuples are copied to the brain I'm not at all sure adding a filter when copying the tuple will optimize thing, because of the overhead in the filter process. (The way that I solved the group/calc part of my project, I don't think it will lead to memory bloat. I'm going to implement a LacyGroupMap which take an extra parameter (a list of IISet). Each brain created in the LacyMap will have methods for calculations directly on the self.data in the Catalog. The data it self will not be stored. There will most probably be a pre calculate method that calculate all variables that are applicable and caches the result.) One way to reduce memory consumption in wide Catalogs would be to have LacyBrains (vertical lacyness, there might be reasons why that would be a bad idea, which I'm not aware of) Another way would be to have multiple data attributes in the Catalog, like tables, and to join the tuples from them with a from table1, table2 statement. In this way it would be possible to control the width of the brains. It would also be possible for the object indexing it self to tell the catalog in which tables it should store meta data. There have been some proposals (ObjectHub et al) which I read some time ago. I didn't feel then that we what I was looking for. Please tell me if there's been any proposals or discussions regarding this. Regards, Johan Carlsson -- Torped Strategi och Kommunikation AB Johan Carlsson [EMAIL PROTECTED] Mail: Birkagatan 9 SE-113 36 Stockholm Sweden Visit: Västmannagatan 67, Stockholm, Sweden Phone +46-(0)8-32 31 23 Fax +46-(0)8-32 31 83 Mobil +46-(0)70-558 25 24 http://www.torped.se http://www.easypublisher.com ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] NOT in Field and Keyword Indexes
At 11:04 2002-07-06 +0200, Johan Carlsson [Torped] said:
I intend to release my code on zope.org, but first I though I'd write some
unit tests.
Ok, writing the UnitTest took more time that writing the Index :-)
Here it is any way:
http://www.zope.org/Members/johanc/EasyIndexes
Usage:
Not 'a' in foo:
record = { 'foo' : { 'query' : ['a']
, 'operator' : 'not'
}
}
Not ( 'b' or 'e' ) in foo:
record = { 'foo' : { 'query' : ['b','e']
, 'operator' : 'notor'
}
}
Not ( 'b' and 'e' ) in foo:
record = { 'foo' : { 'query' : ['b','e']
, 'operator' : 'notand'
}
}
--
Torped Strategi och Kommunikation AB
Johan Carlsson
[EMAIL PROTECTED]
Mail:
Birkagatan 9
SE-113 36 Stockholm
Sweden
Visit:
Västmannagatan 67, Stockholm, Sweden
Phone +46-(0)8-32 31 23
Fax +46-(0)8-32 31 83
Mobil +46-(0)70-558 25 24
http://www.torped.se
http://www.easypublisher.com
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] What happend to sessionapi-prog.stx?
What happend to sessionapi-prog.stx? I found it in the http://cvs.zope.org/Products/Sessions/help/Attic/ Is there a replacement? Regards, Johan -- Torped Strategi och Kommunikation AB Johan Carlsson [EMAIL PROTECTED] Mail: Birkagatan 9 SE-113 36 Stockholm Sweden Visit: Västmannagatan 67, Stockholm, Sweden Phone +46-(0)8-32 31 23 Fax +46-(0)8-32 31 83 Mobil +46-(0)70-558 25 24 http://www.torped.se http://easypublisher.torped.se ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] ts_regex in Zope 2.5.1
Hi, It ts_regex that comes with Zope 2.5.1 seems to rely on regex. Is re thread safe? Also, Plugins from ZPatterns uses ts_regex, is there a later version of Plugins that dones use it? Thanks in advance, Johan Carlsson -- Torped Strategi och Kommunikation AB Johan Carlsson -- [EMAIL PROTECTED] Birkagatan 9 -- SE-113 36 Stockholm -- Sweden Phone +46-(0)8-32 31 23 -- Fax +46-(0)8-32 31 83 -- Mobil +46-(0)70-558 25 24 http://www.torped.se -- http://easypublisher.torped.se ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: [Zope] range:min bug in ZCatalogs fo Zope 2.3.2
At 05:45 2002-03-22 -0500, Andreas Jung wrote: BTrees were rewritten in 2.4.0 so I assume you will run into problems. Please check the release notes for the latest 2.3.X version if there were some fixes for your problem since 2.3.2. -aj Problem solved by replacing BTree with Z 2.5.0 version. Thanks, Johan Carlsson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: [Zope] range:min bug in ZCatalogs fo Zope 2.3.2
At 06:08 2002-03-21 -0500, you wrote: Try to reproduce the same problem under 2.5.X. Or upgrade to 2.5.X. Andreas The problem is reproducible in Zope 2.3.2 but doesn't appear in Zope 2.5.0. What I have tested is a OOBTree with DateTime values (or None) as keys and IITreeSet or integers as values (as in the case with FieldIndexes). The OOBTrees items method gets called with one argument (a DateTime) resulting in a list of sets containing items with keys larger than the passed DateTime value. When tested on Zope 2.3.2 some dates result in a empty list. So, upgrading our software to 2.5.0 would take some time and I need a quick fix. Would it be possible to run the BTree code from Zope 2.5.0 in Zope 2.3.2? Best Regards, Johan Carlsson Hi, I have encountered a strange problem with ZCatalogs in Zope 2.3.2. We have objects having DateTime arguments effective_date and expiration_date that gets indexed in different ZCatalogs. (The index is a field index). We then tries to get all published objects that have not yet been archived: now=_.DateTime() Service.searchResults( effective_date =now, effective_date_usage='range:max' , expiration_date =now, expiration_date_usage='range:min' ) Now this works fine, until one day when result searchResults returns an empty result list which is not expected. It seems to be that some days the range:min statement doesn't work correctly. (I tried with different range:* combinations and the only thing I found not working is the range:min statement (some days). If I instead use a different value for now: now2 = _.DateTime(now.Date()) # now.Date() returns somethong like 2002/03/21 which in turn makes the now2 equal to the DateTime value 2002/03/21 00:00:000 GMT+1 (GMT+1 is my local TimeZone) Now using now2 in: Service.searchResults( effective_date =now, effective_date_usage='range:max' , expiration_date =now2, expiration_date_usage='range:min' ) will always result in a list (all though it gets quantized to midnight the resulting list will be somewhat biased). What I don't seem to understand is that DateTime() and DateTime(DateTime().Date) gets so different results. Have anyone seen something like this before? Best Regards, Johan Carlsson -- Johan Carlsson, Torped AB http://www.torped.se -- http://easypublisher.torped.se ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev ) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: [Zope] range:min bug in ZCatalogs fo Zope 2.3.2
At 05:45 2002-03-22 -0500, you wrote: BTrees were rewritten in 2.4.0 so I assume you will run into problems. Please check the release notes for the latest 2.3.X version if there were some fixes for your problem since 2.3.2. -aj The interface seems to be equal with the old one? Comparing the Components/BTree folder there doesn't seem to be an enormous amount of changes. Has it been rewritten or has it only been fixed? Reading the Change files from 2.4.0 and forth indicates 3 bug fixes related to BTree? It seems to be possible to compile the BTree extention module and just replace that. Are there any don't thats I should be where of? Like changes in python extensions between Python 1.5.2 and Python 2.1.2? Regards, Johan Carlsson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] range:min bug in ZCatalogs fo Zope 2.3.2
Hi, I have encountered a strange problem with ZCatalogs in Zope 2.3.2. We have objects having DateTime arguments effective_date and expiration_date that gets indexed in different ZCatalogs. (The index is a field index). We then tries to get all published objects that have not yet been archived: now=_.DateTime() Service.searchResults( effective_date =now, effective_date_usage='range:max' , expiration_date =now, expiration_date_usage='range:min' ) Now this works fine, until one day when result searchResults returns an empty result list which is not expected. It seems to be that some days the range:min statement doesn't work correctly. (I tried with different range:* combinations and the only thing I found not working is the range:min statement (some days). If I instead use a different value for now: now2 = _.DateTime(now.Date()) # now.Date() returns somethong like 2002/03/21 which in turn makes the now2 equal to the DateTime value 2002/03/21 00:00:000 GMT+1 (GMT+1 is my local TimeZone) Now using now2 in: Service.searchResults( effective_date =now, effective_date_usage='range:max' , expiration_date =now2, expiration_date_usage='range:min' ) will always result in a list (all though it gets quantized to midnight the resulting list will be somewhat biased). What I don't seem to understand is that DateTime() and DateTime(DateTime().Date) gets so different results. Have anyone seen something like this before? Best Regards, Johan Carlsson -- Johan Carlsson, Torped AB http://www.torped.se -- http://easypublisher.torped.se ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] The future of ZPatterns and LoginManager
Hi everybody, (I did ask this question to the ZPatterns list, but without any answer. So I hope one better luck here.) I finally got the opportunity to use ZPatterns in a project and am quite please with the result (after the mandatory wrestling :) Anyway, I would like to use both ZPatterns again, as well as LoginManager, but I feel somewhat of a doubt what they will be around in the future and in Zope 3. So, are they? Or what I really want to know, is it safe to base my applications on ZP and/or LM or is it something I'm going to regret? Best Regards, Johan Carlsson torped strategi och kommunikation ab johan carlsson [EMAIL PROTECTED] birkagatan 9 113 36 stockholm västmannagatan 67 http://www.torped.se http://easypublisher.torped.se ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] unrestrictedTraverse unable to access the root object
Hi all,
The collector seems to still be offline?
I just realized that the unrestrictedTraverse returns self
for the empty-string url: ''
I did expect it to return the root object because the empty-string
is equal to the path tuple ('',).
Is there any reason for the current behaviour?
Does anybody know any bad things that might happen if I
change this behaviour?
Best Regards,
Johan Carlsson
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] RFC: Date property requiers valid date (no more)
Johan - I have done a similiar hotfix by using None - which I think is a better alternative than ''. If you want to see all of the source, download the following Product, http://www.zope.org/Members/natsukashi/Products/CMFPropertyCore, and look inside the file CMFPropertyCore/LinkPropertyManager.py Thanks Joseph, Great input. Regards, Johan Carlsson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
