Re: [Zope-dev] Xron fragility
Yes, using Client.py was the easiest way to do that. The biggest problem with Client.py is its dependency on HTTP. If your server is set up to only accept HTTPS, then you can't use Xron -- not a desirable trade-off. If security is the problem, you could configure the HTTP part to listen only on a loopback interface. (yes, this is supported on NT and windows 95 too) We need another mechanism that achieves the same goal -- simulating the environment of a request -- without going all the way back to the socket. I always thought the use of Client was very elegent. The performance hit is negligible, and all the plausible alternatives need many more lines of code. Another advantage is that it promotes testability of the scheduled events since it is easy (trivial) to reproduce the environment in which they run when scheduled. Toby Dickenson [EMAIL PROTECTED] ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Xron fragility
From: "Kapil Thangavelu" [EMAIL PROTECTED] [snip] Xron has a single dispatcher thread. This thread knows how long to sleep for until the next job needs to run. this model has problems when new jobs (for immeadiate execution) are entered after the system sleeps. i think it should it taken into account that people will use this dispatching thread to achieve async operations on a request, and that these might need to be done immeadiately. This is not (or should not be) a problem in current Xron design. The Dispatcher wakes up early if the Schedule catalog is changed. -- Loren ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Xron fragility
What if you want a small job to be done every 30 seconds? I generally don't... I'm satisfied with cron granularity, although others might not be. Do you have this requirement? I thought a little about the granularity issue and decided to go with the most general solution. I have no requirement for 30-second intervals, but I wanted to avoid support requests from those who might have such a need. Also, I have no need for large numbers of scheduled methods. But I used a heavy-duty data structure (Catalog) and the sleeping dispatcher model, just so there would be no arbitrary limits that would cause me support headaches later. It uses Client.py to run a job. Why is this? This doesn't make any sense to me on its face. Why not just call the method from a separate thread? Which separate thread? Do you create a new ZODB thread? That sounds as if it might use quite a few resources. Is there a Zope API for making ZODB threads call methods on objects, as signalled by other threads? I'm guessing the easiest way to do that is to use Client.py to make a new http request. Yes, using Client.py was the easiest way to do that. The biggest problem with Client.py is its dependency on HTTP. If your server is set up to only accept HTTPS, then you can't use Xron -- not a desirable trade-off. We need another mechanism that achieves the same goal -- simulating the environment of a request -- without going all the way back to the socket. -- Loren ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] Xron fragility
The Xron product seems rather "fragile" in use. That is, if things go wrong when an Xron DTML Method is triggered, that method doesn't get rescheduled. An example of this is that my intranet DNS server needed to be rebooted. Xron couldn't look up the appropriate domain name, and stopped working. -- 2000-10-11T08:16:34 PROBLEM(100) Products.Xron.Loggerr Trigger event: http://my.development.server:4080/my_xron_method Trigger time: 2000/10/11 09:14:00 GMT+1 Failed to trigger event. Type=bci.NotAvailable Val=host not found (File: http://my.development.server/my_xron_method/trigger Line: []) None None for None -- 2000-10-11T08:19:04 PROBLEM(100) Products.Xron.Loggerr Failed to disarm event Also, if something bad happens in the Dispatcher thread, the thread dies. Before I leap in and start patching Xron, I'd like to have a discussion about how Xron should handle problems. My first thought is that on errors in the dispatcher thread, Xron should enter an "error state" where it probes every so often to see whether it can resume normal operation. The length of time between probes could increase with each probe, to give good performance with transient problems whilst preventing Xron from choking resources. -- Steve Alexander Software Enigneer Cat-Box limited http://www.cat-box.net ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Xron fragility
Steve, I am also interested in scheduling... though I haven't looked closely at Xron. The Xron product seems rather "fragile" in use. That is, if things go wrong when an Xron DTML Method is triggered, that method doesn't get rescheduled. Does Xron take an optimistic approach to repeating jobs? In other words, does it assume every job is a one-time job and that the last duty that a repeating job performs is to reschedule itself? An example of this is that my intranet DNS server needed to be rebooted. Xron couldn't look up the appropriate domain name, and stopped working. -- 2000-10-11T08:16:34 PROBLEM(100) Products.Xron.Loggerr Trigger event: http://my.development.server:4080/my_xron_method Trigger time: 2000/10/11 09:14:00 GMT+1 Failed to trigger event. Type=bci.NotAvailable Val=host not found (File: http://my.development.server/my_xron_method/trigger Line: []) None None for None -- 2000-10-11T08:19:04 PROBLEM(100) Products.Xron.Loggerr Failed to disarm event Also, if something bad happens in the Dispatcher thread, the thread dies. What does the Dispatcher thread do? Does it fire off worker threads? Before I leap in and start patching Xron, I'd like to have a discussion about how Xron should handle problems. My first thought is that on errors in the dispatcher thread, Xron should enter an "error state" where it probes every so often to see whether it can resume normal operation. The length of time between probes could increase with each probe, to give good performance with transient problems whilst preventing Xron from choking resources. What are other threads in Xron doing while the Dispatcher thread is hosed? What are the other threads? ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Xron fragility
Chris McDonough wrote: Steve, I am also interested in scheduling... though I haven't looked closely at Xron. The Xron product seems rather "fragile" in use. That is, if things go wrong when an Xron DTML Method is triggered, that method doesn't get rescheduled. Does Xron take an optimistic approach to repeating jobs? In other words, does it assume every job is a one-time job and that the last duty that a repeating job performs is to reschedule itself? It is the Xron DTML Method's responsibility to reschedule itself. It could be its first duty, however. Then again, as all the work happens inside a ZODB transaction, it doesn't matter when the reschedule happens. What does the Dispatcher thread do? Does it fire off worker threads? Xron has a single dispatcher thread. This thread knows how long to sleep for until the next job needs to run. It uses Client.py to run a job. Jobs are Xron DTML Methods. They have a "trigger" method that gets called through the web from localhost, by the Dispatcher thread. Before I leap in and start patching Xron, I'd like to have a discussion about how Xron should handle problems. My first thought is that on errors in the dispatcher thread, Xron should enter an "error state" where it probes every so often to see whether it can resume normal operation. The length of time between probes could increase with each probe, to give good performance with transient problems whilst preventing Xron from choking resources. What are other threads in Xron doing while the Dispatcher thread is hosed? What are the other threads? There are no other threads in Xron. Xron creates a single thread to do all event dispatching. The work is done in Zope ZODB threads (or whatever the correct terminology is). The dispatcher thread communicates with a ZODB thread by using Client to make a web request. Xron usually makes requests as the anonymous user. However, I've patched my Xron to make requests as XronUser by hardcoding (!) the username and password for that in the RPC.py module of Xron. Then, I've manually created a XronUser user in my root user folder. I give XronUser the local roles needed to run methods I want to be scheduled via Xron. This is not the ideal way to make Xron use authentication. I posted a proposed design for that to zope-dev in August. http://lists.zope.org/pipermail/zope-dev/2000-August/006548.html http://lists.zope.org/pipermail/zope-dev/2000-August/006549.html http://lists.zope.org/pipermail/zope-dev/2000-August/006550.html -- Steve Alexander Software Engineer Cat-Box limited http://www.cat-box.net ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Xron fragility
Chris McDonough wrote: As a side note, I do not like the fact that Xron requires you to use special DTML methods. I suppose this is a requirement in this architecture due to the fact they need to be autocataloged, but I don't really like that feature either :-). I think Xron could work with anything that has "trigger" and "disarm" methods, that gets put into the Schedule catalog. What does the Dispatcher thread do? Does it fire off worker threads? Xron has a single dispatcher thread. This thread knows how long to sleep for until the next job needs to run. Do you think this is this any more effective than having a producer thread do a lookup every minute to see what jobs are current, after which it should put the current jobs in a queue? What if you want a small job to be done every 30 seconds? It uses Client.py to run a job. Why is this? This doesn't make any sense to me on its face. Why not just call the method from a separate thread? Which separate thread? Do you create a new ZODB thread? That sounds as if it might use quite a few resources. Is there a Zope API for making ZODB threads call methods on objects, as signalled by other threads? I'm guessing the easiest way to do that is to use Client.py to make a new http request. I see... would it be naive if I were to say that I think that a scheduler should fire off threads of its own and not rely on an external facility to run the jobs? I'd really like to see a scheduler as a standard part of Zope. I think it would be good to have scheduled methods called from within Zope, rather then going through http. I don't know enough about how ZPublisher/ZODB works to know where to start this though. -- Steve Alexander Software Engineer Cat-Box limited http://www.cat-box.net ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Xron fragility
As a side note, I do not like the fact that Xron requires you to use special DTML methods. I suppose this is a requirement in this architecture due to the fact they need to be autocataloged, but I don't really like that feature either :-). I think Xron could work with anything that has "trigger" and "disarm" methods, that gets put into the Schedule catalog. I guess I'm not real real hot on the dependency on a catalog at all. I'm having a hard time imagining a situation where you have so many events that you need to index them -- especially using a ZCatalog. It just seems like too many moving parts and overkill for the job. :-) Xron has a single dispatcher thread. This thread knows how long to sleep for until the next job needs to run. Do you think this is this any more effective than having a producer thread do a lookup every minute to see what jobs are current, after which it should put the current jobs in a queue? What if you want a small job to be done every 30 seconds? I generally don't... I'm satisfied with cron granularity, although others might not be. Do you have this requirement? It uses Client.py to run a job. Why is this? This doesn't make any sense to me on its face. Why not just call the method from a separate thread? Which separate thread? Do you create a new ZODB thread? That sounds as if it might use quite a few resources. Is there a Zope API for making ZODB threads call methods on objects, as signalled by other threads? I'm guessing the easiest way to do that is to use Client.py to make a new http request. Well... AFAIK, there is no such thing as a ZODB thread... there are a number of Python threads used by Zope. We'd just fire off another one to perform a task as part of the thing that takes an event off the queue. This might be naive. I need to try it. I'd really like to see a scheduler as a standard part of Zope. I think it would be good to have scheduled methods called from within Zope, rather then going through http. I don't know enough about how ZPublisher/ZODB works to know where to start this though. I'll need to try to call get_transaction().commit() from a manually-constructed Python thread fired off from Zope and see what happens. I think it'll just work. I'll find out in a few minutes. :-) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Xron fragility
I'll need to try to call get_transaction().commit() from a manually-constructed Python thread fired off from Zope and see what happens. I think it'll just work. I'll find out in a few minutes. :-) Yes, it just works. :-) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Xron fragility
Do you think this is this any more effective than having a producer thread do a lookup every minute to see what jobs are current, after which it should put the current jobs in a queue? E.g., as the "run" method of a separate producer thread: the same problem applies, a solution might be a timer and a signal handler to catch a signal event when a new job is entered into the system. events could be managed as some sort of collection of linked lists (or whatever ds). I think this problem is solved by: - polling for "current" events every 60 seconds *exactly on the 0th second of the minute* (this ensures that stopping and starting the scheduler won't accidently cause a "current" repeating event to fire twice). - allowing one-time events (which are most likely to need to be executed immediately) to be "current" for the minute implied by the event's timestamp as well as the minute just before that one. This ensures that an event that has a timestamp that was "current" when entered, but has subsequently been passed due to event polling overhead will be executed (even if it is entered at hh:mm:59.999 or similar). It also assumes that the scheduling poller will never take more than 60 seconds to poll all events, which is a fairly safe assumption. - the executor of the event is required to invalidate one-time events once they've been executed (or before they're executed) so they don't get fired twice. I don't believe there is any need for a signal handler or anything of the sort, the problem seems to just require careful synchronization. i played around with this in a monitor thread and you just can't achieve the proper semantics to execute an arbitray zope method. you have to recreate the request, response as well as have all the required support modules (DTML, tracebacks, etc) to successfully call a method, you are in essence reduplicating the publishing machinery. i tried to think of a way to get around this but i came up short. you can still access variables and simple methods. This would need to be thought through... the same problem exists for unit testing. :-( I see... would it be naive if I were to say that I think that a scheduler should fire off threads of its own and not rely on an external facility to run the jobs? the dispatcher thread itself has a connection to the ZODB, its using client as RPC to invoke the methods on Publishing threads. Since there is a hardcoded limit to ZODB connections firing off more connections does not seem like a good idea. AFAIK, threads will block if they can't get a DB connection. What risks does that imply? Is it a problem? ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
