Re: [Zope-dev] Running methods on Zope startup.

[grahamd]

On Apr 25 22:19, Myroslav Opyr <[EMAIL PROTECTED]> wrote:
>
> Subject: Re: [Zope-dev] Running methods on Zope startup.
>
> Chris McDonough wrote:
> 
> >[snip]
> >
> There is product ZExternalNews which is working in background from Zope
> startup. You can take a look there. There are several Startup issues in
> the code.

Yes, this product is doing a similar thing to what I want to do. I
note though that it does:

  app=context._ProductContext__app

Ie., it breaks abstraction and accesses the private application context
which is available in the product context. Yes, it is possible, but it
is (is it) cheating. :-)

Is there any reason why the application context in the product context
just can't be public in the first place? It would make things easier
and doesn't seem that big a deal to change it. At least Zope could
provide a public method which gives access to it.

I also note that this product does what it needs direct from the product
initialize routine. As long as this product doesn't rely in anyway on
other products this would be okay, but if it did, it could be a problem
as threads created by the product might try to access another product
or even part of Zope which hadn't been initialised yet.

Anyway, I might just break the encapsulation in the same way for now, will
save a lot of work.

Thanks for the pointer to this package.

--
Graham Dumpleton ([EMAIL PROTECTED])


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )

Re: [Zope-dev] Running methods on Zope startup.

[grahamd]

On Apr 25 10:02, Toby Dickenson <[EMAIL PROTECTED]> wrote:
>
> Subject: Re: [Zope-dev] Running methods on Zope startup.
>
> On Wed, 24 Apr 2002 21:56:52 -0600 (MDT), [EMAIL PROTECTED] wrote:
> 
> >Imagine a situation similar to database connections. With database connections
> >when you create it you can indicate that you want the connection established.
> >Now when Zope is restarted, those database connections will be automatically
> >restarted when Zope restarts. Although, I am not sure whether this is when
> >Zope actually starts or when the next request using that database happens.
> 
> The connection is made when the next request comes in, and the
> connection is broken when ZODB decides it is time to deactivate the DA
> object; which is hopefully only happens if the connection has not been
> used for a while.

Ok, wasn't quite sure which way it did it.

> >I can't rely on
> >the connection only starting when someone tries to access the resource, I
> >want it to happen straight away upon Zope restart.
> 
> I think you will have to fight ZODB if you need:
> * The connection to always stay up
> * one connection per instance of a class stored in ZODB
>   (like database adapters)

On the latter point of searching automatically for all instances of a
particular type and automatically starting up something for it, that was
my first thought and in the end I figured that could get real inefficient.
That is when I came to the lowest common denominator of a "Startup"
folder and figured that a manual association could be made. At least
an association with a product instance which could then do the hard
work.

Anyway, I know I will potentially have to fight ZODB though, as I can see
lots of instances where I will need to recognise existance of objects in ZODB
and create parallel transient objects which do something in relation to
those instances. Given I am new to Zope this might turn out to be messier
than I think it will.

> >Can anyone point me to such a Zope product. I have started looking at how
> >to do it myself and have some of the bits in place. The first issue is that
> >when the "initialize()" method in the product "__init__.py" file is called
> >not all Zope startup has occured, so it is too early to trigger things from
> >that point.
> 
> Initialize looks like the right place to me. What do you think this is
> too early for?

Not all products might have been initialized at that point, plus possibly
other stuff as well in Zope itself. The asyncore loop also isn't running
at that point. For reasons unknown I have also already found that if I
create some separate code of my own from a thread started in the initialize
routine, that even though the main thread of control returned back into
Zope to finishing doing everything and start up asyncore, Zope wouldn't then
service any requests. Was most strange, would only work if I created the
separate thread from callback in asyncore once it had been started.

> One thing that might be nice to add is *dependencies* between product
> initialization. Today you have to rely on Zope sorting product names
> alphabetically before initialization, which slightly sucks.

This is an issue with the idea of a "Startup" folder as well as what
order to you execute register methods. First thought was to have ability
to say that method relied on other method being run first, but then
figure that for simplicity, might be easier to let user indicate specific
order they were run through management interface.

> >The next problem is getting an appropriate context to be able to start making
> >calls into Zope objects. The "initialize()" method gets a product context,
> >but doesn't have public access to the application context. Is there some way
> >of accessing the Zope application context and thus root folder where you
> >don't otherwise have a handle to it?
> 
> Public access? no. currently you have to hack through the jar.
> 
> I agree there should be an API for this.

But then, as suggested by someone else, if a startup mechanism is inbuilt
the hook would be placed in the Application.py file which has access to
the application context anyway.

Feedback appreciated.

--
Graham Dumpleton ([EMAIL PROTECTED])


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )

Re: [Zope-dev] Running methods on Zope startup.

[grahamd]

On Apr 25 00:57, "Chris McDonough" <[EMAIL PROTECTED]> wrote:
>
> Subject: Re: [Zope-dev] Running methods on Zope startup.
>
> Something like this is slated for Zope 2.6.  If possible (and that's a big
> if at the moment, with the work we've got lined up), it is my intention to
> create a service which does the following:
> 
> ...
> 

> The two things that I think are different about this than what you specified
> is that in my conception, the startup folder is not a persistent Zope
> object, and you would need to place files in a directory on disk to hook the
> startup process.  It also solves the problem of needing to have a handle to

The important thing as far as I can see is that the hook is at least
there in such a way that the application context is available. As long
as one has that, it would then be very easy to add in a method which went
looking for the "Startup" folder in the database and went from there.

> I think your idea of registering a callback to ensure that startup is
> complete before performing any actions is very sound.  I think a callback
> can be registered almost anywhere, but a plausible place to create the
> registration method might be in lib/python/OFS/Application.py.  Then the
> callback should probably be called by lib/python/Zope/__init__.py (somewhere
> after "OFS.Application.initialize(c)", after which point all Products should
> be initialized).

Yes, worked out where it could be done, for now I was trying to work out
how it might be done without modifying the Zope code.

> This is a little brittle, though.  A better way to do this would be to
> create a generic Zope scheduling module.  This is where hooking the asyncore
> mainloop would be very appropriate.  Unfortunately, the implementation is a
> bit more complex because you'd probably need to make sure the scheduler
> didn't hold up the mainloop, and that means kicking off one or more threads
> to do the work instead of using the main thread just at startup.  However,
> it would be much more generally useful and forward-thinking to implement it
> this way, even if it was a stub that didn't actually do anything but startup
> stuff at first.

As long as one has the startup hook, a scheduler can then be a Zope product.
Steps would then be for someone to install "Startup" product and if need be
manually register callback in internal Zope startup directory to execute
"Startup" folder in known location. Someone installing the "Scheduler"
product would then register instance of it with "Startup", or it could even
be automatic in some way by using registration ability of ZCatalog.

In respect of a separate thread, I agree. I actually already have a full
scheduler/dispatcher written I can hook in. It actually has its own select
loop and thus must be run in a separate thread. It even has mechanism to
be woken up from select if new job was scheduled to be running but dispatcher
was otherwise blocking. The only issue with this separate dispatcher is
that the core is actually written in C++ and isn't pure Python but relying
on Python wrappers that already exist.

> If I were to do this, I would:
> 
>   - Implement a stub scheduler module that was really dumb and only
> knew how to do "run once, now" jobs (no run-many jobs, no run-then
> jobs).  Do this so as to not go insane trying to reinvent cron and at
> at the moment. ;-)

I already have the cron code in the dispatcher system I have. :-)

>   - Implement a non-Product "Startup" module that did what I explain above,
> but registered a callback with the scheduler instead of with
> the method called by Zope/__init__.
> 
>   - Change medusa's asyncore so that the "poll" (and poll2, and poll3)
> function kept track of when the scheduler had last been run.

How is that important if it is run in a separate thread?

>   - In asyncore, when either the select call times out or the last time the
> schduler was run was over "X" seconds in the past, call in to
> the scheduler, telling it to wake up, modifying the "last run" time.

Hmmm, still not sure why, but then the dispatcher code I have probably
has a much more developed job model than asyncore does. Not only does it
support i/o activity, timeouts, alarms, cron alarms, it also has three
different types of jobs, ie., priority, standard and idle. Add to that that
it has a separate thread, not sure why they would need to communicate.

>   - The (dumb) scheduler would wake up and notice that it had a job and
> would kick off a thread which would call back in to the startup code.
> Then it would remove the job from its queue and go on with its life,
> which would be absolutely desolate until

Re: [Zope-dev] Running methods on Zope startup.

[Myroslav Opyr]

Chris McDonough wrote:

>[snip]
>
There is product ZExternalNews which is working in background from Zope 
startup. You can take a look there. There are several Startup issues in 
the code.

Regards,

Myroslav
-- 
Myroslav Opyr
zope.net.ua  ° Ukrainian Zope Hosting
e-mail: [EMAIL PROTECTED] 
cell: +380 50.3174578






___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists -
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )

Re: [Zope-dev] Running methods on Zope startup.

[Toby Dickenson]

On Wed, 24 Apr 2002 21:56:52 -0600 (MDT), [EMAIL PROTECTED] wrote:

>Imagine a situation similar to database connections. With database connections
>when you create it you can indicate that you want the connection established.
>Now when Zope is restarted, those database connections will be automatically
>restarted when Zope restarts. Although, I am not sure whether this is when
>Zope actually starts or when the next request using that database happens.

The connection is made when the next request comes in, and the
connection is broken when ZODB decides it is time to deactivate the DA
object; which is hopefully only happens if the connection has not been
used for a while.

>I can't rely on
>the connection only starting when someone tries to access the resource, I
>want it to happen straight away upon Zope restart.

I think you will have to fight ZODB if you need:
* The connection to always stay up
* one connection per instance of a class stored in ZODB
  (like database adapters)

>Can anyone point me to such a Zope product. I have started looking at how
>to do it myself and have some of the bits in place. The first issue is that
>when the "initialize()" method in the product "__init__.py" file is called
>not all Zope startup has occured, so it is too early to trigger things from
>that point.

Initialize looks like the right place to me. What do you think this is
too early for?

One thing that might be nice to add is *dependencies* between product
initialization. Today you have to rely on Zope sorting product names
alphabetically before initialization, which slightly sucks.

>The next problem is getting an appropriate context to be able to start making
>calls into Zope objects. The "initialize()" method gets a product context,
>but doesn't have public access to the application context. Is there some way
>of accessing the Zope application context and thus root folder where you
>don't otherwise have a handle to it?

Public access? no. currently you have to hack through the jar.

I agree there should be an API for this.



Toby Dickenson
[EMAIL PROTECTED]


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists -
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )

Re: [Zope-dev] Running methods on Zope startup.

[Chris McDonough]

Something like this is slated for Zope 2.6.  If possible (and that's a big
if at the moment, with the work we've got lined up), it is my intention to
create a service which does the following:

- Searches for an appropriately-named on-disk directory (var/startup
probably).

- Iterates over the contents of the directory, finding all .py files and
turns them into python modules.

- Searches each module for a well-known top-level function (named "onstart"
or something similar) and calls the function with a single argument, which
will be the Zope "application" (root) object.

- The code does whatever arbitrary thing that it needs to do, using the
application object as a gateway with which it can manipulate
persistently-stored objects in the ZODB if necessary.

- Optionally, the code deletes itself (if this is a run-once sort of thing)
or places a marker in the database (or on the filesystem) that says it
should not run again at startup.

- The code finishes.

The two things that I think are different about this than what you specified
is that in my conception, the startup folder is not a persistent Zope
object, and you would need to place files in a directory on disk to hook the
startup process.  It also solves the problem of needing to have a handle to

I think your idea of registering a callback to ensure that startup is
complete before performing any actions is very sound.  I think a callback
can be registered almost anywhere, but a plausible place to create the
registration method might be in lib/python/OFS/Application.py.  Then the
callback should probably be called by lib/python/Zope/__init__.py (somewhere
after "OFS.Application.initialize(c)", after which point all Products should
be initialized).

This is a little brittle, though.  A better way to do this would be to
create a generic Zope scheduling module.  This is where hooking the asyncore
mainloop would be very appropriate.  Unfortunately, the implementation is a
bit more complex because you'd probably need to make sure the scheduler
didn't hold up the mainloop, and that means kicking off one or more threads
to do the work instead of using the main thread just at startup.  However,
it would be much more generally useful and forward-thinking to implement it
this way, even if it was a stub that didn't actually do anything but startup
stuff at first.

If I were to do this, I would:

  - Implement a stub scheduler module that was really dumb and only
knew how to do "run once, now" jobs (no run-many jobs, no run-then
jobs).  Do this so as to not go insane trying to reinvent cron and at
at the moment. ;-)

  - Implement a non-Product "Startup" module that did what I explain above,
but registered a callback with the scheduler instead of with
the method called by Zope/__init__.

  - Change medusa's asyncore so that the "poll" (and poll2, and poll3)
function kept track of when the scheduler had last been run.

  - In asyncore, when either the select call times out or the last time the
schduler was run was over "X" seconds in the past, call in to
the scheduler, telling it to wake up, modifying the "last run" time.

  - The (dumb) scheduler would wake up and notice that it had a job and
would kick off a thread which would call back in to the startup code.
Then it would remove the job from its queue and go on with its life,
which would be absolutely desolate until the next time Zope started up.

This architecture would facilitate innovations (like a *real* scheduling
facility ;-) in the future.

HTH,

- C


- Original Message -
From: <[EMAIL PROTECTED]>
To: <[EMAIL PROTECTED]>
Sent: Wednesday, April 24, 2002 11:56 PM
Subject: [Zope-dev] Running methods on Zope startup.


> Is there a Zope product out there which implements a "Startup" like folder
> for Zope? That is, I want to be able to have a folder like object called
> "Startup" in the root directory, and when Zope is first started I want
> Zope to automatically make a call against the "Startup" object which could
> then in turn run methods of any user defined objects which have been added
> into the "Startup" folder to perform special processing only when Zope is
> started.
>
> Imagine a situation similar to database connections. With database
connections
> when you create it you can indicate that you want the connection
established.
> Now when Zope is restarted, those database connections will be
automatically
> restarted when Zope restarts. Although, I am not sure whether this is when
> Zope actually starts or when the next request using that database happens.
>
> Anyway, I ultimately want to be able to have a Zope product which has a
> persistent interprocess messaging connection to another process. Although
> when first added into Zope the connection can be initiated, I know of no
> way to have Zope restart that connection when it restarts. I can't rely on
> the connection only starting when someone tries to access the resource, I
> want it to happen