python-rq is more "engineered" towards speedier "I queued it and .. voilà is executed" kind of environments.
Scheduler needs to "balance" between that and "please run this every hour" scenarios (periodic) ... additionally leaving your db alone as much as possible ^_^ python-rq has an external "addon" to be able to schedule periodic function, that needs to be active to do the work (move periodic to the actual "please execute me now" queue) In any case, for a job that needs to be executed once or twice a week (and probably takes less than 3 minutes), I'd go for a standard cron: having an external process to monitor continously your queue for doing nothing for the most part of the week (plus the eventual worker processes standing by for python-rq) is frankly a waste of resources, with one or the other system ^_^ On Friday, April 19, 2013 3:27:48 PM UTC+2, Richard wrote: > > Hello, > > First, thanks Bruno to share your experiments with python-rq, redis and > queing stuff... > > I would like to know if this is the way to go over Scheduler for emails > shippement (around 200 emails each time once or twice a weeks)? > > Thanks > > Richard > > > On Fri, Apr 19, 2013 at 6:23 AM, Ricardo Pedroso > <[email protected]<javascript:> > > wrote: > >> On Thu, Apr 18, 2013 at 10:02 PM, Niphlod <[email protected] <javascript:>> >> wrote: >> > PS: on the web2py process side, it seems that at most 10 connections are >> > istantiated with Redis: is there a connection pool? >> >> I guess, because you are running the web server with 10 threads. >> python redis by default use a connection pool... see below >> >> > On the worker side, instead, I noticed that each job generates a new >> fresh >> > connection to Redis: is this expected ? >> >> I guess because, before starts a work, RQ forks a process. >> >> >> >> This is how understand it, I may be wrong: >> >> If a request for a connection is in a different process a new empty >> connection pool is created. >> >> If the request for a connection is in the same process but none >> connections >> is available python redis starts another one. >> >> If someone knows more details, please share. >> >> This is the relevant bits I guess, on how python redis manage the >> connections: >> >> def _checkpid(self): >> if self.pid != os.getpid(): >> self.disconnect() >> self.__init__(self.connection_class, self.max_connections, >> **self.connection_kwargs) >> >> def get_connection(self, command_name, *keys, **options): >> "Get a connection from the pool" >> self._checkpid() >> try: >> connection = self._available_connections.pop() >> except IndexError: >> connection = self.make_connection() >> self._in_use_connections.add(connection) >> return connection >> >> def disconnect(self): >> "Disconnects all connections in the pool" >> all_conns = chain(self._available_connections, >> self._in_use_connections) >> for connection in all_conns: >> connection.disconnect() >> >> Ricardo >> >> -- >> >> --- >> You received this message because you are subscribed to the Google Groups >> "web2py-users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> For more options, visit https://groups.google.com/groups/opt_out. >> >> >> > -- --- You received this message because you are subscribed to the Google Groups "web2py-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
