Re: [pylons-devel] Re: Scripting requests - Pyramid + Celery

[Robin Harms Oredsson]

Good point about the debug_toolbar. That might be it. Thanks!

Any other celery examples out there? Especially around scripting requests?

Thanks,
Robin


2015-11-18 22:12 GMT+01:00 Jonathan Vanasco :

> Ok, this use case is somewhat similar to ours... in that it doesn't
> require pyramid at all.  We use sqlalchemy, but zodb shouldn't be much
> different.
>
> We have 2 packages:
>
> * `myapp_pyramid`
> * `myapp_celery`
>
> `myapp_pyramid` will import `myapp_celery` to call it's `@task` decorated
> functions.  these functions are just a workhorse.
>
> `myapp_celery` is also invoked as a worker daemon.  it imports
> `myapp_pyramid`, but only to use the models and some library utilities.
>  (in a more perfect world, the models would be in a third package).
>
> each celery `@task` starts/manages it's own transaction and database
> connection.  it doesn't run any pyramid code, and isn't bound to a request
> or anything like that.
>
> quick sidenote, if you think there is a memory leak -- are you running
> debugtoolbar?  that stashes the requests, so the process will keep growing
> and it will look like a memory leak... but it's really a conscious decision
> to keep the memory.
>
> --
> You received this message because you are subscribed to the Google Groups
> "pylons-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pylons-devel+unsubscr...@googlegroups.com.
> To post to this group, send email to pylons-devel@googlegroups.com.
> Visit this group at http://groups.google.com/group/pylons-devel.
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Betahaus
phone: +46 70-333 00 10
web: http://www.betahaus.net

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-devel+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-devel@googlegroups.com.
Visit this group at http://groups.google.com/group/pylons-devel.
For more options, visit https://groups.google.com/d/optout.

[pylons-devel] Re: Scripting requests - Pyramid + Celery

[Jonathan Vanasco]

What exactly are you trying to do with Celery - can you give an example?

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-devel+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-devel@googlegroups.com.
Visit this group at http://groups.google.com/group/pylons-devel.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-devel] Re: Scripting requests - Pyramid + Celery

[Robin Harms Oredsson]

Fair point :)

I have an application with a ZODB. Some parts of the app will require
complex calculations that will take a really long time to run. After they
finish, they'll want to write something. So i essentially imagine something
like this:

- Pyramid frontend, read access
- Celery long running tasks with read access.
- Celery write tasks

The rationale for this is that the ZCatalog doesn't handle conflict
resolution well, so I'll want the write tasks to be queued and written in
sequence. Also, I want all the write tasks to use their own transactions.
This is currently accomplished by creating a new request before the task is
run, and running the closer when it's finished. That seems to cause a
memory leak though.

Anyone doing something similar?

Thanks a lot,
Robin


2015-11-18 18:06 GMT+01:00 Jonathan Vanasco :

> What exactly are you trying to do with Celery - can you give an example?
>
> --
> You received this message because you are subscribed to the Google Groups
> "pylons-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pylons-devel+unsubscr...@googlegroups.com.
> To post to this group, send email to pylons-devel@googlegroups.com.
> Visit this group at http://groups.google.com/group/pylons-devel.
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Betahaus
phone: +46 70-333 00 10
web: http://www.betahaus.net

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-devel+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-devel@googlegroups.com.
Visit this group at http://groups.google.com/group/pylons-devel.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-devel] Re: Scripting requests - Pyramid + Celery

[Jonathan Vanasco]

Ok, this use case is somewhat similar to ours... in that it doesn't require 
pyramid at all.  We use sqlalchemy, but zodb shouldn't be much different.

We have 2 packages:

* `myapp_pyramid`
* `myapp_celery`

`myapp_pyramid` will import `myapp_celery` to call it's `@task` decorated 
functions.  these functions are just a workhorse.

`myapp_celery` is also invoked as a worker daemon.  it imports 
`myapp_pyramid`, but only to use the models and some library utilities. 
 (in a more perfect world, the models would be in a third package).  

each celery `@task` starts/manages it's own transaction and database 
connection.  it doesn't run any pyramid code, and isn't bound to a request 
or anything like that.  

quick sidenote, if you think there is a memory leak -- are you running 
debugtoolbar?  that stashes the requests, so the process will keep growing 
and it will look like a memory leak... but it's really a conscious decision 
to keep the memory.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-devel+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-devel@googlegroups.com.
Visit this group at http://groups.google.com/group/pylons-devel.
For more options, visit https://groups.google.com/d/optout.