[pylons-devel] ZODB transaction callback problem

[Robin Harms Oredsson]

Hi everyone,

I'm encountering problems with transactions when using ZODB. On very rare
occasions the primary connection seems to have dumped the transaction
manager before running the request callbacks. As far as I understand the
code, transaction_manager on the primary connection is set to None once
it's finished.

Since I haven't seen this problem before and I've been running ZODB
instances with roughly the same code for 5 years I'm guessing something
else is new.

Does anyone have any advice on what could be going on?

Some relevant(?) versions:
pyramid_zodbconn 0.7
pyramid_tm 1.0.2
ZODB 4.4.4
transaction 1.7.0
Pyramid 1.7.3


The only pattern I've really found seems to be:
- It always happens on aborts
- It seems to be during some  server load
- It only seems to happen during requests that only read data

If anyone have some insight in this I'd really appreciate it.

All the best,
Robin

ps
Basically logged exceptions will look something like this:

<...>
  File "./eggs/pyramid-1.7.3-py2.7.egg/pyramid/request.py", line 147, in
_process_finished_callbacks
callback(self)
  File
"./eggs/pyramid_zodbconn-0.7-py2.7.egg/pyramid_zodbconn/__init__.py", line
56, in finished
primary_conn.transaction_manager.abort()
AttributeError: 'NoneType' object has no attribute 'abort'

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-devel+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-devel@googlegroups.com.
Visit this group at https://groups.google.com/group/pylons-devel.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-devel] Re: Deployment with supervisor + ZEO

[Robin Harms Oredsson]

OK thanks, I'll try to increase logging verbosity some way. The permission
part is something I've managed to rule out :/

Regards,
Robin

2015-12-16 4:54 GMT+01:00 Jonathan Vanasco :

> Then the only things I can think of are:
>
> * user/permissions error on supervisor starting the process
> * an environment variable missing/unset/overwritten in supervisor
>
> --
> You received this message because you are subscribed to the Google Groups
> "pylons-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pylons-devel+unsubscr...@googlegroups.com.
> To post to this group, send email to pylons-devel@googlegroups.com.
> Visit this group at https://groups.google.com/group/pylons-devel.
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Betahaus
phone: +46 70-333 00 10
web: http://www.betahaus.net

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-devel+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-devel@googlegroups.com.
Visit this group at https://groups.google.com/group/pylons-devel.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-devel] Re: Scripting requests - Pyramid + Celery

[Robin Harms Oredsson]

Good point about the debug_toolbar. That might be it. Thanks!

Any other celery examples out there? Especially around scripting requests?

Thanks,
Robin


2015-11-18 22:12 GMT+01:00 Jonathan Vanasco :

> Ok, this use case is somewhat similar to ours... in that it doesn't
> require pyramid at all.  We use sqlalchemy, but zodb shouldn't be much
> different.
>
> We have 2 packages:
>
> * `myapp_pyramid`
> * `myapp_celery`
>
> `myapp_pyramid` will import `myapp_celery` to call it's `@task` decorated
> functions.  these functions are just a workhorse.
>
> `myapp_celery` is also invoked as a worker daemon.  it imports
> `myapp_pyramid`, but only to use the models and some library utilities.
>  (in a more perfect world, the models would be in a third package).
>
> each celery `@task` starts/manages it's own transaction and database
> connection.  it doesn't run any pyramid code, and isn't bound to a request
> or anything like that.
>
> quick sidenote, if you think there is a memory leak -- are you running
> debugtoolbar?  that stashes the requests, so the process will keep growing
> and it will look like a memory leak... but it's really a conscious decision
> to keep the memory.
>
> --
> You received this message because you are subscribed to the Google Groups
> "pylons-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pylons-devel+unsubscr...@googlegroups.com.
> To post to this group, send email to pylons-devel@googlegroups.com.
> Visit this group at http://groups.google.com/group/pylons-devel.
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Betahaus
phone: +46 70-333 00 10
web: http://www.betahaus.net

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-devel+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-devel@googlegroups.com.
Visit this group at http://groups.google.com/group/pylons-devel.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-devel] Re: Scripting requests - Pyramid + Celery

[Robin Harms Oredsson]

Fair point :)

I have an application with a ZODB. Some parts of the app will require
complex calculations that will take a really long time to run. After they
finish, they'll want to write something. So i essentially imagine something
like this:

- Pyramid frontend, read access
- Celery long running tasks with read access.
- Celery write tasks

The rationale for this is that the ZCatalog doesn't handle conflict
resolution well, so I'll want the write tasks to be queued and written in
sequence. Also, I want all the write tasks to use their own transactions.
This is currently accomplished by creating a new request before the task is
run, and running the closer when it's finished. That seems to cause a
memory leak though.

Anyone doing something similar?

Thanks a lot,
Robin


2015-11-18 18:06 GMT+01:00 Jonathan Vanasco :

> What exactly are you trying to do with Celery - can you give an example?
>
> --
> You received this message because you are subscribed to the Google Groups
> "pylons-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pylons-devel+unsubscr...@googlegroups.com.
> To post to this group, send email to pylons-devel@googlegroups.com.
> Visit this group at http://groups.google.com/group/pylons-devel.
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Betahaus
phone: +46 70-333 00 10
web: http://www.betahaus.net

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-devel+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-devel@googlegroups.com.
Visit this group at http://groups.google.com/group/pylons-devel.
For more options, visit https://groups.google.com/d/optout.

[pylons-devel] Zope object events for colander schemas?

[Robin Harms Oredsson]

Hi,
I'm just curious of what the preferred method of doing changes to a schema
would be?

Usecase:
base_package has the schema:

class BaseSchema(colander.Schema):
some field = etc...

Another package is a plugin, and would like to add a field to that schema.

At least in my head, this would be quite simple with a zope object event,
something like SchemaCreated that would fire every time a schema was
instantiated. This would work with Pyramids subscribers.
(Pretty much like repoze.folder does when something is added)

This won't work AFAIK, since the schema will be of the SchemaNode class
once instantiated. Any relation to the initial class is gone for this
object.

Any thoughts on a smart solution for this that doesn't require monkeys with
wrenches? :)

Thanks in advance,
Robin

-- 
You received this message because you are subscribed to the Google Groups 
pylons-devel group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-devel+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-devel@googlegroups.com.
Visit this group at http://groups.google.com/group/pylons-devel.
For more options, visit https://groups.google.com/groups/opt_out.