[Repoze-dev] [repoze.bfg.jinja2] Many jinja2.Environment object are created, one by view. It's a mistake ?
Hi, I use ``repoze.bfg.jinja2`` in a project. I've read the source code and I wonder why a new ``jinja2.Environment`` object are created by view ? It's a bug ? or a feature ? Thanks for your help, Stephane -- Stéphane Kleinsteph...@harobed.org - French blog: http://stephane-klein.info twitter: http://twitter.com/klein_stephane pro: http://www.is-webdesign.com ___ Repoze-dev mailing list Repoze-dev@lists.repoze.org http://lists.repoze.org/listinfo/repoze-dev
[Repoze-dev] repoze.zodbconn connection handling
I am using repoze.zodbconn in a BFG application but I suspect it is not working as it should be: as soon as multiple requests come in at the same time I get a lot of locking errors and broken results: LockError: Couldn't lock '/Users/wichert/Work/code/buildout/blackbox/var/Data.fs.lock' The way I've setup repoze.zodbconn is from repoze.zodbconn.finder import PersistentApplicationFinder from myproject import appmaker finder = PersistentApplicationFinder(settings[zodb.url], appmaker) def get_root(request): return finder(request.environ) return get_root the get_root method then serves as the root factory for BFG. What I think is happening is that get_root is called for every request. get_root then calls the PersistentApplicationFinder which tries to open a new ZODB connection, which fails if another thread already has an open connection to the same FileStorage. What I think should be done is to open the ZODB connection on app startup and only appmaker(conn.root90) per request. ZODB should then care of per-thread connection management itself. I see no way to accomplish that using the code in repoze.zodbconn, so I am wondering if I missed something critical or if this is indeed a bug in repoze.zodbconn. Wichert. ___ Repoze-dev mailing list Repoze-dev@lists.repoze.org http://lists.repoze.org/listinfo/repoze-dev
Re: [Repoze-dev] repoze.zodbconn connection handling
On 6/25/10 20:03 , Wichert Akkerman wrote: The way I've setup repoze.zodbconn is from repoze.zodbconn.finder import PersistentApplicationFinder from myproject import appmaker finder = PersistentApplicationFinder(settings[zodb.url], appmaker) def get_root(request): return finder(request.environ) return get_root [.. snip ..] What I think should be done is to open the ZODB connection on app startup and only appmaker(conn.root90) per request. ZODB should then care of per-thread connection management itself. I see no way to accomplish that using the code in repoze.zodbconn, so I am wondering if I missed something critical or if this is indeed a bug in repoze.zodbconn. I've rewritten my code to look like this: from repoze.zodbconn.uri import db_from_uri db=db_from_uri(settings[zodb.url]) conn=db.open() def get_root(request): return appmaker(conn.root()) return get_root and removed the EnvironmentDeleterMiddleware from the WSGI chain since I no longer put anything in environ. So far it looks like this works, but I'ld like some confirmation from the gurus here. Wichert. ___ Repoze-dev mailing list Repoze-dev@lists.repoze.org http://lists.repoze.org/listinfo/repoze-dev
Re: [Repoze-dev] cogbin is back - please add/update keyword turbogears2 in your setup.py and upload to pypi.
On Fri, Jun 25, 2010 at 12:40 PM, Wichert Akkerman wich...@wiggy.net wrote: On 6/25/10 16:50 , Lukasz Szybalski wrote: django-lazysignup 0.1.2 django-lockdown 0.1.1 django-paranoid-sessions 0.2.0 grokproject 2.0.1 So unfortunately the Topic :: Internet :: WWW/HTTP :: WSGI :: Middleware gives out even more projects that are not related. If you see packages with incorrect Trove classifiers please contact their maintainer and ask them to correct it. Please don't make their fault our problem :) Just so I'm clear here. Assuming the trove classifiers are correct for the packages above; that means they should work with turbogears2? The package above do not have the WSGI middleware trover classifier because you did a completely different search. All proper WSGI middleware packages should work fine with turbogears. I see. Thanks. So I guess proper search is this then: http://pypi.python.org/pypi?:action=browseshow=allc=319c=326c=506c=509 I'll see if that can be incorporated in easily. Thanks, Lucas ___ Repoze-dev mailing list Repoze-dev@lists.repoze.org http://lists.repoze.org/listinfo/repoze-dev
Re: [Repoze-dev] repoze.zodbconn connection handling
Am 25.06.2010, 20:03 Uhr, schrieb Wichert Akkerman wich...@wiggy.net: I am using repoze.zodbconn in a BFG application but I suspect it is not working as it should be: as soon as multiple requests come in at the same time I get a lot of locking errors and broken results: LockError: Couldn't lock '/Users/wichert/Work/code/buildout/blackbox/var/Data.fs.lock' Hiya Wiggy, I've been getting similar issues when I run checks with ab on a site. I thought it was down to Snow Leopard's notoriously sluggish file system. Is this similar to your environment? Charlie -- Charlie Clark Managing Director Clark Consulting Research German Office Helmholtzstr. 20 Düsseldorf D- 40215 Tel: +49-211-600-3657 Mobile: +49-178-782-6226 ___ Repoze-dev mailing list Repoze-dev@lists.repoze.org http://lists.repoze.org/listinfo/repoze-dev
Re: [Repoze-dev] repoze.zodbconn connection handling
On 6/25/10 20:18 , Charlie Clark wrote: Am 25.06.2010, 20:03 Uhr, schrieb Wichert Akkermanwich...@wiggy.net: I am using repoze.zodbconn in a BFG application but I suspect it is not working as it should be: as soon as multiple requests come in at the same time I get a lot of locking errors and broken results: LockError: Couldn't lock '/Users/wichert/Work/code/buildout/blackbox/var/Data.fs.lock' Hiya Wiggy, I've been getting similar issues when I run checks with ab on a site. I thought it was down to Snow Leopard's notoriously sluggish file system. Is this similar to your environment? Yes, but I do not think that is the culprit. Chris convinced me that my reasoning was wrong: PersistentApplicationFinder does open the database only once and stores the database in a class instance variable, so the problem is elsewhere. The current thinking is that this is a race with multiple threads trying to open the database for the first time. Wichert. ___ Repoze-dev mailing list Repoze-dev@lists.repoze.org http://lists.repoze.org/listinfo/repoze-dev
Re: [Repoze-dev] repoze.zodbconn connection handling
On Fri, 2010-06-25 at 12:55 -0600, Shane Hathaway wrote: On 06/25/2010 12:03 PM, Wichert Akkerman wrote: I am using repoze.zodbconn in a BFG application but I suspect it is not working as it should be: as soon as multiple requests come in at the same time I get a lot of locking errors and broken results: LockError: Couldn't lock '/Users/wichert/Work/code/buildout/blackbox/var/Data.fs.lock' The way I've setup repoze.zodbconn is from repoze.zodbconn.finder import PersistentApplicationFinder from myproject import appmaker finder = PersistentApplicationFinder(settings[zodb.url], appmaker) def get_root(request): return finder(request.environ) return get_root According to the docs[1], your code should look more like this: from repoze.zodbconn.finder import PersistentApplicationFinder from myproject import appmaker finder = PersistentApplicationFinder(settings[zodb.url], appmaker) app = finder(request.environ) return app OTOH, personally, I don't think this is a good pattern. I think the right idea is to open the connection in the WSGI stack [2]. That way, other WSGI components can use the connection, and you can guarantee that the connection gets closed. repoze.zodbconn supports both patterns. The problem Wiggy saw seems to have been related to a race condition, which I've fixed in release 0.12 (http://pypi.python.org/pypi/repoze.zodbconn/0.12). While it's probably trivial overhead, the issue with using the connector middleware that adds a ZODB connection to the environment at ingress is that not all requests actually need the connection. For example, image requests never need a ZODB connection. In Wiggy's application, there are a lot requests to the application which use a relational database, but which never access ZODB. This is a problem shared with other middleware like repoze.who, which adds identification information to the request. In the case of repoze.who, it can add nontrivial processing time to inject identification info into the environ, which can add inappropriate performance penalty for requests for images, and other views which dont require authentication information. We've addressed this in repoze.who by giving r.who's middleware a mode which injects only a factory into the environ instead of more eagerly constructing identity information. We can't really do that for repoze.zodbconn#connector, or at least it doesn't make much sense to do so, given that the entire purpose of opening it early is to be able to close it easily. - C ___ Repoze-dev mailing list Repoze-dev@lists.repoze.org http://lists.repoze.org/listinfo/repoze-dev
Re: [Repoze-dev] repoze.zodbconn connection handling
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Shane Hathaway wrote: On 06/25/2010 12:03 PM, Wichert Akkerman wrote: I am using repoze.zodbconn in a BFG application but I suspect it is not working as it should be: as soon as multiple requests come in at the same time I get a lot of locking errors and broken results: LockError: Couldn't lock '/Users/wichert/Work/code/buildout/blackbox/var/Data.fs.lock' The way I've setup repoze.zodbconn is from repoze.zodbconn.finder import PersistentApplicationFinder from myproject import appmaker finder = PersistentApplicationFinder(settings[zodb.url], appmaker) def get_root(request): return finder(request.environ) return get_root According to the docs[1], your code should look more like this: from repoze.zodbconn.finder import PersistentApplicationFinder from myproject import appmaker finder = PersistentApplicationFinder(settings[zodb.url], appmaker) app = finder(request.environ) return app OTOH, personally, I don't think this is a good pattern. I think the right idea is to open the connection in the WSGI stack [2]. That way, other WSGI components can use the connection, and you can guarantee that the connection gets closed. repoze.zodbconn supports both patterns. Could we make your preferred pattern lazier? E.g., instead of opening the connection and stashing it in the environ, could we stash an opener function, which would then be called by the API used by the app to extract the connection from the environ. That function could arrange to have the closer written back into the environ for the egress-side zodbconn middleware to use, for instance. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkwlB0sACgkQ+gerLs4ltQ5MFACgmZtNQP1nhXR/Cwcg+WjR8qb5 sbIAoLODPOVTmXJDpoRPKmqxvFk/F+7k =nD9V -END PGP SIGNATURE- ___ Repoze-dev mailing list Repoze-dev@lists.repoze.org http://lists.repoze.org/listinfo/repoze-dev
Re: [Repoze-dev] repoze.zodbconn connection handling
On 06/25/2010 01:16 PM, Chris McDonough wrote: We've addressed this in repoze.who by giving r.who's middleware a mode which injects only a factory into the environ instead of more eagerly constructing identity information. We can't really do that for repoze.zodbconn#connector, or at least it doesn't make much sense to do so, given that the entire purpose of opening it early is to be able to close it easily. You brought up a good point. Why couldn't we apply the same pattern here? The connector middleware could look something like this: class ConnectionFactory: def __init__(self, db): self.db = db self.connection = None def __call__(self): if self.connection is None: self.connection = self.db.open() return self.connection def close(self): if self.connection is not None: self.connection.close() class ConnectionMiddleware: def __init__(self, next_app, db): self.next_app = next_app self.db = db def __call__(self, environ, start_response): factory = ConnectionFactory(self.db) self.environ['repoze.zodbconn.connection_factory'] = factory try: return self.next_app(environ, start_response) finally: factory.close() Then we could make PersistentApplicationFinder use 'repoze.zodbconn.connection_factory' from the environment. Shane ___ Repoze-dev mailing list Repoze-dev@lists.repoze.org http://lists.repoze.org/listinfo/repoze-dev
Re: [Repoze-dev] repoze.zodbconn connection handling
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Shane Hathaway wrote: On 06/25/2010 01:16 PM, Chris McDonough wrote: We've addressed this in repoze.who by giving r.who's middleware a mode which injects only a factory into the environ instead of more eagerly constructing identity information. We can't really do that for repoze.zodbconn#connector, or at least it doesn't make much sense to do so, given that the entire purpose of opening it early is to be able to close it easily. You brought up a good point. Why couldn't we apply the same pattern here? The connector middleware could look something like this: class ConnectionFactory: def __init__(self, db): self.db = db self.connection = None def __call__(self): if self.connection is None: self.connection = self.db.open() return self.connection def close(self): if self.connection is not None: self.connection.close() class ConnectionMiddleware: def __init__(self, next_app, db): self.next_app = next_app self.db = db def __call__(self, environ, start_response): factory = ConnectionFactory(self.db) self.environ['repoze.zodbconn.connection_factory'] = factory try: return self.next_app(environ, start_response) finally: factory.close() Then we could make PersistentApplicationFinder use 'repoze.zodbconn.connection_factory' from the environment. That looks lazy enough for my taste. ;) Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkwlClUACgkQ+gerLs4ltQ54ZACgmgZR+Q1qp7YeyMCsMPue40zC bl0An06+EJMmj3XPcVrxNw7y6P2JGAbL =8Fml -END PGP SIGNATURE- ___ Repoze-dev mailing list Repoze-dev@lists.repoze.org http://lists.repoze.org/listinfo/repoze-dev
Re: [Repoze-dev] repoze.zodbconn connection handling
On Fri, 2010-06-25 at 15:58 -0400, Tres Seaver wrote: Then we could make PersistentApplicationFinder use 'repoze.zodbconn.connection_factory' from the environment. That looks lazy enough for my taste. ;) +1 - C ___ Repoze-dev mailing list Repoze-dev@lists.repoze.org http://lists.repoze.org/listinfo/repoze-dev
