2011/2/14 Ben Hoyt <[email protected]>: > Hi folks, > We (Oyster.com) use web.ctx attribute access heavily in our template > rendering for internationalization (our version of gettext accesses > web.ctx.locale to look up the user's language, and we call gettext() with > each UI string we need to translate). At present web.ctx is a ThreadedDict, > which is a roll-your-own thread-local dict class. I found that by > subclassing ThreadedDict from Python's built-in threading.local class it > speeds up web.ctx attribute access by a factor of 50 or so, which in turn > speeds up our template rendering by about 30% (fairly significant). > Here are timings for original vs new (using threading.local). 4.41 / 0.0835 > = 53x as fast. > C:\>python -m timeit -n 1000000 -s "import web; d = > web.utils.OldThreadedDict(); d.x = 1" "d.x" > 1000000 loops, best of 3: 4.41 usec per loop > C:\>python -m timeit -n 1000000 -s "import web; d = > web.utils.ThreadedDict(); d.x = 1" "d.x" > 1000000 loops, best of 3: 0.0835 usec per loop > So I propose ThreadedDict is changed to inherit from Python's > threading.local. This was added in Python 2.4 -- what Python versions is > web.py supposed to run on? If you need to support pre-2.4 versions, you > could always use threading.local if it exists, otherwise revert to the > existing version. > Below is our version of ThreadedDict (it's somewhat verbose with all those > methods, just so I'm fully emulating a dict). > Thanks, > Ben. > ----- > class ThreadedDict(threading.local): > """Thread local storage. > > >>> d = ThreadedDict() > >>> d.x = 1 > >>> d.x > 1 > >>> import threading > >>> def f(): d.x = 2 > ... > >>> t = threading.Thread(target=f) > >>> t.start() > >>> t.join() > >>> d.x > 1 > """ > # Define all these methods to more or less fully emulate dict -- > attribute access > # is built into threading.local. > def __len__(self): > return len(self.__dict__) > def __getitem__(self, key): > return self.__dict__[key] > def __setitem__(self, key, value): > self.__dict__[key] = value > def __delitem__(self, key): > del self.__dict__[key] > def __contains__(self, key): > return key in self.__dict__ > has_key = __contains__ > def clear(self): > self.__dict__.clear() > def copy(self): > return self.__dict__.copy() > def get(self, key, default=None): > return self.__dict__.get(key, default) > def items(self): > return self.__dict__.items() > def iteritems(self): > return self.__dict__.iteritems() > def keys(self): > return self.__dict__.keys() > def iterkeys(self): > return self.__dict__.iterkeys() > iter = iterkeys > def values(self): > return self.__dict__.values() > def itervalues(self): > return self.__dict__.itervalues() > def pop(self, key, *args): > return self.__dict__.pop(key, *args) > def popitem(self): > return self.__dict__.popitem() > def setdefault(self, key, default=None): > return self.__dict__.setdefault(key, default) > def update(self, *args, **kwargs): > self.__dict__.update(*args, **kwargs) > def __repr__(self): > return '<ThreadedDict ' + dict.__repr__(self.__dict__) + '>' > __str__ = __repr__
Cool. There are couple of problems with this. In most wsgi adapters, threads are recycled by using a thread pool. So it is important to clear all the threadeddict objects used in the system to avoid interference with later requests. currently, it is done like this: https://github.com/webpy/webpy/blob/master/web/application.py#L109 I think this can be solved by maintaing the list of instances and providing a clear_all method to clear all instances. And the Session class is extended from ThreadedDict and it is using __dict__ for storing some private data. This will fail because of that. https://github.com/webpy/webpy/blob/master/web/session.py#L45 I'm looking in to these issues right now. Will report back once I have something ready. Anand -- You received this message because you are subscribed to the Google Groups "web.py" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/webpy?hl=en.
