Hi folks
I have a caching/router based question.
I need to cache a site with a large number of pages (think 20 million+). These
pages are generated by a Python application with a fair amount of database
calls, data crunching and template rendering, hence the need to cache the
responses.
First I looked into uWSGI's Cache2 setup, which is awesome but "only" memory
based. With a potential of 20 million pages at ~30 Kb each ... well I do not
have that kind of memory :)
File based caching it is. As far as I could find in the docs Cache2 does not
support dumping responses into files, only into memory (tell me if I'm wrong
here...). The only way to accomplish file based caching is through the router
and the "transformation_tofile" plugin as demonstrated in the caching cookbook.
I have tested this and it works very well. Not as fast as memory based caching,
but still a considerable speed boost.
Here comes the crux: If more than 20 million files could potentially be dumped
into a single cache directory on disk this could wreak havoc on filesystem read
times in that specific directory.
So here is what I try to do: I would like to create subdirectories inside the
main cache directory based on the URL path. Each path in the application points
to a different type of page so I'm looking at the following cache setup:
- If the path is: "/path-a/some-title/" the cache on disk
"/main-cache-dir/path-a/some-title/file.html
In essence this is no problem and can be achieved by the following router rule:
route-if = isfile:/disk1/uwsgi_cache/files${PATH_INFO}file.html
static:/disk1/uwsgi_cache/files${PATH_INFO}file.html
However, uWSGI needs to create this subdirectory structure (and the arbitrary
named "file.html" file) first. I know uWSGI can do this with the "exec-asap"
directive:
exec-asap = mkdir -p <some-path>
Yet "exec-asap" is not part of the router setup and does not fire during
routing nor does it carry the needed routing variables (such as ${PATH_INFO}).
Does this sound like a scenario for uWSGI "flat file" caching ... can it cache
to subdirectories instead of flat all in one?
Is it possible to create directories (and their parents) from within the router?
Cheers,
Tim
_______________________________________________
uWSGI mailing list
[email protected]
http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
