Using etags for backend caching

[Tom Eastman]

Hey guys,

My project makes pretty heavy use of etags for browser-level caching, 
but I'd like to go one step further and be able to cache pages on the 
backend keyed by the page's etag.


As far as I can tell, none of the current built-in caching 
middleware/decorators or conditional processing does this, but I don't 
think it will be terribly hard to implement.


The logic would look something like this:

etag = calculate_etag(request)
if etag in cache:
return cache.get(etag)
else:
response = view(request, ...)
cache.set(etag, response)
return response

Does this seem feasible? Are there any major problems I'm missing? Has 
someone already done it?


If no-one's done it before, and you think it sounds super useful, should 
I write a generic enough version that I can put online for people to use?


Thanks for any input you can give.

Cheers!

Tom


--
You received this message because you are subscribed to the Google Groups "Django 
users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

New project: Planning for the future

[Tom Eastman]

Hey guys,

I'm about to embark on a reasonably ambitious Django project that might
need to be maintained for some time into the future. I'm interested in
making the best choices *now* to minimize future pain.

So I'm interested in peoples' experiences and maybe even regrets. What
sort of things do you wish you had decided in early stages that would
have made your life a lot easier?

Some thoughts I'm already having:

- Components: Django, apache, mod-wsgi, memcached, postgres

- My intent is to keep it as horizontally scalable as possible -- I want
to make it easy to simply throw more servers at the system when load
gets high.

- I might need a distributed task queue -- I haven't used one before
myself but I've heard about celery, any suggestions?

- It's a fairly tightly integrated system so I'll keep a central
TEMPLATES and STATIC folder, rather than wanting them per-app.

- test-driven development and an integration test suite are project
requirements. I've used Selenium in the past, but since I'm starting
fresh should I use a newer technology, like PhantomJS?  What about for
the test suite? nose?

- Deciding on how I want to do deployment might be the decision with the
most obvious long-term consequences. I've used setup.py and buildout in
the past, my sysadmin team would far prefer everything comes in nicely
packaged .debs, there's a lot of options, and all of them are fiddly. I
would really appreciate people's thoughts on this.

- Writing future-proof code: I will be writing in Python 2.7, but I want
to remain as future-compatible to Python 3 as possible. How do I go
about achieving this? Some combination of using "from future import
[whatever]" and a compatibility library like 'six'? I've only heard of
'six', I'm yet to use it. The simple fact is that Python 3 is the
'future', but it's not the 'present', at least until Django and 3rd
party libraries are all ready for it.


If you have an opinion on any or many of the above, don't be afraid to
shout out! It might be an interesting discussion.

Cheers!

Tom


-- 
  Tom Eastman  //  Catalyst IT Ltd.  //  +64 4 803 2432



signature.asc
Description: OpenPGP digital signature

Re: Django deployment practices -- do people use setup.py?

[Tom Eastman]

On 14/03/12 21:47, Thomas Guettler wrote:
> Hey,
> 
> I read your post some days ago and waited what other people say. But
> nobody answered.
> 
> I posted a related question some time ago: Staging (dev,test,prod) in
> django. I explain
> my setup there.
> http://markmail.org/thread/wqwvordnlhyizwyp
> 
> A related thread on django-dev:
> http://groups.google.com/group/django-developers/browse_thread/thread/d919da361273dbc0/8de3c5cf9369eca3?#8de3c5cf9369eca3
> 
> 
> I think it would be very good to have the fundamental parts in django:
>  * stages: DEV, TEST, PROD
>  * get next system: user, host, install-prefix
> 
> Several deployment implementation could use these methods and variables.
> Switching
> between a deployment implementation would be easier.
> 
>   Thomas


Hi Thomas,

Yeah, I didn't get a lot of feedback -- but in the intervening time I've
been using zc.buildout with two of my projects and I've quite liked the
way it does things. Doesn't solve all my problems -- but it does seem to
do a nice job of combining what using virtualenv and setup.py gets you.

Cheers,

Tom



-- 
  Tom Eastman  //  Catalyst IT Ltd.  //  +64 4 803 2432



signature.asc
Description: OpenPGP digital signature

Django deployment practices -- do people use setup.py?

[Tom Eastman]

Hey guys,

I'm looking for deployment best practices for Django projects. Google
searches seem to show countless numbers of them, many of them somewhat
contradictory :-)

So as a simple discussion point: I'm curious to know if lots of people
use setup.py to deploy a django project? That'd mean setup.py would
install your python packages to somewhere on sys.path, maybe in your
python site-packages or maybe in a virtualenv.

Is this considered a common/recommended practice? If so -- where do you
put your settings files?

Maybe you want settings files in /etc/myproject? and add that to
sys.path? Do people do this?

Or do people skip the setup.py thing, and just check out their source to
some directory and add that directory to sys.path?

What I'm doing at the moment:

 - Fabric command sets up apache and directories
 - Another one uses setup.py to create an archive, delivers the archive
and runs setup.py install etc.

Is this silly?

What are your thoughts?


-- 
  Tom Eastman  //  Catalyst IT Ltd.  //  +64 4 803 2432



signature.asc
Description: OpenPGP digital signature

Re: Routing to database based on user

[Tom Eastman]

On 31/01/12 11:45, akaariai wrote:
> On Jan 31, 12:01 am, Tom Eastman <t...@catalyst.net.nz> wrote:
>> Hey guys,
>>
>> I'm writing a django project that will require me to route queries to
>> certain large databases based on who the logged in user is.
>>
>> So all the tables for django.contrib.auth and session and stuff will be
>> in the 'central' database, as well as a table that maps users to which
>> database they need to use for the main app.
>>
>> Can you help me come up with a way of routing database queries this way
>> using a Django database router?
>>
>> At the start of the view could I take the logged in user from
>> request.user or wherever it is, and some how provide that variable to my
>> database router for the rest of the request?
>>
>> All suggestions welcome.

> 
> I think the best solution forward is to use threading.local to store
> the request.user. So, something like this should work:
>   - have a middleware that stores the request.user in threading.local
>   - database routers just fetch the request.user from the
> threading.local storage.


Thanks Anssi!

Here is my solution, I wonder if you could just tell me if you think
there's a major problem with it.

In simplistic terms, the goal is "whenever a model from wxdatabase is
accessed, route the query to the database specified in the user's
organization's field".

It consists of a piece of django middleware and a db_router.

I guess my main question is: am I using threading.local() correctly?

Cheers!

Tom


##
##
import threading

_local = threading.local()

class WXDatabaseMiddleware(object):
def process_request(self, request):
_local.wx_database_label = None

try:
profile = request.user.get_profile()
_local.wx_database_label = profile.organization.database.label
except:
## This exception needs to be logged!
pass


class WxRouter(object):
def _get_database_label(self):
return _local.wx_database_label

def read_and_write(self, model, **hints):
if model._meta.app_label == "wxdatabase":
return self._get_database_label()
else:
return None

db_for_read  = read_and_write
db_for_write = read_and_write

###
###



signature.asc
Description: OpenPGP digital signature

Routing to database based on user

[Tom Eastman]

Hey guys,

I'm writing a django project that will require me to route queries to
certain large databases based on who the logged in user is.

So all the tables for django.contrib.auth and session and stuff will be
in the 'central' database, as well as a table that maps users to which
database they need to use for the main app.

Can you help me come up with a way of routing database queries this way
using a Django database router?

At the start of the view could I take the logged in user from
request.user or wherever it is, and some how provide that variable to my
database router for the rest of the request?

All suggestions welcome.

Cheers!

Tom



signature.asc
Description: OpenPGP digital signature

FileWrapper not working for me.

[Tom Eastman]

Hey guys, I'm using the Maverick packages, so Django 1.2.3

I need to serve a large binary (an mp3 in this case) directly through 
Django to ensure good access control independent of the web server itself.


I have the following snippet of code, to download binary files via django:


filename = os.path.join(settings.MEDIA_ROOT,  diary.file.name)
filesize = os.path.getsize(filename)
mimetype, encoding = mimetypes.guess_type(filename)

lastmodified = http_date(os.path.getmtime(filename))

wrapper = FileWrapper(file(filename))
response = HttpResponse(wrapper, content_type=mimetype)
#response = HttpResponse(open(filename).read(), content_type=mimetype)

response['Content-Length'] = filesize
response['Last-Modified'] = lastmodified

return response


For some reason, I only ever get a zero-length response out of this. But 
if I uncomment the other HttpResponse, the one that reads it all into 
memory first (yuck!) it works just fine.  Why would the FileWrapper not 
be doing what it's supposed to do?


--
You received this message because you are subscribed to the Google Groups "Django 
users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Model Inheritance.

[Tom Eastman]

On 29/10/10 11:06, Carles Barrobés wrote:

Your "place" object will contain an attribute called "restaurant" to
access
the object as an instance of the subclass.

If this place object is an instance of another place subclass,
accessing the
restaurant attribute will raise a DoesNotExist error.

Carles.


That's correct, but I want to take a 'Place' object, that doesn't have a 
'restaurant', and turn it *in to* a 'Restaurant' by adding the 
corresponding row to the Restaurant table.


--
You received this message because you are subscribed to the Google Groups "Django 
users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Model Inheritance.

[Tom Eastman]

I'm working with model inheritance, and I need to convert my objects 
into instances of its subclass.  Is there an easy way to do this?



Looking at the example in:


http://docs.djangoproject.com/en/1.2/topics/db/models/#multi-table-inheritance


I have a whole bunch of 'Place' objects that are in my database, and I 
want to turn them into 'Restaurant' objects.  How can I achieve this?


(I want to end up with the implicit one-to-one relation in 'Restaurant' 
to be pointing to the already existing object in 'Place')


Cheers!

Tom

--
You received this message because you are subscribed to the Google Groups "Django 
users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Disabling the use of certain template tags

[Tom Eastman]

Hey guys,

I'm planning on writing a django app that will serve as a tool for 
writing mail-merge type form letters.  Well, not actually letters, but 
documents which will have variable substitution in them, to either 
rendered either as HTML or LaTeX documents or some other markup language.


Of course, I would love to be able to take advantage of the Django 
template system.  I imagine users being able to create their own 
templates which are then rendered with contexts to produce the output 
documents.


That part is pretty easy -- I've done something similar before, and 
there's also the django-dbtemplates app which appears to do something 
similar.


But I want to ensure that my users can't access anything in the template 
*loader*, to prevent them including system templates or other 
potentially sensitive things into their own templates.


To that end, is there a way I can load and render templates, but disable 
any occurrences of the '{% include %} or {% extends %} tags or things of 
that nature?


Cheers,

Tom

--
You received this message because you are subscribed to the Google Groups "Django 
users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Accessing Multiple Databases.

[Tom Eastman]

Malcolm Tredinnick wrote:

> In hindsight, I probably wasn't clear as to why I'm suggesting this:
> there are a lot of places where Django would like to write to your
> database (sessions, auth lookups, ...). You can configure around and
> away from each of those in turn, but, if this were me, I'd save on brain
> wear-and-tear by letting Django behave normally until and unless I
> wanted to talk to this other read-only database and then I'd explicitly
> point my database connection in that direction. I have no stronger
> technical reason for my suggestion than that; it's primarily intuition
> and gut-feeling, but you don't have to tell anybody that.

Thanks Malcolm,  that vague use-case was pretty much what I was thinking 
too, I'll read through the article and see what insights it has :-)

Cheers!

    Tom


-- 
Tom Eastman / +64 3 479 7073 / tom.east...@stonebow.otago.ac.nz


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Accessing Multiple Databases.

[Tom Eastman]

Hey Guys,

I'm about to start developing a web front-end for a large database being 
developed by a colleague.  The web front-end is only going to be 
accessing a sub-set of the database, and mostly via 'views' that will be 
specific to the requirements of the web frontend.

I want to use Django, since I'm vaguely familiar with it (and it rocks). 
But what I want to do is have one database for all the django-specific 
stuff, so that I don't have to create any django-specific tables in the 
main database.

Can anyone point me in the direction of achieving this?  Will I still be 
able to take advantage of Django's database API or will I end up having 
to drop to raw SQL a lot more?

I'm just kind of testing the waters at the moment.  Anyone have any 
experience in similar situations?

Thanks!!

 Tom


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Accessing Multiple Databases

[Tom Eastman]

Hey Guys,

I'm about to start developing a web front-end for a large database being 
developed by a colleague.  The web front-end is only going to be 
accessing a sub-set of the database, and mostly via 'views' that will be 
specific to the requirements of the web frontend.

I want to use Django, since I'm vaguely familiar with it (and it rocks). 
But what I want to do is have one database for all the django-specific 
stuff, so that I don't have to create any django-specific tables in the 
main database.

Can anyone point me in the direction of achieving this?  Will I still be 
able to take advantage of Django's database API or will I end up having 
to drop to raw SQL a lot more?

I'm just kind of testing the waters at the moment.  Anyone have any 
experience in similar situations?

Thanks!!

 Tom


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---