Re: One Django instance, hundreds of websites

[Xavier Ordoquy]

Le 31 janv. 2011 à 07:30, James Hancock a écrit :

> This post is getting pretty long. But I had a simple Django fix that would 
> make it work a lot easier for me, and might help others. (I say this because 
> of how I implemented it, I am working with about 60 different sites and it is 
> a pretty simple arrangement)
> 
> Imagine you were able to set a site_id per request rather than relying on the 
> settings SITE_ID. Django would then checked for a request's site_id first and 
> then second check the settings one.
> 
> It is really simple, but it would fix a lot of my problems and avoid having 
> to switch around the settings SITE_ID per request. Setting the requests 
> site_id with middleware is straightforward enough and further customizations 
> to the request. Changing the urls for example.
> 
> This approach would avoid:
> Threading issus, and global variables
> Adding new functions to work with (Saves time and pain, documentation, 
> testing, releases so forth)
> Doesn't break things that are tied into the Sites Framework(site-maps, 
> comments, etc...)
> It also feels a little more DRY to me.
> 
> Let me know if I have assumed something I shouldn't have. I don't know much 
> about how the current implementation and use of SITE_ID in the backend.
> 
> Cheers,
> James Hancock

The thread is pretty long because there are also 2 threads in one:
 - one for simply changing the site_id per request
 - one for changing the all setting per request

This being said, your solution sound pretty simple  but would probably require 
all the applications to be rewritten.
If your own applications aren't much an issue, other might be.

Regards,
Xavier.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: One Django instance, hundreds of websites

[James Hancock]

This post is getting pretty long. But I had a simple Django fix that would
make it work a lot easier for me, and might help others. (I say this because
of how I implemented it, I am working with about 60 different sites and it
is a pretty simple arrangement)

Imagine you were able to set a site_id per request rather than relying on
the settings SITE_ID. Django would then checked for a request's site_id *first
*and then *second *check the settings one.

It is really simple, but it would fix a lot of my problems and avoid having
to switch around the settings SITE_ID per request. Setting the requests
site_id with middleware is straightforward enough and further customizations
to the request. Changing the urls for example.

This approach would avoid:

   - Threading issus, and global variables
   - Adding new functions to work with (Saves time and pain, documentation,
   testing, releases so forth)
   - Doesn't break things that are tied into the Sites Framework(site-maps,
   comments, etc...)

It also feels a little more DRY to me.

Let me know if I have assumed something I shouldn't have. I don't know much
about how the current implementation and use of SITE_ID in the backend.

Cheers,
James Hancock


On Mon, Jan 31, 2011 at 6:09 AM, lwcy...@gmail.com wrote:

> I believe this ticket: http://code.djangoproject.com/ticket/14628
>
> which was created during this chat session
>
> http://www.revsys.com/officehours/2010/nov/05/#question5
>
> is also relevant to the issue at hand.
>
> An interesting bit of that chat is:
> jacobkmnicoechaniz: one hint is that although the documentation says that
> you shouldn't modify settings at runtime, in fact many of them *can* be
> changed at runtime without problems. Unfortunately there's a bit of a
> trial-and-error in figuring out which.Ticket #14628 describes the
> situation.
>
>
> The proposals in Ticket #15089 (and the related thread) approach the
> multi-tenancy problem by providing the means to determine the current site
> dynamically, but this, IMHO, does not fully solve the issue.
>
> In our case (it's described in the chat-log) we have 1500 sites which would
> benefit from having separate settings files as there are configuration
> details which change from one to the other. These are not only core and
> contrib but also external apps, like django-filebrowser for example, for
> which we have different upload limits per client; we use some 10 external
> apps or more.
>
> What we have been working on makes use of the threadlocals approach and a
> proxy object for settings values. This way, whenever some code is trying to
> get the value for a setting, the actual value is determined from the
> settings corresponding to the site being requested.
> All of this has worked very well to a certain extent, but we have had a
> very hard time figuring out which settings can actually be overridden at
> runtime and which can't.
>
> I believe that to achieve true multi-tenancy it should be made very clear
> when a setting is meant to stay unchanged and when it's OK to change it at
> runtime. The above mentioned ticket proposes this distinction for django
> settings but a complete resolution should also involve a strategy to allow
> third party app developers to easily make this same distinction in their own
> code. Eventually, apps could state if they are multi-tenancy compatible (all
> settings can be changed at runtime).
>
> Maybe an easy way to accomplish this separation would be to have dynamic
> and static settings live in different files, which would make it self
> explanatory. It would be up to the developer of each app to understand which
> of his settings can actually be changed at runtime ad which can't.
>
>
> What we have implemented ATM for our specific problem is a "man in the
> middle" (using twisted) which spawns server processes based on the requested
> site and only keeps alive a number of them and discards those that have been
> idle for a while. This is only useful in a situation like ours where most of
> the sites get very little hits per day, but it's been working just fine so
> far. The code (needs some love and generalization) is available at:
> http://bitbucket.org/san/luisito
>
>
>
>
>
> On Sun, Jan 30, 2011 at 3:39 PM, Daniel Moisset 
> wrote:
>
>> On Sun, Jan 30, 2011 at 2:20 AM, Russell Keith-Magee
>>  wrote:
>> >
>> > Every single problem associated with using global variables exists
>> > with threadlocals -- and then a few more. They *can* be used
>> > successfully. However, in almost every case, they can also be avoided
>> > entirely with a good dose of rigorous engineering.
>> >
>> > I *strongly* advise against the use of this technique.
>> >
>>
>> I understand (and completely support) your objection, specially when
>> someone says «one could save even the request object to thread
>> "global" and it would be accessible anywhere.» (which would make code
>> using requests, i.e. a lot 

Re: One Django instance, hundreds of websites

[lwcy...@gmail.com]

I believe this ticket: http://code.djangoproject.com/ticket/14628

which was created during this chat session

http://www.revsys.com/officehours/2010/nov/05/#question5

is also relevant to the issue at hand.

An interesting bit of that chat is:
jacobkmnicoechaniz: one hint is that although the documentation says that
you shouldn't modify settings at runtime, in fact many of them *can* be
changed at runtime without problems. Unfortunately there's a bit of a
trial-and-error in figuring out which.Ticket #14628 describes the situation.


The proposals in Ticket #15089 (and the related thread) approach the
multi-tenancy problem by providing the means to determine the current site
dynamically, but this, IMHO, does not fully solve the issue.

In our case (it's described in the chat-log) we have 1500 sites which would
benefit from having separate settings files as there are configuration
details which change from one to the other. These are not only core and
contrib but also external apps, like django-filebrowser for example, for
which we have different upload limits per client; we use some 10 external
apps or more.

What we have been working on makes use of the threadlocals approach and a
proxy object for settings values. This way, whenever some code is trying to
get the value for a setting, the actual value is determined from the
settings corresponding to the site being requested.
All of this has worked very well to a certain extent, but we have had a very
hard time figuring out which settings can actually be overridden at runtime
and which can't.

I believe that to achieve true multi-tenancy it should be made very clear
when a setting is meant to stay unchanged and when it's OK to change it at
runtime. The above mentioned ticket proposes this distinction for django
settings but a complete resolution should also involve a strategy to allow
third party app developers to easily make this same distinction in their own
code. Eventually, apps could state if they are multi-tenancy compatible (all
settings can be changed at runtime).

Maybe an easy way to accomplish this separation would be to have dynamic and
static settings live in different files, which would make it self
explanatory. It would be up to the developer of each app to understand which
of his settings can actually be changed at runtime ad which can't.


What we have implemented ATM for our specific problem is a "man in the
middle" (using twisted) which spawns server processes based on the requested
site and only keeps alive a number of them and discards those that have been
idle for a while. This is only useful in a situation like ours where most of
the sites get very little hits per day, but it's been working just fine so
far. The code (needs some love and generalization) is available at:
http://bitbucket.org/san/luisito




On Sun, Jan 30, 2011 at 3:39 PM, Daniel Moisset wrote:

> On Sun, Jan 30, 2011 at 2:20 AM, Russell Keith-Magee
>  wrote:
> >
> > Every single problem associated with using global variables exists
> > with threadlocals -- and then a few more. They *can* be used
> > successfully. However, in almost every case, they can also be avoided
> > entirely with a good dose of rigorous engineering.
> >
> > I *strongly* advise against the use of this technique.
> >
>
> I understand (and completely support) your objection, specially when
> someone says «one could save even the request object to thread
> "global" and it would be accessible anywhere.» (which would make code
> using requests, i.e. a lot of it harder to reuse and to test)
>
> But the core problem being solved here, about moving SITE_ID to a
> threadlocal, doesn't look like bad engineering to me. In fact is
> moving something that is already global (everything in settings.py is)
> to alocation which is *less* global (thread-wise instead of
> process-wise). So this is an increase in locality, if I got it
> right...
>
> In which way is this worse than the current state of a global
> variable? (and in general, do you have some reference to explain
> «Every single problem associated with using global variables exists
> with threadlocals -- and then a few more»? I tend to think that a
> thread local is generally better than a global, even if it tends to be
> worse that good API design/argument passing)
>
> Regards,
>D.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To post to this group, send email to django-developers@googlegroups.com.
> To unsubscribe from this group, send email to
> django-developers+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/django-developers?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To 

Re: One Django instance, hundreds of websites

[Jari Pennanen]

Notice that I never suggested *django* to implement thread local hack,
it just  allowes me to continue. The thread local hack is just that
hack, it hides the real problem for now since Django does not support
the stuff I need it to.

Settings object should be considered mainly read-only, if the stuff
saved to there is not read-only, then it probably is in wrong place.

I've put my *non* thread local proposal here: 
http://ciantic.github.com/multisited/README.html
I would like comments on that from Russ or someone who has influence
on Django core.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: One Django instance, hundreds of websites

[Daniel Moisset]

On Sun, Jan 30, 2011 at 2:20 AM, Russell Keith-Magee
 wrote:
>
> Every single problem associated with using global variables exists
> with threadlocals -- and then a few more. They *can* be used
> successfully. However, in almost every case, they can also be avoided
> entirely with a good dose of rigorous engineering.
>
> I *strongly* advise against the use of this technique.
>

I understand (and completely support) your objection, specially when
someone says «one could save even the request object to thread
"global" and it would be accessible anywhere.» (which would make code
using requests, i.e. a lot of it harder to reuse and to test)

But the core problem being solved here, about moving SITE_ID to a
threadlocal, doesn't look like bad engineering to me. In fact is
moving something that is already global (everything in settings.py is)
to alocation which is *less* global (thread-wise instead of
process-wise). So this is an increase in locality, if I got it
right...

In which way is this worse than the current state of a global
variable? (and in general, do you have some reference to explain
«Every single problem associated with using global variables exists
with threadlocals -- and then a few more»? I tend to think that a
thread local is generally better than a global, even if it tends to be
worse that good API design/argument passing)

Regards,
   D.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: One Django instance, hundreds of websites

[Jari Pennanen]

In above I have error:

  authbackend.save(session, user) -> bool
  authbackend.load(session) -> user object

should be:

  authbackend.save(request, user) -> bool
  authbackend.load(request) -> user object

Since getting site id from request is the thing I need to do and save
it to session.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: One Django instance, hundreds of websites

[Jari Pennanen]

With globals:

No patches to Django required. Flatpages, media urls, media roots can
be customed by request and works without single problem. Mostly
because settings are used like settings.SITE_ID etc. and not like
getattr(settings, 'SITE_ID') in apps.

If Django ever is patched to work without this thread local trick, I
can very simply change it to use the interfaces I speaked about.
*Using* these interfaces I proposed (instead of globals) is simple,
but patching Django to use them is slow process. I'm willing to work
on it, but I don't think there is enough momentum to make for instance
the:

  get_request_media_root(request)
  get_request_media_url(request)

I proposed.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: One Django instance, hundreds of websites

[Jari Pennanen]

On Jan 30, 7:20 am, Russell Keith-Magee 
wrote:
> On Sat, Jan 29, 2011 at 8:55 PM, Jari Pennanen  
> wrote:
> If an engineer came to their supervisor with a problem and said "I'm
> going to fix this problem with a global variable", they would be
> soundly beaten by any supervisor worth their salt. Somehow, because
> the name has been changed to "threadlocal", global variables have
> suddenly become acceptable.
>
> Every single problem associated with using global variables exists
> with threadlocals -- and then a few more. They *can* be used
> successfully. However, in almost every case, they can also be avoided
> entirely with a good dose of rigorous engineering.

Without the globals:

What are the chances to get the load of patches to Django which would
be required to implement following:

  get_request_site_id(request)
  get_request_media_root(request)
  get_request_media_url(request)

Even these would require big patches, and thats not going to happen,
it literally takes *years* to get those interfaces to Django and make
all the apps working with the above interfaces even though they are
simple changes.

On top of that there had to be at least a second auth backend method
(if not altered completly see below):

  get_user_request(user_id, request=None)

And replace the django.contrib.auth.get_user() backend get_user() call
with get_user_request()

-
Altering the auth backend role.

In fact I think this module level django.contrib.auth.get_user(),
login() is currently wrong way to do things in the first place IMO, it
cuts out the authentication backend all together. It would be simpler
to imagine this login / get_user as following:

- Saving the user to request.session (currently handled by the module
level login(), and backend.authenticate())
- Loading user from request.session (requests after login) (currently
handled by the module level get_user(), and backend.get_user())

There is no simple way one could define own behavior for this saving
and loading the user to session in auth backends. That simply is odd
to me. It sounds like the thing I would like to define myself.

If there were a way to define saving and loading user one could do
cool stuff like:

1. per site login system (saving site_id to session during saving and
fetching it from session during loading)
2. during login caching of user permissions to session (no need to hit
database after login for permissions!)

One could define the saving and loading of user to session in
authentication backend with simple methods like:

  authbackend.save(session, user) -> bool
  authbackend.load(session) -> user object

Then:

- django.contrib.auth.login would become the caller for
backend.save(session, user) of course the login() could still use the
backend_session_key and user_id there
- django.contrib.auth.get_user would become the caller for
backend.load(session)

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: ANN: Server upgrade on djangoproject.com

[Russell Keith-Magee]

On Sun, Jan 30, 2011 at 2:11 PM, Russell Keith-Magee
 wrote:
> On Sat, Jan 29, 2011 at 5:45 AM, Jacob Kaplan-Moss  wrote:
>> On Fri, Jan 28, 2011 at 3:33 PM, Jacob Kaplan-Moss  
>> wrote:
>>> I'm starting the switchover to the new djangoproject.com server right
>>> now. Might be around 5 mins of downtime or so.
>>
>> Migration should be complete now. I'm still cleaning up a few
>> last-minute things, but if you notice anything wrong please pop into
>> #django-dev on freenode and let me know.
>
> You're not around in IRC (I'm guessing you're in bed) so here's a
> couple of issues:

One more -- it looks like SVN checkin triggers aren't working. I've
just checked in a couple of tickets, and they weren't autoclosed by
the commits.

Russ %-)

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.