Re: Scalability where file uploads are involved
On Jun 22, 9:51 pm, "[EMAIL PROTECTED]" <[EMAIL PROTECTED]> wrote: > I've done some preliminary work integrating Tramline with django > applications. It wasn't difficult to set up and the performance was > impressive. We are looking at this solution now for all of our media > handling in Django going forward. > > http://infrae.com/products/tramline If using Tramline, just take note of: https://issues.apache.org/jira/browse/MODPYTHON-76 The author of Tramline recommends a patch be made to mod_python but in the bigger scheme of things haven't been able to work out definitively why it should be necessary and whether it is just a workaround for problems in certain versions of underlying Apache code. We have identified another issue in mod_python filter code which was fixed, but author seemed to fix this itself didn't resolve problem. Anyway, this is not to deter you from trying Tramline as I think the concept is quite interesting and could help to improve performance. Personally I would have written a dedicated C based Apache module to do it, but then things like this are much easier to do in mod_python that using Apache C module API. :-) Graham > On Jun 7, 12:38 pm, "Nimrod A. Abing" <[EMAIL PROTECTED]> wrote: > > > Hello, > > > I am currently doing research on scaling one of the sites I made using > > Django. I have looked at Chapter 21 of the in-progress Django Book. I > > plan to follow the track recommending 3 django servers behind perlbal > > + 1 dedicated media server + 1 dedicated database server. > > > The current setup for the site I am working on uses a single machine > > for the whole site. I have Apache +mod_pythonon one IP address > > serving Django requests. For media, I have lighttpd running on another > > IP address on the same machine. Postgresql is also running on the same > > machine. > > > This site allows users to upload pictures. Those uploaded pictures are > > then served by lighttpd. While this works quite well in our single > > machine setup I am dumbfounded as to how to handle file/image uploads > > when we finally move to a load-balanced setup with a dedicated media > > server. > > > The only thing I can think of right now is to share a directory on the > > dedicated media server and have each server in the Django cluster > > mount that share either through NFS or Samba. But I don't think it's a > > good idea to do it this way. > > > Does anyone know how else this can be done? > > > Thanks in advance. > > -- > > _nimrod_a_abing_ > > >http://www.preownedcar.com/http://abing.gotdns.com/ --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Scalability where file uploads are involved
I've done some preliminary work integrating Tramline with django applications. It wasn't difficult to set up and the performance was impressive. We are looking at this solution now for all of our media handling in Django going forward. http://infrae.com/products/tramline On Jun 7, 12:38 pm, "Nimrod A. Abing" <[EMAIL PROTECTED]> wrote: > Hello, > > I am currently doing research on scaling one of the sites I made using > Django. I have looked at Chapter 21 of the in-progress Django Book. I > plan to follow the track recommending 3 django servers behind perlbal > + 1 dedicated media server + 1 dedicated database server. > > The current setup for the site I am working on uses a single machine > for the whole site. I have Apache + mod_python on one IP address > serving Django requests. For media, I have lighttpd running on another > IP address on the same machine. Postgresql is also running on the same > machine. > > This site allows users to upload pictures. Those uploaded pictures are > then served by lighttpd. While this works quite well in our single > machine setup I am dumbfounded as to how to handle file/image uploads > when we finally move to a load-balanced setup with a dedicated media > server. > > The only thing I can think of right now is to share a directory on the > dedicated media server and have each server in the Django cluster > mount that share either through NFS or Samba. But I don't think it's a > good idea to do it this way. > > Does anyone know how else this can be done? > > Thanks in advance. > -- > _nimrod_a_abing_ > > http://www.preownedcar.com/http://abing.gotdns.com/ --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Scalability where file uploads are involved
On 6/9/07, ZebZiggle <[EMAIL PROTECTED]> wrote: > > If your focus is on images, I can't recommend this utility strong > enough: > > http://www.aurigma.com/ Bookmarked :) Although we only allow as many as four images per upload session, I have to entertain the possiblity that my employer will ask for the ability to upload more than one image. Thanks! -- _nimrod_a_abing_ http://abing.gotdns.com/ http://www.preownedcar.com/ --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Scalability where file uploads are involved
On 6/9/07, Jacob Kaplan-Moss <[EMAIL PROTECTED]> wrote: > > On 6/7/07, Nimrod A. Abing <[EMAIL PROTECTED]> wrote: > > The only thing I can think of right now is to share a directory on the > > dedicated media server and have each server in the Django cluster > > mount that share either through NFS or Samba. But I don't think it's a > > good idea to do it this way. > > We do it this way, and it works just fine. NFS over a dedicated GigE > link is nearly indistinguishable from a local drive. What about write deadlocks? How do you deal with them if they occur? -- _nimrod_a_abing_ http://abing.gotdns.com/ http://www.preownedcar.com/ --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Scalability where file uploads are involved
If your focus is on images, I can't recommend this utility strong enough: http://www.aurigma.com/ I believe Facebook uses it and I have no relation to it. It's a client-side compression utility for images and it supports a Java / ActiveX client model ... whatever works. Of course, you need to concern yourself with the other ideas offered in this thread, but if you can simply reduce the amount of data passed across the wire, you'll be much better off. Hope it helps, Sandy --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Scalability where file uploads are involved
On 6/7/07, Nimrod A. Abing <[EMAIL PROTECTED]> wrote: > The only thing I can think of right now is to share a directory on the > dedicated media server and have each server in the Django cluster > mount that share either through NFS or Samba. But I don't think it's a > good idea to do it this way. We do it this way, and it works just fine. NFS over a dedicated GigE link is nearly indistinguishable from a local drive. Jacob --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Scalability where file uploads are involved
Thanks for this suggestion. Although there would be a slight delay while the media server syncs up with the Django servers I think this solution is more feasible. On 6/8/07, Malcolm Tredinnick <[EMAIL PROTECTED]> wrote: > > On Fri, 2007-06-08 at 00:38 +0800, Nimrod A. Abing wrote: > > The only thing I can think of right now is to share a directory on the > > dedicated media server and have each server in the Django cluster > > mount that share either through NFS or Samba. But I don't think it's a > > good idea to do it this way. > > Or just copy the images across to the target machine. You could either > do this as part of your model's save() method or initially store them on > the Django server and have a cronjob periodically copy them across to > the analogous directory on the image server. -- _nimrod_a_abing_ http://abing.gotdns.com/ http://www.preownedcar.com/ --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Scalability where file uploads are involved
On Fri, 2007-06-08 at 00:38 +0800, Nimrod A. Abing wrote: > Hello, > > I am currently doing research on scaling one of the sites I made using > Django. I have looked at Chapter 21 of the in-progress Django Book. I > plan to follow the track recommending 3 django servers behind perlbal > + 1 dedicated media server + 1 dedicated database server. > > The current setup for the site I am working on uses a single machine > for the whole site. I have Apache + mod_python on one IP address > serving Django requests. For media, I have lighttpd running on another > IP address on the same machine. Postgresql is also running on the same > machine. > > This site allows users to upload pictures. Those uploaded pictures are > then served by lighttpd. While this works quite well in our single > machine setup I am dumbfounded as to how to handle file/image uploads > when we finally move to a load-balanced setup with a dedicated media > server. > > The only thing I can think of right now is to share a directory on the > dedicated media server and have each server in the Django cluster > mount that share either through NFS or Samba. But I don't think it's a > good idea to do it this way. Or just copy the images across to the target machine. You could either do this as part of your model's save() method or initially store them on the Django server and have a cronjob periodically copy them across to the analogous directory on the image server. Regards, Malcolm --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Scalability where file uploads are involved
Hello, I am currently doing research on scaling one of the sites I made using Django. I have looked at Chapter 21 of the in-progress Django Book. I plan to follow the track recommending 3 django servers behind perlbal + 1 dedicated media server + 1 dedicated database server. The current setup for the site I am working on uses a single machine for the whole site. I have Apache + mod_python on one IP address serving Django requests. For media, I have lighttpd running on another IP address on the same machine. Postgresql is also running on the same machine. This site allows users to upload pictures. Those uploaded pictures are then served by lighttpd. While this works quite well in our single machine setup I am dumbfounded as to how to handle file/image uploads when we finally move to a load-balanced setup with a dedicated media server. The only thing I can think of right now is to share a directory on the dedicated media server and have each server in the Django cluster mount that share either through NFS or Samba. But I don't think it's a good idea to do it this way. Does anyone know how else this can be done? Thanks in advance. -- _nimrod_a_abing_ http://www.preownedcar.com/ http://abing.gotdns.com/ --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
