Re: Scaling Django

Normalization is something like that : http://www.studytonight.com/dbms/database-normalization.php hardware for this Mysql was : serg@anomehost:~$ free -m total used free sharedbuffers cached Mem: 4049 3920129 0338

Re: Scaling Django

Hi Sergiy, are you referring to my post or to the OP? On Sunday, February 7, 2016 at 6:03:11 AM UTC+8, Sergiy Khohlov wrote: > > Print database structure. > Check possibility of DB normalization. > You might have meant "denormalization" here (?), especially when operating at such scale. We do

Re: Scaling Django

Print database structure. Check possibility of DB normalization. 100 GB (my "record" is 452 GB )is not so high but this size requires some attention. (Look like you Mysql used only one db file: try to set table per file. Check index size , and verify that indexes are working corectly) Review

Scaling Django

Lots of great replies already. I also want to add a few (random) things ... - have you clearly defined and isolated what issue(s) are you facing? - you mentioned using DRF in a service, with a large JSON reponse taking seconds to finish, how did you troubleshoot/profile this? Seconds to process

Re: Scaling Django

" It's hard to hire Django engineers " I don't think that this is a problem because good software developer can learn Django faster than other frameworks. For example I have a Costa Rican startup that develop in Django, as small company in small country we don't have inversor that allows to hire

Re: Scaling Django

This is a sidelight to the OP, but he did mention django forms in one message. They are a dog. I have profiled a couple of slow pages with a lot of small forms and that's where all the time was spent (rendering forms on the server). We're moving those to DRF-serving-json to a javascript

Re: Scaling Django

> Den 3. feb. 2016 kl. 22.30 skrev Joshua Pokotilow : > > At the startup where I work, we've written a lot of our server code in > Django. So far, we've adopted a "build it fast" mentality, so we invested > very little time in optimizing our code. A small amount of load

Re: Scaling Django

On Wed, Feb 3, 2016 at 11:30 PM, Joshua Pokotilow wrote: > At the startup where I work, we've written a lot of our server code in > Django. So far, we've adopted a "build it fast" mentality, so we invested > very little time in optimizing our code. A small amount of load

Re: Scaling Django

that Django will > scale. Anything compelling (and preferably not obvious) that would help > shed some light on Django's ability to scale would be *greatly* > appreciated, as it's very difficult for me to defend my position that > Django is a viable long-term solution without soli

Re: Scaling Django

rrero" <rafael.ferr...@gmail.com> writes: > Maybe I don't understand you very well, and for shure you have a very specific > problem to solve... but... do you read something of this? > > http://blog.disqus.com/post/62187806135/scaling-django-to-8-billion-page-views > https:/

Re: Scaling Django

http://blog.disqus.com/post/62187806135/scaling-django-to-8-billion-page-views Also caching of template pages via template tag {% cache %} Or is this just part of standard Django caching?

Re: Scaling Django

dpoint that returns >>>> a >>>> boatload of user data, and so there's talk about using a new microservice >>>> for users in lieu of our existing Django models. Even if we are able to >>>> fix >>>> bottlenecks we encounter in a ti

Re: Scaling Django

; scale. Anything compelling (and preferably not obvious) that would help >> shed some light on Django's ability to scale would be *greatly* >> appreciated, as it's very difficult for me to defend my position that >> Django is a viable long-term solution without solid evidence to back

Re: Scaling Django

fix >> bottlenecks we encounter in a timely fashion, my colleagues fear that >> Django won't scale with the business. >> >> I'm writing this post to garner additional evidence that Django will >> scale. Anything compelling (and preferably not obvious) that would he

Re: Scaling Django

ngo models. Even if we are able to fix >> bottlenecks we encounter in a timely fashion, my colleagues fear that >> Django won't scale with the business. >> >> I'm writing this post to garner additional evidence that Django will >> scale. Anything compelling (and preferab

Re: Scaling Django

on, my colleagues fear that Django >>> won't scale with the business. >>> >>> I'm writing this post to garner additional evidence that Django will >>> scale. Anything compelling (and preferably not obvious) that would help shed >>> some light on Django's ability

Re: Scaling Django

t;>> bottlenecks we encounter in a timely fashion, my colleagues fear that >>> Django won't scale with the business. >>> >>> I'm writing this post to garner additional evidence that Django will >>> scale. Anything compelling (and preferably not obvious) th

Re: Scaling Django

o's ability to scale would be *greatly* > appreciated, as it's very difficult for me to defend my position that > Django is a viable long-term solution without solid evidence to back up my > claims. It certainly doesn't help that I don't have any experience scaling > Django myself!

Re: Scaling Django

my colleagues fear that >> Django won't scale with the business. >> >> I'm writing this post to garner additional evidence that Django will >> scale. Anything compelling (and preferably not obvious) that would help >> shed some light on Django's ability to scale w

Re: Scaling Django

er additional evidence that Django will > scale. Anything compelling (and preferably not obvious) that would help > shed some light on Django's ability to scale would be *greatly* > appreciated, as it's very difficult for me to defend my position that > Django is a viable long-term s

Re: Scaling Django

s post to garner additional evidence that Django will > scale. Anything compelling (and preferably not obvious) that would help > shed some light on Django's ability to scale would be *greatly* > appreciated, as it's very difficult for me to defend my position that > Django is a v

Re: Scaling Django

Maybe I don't understand you very well, and for shure you have a very specific problem to solve... but... do you read something of this? http://blog.disqus.com/post/62187806135/scaling-django-to-8-billion-page-views https://www.digitalocean.com/community/tutorials/how-to-scale-django-beyond

Re: Scaling Django

) that would help > shed some light on Django's ability to scale would be *greatly* > appreciated, as it's very difficult for me to defend my position that > Django is a viable long-term solution without solid evidence to back up my > claims. It certainly doesn't help that I don't have any exp

Scaling Django

us) that would help shed some light on Django's ability to scale would be *greatly* appreciated, as it's very difficult for me to defend my position that Django is a viable long-term solution without solid evidence to back up my claims. It certainly doesn't help that I don't have any experience

Re: Scaling django (nginx + apache + mod_wsgi + postgresql)

Just to chime in on this.. In terms of commercial options, we have sometimes gone with ZXTM (now known as StingRay Traffic Manager) , it has some truly amazing features and you should definitely check it out. I believe that RiverBed have since started issuing free commercial licences for up to a

Re: Scaling django (nginx + apache + mod_wsgi + postgresql)

On Tue, Oct 30, 2012 at 7:35 AM, Isaac XXX wrote: > Hi Tom, > > you're right, I was not really explicit about what were my lacks of > information. Right now, the following points are the ones I can't found a > howto for the desired deployment: > > - Create a master-slave system

Re: Scaling django (nginx + apache + mod_wsgi + postgresql)

The easiest thing I've found to use is simply uWSGI with Nginx. It's easy to just create new Django servers on the fly. You simply include the IPs in a list and it will use various algorithms (optional) to distribute the requests appropriately. As a lot of applications are IO bound, you could

Re: Scaling django (nginx + apache + mod_wsgi + postgresql)

Hi there, maybe you're right, but I'm not really worried about RAM footprint, or resources consumption. I'm concerned now on architecture, setting a right scalable system, and a right cluster of systems, without lacks of communications between them. Underlaying technologies can be easily

Re: Scaling django (nginx + apache + mod_wsgi + postgresql)

Thank you so much for tips. I will keep them when I start to test environment for performance. Cheers Isaac On 10/29/2012 06:44 PM, Cal Leeming [Simplicity Media Ltd] wrote: Hi Isaac, If there is one thing I have learnt about scaling apps, it's about trying things out for yourself. Sure

Re: Scaling django (nginx + apache + mod_wsgi + postgresql)

Hi Tom, you're right, I was not really explicit about what were my lacks of information. Right now, the following points are the ones I can't found a howto for the desired deployment: - Create a master-slave system on postgresql, maintaining all systems up to date, distributing reads, and

Re: Scaling django (nginx + apache + mod_wsgi + postgresql)

The old rule of thumb is to avoid premature optimization. I'd build, profile, then scale or otherwise optimize as needed. This isn't a tutorial or really a discussion on all points of scaling a system; but identifying bottle necks will do wonders when it comes to deciding what, where, and how to

Re: Scaling django (nginx + apache + mod_wsgi + postgresql)

Hi Isaac, If there is one thing I have learnt about scaling apps, it's about trying things out for yourself. Sure there are some best practice guidelines (i.e. serving files from nginx, or using apache's X-SendFile rather than streaming out via the webapp), but if someone comes along and tells

Re: Scaling django (nginx + apache + mod_wsgi + postgresql)

On 29/10/2012 16:03, Isaac XXX wrote: Hi there, thank you for response Tom. Actually, I've a complete idea at how to build this system, but I lack the exact information about how to join systems, and what I was looking for was a source of cohesive information on all systems. At least, when

Re: Scaling django (nginx + apache + mod_wsgi + postgresql)

On Mon, Oct 29, 2012 at 4:03 PM, Isaac XXX wrote: > Hi there, > > thank you for response Tom. > > Actually, I've a complete idea at how to build this system, but I lack the > exact information about how to join systems, and what I was looking for was > a source of cohesive

Re: Scaling django (nginx + apache + mod_wsgi + postgresql)

Hi there, thank you for response Tom. Actually, I've a complete idea at how to build this system, but I lack the exact information about how to join systems, and what I was looking for was a source of cohesive information on all systems. At least, when I finish to build that system, I will

Re: Scaling django (nginx + apache + mod_wsgi + postgresql)

On Mon, Oct 29, 2012 at 2:42 PM, Isaac XXX wrote: > Hi folks, > > I'm developing a new application that should get high traffic. Right now, > I've other projects with the follow architecture: > > Nginx on front: serving static content and redirecting to apache for dynamic >

Scaling django (nginx + apache + mod_wsgi + postgresql)

Hi folks, I'm developing a new application that should get high traffic. Right now, I've other projects with the follow architecture: Nginx on front: serving static content and redirecting to apache for dynamic data Apache+mod_wsgi: serving dynamic pages PostgreSQL: backend for data storage

Re: Scaling django installation

> I guess it was 16. Sounds good. > Separation of django and static content is part of the deployment/setup > change anyway. Yes that would definitely help. As mentioned before look at varnish. rgds vivek -- You received this message because you are subscribed to the Google Groups "Django

Re: Scaling django installation

On Mon, Jun 4, 2012 at 12:15 PM, vivek wrote: > > > > > > That's aggregated load time, and not a single page loading time. The test > > comprised of navigating to multiple pages to generate more real life > > scenario. > > > > How many pages? > I guess it was 16. > > > >

Re: Scaling django installation

> > That's aggregated load time, and not a single page loading time. The test > comprised of navigating to multiple pages to generate more real life > scenario. > How many pages? > > > 3. text/html , which is the output of django app, is taking 62.74 % > > time. > > This number might not be

Re: Scaling django installation

On Monday, June 4, 2012, vivek wrote: > Hi, > > > To load test I used loadimpact.com and the results of which can be > found on: > http://loadimpact.com/load-test/www.reviews42.com-18774e46e8f562a6eb4... > > The test configuration consisted of 600 VUs with 10 mins step duration. > > Got around .1

Re: Scaling django installation

Hi, > To load test I used loadimpact.com and the results of which can be found > on:http://loadimpact.com/load-test/www.reviews42.com-18774e46e8f562a6eb4... > The test configuration consisted of 600 VUs with 10 mins step duration. > Got around .1 millions requests and around 200+ requests/sec

Re: Scaling django installation

On Sat, Jun 2, 2012 at 7:14 AM, Tim Chase wrote: > On 06/01/12 09:17, Subhranath Chunder wrote: > > (Given the fact that the server is deployed in Amazon EC2 > > Singapore location, as m1.xlarge with all it's network, memory > > constrains in place) > > A couple of

Re: Scaling django installation

On 06/01/12 09:17, Subhranath Chunder wrote: > (Given the fact that the server is deployed in Amazon EC2 > Singapore location, as m1.xlarge with all it's network, memory > constrains in place) A couple of the other aspects that occurred to me: Is there geographical separation between your

Re: Scaling django installation

On Fri, Jun 1, 2012 at 8:20 PM, Kurtis Mullins wrote: > To me, the biggest bottleneck in a "Django Application Installation" (not > application) is not going to be Django at all. It's going to be I/O -- > typically to the database and/or file system. Yup. > Another

Re: Scaling django installation

at the app level. > > Just to keep the thread a bit more focused on it's purpose, I would like > to remind ourselves that, the discussion is on "Scaling django > installation" and not "Scaling django application". > For e.g. (with random number representations) >

Re: Scaling django installation

aling django installation" and not "Scaling django application". For e.g. (with random number representations) - Setup 1: Single server setup: x1 Computation Units, x2 GB memory, n Geographical location, Max Serves 2000 requests/sec - Setup 2: 2 servers cluster setup (1 server serving

Re: Scaling django installation

Check out django-cache-machine. It uses memcache to cache your ORM qureies and updates (invalidates) that cache when they change. On Fri, Jun 1, 2012 at 10:35 AM, Tim Chase wrote: > On 06/01/12 09:17, Subhranath Chunder wrote: > > On Fri, Jun 1, 2012 at 6:57 PM,

Re: Scaling django installation

On 06/01/12 09:17, Subhranath Chunder wrote: > On Fri, Jun 1, 2012 at 6:57 PM, Tim Chase > wrote: >> 2) I/O >> 2a) disk >> 2b) network >> 2c) memory >> > Don't think these might be creating much bottleneck in my scenario. But > still, nothing like getting to exact

Re: Scaling django installation

On Fri, Jun 1, 2012 at 6:57 PM, Tim Chase wrote: > On 06/01/12 03:56, Subhranath Chunder wrote: > > With that in mind, how should we measure response complexity? > > Any particular parameter, scale? Probably I can measure against > > that, and share the numbers to

Re: Scaling django installation

On Fri, Jun 1, 2012 at 3:56 AM, Subhranath Chunder wrote: > how should we measure response complexity? a simple first approximation is the number of DB queries per page. the debug toolbar nicely gives that figure while developing. -- Javier -- You received this message

Re: Scaling django installation

On 06/01/12 03:56, Subhranath Chunder wrote: > With that in mind, how should we measure response complexity? > Any particular parameter, scale? Probably I can measure against > that, and share the numbers to shed more light on how many > requests can be handled in with a particular hardware

Re: Scaling django installation

On Fri, Jun 1, 2012 at 1:39 AM, Doug Ballance wrote: > I don't think anyone will be able to give you a good evaluation > without knowing more about the requests. Django itself could probably > handle 10k requests per second returning a simple "hello world" > response, or

Re: Scaling django installation

I don't think anyone will be able to give you a good evaluation without knowing more about the requests. Django itself could probably handle 10k requests per second returning a simple "hello world" response, or less than 10 if you are returning very large/difficult to generate responses. It is

Re: Scaling django installation

Bump! On Wednesday, May 30, 2012, Subhranath Chunder wrote: > As the subject suggests, wanted to discuss, acquire and share some > knowledge on scaling django installation. > > Firstly, my current project is a product Reviews platform, and I wanted to > benchmark or load t

Scaling django installation

As the subject suggests, wanted to discuss, acquire and share some knowledge on scaling django installation. Firstly, my current project is a product Reviews platform, and I wanted to benchmark or load test the current deployment. Currently the deployment/installation stands on a single server