About db bottlenecks: I think that using executesql gives a performance gain of 2 orders of magnitude over DAL on datasets with more than 1000 records. The DAL is a huge bottleneck if used inappropriately.
I suggest using posgtresql with proper indexing. How? Just take your query and run it inside an EXPLAIN statement, see where indexes are needed and *create those indexes*. Speed improvements are *dramatic*. Pass to the DAL only small datasets: even if python has the best sorting and filtering algorithms around (no kidding), postgresql having those same algorithims is simply faster, because of the way objects are handled in memory. When Massimo says the db is the bottleneck, he does not mean that relational DBs are slow in general, IMHO he means that _your_ DB is slow, badly designed and/or badly queried. I also suggest to pay attention to one more thing. When you write the application and you do some test you do it in a quasi-singlethreaded setup. Well everything works fine. Then you go in production and after you reach 20 hits/sec your db becomes a bottleneck whatever hardware you have. If you are using postgresql use a concurrency model based on procesess, *avoid threads*. i.e. configure web2py to work as uwsgi, scgi or fcgi service. mic 2012/1/27 Vinicius Assef <[email protected]>: > It really depends. > > Querying and returning large datasets can become a huge bottleneck. > More than letting db server work on its own relations. If you use a > real RDBMS, it's designed to work on it. > > Web2py doesn't create indexes. You have to make it manually. Making > so, RDBMS thanks you a lot and can be your close friend. > > -- > Vinicius Assef. > > > > On Fri, Jan 27, 2012 at 4:24 PM, Bruno Rocha <[email protected]> wrote: >> I also noted best performance directly rendering the templates and caching >> views where it can be cached. >> >> just replacing >> >> return dict() >> >> with >> >> return response.render(filename, context) >> >> in conttrollers and also usins @cached controllers and cached queries. >> >> the bottleneck is always server and database, so it is better to use pure >> Python to sort, find, filter Rows objects than using a lot of database >> requests. >> >> DAL provides .sort .find .exclude and Python has a lot of good things like >> map, reduce, filter. With one db request you can fetch records put them in >> cache and use Python in some cases to avoid more sql queries. Even >> paginations can be done without the need to go to db again. >> >> Redis is being a good cache solution. >> >> http://zerp.ly/rochacbruno >> >> Em 27/01/2012 13:40, "Alfonso de la Guarda" <[email protected]> escreveu: >> >>> Hi, >>> >>> I'm about to start a major project development and although I am a >>> regular user of web2py in my development, they have been mostly in >>> intranets, what worries me is if anyone has had previous experience of >>> putting an application based on web2py with several thousand users, in >>> which case I would like your job with this. >>> >>> Previously, I have dealt with a project based on django super (500000 >>> unique users month) and problems of performance, scalability and >>> others were so serious that they chose to pass it to php, I would not >>> do the same with this application in web2py. >>> >>> >>> Thanks.... >>> >>> Saludos, >>> >>> -------------------------------- >>> Alfonso de la Guarda >>> Centro Open Source(COS) >>> http://www.cos-la.net >>> http://alfonsodg.net >>> Twitter: @alfonsodg >>> Redes sociales: alfonsodg >>> Telef. 991935157 >>> 1024D/B23B24A4 >>> 5469 ED92 75A3 BBDB FD6B 58A5 54A1 851D B23B 24A4
