Hi laurent
Thank you for all your valid remarks. We should really discuss these
issues a bit further when we have some spare time because they could
improve the performance drastically.
I do want to note that using memcache is very good but should not be an
obligation. We may not forget that we are also supporting chamilo on
(free) webhosts and requiring extra modules to be installed is not
really the way to go. However, it should be implemented in the way that
when the possiblity is there to install memcache on your server that the
code can also use memcache.
About the global.inc: I'm currently diving into the code and splitting
the code in 2 because there are some events that have been triggered
that should only been triggerd on the actual request (not on the ajax
request etc). I have also noticed that there are functions for login /
logout / go back from user view to admin view etc in that class that
should be converted to user application components. As far as i have
seen the settings are the last bottleneck that is currently in my
splitted code. I believe that this can be solved with a caching
mechanism like memcache (maybe we can do this to for registrations, menu
items etc).
For the lazy loading: as you may or may nog have seen, the settings are
only loaded the first time the settings are called for (this is the case
for platformsetting, localsetting and even registrations). So the lazy
loading is not really an issue. It's more that at this point the
platformsetting is called every global.inc for the following checks:
server type, timezone, language. This means that every time the
global.inc is loaded all the platformsettings are loaded to.
I'm also really really interested in your work about the message queue
for tracking / events. I believe that an asynchronous system could
really speed up the performance. But again it should be possible to use
both ways, either the current system (for people on hosts) or the
asynchronous system (for people that manage their own servers).
Best regards and thank you for your feedback
Sven Vanpoucke
Op 10/03/11 09:38, Laurent Opprecht schreef:
Cheers,
I would like to add a few comments on top of what Sven already said.
First tracking some simple ajax calls with a profiler - i.e. user leave
- we have about 90% of the work done by global and 10% of actual work.
Digging into "global" a lot of the work is taken by admin calls - i.e.
platform settings. Which obviously do not change a lot.
The pages that generates the css and js files do not answer the "if
modified since" html header and do not generate a cache header in the
far future. They do but in the very near future.
There are a lot of small css and js files generating a lot of chatty
calls. Even with the cache turned on for css and js the browser seems to
suffer. Plus we have several page headers for js that takes uncesseray
space.
Js files are not minimized.
Possible solutions:
- If not already done apache can be configured to emit a expiracy header
in the far future for static files - i.e. css and js. If you hit the
refresh in the browser - F5 - it will refresh cached files by emiting a
"if modified since" header. In this case apache will return either a
unmodified html header or the modififed file. So it is always possible
to refresh those files in dev.
- The php files that dynamically emit css and js should emit an html
expiracy header in the - very - far future in production and in not so
far future in dev. There is already a parameter for that. Plus they
should check the "if modified since" header and should return either 304
or file's content accordingly
- js files are not minified. That is remove "unnecessary" chars like
spaces, comments, etc for production. Several facilities exist for that.
- Have one entry point for js as well as css. That would avoid having
too many html headers in the page.
- I believe it should be possible to generate one big file for css, and
possibly for js. That may be done dynamicaly if we emit the expiracy
headers and handle the if modified since request.
- Cache admin content. That would require to install an object cache -
i.e. memcache, etc - We could cache there frequent queries such as
platform settings instead of retrieving from the db. For that to work we
would need to invalid the cache when modifying platform parameters. A
possible approach would be create an Cache class to abstract possible
cache implementations. Add cache pools - i.e. admin, weblcms, user,
whatever. Another approach could be to cache the user object in session
with frequently accessed parameters.
- implement lazy loading in Global to avoid unnecessary calls. That is
retrieve data from cache/db on first access instead of retrieving them
upfront.
- implement a message queue and process events asynchronously.
- I agree with Sven's proposition we certainly need a lighgter "global"
include for light calls such as ajax. I am only wondering if lazy
loading would be enough or if we actually need to split the file in two.
Talking about Message Queue I would like to take the opportunity to
introduce Daniel to the community. Daniel is a student here at the
University of Geneva and will work on how we can implement a Message
Queue for events. I already sent him the remarks I got but feel free to
add more requirements.
Le 09.03.2011 16:30, Sven Vanpoucke a écrit :
Hello all
We at hogent are currently working at improving the performance of the
chamilo platform. We have found a very good software package which
makes it possible to profile our database queries which is called
Jetprofiler. We stumbled upon some issues where a lot of queries were
used to render one page (for example the browser of the repository
application). A lot of these issues where due to some basic structures
like registrations (which were contacted for each active application
separately), menu items (same as with registrations) and some other
unnecessary calls to the database. Adding a bit of caching here and
there (in tables where there are no more then 150 records) we were
able to limit the amount of queries drastically (for example in the
repository we went from 150+ queries for 10 content objects to only 37
queries).
Although optimizing these things improved the performance already by
30% (measured by stress tests with JMeter) we noticed in jetprofiler
that the top used queries where still the administration / settings
tables. After a deep analysis of the problem we noticed that the
queries as such were good queries, but they were listed as top queries
because they where called every often. More then once per request...
So we started digging into the code and soon noticed that global.inc
is called for a lot of side scripts that are launched either through
css files or javascript ajax. For example in the repository there are
3 calls to css files which are dynamically built (and even work
recursively) and there are 4 calls to an ajax file => utilities.php.
This means that global.inc.php is loaded at least 8 times per request,
thus retrieving all the settings, all the registrations and even add
some tracking that is used on each page. This means that there is a
lot of overhead created for one request. We think it should be
possible to minimize this overhead, especially since it's something
very generic that will happen on almost every page throughout the
system. We propose the following steps in order to optimize these
problems.
1) global.inc seems to be a bit to heavy for use in ajax code etc. We
could try to split the global.inc in for example a common.inc and a
real global.inc. The common.inc could then be responsible for the
oblivious code from the global.inc and could require the global.inc.
Through this way the real pages on the system would use common.inc and
have all the code, and the ajax and other requests are using
global.inc which would only define the basic paths, autoloaders,
install check etc. This would avoid having the online, visit events to
be triggered on every ajax call. It's enough to trigger them for each
request.
2) The css files are loaded dynamically each time. Though they are
never changed, except when the registrations change. They are also
built on php files which reads the active registrations and then
render the css files with the import statement. It could be a solution
to statically generate these import files when the registrations are
changing (which is not very often) so that the css file can be sent
statically. Through this way we could already save 3 calls to the
global.inc and the registrations table.
3) There are a lot of javascript / ajax calls that are called already
when the page is rendered the first time. For example for paths,
translations, themes etc. We think this should be optimized by caching
the utilities.js file and remove some oblivious calls to some
settings. These settings (like web path, theme etc) could be given to
the javascript file by printing them as html in the header. We could
also optimize this by avoiding that the system would do ajax calls
when a page just finished rendering. In our opinion it should be best
that ajax calls should only be called when a user interaction has
happened.
As we already have a first release we think that this is a really
important topic. We would like to fix these issues but are welcoming
all remarks / feedback.
Best regards
_______________________________________________
Dev mailing list
Dev@lists.chamilo.org
http://lists.chamilo.org/listinfo/dev
--
Met vriendelijke groeten
Sven Vanpoucke
Digitaal Leren
Directie Onderwijs
Hogeschool Gent
http://digitaal-leren.hogent.be/
_______________________________________________
Dev mailing list
Dev@lists.chamilo.org
http://lists.chamilo.org/listinfo/dev
