[
http://mifosforge.jira.com/browse/MIFOS-2744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
johnwoodlock updated MIFOS-2744:
--------------------------------
Description:
Note: I've gone into excessive detail in this description as its likely that
this situation applies in a lot of places in mifos. It might help people to
find and improve potential scalability areas of concern.
I was running on a personal test dataset with Firebug/YSlow turned on. The
dataset contained 92 centers for a particular loan officer.
As I selected the loan officer("Client & Accounts"/Select a Branch
Office/Select a Loan Officer) in order to bring back the list of active
centers, I noticed it was 'a little bit slow' based on firebug saying it took
0.34 seconds. I retried the operation a few times to get make sure I got a
consistent timing. (yep, about .3/.4 secs).
Is .3/.4 secs a long time to retrieve 92 centers? Well, it was an indicator
than something more than a query returning 92 records was involved... but I
wasn't sure... so I decided to have a look.
I kicked of my mysqlproxy program. This proxy program is a bit of a pain
because it requires changing the mysql database port setting for mifos to 4040
from 3306. The reason for mysqlproxy is it allows me to log the sql being
fired at mysql and I can use a mysql browser window to log something (e.g.
"starting loan officer click") so I know the sql that relates just to my
transaction. Any method a person is familiar with, that logs the sql sent
should be good enough.
Anyways, looking at the sql log showed that the "select a loan officer"
transaction kicked off a few queries to check permission, one query which was
the main 'get me my active centers' and, surprisingly, 92 sets of four queries
that picked up customer related tables.
After checking that the "select a loan officer" transaction didn't require the
data in the 'extra queries' I found it was due to hibernate configuration. A
mixture of one-to-one's, non-lazy loading and other many-to-one settings. It
would be good to resolve the hibernate configuration issues but I wasn't sure
of the impact e.g. AFAICS, hibernate doesn't have a way to lazy load one to
ones so the only way of avoiding the read would be to change to one to many
(possible but impact too much for me right now).
So, for me, it was doing 368 too many queries. All in under 0.4 secs...
impressively quick but a clue to a potential scalability problem (too many
database requests and they won't always be in memory) .
I set up a dto to cater for the UI needs and changed code/queries to use this
instead.
Results: (all the data was in memory)
Avg time before dto: 0.312 secs
Avg time after dto: 0.042 secs
So, nearly an 8 fold improvement (probably more if the fixed cost of the
browser's part is taken out).
Does it matter? Yes. In concurrent usage, I believe this discrepancy would
become far more noticeable (that's my story).
Take away points (for me):
It can be hard to see potential scalability issues even when running with a
large dataset in single user mode. Sometimes you have to look for anomolies in
the underlying code/database access and make a judgement.
Refactoring (e.g. moving business objects out of UI and struts actions and
inserting dto's instead) will improve scalability in some cases. I'm not
against naked objects but don't believe that approach is useful to mifos right
now (if there was a solid domain model they would be interesting).
was:
Note: I've gone into excessive detail in this description as its likely that
this situation applies in a lot of places in mifos. It might help people to
find and improve potential scalability areas of concern.
I was running on a personal test dataset with Firebug/YSlow turned on. The
dataset contained 92 centers for a particular loan officer.
As I selected the loan officer("Client & Accounts"/Select a Branch
Office/Select a Loan Officer) in order to bring back the list of active
centers, I noticed it was 'a little bit slow' based on firebug saying it took
0.34 seconds. I retried the operation a few times to get make sure I got a
consistent timing. (yep, about .3/.4 secs).
Is .3/.4 secs a long time to retrieve 92 centers? Well, it was an indicator
than something more than a query returning 92 records was involved... but I
wasn't sure... so I decided to have a look.
I kicked of my mysqlproxy program. This proxy program is a bit of a pain
because it requires changing the mysql database port setting for mifos to 4040
from 3306. The reason for mysqlproxy is it allows me to log the sql being
fired at mysql and I can use a mysql browser window to get to write something
(like "starting loan officer click") so I know when my transaction started and
finished. Any method a person is familiar with that gets the sql written is
good enough.
Anyways, looking at the sql log showed that the "select a loan officer"
transaction kicked off a few queries to check permission, one query which was
the main 'get me my active centers' and 92 sets of four queries that picked up
tables related to customer/center.
This was due to how CustomerBO was configured in hibernate.
CustomerAddressDetailEntity
are configured as one to one. AFAICS, hibernate doesn't have a way to lazy
load one to ones so the only way of avoiding the read would be to change to one
to many (possible but impact too much for me right now).
So, for me, it was doing 368 too many queries. All in under 0.4 secs...
impressively quick but a clue to a potential scalability problem (too many
database requests)
> Using a dto to get a list of centers/groups in order to avoid hibernate n+1
> situation (when changing hibernate configuration deemed too risky)
> ----------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MIFOS-2744
> URL: http://mifosforge.jira.com/browse/MIFOS-2744
> Project: mifos
> Issue Type: Improvement
> Components: Performance
> Affects Versions: Release 1.4
> Reporter: johnwoodlock
> Assignee: johnwoodlock
> Fix For: Gazelle C
>
>
> Note: I've gone into excessive detail in this description as its likely that
> this situation applies in a lot of places in mifos. It might help people to
> find and improve potential scalability areas of concern.
> I was running on a personal test dataset with Firebug/YSlow turned on. The
> dataset contained 92 centers for a particular loan officer.
> As I selected the loan officer("Client & Accounts"/Select a Branch
> Office/Select a Loan Officer) in order to bring back the list of active
> centers, I noticed it was 'a little bit slow' based on firebug saying it took
> 0.34 seconds. I retried the operation a few times to get make sure I got a
> consistent timing. (yep, about .3/.4 secs).
> Is .3/.4 secs a long time to retrieve 92 centers? Well, it was an indicator
> than something more than a query returning 92 records was involved... but I
> wasn't sure... so I decided to have a look.
> I kicked of my mysqlproxy program. This proxy program is a bit of a pain
> because it requires changing the mysql database port setting for mifos to
> 4040 from 3306. The reason for mysqlproxy is it allows me to log the sql
> being fired at mysql and I can use a mysql browser window to log something
> (e.g. "starting loan officer click") so I know the sql that relates just to
> my transaction. Any method a person is familiar with, that logs the sql sent
> should be good enough.
> Anyways, looking at the sql log showed that the "select a loan officer"
> transaction kicked off a few queries to check permission, one query which was
> the main 'get me my active centers' and, surprisingly, 92 sets of four
> queries that picked up customer related tables.
> After checking that the "select a loan officer" transaction didn't require
> the data in the 'extra queries' I found it was due to hibernate
> configuration. A mixture of one-to-one's, non-lazy loading and other
> many-to-one settings. It would be good to resolve the hibernate
> configuration issues but I wasn't sure of the impact e.g. AFAICS, hibernate
> doesn't have a way to lazy load one to ones so the only way of avoiding the
> read would be to change to one to many (possible but impact too much for me
> right now).
> So, for me, it was doing 368 too many queries. All in under 0.4 secs...
> impressively quick but a clue to a potential scalability problem (too many
> database requests and they won't always be in memory) .
> I set up a dto to cater for the UI needs and changed code/queries to use this
> instead.
> Results: (all the data was in memory)
> Avg time before dto: 0.312 secs
> Avg time after dto: 0.042 secs
> So, nearly an 8 fold improvement (probably more if the fixed cost of the
> browser's part is taken out).
> Does it matter? Yes. In concurrent usage, I believe this discrepancy would
> become far more noticeable (that's my story).
> Take away points (for me):
> It can be hard to see potential scalability issues even when running with a
> large dataset in single user mode. Sometimes you have to look for anomolies
> in the underlying code/database access and make a judgement.
> Refactoring (e.g. moving business objects out of UI and struts actions and
> inserting dto's instead) will improve scalability in some cases. I'm not
> against naked objects but don't believe that approach is useful to mifos
> right now (if there was a solid domain model they would be interesting).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://mifosforge.jira.com/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
Mifos-issues mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/mifos-issues
