Reporting ETLs are taking too long on RDS set-up
------------------------------------------------
Key: MIFOSBI-306
URL: http://mifosforge.jira.com/browse/MIFOSBI-306
Project: mifos business intelligence
Issue Type: Bug
Components: Data Warehouse
Reporter: emilytucker
Priority: Major
>From John W:
> I just want to highlight an issue with running ETL against the 'RDS'
> implementations of mysql
>
> Basically the jobs take 4 times as long as opposed to when the ETL is
> running on the same machine as mysql because masses of data gets moved
> from mysql to java box (if mifos is chatty then ETL hogs the whole
> conversation)
>
> e.g. full ETL for Keef data on my laptop or the dev sandbox takes
> about 1.25 hrs but took nearly 5 hours against the RDS in test keef
> today
Also note that the SECDEP ETL takes about 9 hours
Johns asks: Is there anyway to get ETL runnning on same box as RDS?
AdamM responds:
No, because RDS instances only provide the MySQL service. They may just be EC2
instances under the hood, but we don't have low-level
(ie: shell) access to those machines.
Of course we can go back to running MySQL ourselves on EC2 instances, but I'm
not keen on that since we'd have to once again shoulder that admin burden, and
with fewer staff.
Some ideas off the top of my head:
* implement incremental ETL (or is this in-progress? I forget)
* test, then change back to the setup where ETL reads from the read
replica and writes to the master (comes back to understanding why
ETL is slow w/RDS, though; also, replication appears to have been
working fine for the last week)
* optimize the MySQL server on RDS
* ask Oracle/MySQL support for help (we're paying for it, and Todd
Farmer is still around)
* move to a larger size RDS instance
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
------------------------------------------------------------------------------
What Every C/C++ and Fortran developer Should Know!
Read this article and learn how Intel has extended the reach of its
next-generation tools to help Windows* and Linux* C/C++ and Fortran
developers boost performance applications - including clusters.
http://p.sf.net/sfu/intel-dev2devmay
_______________________________________________
Mifos-issues mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/mifos-issues
