[
http://mifosforge.jira.com/browse/MIFOSBI-306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=67733#comment-67733
]
johnwoodlock commented on MIFOSBI-306:
--------------------------------------
I think for the MFI still using the data warehouse we should use the
incremental ETL
I don't have access to the machines to make that switch but I can help whoever
has (Jakub for example) to easily do it.
http://mifosforge.jira.com/wiki/display/MIFOS/Populating+Data+Warehouse+Via+ETL+-+MifosBI_1.2+Onwards#PopulatingDataWarehouseViaETL-MifosBI_1.2Onwards-Incremental%26nbsp%3BETL
> Reporting ETLs are taking too long on RDS set-up
> ------------------------------------------------
>
> Key: MIFOSBI-306
> URL: http://mifosforge.jira.com/browse/MIFOSBI-306
> Project: mifos business intelligence
> Issue Type: Bug
> Components: Data Warehouse
> Reporter: emilytucker
> Priority: Major
>
> From John W:
> > I just want to highlight an issue with running ETL against the 'RDS'
> > implementations of mysql
> >
> > Basically the jobs take 4 times as long as opposed to when the ETL is
> > running on the same machine as mysql because masses of data gets moved
> > from mysql to java box (if mifos is chatty then ETL hogs the whole
> > conversation)
> >
> > e.g. full ETL for Keef data on my laptop or the dev sandbox takes
> > about 1.25 hrs but took nearly 5 hours against the RDS in test keef
> > today
> Also note that the SECDEP ETL takes about 9 hours
> Johns asks: Is there anyway to get ETL runnning on same box as RDS?
> AdamM responds:
> No, because RDS instances only provide the MySQL service. They may just be
> EC2 instances under the hood, but we don't have low-level
> (ie: shell) access to those machines.
> Of course we can go back to running MySQL ourselves on EC2 instances, but I'm
> not keen on that since we'd have to once again shoulder that admin burden,
> and with fewer staff.
> Some ideas off the top of my head:
> * implement incremental ETL (or is this in-progress? I forget)
> * test, then change back to the setup where ETL reads from the read
> replica and writes to the master (comes back to understanding why
> ETL is slow w/RDS, though; also, replication appears to have been
> working fine for the last week)
> * optimize the MySQL server on RDS
> * ask Oracle/MySQL support for help (we're paying for it, and Todd
> Farmer is still around)
> * move to a larger size RDS instance
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
------------------------------------------------------------------------------
Simplify data backup and recovery for your virtual environment with vRanger.
Installation's a snap, and flexible recovery options mean your data is safe,
secure and there when you need it. Discover what all the cheering's about.
Get your free trial download today.
http://p.sf.net/sfu/quest-dev2dev2
_______________________________________________
Mifos-issues mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/mifos-issues
