[
http://mifosforge.jira.com/browse/MIFOSBI-119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=62836#action_62836
]
johnwoodlock commented on MIFOSBI-119:
--------------------------------------
Just itemising some work I've done (in spare time ) with Infobright.
Loading data into Infobright is impressive. First there were some minor
alterations needed to the data types.. the main one being to reset
decimal(21,4) to (18,4) but then I bulk loaded our data in.
1/ using Load Data Infile on Innodb engine took about 9 mins.
Infobright took 80 seconds.
Infobright really expects all data to be loaded in.
2/ The Innodb engine was 1.6Gb
Infobright was 22Mb (very good at compression)
3/ With all Innodb data cached in memory:
some simple aggregate type queries were very very quick on Infobright... maybe
10 - 20 times quicker than Innodb
However some of the queries from our standard reports were either slightly
slower on Infobright or not much more than twice as quick.
Infobright did seem to 'scale' better tho when queries all branches rather than
one.
Infobright is not the only option in the 'get a better datawarehouse/reporting
db engine'... In fact there seems to be several better ones. Although some of
them are 'young' and may take a while to mature. Key message is that there's
plenty of improvement available out there when we need it for MFI's of the size
of GK without building too many 'summary tables'
http://pentahomusings.blogspot.com/2010/12/my-very-dodgy-col-store-database.html
- comparison by Tom Barber of Infobright, InfiniDb, VectorWise and LucidDB
with some good comments after.
> Spike to incorporate InfoBright column db into DW for Large MFI's
> -----------------------------------------------------------------
>
> Key: MIFOSBI-119
> URL: http://mifosforge.jira.com/browse/MIFOSBI-119
> Project: mifos business intelligence
> Issue Type: Story
> Components: Data Warehouse
> Reporter: johnwoodlock
> Assignee: johnwoodlock
> Priority: Major
>
> This is a placeholder/reminder to revisit once we are comfortable with the DW
> and the reports running on it.
> Current DW tables are 'InnoDB' (like mifos production schema). This should
> give reasonable performance for mid size MFIs for quite some time e.g. Secdep
> and smaller. As data volumes increase... we can still get boosts by tuning
> various options including queries, indexes, by denormalising in the ETL and
> by running reports out of hours. All valid but do involve work.
> MyIsam was the other MySQL engine option but in my testing didn't perform
> better than InnoDB as long as reasonable memory was allocated to InnoDB
> buffer.
> Infobright (infobright.com) is a column database which is more geared to data
> warehousing type use (expected to be factors quicker). Also it doesn't
> require much admin mainenance (like indexes because there aren't any) I
> think we should look into it for big MFI's like GK rather than doing a lot of
> 'summary tables' (aggregates) which is the normal route if time becomes a
> major issue.
> There are some things to ponder with ICE the community edition of InfoBright
> e.g. you can't do insert/update/delete sql statements. This seems like a
> show-stopper but not really as long as we can add to the main fact tables
> using the ETL infobright bulk loading plugin.
> Also there is an Infobright/Pentaho integration
> http://www.infobright.com/Event/infobright_and_pentaho_deliver_virtual_machine_that_integrates_open_source_/
> but I'm not sure if this is more useful to us than just installing the
> MySQL/Infobright combo by itself (would need sys admin view).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://mifosforge.jira.com/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
------------------------------------------------------------------------------
Oracle to DB2 Conversion Guide: Learn learn about native support for PL/SQL,
new data types, scalar functions, improved concurrency, built-in packages,
OCI, SQL*Plus, data movement tools, best practices and more.
http://p.sf.net/sfu/oracle-sfdev2dev
_______________________________________________
Mifos-issues mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/mifos-issues
