Thanks for the reply. I'll check our Java setup, retry the import and report
back.
However there is still a larger issue. Taking 16 minutes as a representative
timing and assuming linear scaling (unlikely but for the sake of discussion),
importing the full dataset ("District Outpatient monthly report"), of which the
previous data is just a small part, would take at least 10 hours of solid
computation. I'm not sure our servers would survive that kind of abuse.
And then there is "Inpatients Monthly", "Weekly Disease Outbreak" and about 4
smaller (approx. an order of magnitude) reports. This would be about 35 hours
of computation, with a longer "wall clock" time. Are any of the other import
methods any faster ? Do we have to resort to generating SQL statements instead ?
- Edward -
________________________________
From: Lars Helge Øverland <[email protected]>
To: Edward Ari Bichetero <[email protected]>
Cc: "[email protected]" <[email protected]>
Sent: Monday, May 7, 2012 10:57 AM
Subject: Re: [Dhis2-users] DHIS version 2.8 released
Hello,
I did a test here with a CSV file containing 50 000 records and it took 16
minutes.
I suspect this has to do with your Java configuration - have you e.g. set the
environment variable JAVA_OPTS to allocate memory to Java ? A reasonable value
would be
-Xms500m -Xmx1000m -XX:PermSize=250m -XX:MaxPermSize=500m
Lars
On Sat, May 5, 2012 at 9:11 AM, Edward Ari Bichetero <[email protected]> wrote:
The import file contains 46800 records (individual CSV lines).
>
>
>- Edward -
>
>
>________________________________
>From: Lars Helge Øverland <[email protected]>
>To: Edward Ari Bichetero <[email protected]>
>Cc: "[email protected]" <[email protected]>
>Sent: Friday, May 4, 2012 6:07 PM
>Subject: Re: [Dhis2-users] DHIS version 2.8 released
>
>
>
>
>
>
>On Fri, May 4, 2012 at 4:37 PM, Edward Ari Bichetero <[email protected]>
>wrote:
>
>Congratulation on the new release, it is nice to see the new features
>especially the CSV import functionality.
>>However ...
>>
>>
>>I've just had a go at importing a small set of historical data (Monthly
>>outpatient attendance by district) going back six years (2005-2011). This is
>>just two data elements (Outpatient attendance, Outpatient reattendance) with
>>four combo categories (Male, Female, Below 5 yrs old, 5 yrs and older) for
>>each of our 112 districts. The CSV data file is about 4.6 megabytes in size.
>>
>>I gave up on watching the import process after an hour. At that point it had
>>been using 90% of our test servers memory (4GB) and burning 100% of one
>>cpu/core almost the entire time. This is just one of the smallest datasets we
>>would be looking to import. It appears that the CSV import in it's current
>>state is not able to cope with reasonably large data. Or am I getting this
>>wrong ? Do you have any ideas/workarounds ?
>>
>>
>
>Hi, approximately how many records are in your import file?
>
>Lars
>
_______________________________________________
Mailing list: https://launchpad.net/~dhis2-users
Post to : [email protected]
Unsubscribe : https://launchpad.net/~dhis2-users
More help : https://help.launchpad.net/ListHelp
