Re: [Dhis2-devs] Import of Events is extremely slow - any plans to improve it?

[Morten Olav Hansen]

Hi Calle

Depending on the configuration of the events (how many org units, data
elements etc), yes, event import can still feel a bit slow. We did some
updates in 225/226, but as you are already using 226 this didn't help your
case.

Yes. when we did the metadata importer we managed to speedup it a lot, but
this required a complete rewrite, and to have the same gains in event
import we might have to do the same (which means it will have to wait a
while).

Feel free to write a issue for it.

-- 
Morten Olav Hansen
Senior Engineer, DHIS 2
University of Oslo
http://www.dhis2.org

On Sat, Jul 1, 2017 at 2:20 PM, Calle Hedberg 
wrote:

> David,
>
> I'm importing into 2.26, postgresql is tuned as far as possible (the test
> was on my laptop), and 12GB RAM. I was using CSV, put there should not be
> any significant difference between JSON and CSV.
>
> I will import the 2.5 mill events directly and see what time that takes,
> comparatively.
>
> Regards
> Calle
>
> On 30 June 2017 at 18:05, David Huser  wrote:
>
>> Hi Calle
>>
>> When some conditions are met, the event import can be really speedy
>> (imports of ~100 000 in <30 minutes).
>>
>> - Recent version (2.25+)
>> - PostgreSQL properly tuned (this is important)
>> - Enough RAM (8 GB+)
>>
>> I usually use JSON files and post them against /api/events.
>>
>> David
>>
>> On Fri, Jun 30, 2017 at 12:58 PM, Calle Hedberg 
>> wrote:
>>
>>> Ime,
>>>
>>> I was using the UI - have not looked at the API endpoint.
>>>
>>> For now I will use sql scripts as I've done before - I'm just trying to
>>> find out if anybody is working on improving what is (now) an import
>>> function that in reality only caters for small data sets. There is no
>>> question about the core developers ability to speed this up - I saw that
>>> with the import of OU meta-data - but not sure if anybody has done anything.
>>>
>>> If no answers from core developers, the I will write a JIRA issue for it.
>>>
>>> Regards
>>> Calle
>>>
>>> On 29 June 2017 at 02:25, Ime Asangansi  wrote:
>>>
 Hi Calle,

 Thanks for your question.
 Just curious (I don't have answer). As we are planning to do this soon.

 Are you doing this through UI or API?
 If API, which endpoint enables this?

 Thanks

 Ime



 On Jun 28, 2017 19:15, "Calle Hedberg"  wrote:

> Hi
>
> I started importing around 700,000 events 2 days ago (about 5 mill
> individual values) - and the import is still running, 48 hours later.
> Import is slowing - it seemed to be importing around 7-8 values per 
> second,
> now down to 2-3 per second. It looks like that import might take around 
> 200
> hours - which makes no sense from a usability point of view. Importing
> OrgUnits used to be equally slow, but a revamp recently change a 10,000
> OrgUnit import from hours to something like 5 seconds.
>
> It seems obvious that event imports have been designed to import
> values one by one, which is equivalent to asking people to fetch water
> using a leaking tea-spoon.
>
> I've searched JIRA but cannot find any issue related to this, thus
> this post: Are there any plans underway to make import of Event data 
> usable
> for larger record sets?
>
> Regards
> Calle
>
> ***
>
> Calle Hedberg
>
> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>
> Tel/fax (home): +27-21-685-6472 <+27%2021%20685%206472>
>
> Cell: +27-82-853-5352 <+27%2082%20853%205352>
>
> Iridium SatPhone: +8816-315-19119 <+881%206%20315%2019119>
>
> Email: calle.hedb...@gmail.com
>
> Skype: calle_hedberg
>
> ***
>
>
> ___
> Mailing list: https://launchpad.net/~dhis2-devs
> Post to : dhis2-devs@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~dhis2-devs
> More help   : https://help.launchpad.net/ListHelp
>
>
>>>
>>>
>>> --
>>>
>>> ***
>>>
>>> Calle Hedberg
>>>
>>> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>>>
>>> Tel/fax (home): +27-21-685-6472 <+27%2021%20685%206472>
>>>
>>> Cell: +27-82-853-5352 <+27%2082%20853%205352>
>>>
>>> Iridium SatPhone: +8816-315-19119 <+881%206%20315%2019119>
>>>
>>> Email: calle.hedb...@gmail.com
>>>
>>> Skype: calle_hedberg
>>>
>>> ***
>>>
>>>
>>> ___
>>> Mailing list: https://launchpad.net/~dhis2-devs
>>> Post to : dhis2-devs@lists.launchpad.net
>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>> More help   : https://help.launchpad.net/ListHelp
>>>
>>>
>>
>>
>> --
>>
>>
>> *David Huser*
>> DHIS2 Support Specialist
>> 

Re: [Dhis2-devs] Import of Events is extremely slow - any plans to improve it?

[Calle Hedberg]

David,

I'm importing into 2.26, postgresql is tuned as far as possible (the test
was on my laptop), and 12GB RAM. I was using CSV, put there should not be
any significant difference between JSON and CSV.

I will import the 2.5 mill events directly and see what time that takes,
comparatively.

Regards
Calle

On 30 June 2017 at 18:05, David Huser  wrote:

> Hi Calle
>
> When some conditions are met, the event import can be really speedy
> (imports of ~100 000 in <30 minutes).
>
> - Recent version (2.25+)
> - PostgreSQL properly tuned (this is important)
> - Enough RAM (8 GB+)
>
> I usually use JSON files and post them against /api/events.
>
> David
>
> On Fri, Jun 30, 2017 at 12:58 PM, Calle Hedberg 
> wrote:
>
>> Ime,
>>
>> I was using the UI - have not looked at the API endpoint.
>>
>> For now I will use sql scripts as I've done before - I'm just trying to
>> find out if anybody is working on improving what is (now) an import
>> function that in reality only caters for small data sets. There is no
>> question about the core developers ability to speed this up - I saw that
>> with the import of OU meta-data - but not sure if anybody has done anything.
>>
>> If no answers from core developers, the I will write a JIRA issue for it.
>>
>> Regards
>> Calle
>>
>> On 29 June 2017 at 02:25, Ime Asangansi  wrote:
>>
>>> Hi Calle,
>>>
>>> Thanks for your question.
>>> Just curious (I don't have answer). As we are planning to do this soon.
>>>
>>> Are you doing this through UI or API?
>>> If API, which endpoint enables this?
>>>
>>> Thanks
>>>
>>> Ime
>>>
>>>
>>>
>>> On Jun 28, 2017 19:15, "Calle Hedberg"  wrote:
>>>
 Hi

 I started importing around 700,000 events 2 days ago (about 5 mill
 individual values) - and the import is still running, 48 hours later.
 Import is slowing - it seemed to be importing around 7-8 values per second,
 now down to 2-3 per second. It looks like that import might take around 200
 hours - which makes no sense from a usability point of view. Importing
 OrgUnits used to be equally slow, but a revamp recently change a 10,000
 OrgUnit import from hours to something like 5 seconds.

 It seems obvious that event imports have been designed to import values
 one by one, which is equivalent to asking people to fetch water using a
 leaking tea-spoon.

 I've searched JIRA but cannot find any issue related to this, thus this
 post: Are there any plans underway to make import of Event data usable for
 larger record sets?

 Regards
 Calle

 ***

 Calle Hedberg

 46D Alma Road, 7700 Rosebank, SOUTH AFRICA

 Tel/fax (home): +27-21-685-6472 <+27%2021%20685%206472>

 Cell: +27-82-853-5352 <+27%2082%20853%205352>

 Iridium SatPhone: +8816-315-19119 <+881%206%20315%2019119>

 Email: calle.hedb...@gmail.com

 Skype: calle_hedberg

 ***


 ___
 Mailing list: https://launchpad.net/~dhis2-devs
 Post to : dhis2-devs@lists.launchpad.net
 Unsubscribe : https://launchpad.net/~dhis2-devs
 More help   : https://help.launchpad.net/ListHelp


>>
>>
>> --
>>
>> ***
>>
>> Calle Hedberg
>>
>> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>>
>> Tel/fax (home): +27-21-685-6472 <+27%2021%20685%206472>
>>
>> Cell: +27-82-853-5352 <+27%2082%20853%205352>
>>
>> Iridium SatPhone: +8816-315-19119 <+881%206%20315%2019119>
>>
>> Email: calle.hedb...@gmail.com
>>
>> Skype: calle_hedberg
>>
>> ***
>>
>>
>> ___
>> Mailing list: https://launchpad.net/~dhis2-devs
>> Post to : dhis2-devs@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~dhis2-devs
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>
>
> --
>
>
> *David Huser*
> DHIS2 Support Specialist
> dhu...@baosystems.com | https://baosystems.com | Skype: dafhus | 2900 K
> Street, Suite 406, Washington D.C. 20007
>



-- 

***

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedb...@gmail.com

Skype: calle_hedberg

***
___
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help   : https://help.launchpad.net/ListHelp

Re: [Dhis2-devs] Import of Events is extremely slow - any plans to improve it?

[David Huser]

Hi Calle

When some conditions are met, the event import can be really speedy
(imports of ~100 000 in <30 minutes).

- Recent version (2.25+)
- PostgreSQL properly tuned (this is important)
- Enough RAM (8 GB+)

I usually use JSON files and post them against /api/events.

David

On Fri, Jun 30, 2017 at 12:58 PM, Calle Hedberg 
wrote:

> Ime,
>
> I was using the UI - have not looked at the API endpoint.
>
> For now I will use sql scripts as I've done before - I'm just trying to
> find out if anybody is working on improving what is (now) an import
> function that in reality only caters for small data sets. There is no
> question about the core developers ability to speed this up - I saw that
> with the import of OU meta-data - but not sure if anybody has done anything.
>
> If no answers from core developers, the I will write a JIRA issue for it.
>
> Regards
> Calle
>
> On 29 June 2017 at 02:25, Ime Asangansi  wrote:
>
>> Hi Calle,
>>
>> Thanks for your question.
>> Just curious (I don't have answer). As we are planning to do this soon.
>>
>> Are you doing this through UI or API?
>> If API, which endpoint enables this?
>>
>> Thanks
>>
>> Ime
>>
>>
>>
>> On Jun 28, 2017 19:15, "Calle Hedberg"  wrote:
>>
>>> Hi
>>>
>>> I started importing around 700,000 events 2 days ago (about 5 mill
>>> individual values) - and the import is still running, 48 hours later.
>>> Import is slowing - it seemed to be importing around 7-8 values per second,
>>> now down to 2-3 per second. It looks like that import might take around 200
>>> hours - which makes no sense from a usability point of view. Importing
>>> OrgUnits used to be equally slow, but a revamp recently change a 10,000
>>> OrgUnit import from hours to something like 5 seconds.
>>>
>>> It seems obvious that event imports have been designed to import values
>>> one by one, which is equivalent to asking people to fetch water using a
>>> leaking tea-spoon.
>>>
>>> I've searched JIRA but cannot find any issue related to this, thus this
>>> post: Are there any plans underway to make import of Event data usable for
>>> larger record sets?
>>>
>>> Regards
>>> Calle
>>>
>>> ***
>>>
>>> Calle Hedberg
>>>
>>> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>>>
>>> Tel/fax (home): +27-21-685-6472 <+27%2021%20685%206472>
>>>
>>> Cell: +27-82-853-5352 <+27%2082%20853%205352>
>>>
>>> Iridium SatPhone: +8816-315-19119 <+881%206%20315%2019119>
>>>
>>> Email: calle.hedb...@gmail.com
>>>
>>> Skype: calle_hedberg
>>>
>>> ***
>>>
>>>
>>> ___
>>> Mailing list: https://launchpad.net/~dhis2-devs
>>> Post to : dhis2-devs@lists.launchpad.net
>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>> More help   : https://help.launchpad.net/ListHelp
>>>
>>>
>
>
> --
>
> ***
>
> Calle Hedberg
>
> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>
> Tel/fax (home): +27-21-685-6472 <+27%2021%20685%206472>
>
> Cell: +27-82-853-5352 <+27%2082%20853%205352>
>
> Iridium SatPhone: +8816-315-19119 <+881%206%20315%2019119>
>
> Email: calle.hedb...@gmail.com
>
> Skype: calle_hedberg
>
> ***
>
>
> ___
> Mailing list: https://launchpad.net/~dhis2-devs
> Post to : dhis2-devs@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~dhis2-devs
> More help   : https://help.launchpad.net/ListHelp
>
>


-- 


*David Huser*
DHIS2 Support Specialist
dhu...@baosystems.com | https://baosystems.com | Skype: dafhus | 2900 K
Street, Suite 406, Washington D.C. 20007
___
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help   : https://help.launchpad.net/ListHelp

Re: [Dhis2-devs] Import of Events is extremely slow - any plans to improve it?

[Calle Hedberg]

Ime,

I was using the UI - have not looked at the API endpoint.

For now I will use sql scripts as I've done before - I'm just trying to
find out if anybody is working on improving what is (now) an import
function that in reality only caters for small data sets. There is no
question about the core developers ability to speed this up - I saw that
with the import of OU meta-data - but not sure if anybody has done anything.

If no answers from core developers, the I will write a JIRA issue for it.

Regards
Calle

On 29 June 2017 at 02:25, Ime Asangansi  wrote:

> Hi Calle,
>
> Thanks for your question.
> Just curious (I don't have answer). As we are planning to do this soon.
>
> Are you doing this through UI or API?
> If API, which endpoint enables this?
>
> Thanks
>
> Ime
>
>
>
> On Jun 28, 2017 19:15, "Calle Hedberg"  wrote:
>
>> Hi
>>
>> I started importing around 700,000 events 2 days ago (about 5 mill
>> individual values) - and the import is still running, 48 hours later.
>> Import is slowing - it seemed to be importing around 7-8 values per second,
>> now down to 2-3 per second. It looks like that import might take around 200
>> hours - which makes no sense from a usability point of view. Importing
>> OrgUnits used to be equally slow, but a revamp recently change a 10,000
>> OrgUnit import from hours to something like 5 seconds.
>>
>> It seems obvious that event imports have been designed to import values
>> one by one, which is equivalent to asking people to fetch water using a
>> leaking tea-spoon.
>>
>> I've searched JIRA but cannot find any issue related to this, thus this
>> post: Are there any plans underway to make import of Event data usable for
>> larger record sets?
>>
>> Regards
>> Calle
>>
>> ***
>>
>> Calle Hedberg
>>
>> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>>
>> Tel/fax (home): +27-21-685-6472 <+27%2021%20685%206472>
>>
>> Cell: +27-82-853-5352 <+27%2082%20853%205352>
>>
>> Iridium SatPhone: +8816-315-19119 <+881%206%20315%2019119>
>>
>> Email: calle.hedb...@gmail.com
>>
>> Skype: calle_hedberg
>>
>> ***
>>
>>
>> ___
>> Mailing list: https://launchpad.net/~dhis2-devs
>> Post to : dhis2-devs@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~dhis2-devs
>> More help   : https://help.launchpad.net/ListHelp
>>
>>


-- 

***

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedb...@gmail.com

Skype: calle_hedberg

***
___
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help   : https://help.launchpad.net/ListHelp

Re: [Dhis2-devs] Import of Events is extremely slow - any plans to improve it?

[Ime Asangansi]

Hi Calle,

Thanks for your question.
Just curious (I don't have answer). As we are planning to do this soon.

Are you doing this through UI or API?
If API, which endpoint enables this?

Thanks

Ime



On Jun 28, 2017 19:15, "Calle Hedberg"  wrote:

> Hi
>
> I started importing around 700,000 events 2 days ago (about 5 mill
> individual values) - and the import is still running, 48 hours later.
> Import is slowing - it seemed to be importing around 7-8 values per second,
> now down to 2-3 per second. It looks like that import might take around 200
> hours - which makes no sense from a usability point of view. Importing
> OrgUnits used to be equally slow, but a revamp recently change a 10,000
> OrgUnit import from hours to something like 5 seconds.
>
> It seems obvious that event imports have been designed to import values
> one by one, which is equivalent to asking people to fetch water using a
> leaking tea-spoon.
>
> I've searched JIRA but cannot find any issue related to this, thus this
> post: Are there any plans underway to make import of Event data usable for
> larger record sets?
>
> Regards
> Calle
>
> ***
>
> Calle Hedberg
>
> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>
> Tel/fax (home): +27-21-685-6472
>
> Cell: +27-82-853-5352
>
> Iridium SatPhone: +8816-315-19119
>
> Email: calle.hedb...@gmail.com
>
> Skype: calle_hedberg
>
> ***
>
>
> ___
> Mailing list: https://launchpad.net/~dhis2-devs
> Post to : dhis2-devs@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~dhis2-devs
> More help   : https://help.launchpad.net/ListHelp
>
>
___
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help   : https://help.launchpad.net/ListHelp

[Dhis2-devs] Import of Events is extremely slow - any plans to improve it?

[Calle Hedberg]

Hi

I started importing around 700,000 events 2 days ago (about 5 mill
individual values) - and the import is still running, 48 hours later.
Import is slowing - it seemed to be importing around 7-8 values per second,
now down to 2-3 per second. It looks like that import might take around 200
hours - which makes no sense from a usability point of view. Importing
OrgUnits used to be equally slow, but a revamp recently change a 10,000
OrgUnit import from hours to something like 5 seconds.

It seems obvious that event imports have been designed to import values one
by one, which is equivalent to asking people to fetch water using a leaking
tea-spoon.

I've searched JIRA but cannot find any issue related to this, thus this
post: Are there any plans underway to make import of Event data usable for
larger record sets?

Regards
Calle

***

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedb...@gmail.com

Skype: calle_hedberg

***
___
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help   : https://help.launchpad.net/ListHelp