[Dhis2-devs] Visualisation of Data Element of Value Type - Organisation Unit

[Archana Chillala]

Hi,

We are now using the latest 2.25 DHIS version. The new feature "*Data
Element of value type organisation unit*" proves to be extremely useful to
us. We have a use-case where we want to record the geo-locations (villages)
of each individual patient at the Heath Centre they attend. And this
feature seems a right approach for us, to be used for this purpose where
these locations will be part of the master organisation unit hierarchy and
can be selected from the list shown for the data element.

We have configured a tracker type data element of value type 'org unit',
assigned to a Program and stored some data. However, when we tried to
visualise this data collected for that data element, through Event Reports
and Event Visualiser app, it gives an error saying "No series items
selected". Also, since the value of the data element is a string (which is
the UID of the org. unit), we are not clear as to how we could visualise
this data element. Ideally, what we need is a COUNT of the number of times
a geo-location (some org. unit) selected.

We also tried to use program indicators to form an expression that can give
the desired result. But, that data element doesn't even come up in the box,
where we choose data elements and form expressions. Only number type data
elements show up here.

Is there any other way we could use aggregation and visualisation for this
new data element type?
Could you also please let us know the road map for the visualisation
capabilities of this new data element.

Any help here would be appreciated.



*Cheers,*

Archana Chillala
Application Developer
Email archa...@thoughtworks.com
Telephone +91 9100960533 <+91+9100960533>
[image: ThoughtWorks]

___
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help   : https://help.launchpad.net/ListHelp

Re: [Dhis2-devs] Configuration of approvals in 2.25

[Lars Helge Øverland]

Hi Jason,

could you tell us the exact build date for this WAR file?

Could you also try again with the latest 2.25 WAR build?

We have introduced this property "categoryOptionGroupSet" in master, but
should not be there in 2.25.

regards,

Lars


On Sun, Oct 23, 2016 at 12:32 AM, Jason Pickering <
jason.p.picker...@gmail.com> wrote:

> I am trying to configure a workflow for population projections in 2.25.
> First, I created a new approval worfklow called "Population" with a yearly
> frequency.
>
>
> I  then created a new approval level in the UI like this
>
> [image: Inline image 1]
>
> We are not using and category options here. In the logs I see:
>
> * INFO  2016-10-23 06:27:23,519 (admin) Import:Start
> (DefaultMetadataImportService.java [http-bio-127.0.0.1-8090-exec-4])
> * INFO  2016-10-23 06:27:23,860 (admin) Import:Preheat[REFERENCE] took
> 0.33 seconds (DefaultPreheatService.java [http-bio-127.0.0.1-8090-exec-4])
> * INFO  2016-10-23 06:27:23,864 (admin) Import:Validation took 0.00
> seconds (DefaultObjectBundleValidationService.java
> [http-bio-127.0.0.1-8090-exec-4])
> * INFO  2016-10-23 06:27:23,890 (admin) Creating 1 object(s) of type
> DataApprovalWorkflow (DefaultObjectBundleService.java
> [http-bio-127.0.0.1-8090-exec-4])
> * INFO  2016-10-23 06:27:24,067 (admin) Import:Commit took 0.20 seconds
> (DefaultMetadataImportService.java [http-bio-127.0.0.1-8090-exec-4])
> * INFO  2016-10-23 06:27:24,067 (admin) Import:Done took 0.55 seconds
> (DefaultMetadataImportService.java [http-bio-127.0.0.1-8090-exec-4])
> * INFO  2016-10-23 06:27:39,034 (admin) Import:Start
> (DefaultMetadataImportService.java [http-bio-127.0.0.1-8090-exec-4])
> * INFO  2016-10-23 06:27:40,309 (admin) Import:Preheat[REFERENCE] took
> 1.27 seconds (DefaultPreheatService.java [http-bio-127.0.0.1-8090-exec-4])
> * INFO  2016-10-23 06:27:40,314 (admin) Import:Validation took 0.00
> seconds (DefaultObjectBundleValidationService.java
> [http-bio-127.0.0.1-8090-exec-4])
> * INFO  2016-10-23 06:27:40,314 (admin) Import:Done took 1.28 seconds
> (DefaultMetadataImportService.java [http-bio-127.0.0.1-8090-exec-4])
>
> which seems to be OK, but in the UI, I get a message that creation of the
> approval level failed and in the JS console I see
>
> {"httpStatus":"Conflict","httpStatusCode":409,"status":"WARNING","message":"One
> more more errors occurred, please see full details in import
> report.","response":{"responseType":"ObjectReport","
> uid":"ADOd9xfvDjH","klass":"org.hisp.dhis.dataapproval.
> DataApprovalLevel","errorReports":[{"message":"Missing required property
> `categoryOptionGroupSet`.","mainKlass":"org.hisp.dhis.dataapproval.
> DataApprovalLevel","errorKlass":"org.hisp.dhis.dataelement.
> CategoryOptionGroupSet","errorCode":"E4000"}]}}
>
> So, I guess the question is, how do I configure a data approval workflow
> with no category option group?
>
> Regards,
> Jason
>
>
>
> --
> Jason P. Pickering
> email: jason.p.picker...@gmail.com
> tel:+46764147049
>
> ___
> Mailing list: https://launchpad.net/~dhis2-devs
> Post to : dhis2-devs@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~dhis2-devs
> More help   : https://help.launchpad.net/ListHelp
>
>


-- 
Lars Helge Øverland
Lead developer, DHIS 2
University of Oslo
Skype: larshelgeoverland
l...@dhis2.org
http://www.dhis2.org 
___
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help   : https://help.launchpad.net/ListHelp

Re: [Dhis2-devs] Removing the limit on the number of users to send a message

[Lars Helge Øverland]

Hi there,

yes we can remove this for 2.26.

regards,

Lars


On Mon, Oct 24, 2016 at 4:02 AM, Calle Hedberg 
wrote:

> Hi,
>
> Just to add one piece of info here: Having the ability to email or
> whatsapp more than 200-250 users is going to be critical for disease
> surveillance systems too. In case of a major disease outbreak, these
> systems will have to broadcast messages via multiple protocols to far more
> than 200-250 people.
>
> Regards
> Calle
>
> On 24 October 2016 at 08:29, Sean Broomhead  wrote:
>
>> Hello Lars and team - thanks for having a look at this for us.
>>
>> The requirement has emerged out of the success of the system Mike built
>> to manage placement allocations for all Interns and Community Service
>> Doctors in South Africa. Newly qualified doctors apply centrally,
>> expressing their priority choices for hospital placements for their two
>> years of internship and one year of community service. We run an algorithm
>> to determine which hospital each doctor gets sent to.
>>
>> Both doctors and government love the system. It's email tool now provides
>> the formal mechanism for communicating with these doctors about the
>> placement process. Unfortunately, with the batch size limits, we're having
>> to setup the small batches manually. The current cohort of doctors is about
>> 4,000 and the next cohort of nurses could reach 10,000.
>>
>> Please advise if this needs to go into Launchpad or GitHub, or if you
>> deal with this sort of request differently.
>>
>> Your assistance will be very helpful.
>>
>> Best regards,
>> Sean
>>
>> On Mon, Oct 24, 2016 at 8:03 AM, Mike Nelushi 
>> wrote:
>>
>>> Hi Lars/Morten,
>>>
>>> We want to remove the limit on the number of users to send a DHIS2(2.23)
>>> message. At the moment the limit is 200 users per group.
>>>
>>> I would like to implement this change since we have a system that need
>>> to send 10 000+ emails to our users for notifications.
>>>
>>> Was the limitation created to resolve server health issues when sending
>>> bulk emails?
>>>
>>> Regards,
>>> Mike
>>>
>>>
>>
>>
>>
>> --
>> Dr Sean Broomhead
>> Director ICT
>> HISP
>> +27 79 496 1993
>> Skype: getsean
>>
>> *This message and any attachments are subject to a disclaimer published
>> at http://www.hisp.org/policies.html#comms_disclaimer
>> .  Please read the
>> disclaimer before opening any attachment or taking any other action in
>> terms of this electronic transmission.  If you cannot access the
>> disclaimer, kindly send an email to disclai...@hisp.org
>>  and a copy will be provided to you. By replying to
>> this e-mail or opening any attachment you agree to be bound by the
>> provisions of the disclaimer.*
>>
>
>
>
> --
>
> ***
>
> Calle Hedberg
>
> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>
> Tel/fax (home): +27-21-685-6472
>
> Cell: +27-82-853-5352
>
> Iridium SatPhone: +8816-315-19119
>
> Email: calle.hedb...@gmail.com
>
> Skype: calle_hedberg
>
> ***
>
>


-- 
Lars Helge Øverland
Lead developer, DHIS 2
University of Oslo
Skype: larshelgeoverland
l...@dhis2.org
http://www.dhis2.org 
___
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help   : https://help.launchpad.net/ListHelp

Re: [Dhis2-devs] [Dhis2-users] 25 hours in completing Analytic

[Lars Helge Øverland]

Hi there,

the analytics table generation is bound by db indexing speed / disk write
speed. It scales almost linearly with the number of CPU cores available and
disk write speed.

regards,

Lars



On Mon, Oct 24, 2016 at 7:33 AM, Neeraj Gupta  wrote:

> Hi Calle,
>
> Thanks for all this.
>
> Yes, we will have a test server available with us by first week of
> November, I can start test scenarios suggested by you after that.
>
> Thanks,
> Neeraj
>
> On Mon, Oct 24, 2016 at 3:50 PM, Calle Hedberg 
> wrote:
>
>> Neeraj,
>>
>> Thanks for that, it is useful info.
>>
>> The instance has a very skewed balance between data elements/catcombos
>> and indicators, but what else is new - outside of South Africa, most
>> countries have the same: they collect a huge number or data elements but
>> turn very few of them into indicators (SA collect very very few data
>> elements that are NOT part of any indicator). But that is a separate
>> discussion.
>>
>> I would NOT expect the number of data elements & catcombos + the number
>> of orgunits to have a significant impact on analytics performance, but that
>> needs to be verified.
>>
>> I WOULD expect the number of indicators to have a significant impact, so
>> if you doubled or quadrupled the number of indicators I would expect a
>> significant impact on analytics time
>>
>> Using your and my numbers as a baseline, I would envision the following
>> tests to be revealing:
>>
>> Baseline system 1: SSD, 4-core CPU: 100 mill values, 400 DEs, 250
>> indicators, 30,000 OUs, few catcombos -> 1 hour for analytics processing
>>
>> Baseline system 2: SSD, 12-core CPU: 500 mill values, 3,500 DEs, 15,000
>> OUs, 25 catcombos -> ~10 hours for analytics processing
>>
>> Test 1: Using system 1, reduce OUs to 15,000 - run analytics (hypothesis:
>> no significant difference)
>> Test 2: Using system 1, reduce indicators to 125 - run analytics, then to
>> 0 - run analytics (hypothesis: analytics reduced to 45 min)
>> Test 3: Using system 1, increase datavalues from 100mill to 500 mill by
>> introducing additional attributecombos (easy) - run analytics (hypothesis:
>> ~15 hours for analytics processing, with 4-cores instead of 12-cores)
>> Test 4: Using system 1, increase datavalues to 1 billion - run analytics
>> (hypothesis: 35 hours for analytics processing).
>>
>> Neeraj, I don't know if you have a similar sandbox server available for
>> testing, but if you do:
>>
>> Test 5: Using system 2, increase number of indicators to 600 (just export
>> the 200 you have, modify the names and uids a bit, and re-import - for the
>> purpose of this test, it does not matter that many of the indicators have
>> identical formulas). Hypothesis: Analytics time up at least 20%
>> Test 6: using system 2 (with 200 indicators), shift half of the 500 mill
>> data values forward or backward in time resulting in twice the number of
>> analytics tables at half the size. (hypothesis: slight reduction in
>> processing time)
>> Test 7: Using system 2 (with 200 indicators), increase datavalues to 1
>> billion using a new attributecombo - run analytics (hypothesis: analytics
>> time up to 25 hours again).
>>
>> As far as I've seen, the bulk of analytics processing time is taken up by
>> indexing - which I would expect to follow a moderately quadratic curve in
>> terms of analytics table sizes.
>>
>> Best regards
>> Calle
>>
>> On 24 October 2016 at 10:39, Neeraj Gupta  wrote:
>>
>>> Calle,
>>>
>>> We have around 500 million record in database with 3666 data elements
>>> having 26 category combinations and 201 indicators and there are 14398
>>> organisation units.
>>>
>>> Thanks,
>>> Neeraj
>>>
>>> On Mon, Oct 24, 2016 at 1:50 PM, Calle Hedberg 
>>> wrote:
>>>
 Neeraj,

 You never stated the number of records you have in the datavalue table
 - what is it?

 In the same context: anybody have a rough idea of how many datavalue
 records there are in the global DATIM database - which I think currently
 might be the largest DHIS2 instance around?

 Given our own recent work on performance + what Neeraj has reported,
 I've been thinking of creating one test instance with let us say 500 mill
 datavalue records and another with let us say 1 billion, then use them to
 identify key bottlenecks in various processes AND use them to ensure that
 DHIS2 analytics performance is as linear as possible in terms of database
 size. Postgresql has introduced a number of new indexing algorithms in
 recent versions, and I'm not sure if DHIS2 is taking full advantage of 
 them.

 Best regards
 Calle

 On 24 October 2016 at 07:53, Brajesh Murari 
 wrote:

> Congratulation Neeraj and team ...it much appreciated
>
> On Mon, Oct 24, 2016 at 11:08 AM, Neeraj Gupta 
> wrote:
>
>> Dear 

[Dhis2-devs] Bug 2.25. Error when entering a program dashboard through the relationship link (for entities enrolled in more than one program)

[Alejandro]

Hi,

Tested against https://play.dhis2.org/demo. In Tracker Capture, when there
is a relationship, and we click in the existing link to go to the entity
program dashboards, if that entity is enrolled in more than one program,
the software does not enter in any of them by default. But, it does not
display any active enrollments either...

[image: Inline image 1]

When we select one of the programs in which the entity is enrolled (see
image below), the other enrollments appear again in the section Other
programs.

[image: Inline image 2]

Cheers!
Alejandro
___
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help   : https://help.launchpad.net/ListHelp

Re: [Dhis2-devs] DHIS 2 V 2.24 - /api/24/currentUser not available

[Lorill Crees]

Hi Morten,

Thanks for this. We are getting the user roles from the user credentials,
and I've confirmed this is available under /api/me and /api/24/me.

Thanks,

Lorill

On Sun, Oct 23, 2016 at 7:49 PM, Morten Olav Hansen 
wrote:

> Hi Lorill
>
> So there's a couple of things to note here:
>
> 1) You probably want to use the name `me` instead of `currentUser` as it'
> the new name for it (and has been for a while, just being aliased to
> `currentUser``
> 2) In 224 we included a new `me` endpoint, which can now be reached at
> `/api/24/me` and `/api/25/me`
>
> There seems to be a bug here though, that new endpoint should also be
> available at `/api/me` (and not `/api/currentUser`)
>
> That said, the new endpoint is a bit different. What are you using this
> endpoint for? is it just to get info about current user, or is it to use
> some of its other capabilities ? like data set, program availability
> (`/api/23/currentUser/programs` `/api/23/currentUser/dataSet`)
>
> --
> Morten Olav Hansen
> Senior Engineer, DHIS 2
> University of Oslo
> http://www.dhis2.org
>
> On Tue, Oct 18, 2016 at 10:46 PM, Lorill Crees  wrote:
>
>> Hi,
>>
>> We are using 2.24 revision e2f8d0e and the endpoint /api/24/currentUser
>> is not available. However, these two endpoints *are* available:
>>
>>- /api/currentUser
>>- /api/23/currentUser
>>
>> Is it possible to add the endpoint /api/24/currentUser, or is there a
>> reason it is missing?
>>
>> Thanks,
>>
>> Lorill
>>
>> --
>> Lorill Crees
>> Project Leader / Senior Developer
>> 2Paths Solutions Ltd. 
>>
>> lcr...@2paths.com
>> skype: lorill2paths
>> (604) 689-4123 x 15
>>
>> ___
>> Mailing list: https://launchpad.net/~dhis2-devs
>> Post to : dhis2-devs@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~dhis2-devs
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>


-- 
Lorill Crees
Project Leader / Senior Developer
2Paths Solutions Ltd. 

lcr...@2paths.com
skype: lorill2paths
(604) 689-4123 x 15
___
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help   : https://help.launchpad.net/ListHelp

[Dhis2-devs] Event report tables added to dashboards are displayed 'blank' when period is selected as Start-End dates (2.25)

[Alejandro]

Hi!


We have noticed (tested in https://play.dhis2.org) that any event report
table that is defined with a period type *'Start/end dates'* instead of*
'Fixed and relative periods' *is displayed 'blank' when added to a
dashboard (see image below).


[image: Inline image 1]

The error generated is:


{"httpStatus":"Conflict","httpStatusCode":409,"status":"ERROR","message":"Start
date or end date is invalid: 2016-07-24T00:00:00.000 -
2016-10-24T00:00:00.000"}


It seems to be related to date format, since removing the bold text in
the original query (see below), it works!

https://play.dhis2.org/demo/api/analytics/events/query/eBAyeGv0exc.json?stage=Zj7UnCAulEk=ou:O6uvpzGd5pu=eMyVanycQSC=qrur9Dvnyt5-Yf6UHoPkdS6=tUdBD1JDxpn=sGna2pquXOO=Kswd1r4qWLh=gWxh7DiRmG7=x7PaHGvgWY2=XCMi7Wvnplm=hlPt8H4bUOQ=Thkx2BnO5Kq=Y7hKDSuqEtH=K6uUAvq500H=msodh3rEMJa=oZg33kd9taw=GieVkTxp4HH-TBxGTceyzwy=HS8QXAJtuKV=fWIAEtYVEGk=SWfdB5lX0fk=vV9UWAZohSf-OrkEzxZEH4X=2016-07-24*T00:00:00.000*=2016-10-24*T00:00:00.000*=50=1=NAME


Cheers!

Alejandro
___
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help   : https://help.launchpad.net/ListHelp

[Dhis2-devs] Aggregation for Program Indicators

[Olivier Cheminat]

Dear all,

MSF-OCP is using two types of data entry:

  *   Aggregate data entry on a weekly basis
  *   Event data based on the date of event

We are currently using the 2.22 version. In that version, we are using 
aggregation queries that aggregate event data into data sets that have a weekly 
time period.

We want to upgrade DHIS2 to 2.23 soon. As aggregation queries are not supported 
anymore in the 2.23 version, we will replace them with program indicators. We 
started to create several program indicators, but we are facing a problem.
When we create monthly reports the monthly totals don't match with the sum of 
the aggregated weekly totals belonging to that month. For example:

  *   Aggregate data for 2016W39 will be aggregated into 2016 September when 
displayed per month.
  *   Events with the date 2016-10-01 will be aggregated into 2016W39 when 
displayed by week but aggregated into 2016 October when displayed by month.
  *   This makes it confusing for end users because for aggregate data 
September 2016 is the sum of the weeks 2016W36-2016W39 (28 days), whereas for 
event data September 2016 is the sum of events between 2016-09-01 and 
2016-09-30 (30 days).

Has anybody else experienced this problem, and is there a way to have these 
program indicators aggregated by month using ISO weeks?

Thank you.
Olivier.

Olivier Cheminat
eHealth Referent & Knowledge Coordinator
Médecins Sans Frontières - OCP

___
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help   : https://help.launchpad.net/ListHelp

Re: [Dhis2-devs] [Dhis2-users] Fwd: Secundary geographical data needed for analysis

[Marc Garnica]

The fact is that we want 2 different geographical data. The one you are
mentioning refers to "where the data comes from" which as you said we can
easily specify it from the org unit tree.

But then we also want also to have information as "where the patient comes
from" which is different from the first one.

So for example entering data at a level of a Village XXX, doesn't mean that
the patient comes from Village XXX. It only means that the data (the
medical registration or smth related) comes from the Village XXX. But it
may be (mostly) that the patient doesn't come from Village XXX, he comes
from from Village YYY. So the user has to specify this Village YYY in the
form.

Once the user specifies the village YYY in the form we want to be able to
analyise and be able to fulfill the requirements I've sent before. We have
these 3 options but none of them are specially accurate.

I don't know if I have explained well, thanks a lot
Marc

2016-10-24 13:55 GMT+02:00 Prosper BT :

> Thanks Marc for sharing the reporting requirements.
>
> Am not sure how many villages you have, but feasible workable solution
> would be to create Villages as Orgunits and have data entered at that level.
>
> Regards
>
>
>
>
>
> Prosper Behumbiize, MPH
> DHIS2 Implementation| HISP Uganda/University Of Oslo
> +256 752 751 776 | +256 776 139 139
> pros...@hispuganda.org  | pros...@dhis2.org | Skype:
> prospertb
>
> On Mon, Oct 24, 2016 at 2:03 PM, Marc Garnica 
> wrote:
>
>> The *two main requirements* are:
>>
>> 1.   To create a table with the aggregated number of cases/patients
>> coming from a village and the corresponding incidence rate.
>>
>> 2.   To create a map showing the number of cases/patient coming from
>> the different villages.
>>
>>
>> Thanks,
>>
>> Marc
>>
>> 2016-10-24 12:52 GMT+02:00 Prosper BT :
>>
>>> Dear Marc,
>>>
>>> Can you share your analysis plan?
>>> How do you intend to analyze and present the data?
>>>
>>> Regards
>>>
>>> Prosper Behumbiize, MPH
>>> DHIS2 Implementation| HISP Uganda/University Of Oslo
>>> +256 752 751 776 | +256 776 139 139
>>> pros...@hispuganda.org  | pros...@dhis2.org | Skype:
>>> prospertb
>>>
>>> On Mon, Oct 24, 2016 at 10:42 AM, Marc Garnica 
>>> wrote:
>>>
 Any ideas on that?

 Thanks a lot,
 Marc

 -- Forwarded message --
 From: Marc Garnica 
 Date: 2016-10-21 17:03 GMT+02:00
 Subject: Secundary geographical data needed for analysis
 To: dhis2-us...@lists.launchpad.net


 Hi all,
 First of all, thank you for the support and the new release, it really
 has some nice new features that will help us a lot!. We want to share with
 you a discussion we have been developing this week to see if you can give
 us some advice.

 For every single data entry (individual with or without registration
 and aggregated data) we always need at least one geographical data which
 refers to where the data comes from. This is information can be selected in
 the interface through the Org unit tree in the left.


 But we sometimes need and additional geographical data referring to
 where the patient comes from. In this case we add a data element to add
 this data. This information is so important to create great analysis and
 reports so we need to think well how we want to collect this geographical
 data.


 The first option we did was just create a typed TEXT data element where
 the user could enter manually the village where the patient came from. But
 obviously, this was not correct enough because the user can write the same
 village in more than one spelling way and also there was no feasible way to
 use this data element as a dimension in the analysis so we were not able to
 map the data in a map of villages or create a table with the cases by
 village. So we need to formulate a new treatment


 *POSSIBILITIES*

 *1.   **CAPTURE THE VILLAGE AS A SET OF COORDINATES*

 This options consists in create a data element with type COORDINATES
 and add it to the form. With this options we can add a the geographical
 information of where the patient comes from by providing the coordinates of
 the village or even better (in new versions of DHIS2) we can click on the
 map, search for the village in the search bar and capture the coordinates
 of it visually.

 Nice feature but then the village information is only stored as a pair
 of coordinates, there is no auxiliary information as for instance its name.
 In the analytical point of view we can now show the cases in a map
 distributed by village (which is nice) but we cannot create the desired
 table with the cases by village.

 Another issue in this option is 

Re: [Dhis2-devs] [Dhis2-users] Fwd: Secundary geographical data needed for analysis

[Prosper BT]

Thanks Marc

You explained well then I guess you have to implement both the Orgunit tree
for where then

For patient location, you may either use the new Orgunit data element type
if it happens that the same villages in the Orgunit tree are the same as
where the patients come from.

Alternatively they have to option sets

Regards

Prosper Behumbiize, MPH
DHIS2 Implementation| HISP Uganda/University Of Oslo
+256 752 751 776 | +256 776 139 139
pros...@hispuganda.org  | pros...@dhis2.org | Skype:
prospertb

On Mon, Oct 24, 2016 at 3:17 PM, Marc Garnica 
wrote:

> The fact is that we want 2 different geographical data. The one you are
> mentioning refers to "where the data comes from" which as you said we can
> easily specify it from the org unit tree.
>
> But then we also want also to have information as "where the patient comes
> from" which is different from the first one.
>
> So for example entering data at a level of a Village XXX, doesn't mean
> that the patient comes from Village XXX. It only means that the data (the
> medical registration or smth related) comes from the Village XXX. But it
> may be (mostly) that the patient doesn't come from Village XXX, he comes
> from from Village YYY. So the user has to specify this Village YYY in the
> form.
>
> Once the user specifies the village YYY in the form we want to be able to
> analyise and be able to fulfill the requirements I've sent before. We have
> these 3 options but none of them are specially accurate.
>
> I don't know if I have explained well, thanks a lot
> Marc
>
> 2016-10-24 13:55 GMT+02:00 Prosper BT :
>
>> Thanks Marc for sharing the reporting requirements.
>>
>> Am not sure how many villages you have, but feasible workable solution
>> would be to create Villages as Orgunits and have data entered at that level.
>>
>> Regards
>>
>>
>>
>>
>>
>> Prosper Behumbiize, MPH
>> DHIS2 Implementation| HISP Uganda/University Of Oslo
>> +256 752 751 776 | +256 776 139 139
>> pros...@hispuganda.org  | pros...@dhis2.org | Skype:
>> prospertb
>>
>> On Mon, Oct 24, 2016 at 2:03 PM, Marc Garnica 
>> wrote:
>>
>>> The *two main requirements* are:
>>>
>>> 1.   To create a table with the aggregated number of cases/patients
>>> coming from a village and the corresponding incidence rate.
>>>
>>> 2.   To create a map showing the number of cases/patient coming
>>> from the different villages.
>>>
>>>
>>> Thanks,
>>>
>>> Marc
>>>
>>> 2016-10-24 12:52 GMT+02:00 Prosper BT :
>>>
 Dear Marc,

 Can you share your analysis plan?
 How do you intend to analyze and present the data?

 Regards

 Prosper Behumbiize, MPH
 DHIS2 Implementation| HISP Uganda/University Of Oslo
 +256 752 751 776 | +256 776 139 139
 pros...@hispuganda.org  | pros...@dhis2.org | Skype:
 prospertb

 On Mon, Oct 24, 2016 at 10:42 AM, Marc Garnica  wrote:

> Any ideas on that?
>
> Thanks a lot,
> Marc
>
> -- Forwarded message --
> From: Marc Garnica 
> Date: 2016-10-21 17:03 GMT+02:00
> Subject: Secundary geographical data needed for analysis
> To: dhis2-us...@lists.launchpad.net
>
>
> Hi all,
> First of all, thank you for the support and the new release, it really
> has some nice new features that will help us a lot!. We want to share with
> you a discussion we have been developing this week to see if you can give
> us some advice.
>
> For every single data entry (individual with or without registration
> and aggregated data) we always need at least one geographical data which
> refers to where the data comes from. This is information can be selected 
> in
> the interface through the Org unit tree in the left.
>
>
> But we sometimes need and additional geographical data referring to
> where the patient comes from. In this case we add a data element to add
> this data. This information is so important to create great analysis and
> reports so we need to think well how we want to collect this geographical
> data.
>
>
> The first option we did was just create a typed TEXT data element
> where the user could enter manually the village where the patient came
> from. But obviously, this was not correct enough because the user can 
> write
> the same village in more than one spelling way and also there was no
> feasible way to use this data element as a dimension in the analysis so we
> were not able to map the data in a map of villages or create a table with
> the cases by village. So we need to formulate a new treatment
>
>
> *POSSIBILITIES*
>
> *1.   **CAPTURE THE VILLAGE AS A SET OF COORDINATES*
>
> This options consists in create a data 

Re: [Dhis2-devs] [Dhis2-users] Fwd: Secundary geographical data needed for analysis

[Prosper BT]

Thanks Marc for sharing the reporting requirements.

Am not sure how many villages you have, but feasible workable solution
would be to create Villages as Orgunits and have data entered at that level.

Regards





Prosper Behumbiize, MPH
DHIS2 Implementation| HISP Uganda/University Of Oslo
+256 752 751 776 | +256 776 139 139
pros...@hispuganda.org  | pros...@dhis2.org | Skype:
prospertb

On Mon, Oct 24, 2016 at 2:03 PM, Marc Garnica 
wrote:

> The *two main requirements* are:
>
> 1.   To create a table with the aggregated number of cases/patients
> coming from a village and the corresponding incidence rate.
>
> 2.   To create a map showing the number of cases/patient coming from
> the different villages.
>
>
> Thanks,
>
> Marc
>
> 2016-10-24 12:52 GMT+02:00 Prosper BT :
>
>> Dear Marc,
>>
>> Can you share your analysis plan?
>> How do you intend to analyze and present the data?
>>
>> Regards
>>
>> Prosper Behumbiize, MPH
>> DHIS2 Implementation| HISP Uganda/University Of Oslo
>> +256 752 751 776 | +256 776 139 139
>> pros...@hispuganda.org  | pros...@dhis2.org | Skype:
>> prospertb
>>
>> On Mon, Oct 24, 2016 at 10:42 AM, Marc Garnica 
>> wrote:
>>
>>> Any ideas on that?
>>>
>>> Thanks a lot,
>>> Marc
>>>
>>> -- Forwarded message --
>>> From: Marc Garnica 
>>> Date: 2016-10-21 17:03 GMT+02:00
>>> Subject: Secundary geographical data needed for analysis
>>> To: dhis2-us...@lists.launchpad.net
>>>
>>>
>>> Hi all,
>>> First of all, thank you for the support and the new release, it really
>>> has some nice new features that will help us a lot!. We want to share with
>>> you a discussion we have been developing this week to see if you can give
>>> us some advice.
>>>
>>> For every single data entry (individual with or without registration and
>>> aggregated data) we always need at least one geographical data which refers
>>> to where the data comes from. This is information can be selected in the
>>> interface through the Org unit tree in the left.
>>>
>>>
>>> But we sometimes need and additional geographical data referring to
>>> where the patient comes from. In this case we add a data element to add
>>> this data. This information is so important to create great analysis and
>>> reports so we need to think well how we want to collect this geographical
>>> data.
>>>
>>>
>>> The first option we did was just create a typed TEXT data element where
>>> the user could enter manually the village where the patient came from. But
>>> obviously, this was not correct enough because the user can write the same
>>> village in more than one spelling way and also there was no feasible way to
>>> use this data element as a dimension in the analysis so we were not able to
>>> map the data in a map of villages or create a table with the cases by
>>> village. So we need to formulate a new treatment
>>>
>>>
>>> *POSSIBILITIES*
>>>
>>> *1.   **CAPTURE THE VILLAGE AS A SET OF COORDINATES*
>>>
>>> This options consists in create a data element with type COORDINATES and
>>> add it to the form. With this options we can add a the geographical
>>> information of where the patient comes from by providing the coordinates of
>>> the village or even better (in new versions of DHIS2) we can click on the
>>> map, search for the village in the search bar and capture the coordinates
>>> of it visually.
>>>
>>> Nice feature but then the village information is only stored as a pair
>>> of coordinates, there is no auxiliary information as for instance its name.
>>> In the analytical point of view we can now show the cases in a map
>>> distributed by village (which is nice) but we cannot create the desired
>>> table with the cases by village.
>>>
>>> Another issue in this option is that for the end-user sometimes it will
>>> be difficult to interact with the map and be able to capture the correct
>>> village.
>>>
>>> *2.   **CAPTURE THE VILLAGE AS A ORGANISATION  UNIT*
>>>
>>> This option appeared once 2.25 DHIS2 version was released with a new
>>> Organisation Unit Type for data elements. Using that, we can create a new
>>> data element with type = Organisation unit to capture the information of
>>> where the patient comes from.
>>>
>>> So in this option we can have a user with data capture permission only
>>> on the Sierra Leona hierarchy of organisation units but able to select a
>>> village for example in Benin hierarchy of organisation units.
>>>
>>> Even though the Organisation unit type data element is the more natural
>>> way to specify where the patient comes from, we experienced some problems.
>>> Firstly, we are not able to use this data as a dimension for aggregate
>>> events information (for instance knowing how many events are registered
>>> with a patient coming from X village). And secondly, this option will mean
>>> to add all the villages included in our area of 

Re: [Dhis2-devs] [Dhis2-users] 25 hours in completing Analytic

[Neeraj Gupta]

Hi Calle,

Thanks for all this.

Yes, we will have a test server available with us by first week of
November, I can start test scenarios suggested by you after that.

Thanks,
Neeraj

On Mon, Oct 24, 2016 at 3:50 PM, Calle Hedberg 
wrote:

> Neeraj,
>
> Thanks for that, it is useful info.
>
> The instance has a very skewed balance between data elements/catcombos and
> indicators, but what else is new - outside of South Africa, most countries
> have the same: they collect a huge number or data elements but turn very
> few of them into indicators (SA collect very very few data elements that
> are NOT part of any indicator). But that is a separate discussion.
>
> I would NOT expect the number of data elements & catcombos + the number of
> orgunits to have a significant impact on analytics performance, but that
> needs to be verified.
>
> I WOULD expect the number of indicators to have a significant impact, so
> if you doubled or quadrupled the number of indicators I would expect a
> significant impact on analytics time
>
> Using your and my numbers as a baseline, I would envision the following
> tests to be revealing:
>
> Baseline system 1: SSD, 4-core CPU: 100 mill values, 400 DEs, 250
> indicators, 30,000 OUs, few catcombos -> 1 hour for analytics processing
>
> Baseline system 2: SSD, 12-core CPU: 500 mill values, 3,500 DEs, 15,000
> OUs, 25 catcombos -> ~10 hours for analytics processing
>
> Test 1: Using system 1, reduce OUs to 15,000 - run analytics (hypothesis:
> no significant difference)
> Test 2: Using system 1, reduce indicators to 125 - run analytics, then to
> 0 - run analytics (hypothesis: analytics reduced to 45 min)
> Test 3: Using system 1, increase datavalues from 100mill to 500 mill by
> introducing additional attributecombos (easy) - run analytics (hypothesis:
> ~15 hours for analytics processing, with 4-cores instead of 12-cores)
> Test 4: Using system 1, increase datavalues to 1 billion - run analytics
> (hypothesis: 35 hours for analytics processing).
>
> Neeraj, I don't know if you have a similar sandbox server available for
> testing, but if you do:
>
> Test 5: Using system 2, increase number of indicators to 600 (just export
> the 200 you have, modify the names and uids a bit, and re-import - for the
> purpose of this test, it does not matter that many of the indicators have
> identical formulas). Hypothesis: Analytics time up at least 20%
> Test 6: using system 2 (with 200 indicators), shift half of the 500 mill
> data values forward or backward in time resulting in twice the number of
> analytics tables at half the size. (hypothesis: slight reduction in
> processing time)
> Test 7: Using system 2 (with 200 indicators), increase datavalues to 1
> billion using a new attributecombo - run analytics (hypothesis: analytics
> time up to 25 hours again).
>
> As far as I've seen, the bulk of analytics processing time is taken up by
> indexing - which I would expect to follow a moderately quadratic curve in
> terms of analytics table sizes.
>
> Best regards
> Calle
>
> On 24 October 2016 at 10:39, Neeraj Gupta  wrote:
>
>> Calle,
>>
>> We have around 500 million record in database with 3666 data elements
>> having 26 category combinations and 201 indicators and there are 14398
>> organisation units.
>>
>> Thanks,
>> Neeraj
>>
>> On Mon, Oct 24, 2016 at 1:50 PM, Calle Hedberg 
>> wrote:
>>
>>> Neeraj,
>>>
>>> You never stated the number of records you have in the datavalue table -
>>> what is it?
>>>
>>> In the same context: anybody have a rough idea of how many datavalue
>>> records there are in the global DATIM database - which I think currently
>>> might be the largest DHIS2 instance around?
>>>
>>> Given our own recent work on performance + what Neeraj has reported,
>>> I've been thinking of creating one test instance with let us say 500 mill
>>> datavalue records and another with let us say 1 billion, then use them to
>>> identify key bottlenecks in various processes AND use them to ensure that
>>> DHIS2 analytics performance is as linear as possible in terms of database
>>> size. Postgresql has introduced a number of new indexing algorithms in
>>> recent versions, and I'm not sure if DHIS2 is taking full advantage of them.
>>>
>>> Best regards
>>> Calle
>>>
>>> On 24 October 2016 at 07:53, Brajesh Murari 
>>> wrote:
>>>
 Congratulation Neeraj and team ...it much appreciated

 On Mon, Oct 24, 2016 at 11:08 AM, Neeraj Gupta 
 wrote:

> Dear Team, Thanks for all your suggestions.
>
> Now the time of analytic is reduced to 10 hours 41 minutes.
>
> We tried to VACUUM as Sam suggested but it didn't help then we
> upgraded postgres from 9.4 to 9.5.4 and as Calle and Bob suggested we made
> some changes in configuration file of postgres and it reduced the time. 
> But
> the database size is still same.
>
> 

[Dhis2-devs] Bug in 2.25. Removing dataelements from a dataset with sections

[Alejandro]

Hi!


I was trying to remove some dataelements from a dataset with sections in
2.25 (through the dataset menu) and I noticed the following:


-  You cannot see them in the dataset menu any more (as expected).

-  You cannot see them in the section menu any more (as expected).

-  They still appear in the data entry, when opening the just
modified dataset (all cache cleanings done).


I checked the database and it seems that the link between section and
dataelement is still there (table sectiondataelements). Possible solutions
would be to take this into account or not allow to remove a dataelement
from a dataset if it is still assigned to a section.


That behaviour has been reproduced in https://play.dhis2.org/


Regards!

Alejandro
___
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help   : https://help.launchpad.net/ListHelp

Re: [Dhis2-devs] [Dhis2-users] Fwd: Secundary geographical data needed for analysis

[Marc Garnica]

The *two main requirements* are:

1.   To create a table with the aggregated number of cases/patients
coming from a village and the corresponding incidence rate.

2.   To create a map showing the number of cases/patient coming from
the different villages.


Thanks,

Marc

2016-10-24 12:52 GMT+02:00 Prosper BT :

> Dear Marc,
>
> Can you share your analysis plan?
> How do you intend to analyze and present the data?
>
> Regards
>
> Prosper Behumbiize, MPH
> DHIS2 Implementation| HISP Uganda/University Of Oslo
> +256 752 751 776 | +256 776 139 139
> pros...@hispuganda.org  | pros...@dhis2.org | Skype:
> prospertb
>
> On Mon, Oct 24, 2016 at 10:42 AM, Marc Garnica 
> wrote:
>
>> Any ideas on that?
>>
>> Thanks a lot,
>> Marc
>>
>> -- Forwarded message --
>> From: Marc Garnica 
>> Date: 2016-10-21 17:03 GMT+02:00
>> Subject: Secundary geographical data needed for analysis
>> To: dhis2-us...@lists.launchpad.net
>>
>>
>> Hi all,
>> First of all, thank you for the support and the new release, it really
>> has some nice new features that will help us a lot!. We want to share with
>> you a discussion we have been developing this week to see if you can give
>> us some advice.
>>
>> For every single data entry (individual with or without registration and
>> aggregated data) we always need at least one geographical data which refers
>> to where the data comes from. This is information can be selected in the
>> interface through the Org unit tree in the left.
>>
>>
>> But we sometimes need and additional geographical data referring to where
>> the patient comes from. In this case we add a data element to add this
>> data. This information is so important to create great analysis and reports
>> so we need to think well how we want to collect this geographical data.
>>
>>
>> The first option we did was just create a typed TEXT data element where
>> the user could enter manually the village where the patient came from. But
>> obviously, this was not correct enough because the user can write the same
>> village in more than one spelling way and also there was no feasible way to
>> use this data element as a dimension in the analysis so we were not able to
>> map the data in a map of villages or create a table with the cases by
>> village. So we need to formulate a new treatment
>>
>>
>> *POSSIBILITIES*
>>
>> *1.   **CAPTURE THE VILLAGE AS A SET OF COORDINATES*
>>
>> This options consists in create a data element with type COORDINATES and
>> add it to the form. With this options we can add a the geographical
>> information of where the patient comes from by providing the coordinates of
>> the village or even better (in new versions of DHIS2) we can click on the
>> map, search for the village in the search bar and capture the coordinates
>> of it visually.
>>
>> Nice feature but then the village information is only stored as a pair of
>> coordinates, there is no auxiliary information as for instance its name. In
>> the analytical point of view we can now show the cases in a map distributed
>> by village (which is nice) but we cannot create the desired table with the
>> cases by village.
>>
>> Another issue in this option is that for the end-user sometimes it will
>> be difficult to interact with the map and be able to capture the correct
>> village.
>>
>> *2.   **CAPTURE THE VILLAGE AS A ORGANISATION  UNIT*
>>
>> This option appeared once 2.25 DHIS2 version was released with a new
>> Organisation Unit Type for data elements. Using that, we can create a new
>> data element with type = Organisation unit to capture the information of
>> where the patient comes from.
>>
>> So in this option we can have a user with data capture permission only on
>> the Sierra Leona hierarchy of organisation units but able to select a
>> village for example in Benin hierarchy of organisation units.
>>
>> Even though the Organisation unit type data element is the more natural
>> way to specify where the patient comes from, we experienced some problems.
>> Firstly, we are not able to use this data as a dimension for aggregate
>> events information (for instance knowing how many events are registered
>> with a patient coming from X village). And secondly, this option will mean
>> to add all the villages included in our area of research which may lead to
>> memory problems in our server.
>>
>> *3.   **CAPTURE THE VILLAGE AS A RELATIONSHIP WITH THE PATIENT*
>>
>> The 3rd option is to use the “Relationship” implemented in Tracker
>> Capture App. This app can create links between different entities
>> registered in the system; originally it was created for tracking a pregnant
>> patient and then registers to the system her new child. But we could use
>> this relationship in another way just registering villages in the system
>> and then linking these registrations with the patient. This is a more
>> complex way to implement our 

Re: [Dhis2-devs] [Dhis2-users] Fwd: Secundary geographical data needed for analysis

[Prosper BT]

Dear Marc,

Can you share your analysis plan?
How do you intend to analyze and present the data?

Regards

Prosper Behumbiize, MPH
DHIS2 Implementation| HISP Uganda/University Of Oslo
+256 752 751 776 | +256 776 139 139
pros...@hispuganda.org  | pros...@dhis2.org | Skype:
prospertb

On Mon, Oct 24, 2016 at 10:42 AM, Marc Garnica 
wrote:

> Any ideas on that?
>
> Thanks a lot,
> Marc
>
> -- Forwarded message --
> From: Marc Garnica 
> Date: 2016-10-21 17:03 GMT+02:00
> Subject: Secundary geographical data needed for analysis
> To: dhis2-us...@lists.launchpad.net
>
>
> Hi all,
> First of all, thank you for the support and the new release, it really has
> some nice new features that will help us a lot!. We want to share with you
> a discussion we have been developing this week to see if you can give us
> some advice.
>
> For every single data entry (individual with or without registration and
> aggregated data) we always need at least one geographical data which refers
> to where the data comes from. This is information can be selected in the
> interface through the Org unit tree in the left.
>
>
> But we sometimes need and additional geographical data referring to where
> the patient comes from. In this case we add a data element to add this
> data. This information is so important to create great analysis and reports
> so we need to think well how we want to collect this geographical data.
>
>
> The first option we did was just create a typed TEXT data element where
> the user could enter manually the village where the patient came from. But
> obviously, this was not correct enough because the user can write the same
> village in more than one spelling way and also there was no feasible way to
> use this data element as a dimension in the analysis so we were not able to
> map the data in a map of villages or create a table with the cases by
> village. So we need to formulate a new treatment
>
>
> *POSSIBILITIES*
>
> *1.   **CAPTURE THE VILLAGE AS A SET OF COORDINATES*
>
> This options consists in create a data element with type COORDINATES and
> add it to the form. With this options we can add a the geographical
> information of where the patient comes from by providing the coordinates of
> the village or even better (in new versions of DHIS2) we can click on the
> map, search for the village in the search bar and capture the coordinates
> of it visually.
>
> Nice feature but then the village information is only stored as a pair of
> coordinates, there is no auxiliary information as for instance its name. In
> the analytical point of view we can now show the cases in a map distributed
> by village (which is nice) but we cannot create the desired table with the
> cases by village.
>
> Another issue in this option is that for the end-user sometimes it will be
> difficult to interact with the map and be able to capture the correct
> village.
>
> *2.   **CAPTURE THE VILLAGE AS A ORGANISATION  UNIT*
>
> This option appeared once 2.25 DHIS2 version was released with a new
> Organisation Unit Type for data elements. Using that, we can create a new
> data element with type = Organisation unit to capture the information of
> where the patient comes from.
>
> So in this option we can have a user with data capture permission only on
> the Sierra Leona hierarchy of organisation units but able to select a
> village for example in Benin hierarchy of organisation units.
>
> Even though the Organisation unit type data element is the more natural
> way to specify where the patient comes from, we experienced some problems.
> Firstly, we are not able to use this data as a dimension for aggregate
> events information (for instance knowing how many events are registered
> with a patient coming from X village). And secondly, this option will mean
> to add all the villages included in our area of research which may lead to
> memory problems in our server.
>
> *3.   **CAPTURE THE VILLAGE AS A RELATIONSHIP WITH THE PATIENT*
>
> The 3rd option is to use the “Relationship” implemented in Tracker
> Capture App. This app can create links between different entities
> registered in the system; originally it was created for tracking a pregnant
> patient and then registers to the system her new child. But we could use
> this relationship in another way just registering villages in the system
> and then linking these registrations with the patient. This is a more
> complex way to implement our needs and it needs a lot of research to know
> if we will be able to show the relationship in a map and to creates tables
> from them.
>
>
>
> As I said, thank you very much. We look forward to your answer.
>
>
> Marc Garnica
>
>
> ___
> Mailing list: https://launchpad.net/~dhis2-users
> Post to : dhis2-us...@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~dhis2-users
> More help   : 

Re: [Dhis2-devs] [Dhis2-users] 25 hours in completing Analytic

[Bob Jolliffe]

Hi Calle

I think Lars would probably know better, but given the kinds of cross
tabulation that is happening with analytics, I doubt that it will scale
linearly.  Would be good to get some empirical data but I think you are
probably going to have something more approaching n^2 time complexity.

On 24 October 2016 at 09:20, Calle Hedberg  wrote:

> Neeraj,
>
> You never stated the number of records you have in the datavalue table -
> what is it?
>
> In the same context: anybody have a rough idea of how many datavalue
> records there are in the global DATIM database - which I think currently
> might be the largest DHIS2 instance around?
>
> Given our own recent work on performance + what Neeraj has reported, I've
> been thinking of creating one test instance with let us say 500 mill
> datavalue records and another with let us say 1 billion, then use them to
> identify key bottlenecks in various processes AND use them to ensure that
> DHIS2 analytics performance is as linear as possible in terms of database
> size. Postgresql has introduced a number of new indexing algorithms in
> recent versions, and I'm not sure if DHIS2 is taking full advantage of them.
>
> Best regards
> Calle
>
> On 24 October 2016 at 07:53, Brajesh Murari 
> wrote:
>
>> Congratulation Neeraj and team ...it much appreciated
>>
>> On Mon, Oct 24, 2016 at 11:08 AM, Neeraj Gupta 
>> wrote:
>>
>>> Dear Team, Thanks for all your suggestions.
>>>
>>> Now the time of analytic is reduced to 10 hours 41 minutes.
>>>
>>> We tried to VACUUM as Sam suggested but it didn't help then we upgraded
>>> postgres from 9.4 to 9.5.4 and as Calle and Bob suggested we made some
>>> changes in configuration file of postgres and it reduced the time. But the
>>> database size is still same.
>>>
>>> Thanks for all your help!
>>>
>>> Thanks,
>>> Neeraj
>>>
>>> On Wed, Oct 19, 2016 at 6:46 PM, Calle Hedberg 
>>> wrote:
>>>
 Neeraj,

 It's always an element of uncertainty linked to database sizes - ref
 Sam's post over. So indicating the number of records you have in the
 datavalue table & key meta-data tables would be useful + indicating whether
 you are running other instances on the same server. Some comments - I've
 been doing a lot of similar optimising work recently:

 1. Upgrading to 9.5.4 is strongly recommended (and don't use 9.6 before
 the worst bugs are fixed and it has stabilised).

 2. Carefully check your postgres.conf against the recommended settings.
 The guide is a bit superficial in the sense that it has recommended "fixed"
 values only and no explanations around ranges below or above those, but you
 can experiment a bit yourself (e.g. the recommended "max_connections = 200"
 might not be sufficient for a really large system like what you have.

 3. If your server is running that single instance only, then 48GB or
 RAM should be sufficient. Our servers are all having 128GB RAM so we
 experimented quite a bit earlier this year with giving a DHIS2 instance
 large amounts or RAM (up to 60-70gb), with negligible impact on
 performance. According to Lars, the DHIS2 cannot really utilize more than
 around 16gb RAM (at least that is how I understood his communication at the
 time). So 48GB should be sufficient for a single instance.

 4. I've been doing performance optimizing recently on an instance with
 - 4-core server with 2x 512gb ssd, 12gb allocated to DHIS2
 - 31,000 Orgunits
 - 420 data elements
 - 250 indicators
 - around 100 mill datavalue records
 - total size around 140gb with analytics tables.

 So the size is only 25% of your 500GB, but RUNNING ANALYTICS ON THAT
 DATABASE INSTANCE IS TAKING JUST OVER 1 HOUR. Fundamentally, if the
 analytics engine is designed well, I would expect a nearly linear
 relationship between database size and the time analytics takes to run. So
 running analytics on your database on our server should in theory take 4-5
 hours.

 We are obviously comparing oranges and nectarines here, in the sense
 that there might be other aspects of our server and database that is
 different from yours (type of CPU, no of OUs, no of DEs/Indicators, whether
 your instance have lots of tracker data, etc etc). I have not seen any
 scientific/quantified comparative performance values related to specific
 parameters like number of CPUs and/or number of cores, but 12 cores SHOULD
 improve analytics performance quite a bit - assuming around 30% then it
 means running analytics on your database/server should take around 3
 hours..

 I tried getting comparative, quantitative data on various
 configurations of hardware and software (e.g. some users prefer CentOS,
 others Ubuntu) during the academy in August, but did not get much - 

Re: [Dhis2-devs] [Dhis2-users] 25 hours in completing Analytic

[Calle Hedberg]

Neeraj,

Thanks for that, it is useful info.

The instance has a very skewed balance between data elements/catcombos and
indicators, but what else is new - outside of South Africa, most countries
have the same: they collect a huge number or data elements but turn very
few of them into indicators (SA collect very very few data elements that
are NOT part of any indicator). But that is a separate discussion.

I would NOT expect the number of data elements & catcombos + the number of
orgunits to have a significant impact on analytics performance, but that
needs to be verified.

I WOULD expect the number of indicators to have a significant impact, so if
you doubled or quadrupled the number of indicators I would expect a
significant impact on analytics time

Using your and my numbers as a baseline, I would envision the following
tests to be revealing:

Baseline system 1: SSD, 4-core CPU: 100 mill values, 400 DEs, 250
indicators, 30,000 OUs, few catcombos -> 1 hour for analytics processing

Baseline system 2: SSD, 12-core CPU: 500 mill values, 3,500 DEs, 15,000
OUs, 25 catcombos -> ~10 hours for analytics processing

Test 1: Using system 1, reduce OUs to 15,000 - run analytics (hypothesis:
no significant difference)
Test 2: Using system 1, reduce indicators to 125 - run analytics, then to 0
- run analytics (hypothesis: analytics reduced to 45 min)
Test 3: Using system 1, increase datavalues from 100mill to 500 mill by
introducing additional attributecombos (easy) - run analytics (hypothesis:
~15 hours for analytics processing, with 4-cores instead of 12-cores)
Test 4: Using system 1, increase datavalues to 1 billion - run analytics
(hypothesis: 35 hours for analytics processing).

Neeraj, I don't know if you have a similar sandbox server available for
testing, but if you do:

Test 5: Using system 2, increase number of indicators to 600 (just export
the 200 you have, modify the names and uids a bit, and re-import - for the
purpose of this test, it does not matter that many of the indicators have
identical formulas). Hypothesis: Analytics time up at least 20%
Test 6: using system 2 (with 200 indicators), shift half of the 500 mill
data values forward or backward in time resulting in twice the number of
analytics tables at half the size. (hypothesis: slight reduction in
processing time)
Test 7: Using system 2 (with 200 indicators), increase datavalues to 1
billion using a new attributecombo - run analytics (hypothesis: analytics
time up to 25 hours again).

As far as I've seen, the bulk of analytics processing time is taken up by
indexing - which I would expect to follow a moderately quadratic curve in
terms of analytics table sizes.

Best regards
Calle

On 24 October 2016 at 10:39, Neeraj Gupta  wrote:

> Calle,
>
> We have around 500 million record in database with 3666 data elements
> having 26 category combinations and 201 indicators and there are 14398
> organisation units.
>
> Thanks,
> Neeraj
>
> On Mon, Oct 24, 2016 at 1:50 PM, Calle Hedberg 
> wrote:
>
>> Neeraj,
>>
>> You never stated the number of records you have in the datavalue table -
>> what is it?
>>
>> In the same context: anybody have a rough idea of how many datavalue
>> records there are in the global DATIM database - which I think currently
>> might be the largest DHIS2 instance around?
>>
>> Given our own recent work on performance + what Neeraj has reported, I've
>> been thinking of creating one test instance with let us say 500 mill
>> datavalue records and another with let us say 1 billion, then use them to
>> identify key bottlenecks in various processes AND use them to ensure that
>> DHIS2 analytics performance is as linear as possible in terms of database
>> size. Postgresql has introduced a number of new indexing algorithms in
>> recent versions, and I'm not sure if DHIS2 is taking full advantage of them.
>>
>> Best regards
>> Calle
>>
>> On 24 October 2016 at 07:53, Brajesh Murari 
>> wrote:
>>
>>> Congratulation Neeraj and team ...it much appreciated
>>>
>>> On Mon, Oct 24, 2016 at 11:08 AM, Neeraj Gupta 
>>> wrote:
>>>
 Dear Team, Thanks for all your suggestions.

 Now the time of analytic is reduced to 10 hours 41 minutes.

 We tried to VACUUM as Sam suggested but it didn't help then we upgraded
 postgres from 9.4 to 9.5.4 and as Calle and Bob suggested we made some
 changes in configuration file of postgres and it reduced the time. But the
 database size is still same.

 Thanks for all your help!

 Thanks,
 Neeraj

 On Wed, Oct 19, 2016 at 6:46 PM, Calle Hedberg  wrote:

> Neeraj,
>
> It's always an element of uncertainty linked to database sizes - ref
> Sam's post over. So indicating the number of records you have in the
> datavalue table & key meta-data tables would be useful + indicating 
> whether
> you are 

Re: [Dhis2-devs] CODE field for Data Set Sections

[Archana Chillala]

Thanks, Nicolay!

Archana Chillala
Application Developer
Email archa...@thoughtworks.com
Telephone +91 9100960533 <+91+9100960533>
[image: ThoughtWorks]


On Fri, Oct 21, 2016 at 7:26 PM, Nicolay Ramm  wrote:

> Hi Archana,
>
> I've fixed this now for 2.25 and 2.24.
>
>
> Best regards,
>
>
> Nicolay Ramm
> Front end developer, DHIS 2
> University of Oslo
> https://www.dhis2.org
>
> On Thu, Oct 20, 2016 at 7:26 AM, Archana Chillala <
> archa...@thoughtworks.com> wrote:
>
>> Hi Lars,
>>
>> Looks like the new version 2.25 still does not have the CODE field
>> exposed in the UI of Data Set Sections (as raised on issue #204). Any
>> update on this?
>>
>>
>> *Cheers,*
>>
>> Archana Chillala
>> Application Developer
>> Email archa...@thoughtworks.com
>> Telephone +91 9100960533 <+91+9100960533>
>> [image: ThoughtWorks]
>> 
>>
>> On Tue, Jul 26, 2016 at 2:54 PM, Lars Helge Øverland 
>> wrote:
>>
>>> Hi Archana,
>>>
>>> yes we can add both code and metadata attributes. (The code field for
>>> section is actually there in the model so you can use it through the API as
>>> of now).
>>>
>>> I have written issues here:
>>>
>>> https://github.com/dhis2/maintenance-app/issues/204
>>>
>>> https://blueprints.launchpad.net/dhis2/+spec/metadata-attributes-section
>>>
>>>
>>> Feel free to take these on ;)
>>>
>>> regards,
>>>
>>> Lars
>>>
>>>
>>>
>>> On Tue, Jul 26, 2016 at 11:00 AM, Archana Chillala <
>>> archa...@thoughtworks.com> wrote:
>>>
 Hi,

 For Data Set sections, there is no '*CODE*' field as of now. Also,
 custom* attributes* cannot be assigned to data set section. Is there
 any scope of incorporating these in the future, as we need an additional
 field to store some kind of code for data set sections.



 *Cheers,*

 Archana Chillala
 Application Developer
 Email archa...@thoughtworks.com
 Telephone +91 9100960533 <+91+9100960533>
 [image: ThoughtWorks]
 

 ___
 Mailing list: https://launchpad.net/~dhis2-devs
 Post to : dhis2-devs@lists.launchpad.net
 Unsubscribe : https://launchpad.net/~dhis2-devs
 More help   : https://help.launchpad.net/ListHelp


>>>
>>>
>>> --
>>> Lars Helge Øverland
>>> Lead developer, DHIS 2
>>> University of Oslo
>>> Skype: larshelgeoverland
>>> l...@dhis2.org
>>> http://www.dhis2.org 
>>>
>>>
>>
>> ___
>> Mailing list: https://launchpad.net/~dhis2-devs
>> Post to : dhis2-devs@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~dhis2-devs
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>
___
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help   : https://help.launchpad.net/ListHelp

[Dhis2-devs] Fwd: [Dhis2-users] GIS 2.25

[Brajesh Murari]

This email was found in my spam directive.

-- Forwarded message --
From: Calle Hedberg 
Date: Mon, Oct 24, 2016 at 1:28 PM
Subject: Re: [Dhis2-users] [Dhis2-devs] GIS 2.25
To: Morten Olav Hansen 
Cc: DHIS 2 Users List , DHIS 2 Developers
List 


Absalom,

Postgis 9.5.3 don't exist... The latest version of postgis to use with
PostgreSQL 9.5.2+ is Postgis 2.3.0

just note this from the download instructions:
*Make sure you are using PostgreSQL 9.5.2+ or PostgreSQL 9.4.7+ or
PostgreSQL 9.6.0+.*

Otherwise follow Morten's advice - press CTRL-SHIFT-J to open the Chrome
console (then check the Console tab), and look for errors which might give
a clue.

Regards
Calle

On 24 October 2016 at 06:54, Morten Olav Hansen  wrote:

> Hi
>
> Are you using chrome? could you open up the developer tools, and see if
> there is anything in the console. Sounds like it can't load because of some
> issues with your configuration.
>
> --
> Morten Olav Hansen
> Senior Engineer, DHIS 2
> University of Oslo
> http://www.dhis2.org
>
> On Mon, Oct 24, 2016 at 11:51 AM, Absolom MURAMIRA 
> wrote:
>
>> Dear Calle,
>>
>> I installed postgis 9.5.3. Would that be the reason?
>>
>> Sent from Yahoo Mail on Android
>> 
>>
>> On Mon, 24 Oct, 2016 at 0:35, Calle Hedberg
>>  wrote:
>> Absalom,
>>
>> You have installed postgis with pg 9.5.4?
>>
>> Regards
>> Calle
>>
>> On 22 October 2016 at 10:36, Absolom MURAMIRA 
>> wrote:
>>
>>> Dear All,
>>>
>>> I am testing release 2.25 and appreciate the developers' contribution to
>>> the good improvements to the DHIS2 application.
>>> Most of the apps are running well apart from the GIS app. It loads
>>> forever and cannot load its interface to enable me generate maps. Can
>>> someone please advise me if there is some configuration I need to enable
>>> first? I checked the demo site which is okay. the pivot tables and data
>>> visualizers are loading normally.
>>>
>>> The testing environment in Linux server 16.04 LTS and postgresql 9.5
>>>
>>> Thank you.
>>>
>>> Absolom
>>> Kigali
>>>
>>> __ _
>>> Mailing list: https://launchpad.net/~dhis2- devs
>>> 
>>> Post to : dhis2-devs@lists.launchpad.net
>>> Unsubscribe : https://launchpad.net/~dhis2- devs
>>> 
>>> More help   : https://help.launchpad.net/ ListHelp
>>> 
>>>
>>>
>>
>>
>> --
>>
>> ***
>>
>> Calle Hedberg
>>
>> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>>
>> Tel/fax (home): +27-21-685-6472
>>
>> Cell: +27-82-853-5352
>>
>> Iridium SatPhone: +8816-315-19119
>>
>> Email: calle.hedb...@gmail.com
>>
>> Skype: calle_hedberg
>>
>> ***
>>
>>
>> ___
>> Mailing list: https://launchpad.net/~dhis2-devs
>> Post to : dhis2-devs@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~dhis2-devs
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>


-- 

***

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedb...@gmail.com

Skype: calle_hedberg

***


___
Mailing list: https://launchpad.net/~dhis2-users
Post to : dhis2-us...@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-users
More help   : https://help.launchpad.net/ListHelp




-- 
Best Regards,

Brajesh Murari,
Postgraduate, Dept of CSE,
Chaudhary Devi Lal University, Sirsa,
India.

The three basic dimensions of human development: a long and healthy life,
access to knowledge, and a decent standard of living.
___
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help   : https://help.launchpad.net/ListHelp

Re: [Dhis2-devs] [Dhis2-users] 25 hours in completing Analytic

[Brajesh Murari]

Much appreciated Bob and Neeraj, much appreciated ...

On Mon, Oct 24, 2016 at 3:27 PM, Knut Staring  wrote:

> It would be very interesting to know of other similarly large
> installations. DATIM, Bangladesh, PSI perhaps - others?
>
> On Mon, Oct 24, 2016 at 10:39 AM, Neeraj Gupta 
> wrote:
>
>> Calle,
>>
>> We have around 500 million record in database with 3666 data elements
>> having 26 category combinations and 201 indicators and there are 14398
>> organisation units.
>>
>> Thanks,
>> Neeraj
>>
>> On Mon, Oct 24, 2016 at 1:50 PM, Calle Hedberg 
>> wrote:
>>
>>> Neeraj,
>>>
>>> You never stated the number of records you have in the datavalue table -
>>> what is it?
>>>
>>> In the same context: anybody have a rough idea of how many datavalue
>>> records there are in the global DATIM database - which I think currently
>>> might be the largest DHIS2 instance around?
>>>
>>> Given our own recent work on performance + what Neeraj has reported,
>>> I've been thinking of creating one test instance with let us say 500 mill
>>> datavalue records and another with let us say 1 billion, then use them to
>>> identify key bottlenecks in various processes AND use them to ensure that
>>> DHIS2 analytics performance is as linear as possible in terms of database
>>> size. Postgresql has introduced a number of new indexing algorithms in
>>> recent versions, and I'm not sure if DHIS2 is taking full advantage of them.
>>>
>>> Best regards
>>> Calle
>>>
>>> On 24 October 2016 at 07:53, Brajesh Murari 
>>> wrote:
>>>
 Congratulation Neeraj and team ...it much appreciated

 On Mon, Oct 24, 2016 at 11:08 AM, Neeraj Gupta 
 wrote:

> Dear Team, Thanks for all your suggestions.
>
> Now the time of analytic is reduced to 10 hours 41 minutes.
>
> We tried to VACUUM as Sam suggested but it didn't help then we
> upgraded postgres from 9.4 to 9.5.4 and as Calle and Bob suggested we made
> some changes in configuration file of postgres and it reduced the time. 
> But
> the database size is still same.
>
> Thanks for all your help!
>
> Thanks,
> Neeraj
>
> On Wed, Oct 19, 2016 at 6:46 PM, Calle Hedberg <
> calle.hedb...@gmail.com> wrote:
>
>> Neeraj,
>>
>> It's always an element of uncertainty linked to database sizes - ref
>> Sam's post over. So indicating the number of records you have in the
>> datavalue table & key meta-data tables would be useful + indicating 
>> whether
>> you are running other instances on the same server. Some comments - I've
>> been doing a lot of similar optimising work recently:
>>
>> 1. Upgrading to 9.5.4 is strongly recommended (and don't use 9.6
>> before the worst bugs are fixed and it has stabilised).
>>
>> 2. Carefully check your postgres.conf against the recommended
>> settings. The guide is a bit superficial in the sense that it has
>> recommended "fixed" values only and no explanations around ranges below 
>> or
>> above those, but you can experiment a bit yourself (e.g. the recommended
>> "max_connections = 200" might not be sufficient for a really large system
>> like what you have.
>>
>> 3. If your server is running that single instance only, then 48GB or
>> RAM should be sufficient. Our servers are all having 128GB RAM so we
>> experimented quite a bit earlier this year with giving a DHIS2 instance
>> large amounts or RAM (up to 60-70gb), with negligible impact on
>> performance. According to Lars, the DHIS2 cannot really utilize more than
>> around 16gb RAM (at least that is how I understood his communication at 
>> the
>> time). So 48GB should be sufficient for a single instance.
>>
>> 4. I've been doing performance optimizing recently on an instance
>> with
>> - 4-core server with 2x 512gb ssd, 12gb allocated to DHIS2
>> - 31,000 Orgunits
>> - 420 data elements
>> - 250 indicators
>> - around 100 mill datavalue records
>> - total size around 140gb with analytics tables.
>>
>> So the size is only 25% of your 500GB, but RUNNING ANALYTICS ON THAT
>> DATABASE INSTANCE IS TAKING JUST OVER 1 HOUR. Fundamentally, if the
>> analytics engine is designed well, I would expect a nearly linear
>> relationship between database size and the time analytics takes to run. 
>> So
>> running analytics on your database on our server should in theory take 
>> 4-5
>> hours.
>>
>> We are obviously comparing oranges and nectarines here, in the sense
>> that there might be other aspects of our server and database that is
>> different from yours (type of CPU, no of OUs, no of DEs/Indicators, 
>> whether
>> your instance have lots of tracker data, etc etc). I have not seen any
>> scientific/quantified 

Re: [Dhis2-devs] [Dhis2-users] 25 hours in completing Analytic

[Knut Staring]

It would be very interesting to know of other similarly large
installations. DATIM, Bangladesh, PSI perhaps - others?

On Mon, Oct 24, 2016 at 10:39 AM, Neeraj Gupta 
wrote:

> Calle,
>
> We have around 500 million record in database with 3666 data elements
> having 26 category combinations and 201 indicators and there are 14398
> organisation units.
>
> Thanks,
> Neeraj
>
> On Mon, Oct 24, 2016 at 1:50 PM, Calle Hedberg 
> wrote:
>
>> Neeraj,
>>
>> You never stated the number of records you have in the datavalue table -
>> what is it?
>>
>> In the same context: anybody have a rough idea of how many datavalue
>> records there are in the global DATIM database - which I think currently
>> might be the largest DHIS2 instance around?
>>
>> Given our own recent work on performance + what Neeraj has reported, I've
>> been thinking of creating one test instance with let us say 500 mill
>> datavalue records and another with let us say 1 billion, then use them to
>> identify key bottlenecks in various processes AND use them to ensure that
>> DHIS2 analytics performance is as linear as possible in terms of database
>> size. Postgresql has introduced a number of new indexing algorithms in
>> recent versions, and I'm not sure if DHIS2 is taking full advantage of them.
>>
>> Best regards
>> Calle
>>
>> On 24 October 2016 at 07:53, Brajesh Murari 
>> wrote:
>>
>>> Congratulation Neeraj and team ...it much appreciated
>>>
>>> On Mon, Oct 24, 2016 at 11:08 AM, Neeraj Gupta 
>>> wrote:
>>>
 Dear Team, Thanks for all your suggestions.

 Now the time of analytic is reduced to 10 hours 41 minutes.

 We tried to VACUUM as Sam suggested but it didn't help then we upgraded
 postgres from 9.4 to 9.5.4 and as Calle and Bob suggested we made some
 changes in configuration file of postgres and it reduced the time. But the
 database size is still same.

 Thanks for all your help!

 Thanks,
 Neeraj

 On Wed, Oct 19, 2016 at 6:46 PM, Calle Hedberg  wrote:

> Neeraj,
>
> It's always an element of uncertainty linked to database sizes - ref
> Sam's post over. So indicating the number of records you have in the
> datavalue table & key meta-data tables would be useful + indicating 
> whether
> you are running other instances on the same server. Some comments - I've
> been doing a lot of similar optimising work recently:
>
> 1. Upgrading to 9.5.4 is strongly recommended (and don't use 9.6
> before the worst bugs are fixed and it has stabilised).
>
> 2. Carefully check your postgres.conf against the recommended
> settings. The guide is a bit superficial in the sense that it has
> recommended "fixed" values only and no explanations around ranges below or
> above those, but you can experiment a bit yourself (e.g. the recommended
> "max_connections = 200" might not be sufficient for a really large system
> like what you have.
>
> 3. If your server is running that single instance only, then 48GB or
> RAM should be sufficient. Our servers are all having 128GB RAM so we
> experimented quite a bit earlier this year with giving a DHIS2 instance
> large amounts or RAM (up to 60-70gb), with negligible impact on
> performance. According to Lars, the DHIS2 cannot really utilize more than
> around 16gb RAM (at least that is how I understood his communication at 
> the
> time). So 48GB should be sufficient for a single instance.
>
> 4. I've been doing performance optimizing recently on an instance with
> - 4-core server with 2x 512gb ssd, 12gb allocated to DHIS2
> - 31,000 Orgunits
> - 420 data elements
> - 250 indicators
> - around 100 mill datavalue records
> - total size around 140gb with analytics tables.
>
> So the size is only 25% of your 500GB, but RUNNING ANALYTICS ON THAT
> DATABASE INSTANCE IS TAKING JUST OVER 1 HOUR. Fundamentally, if the
> analytics engine is designed well, I would expect a nearly linear
> relationship between database size and the time analytics takes to run. So
> running analytics on your database on our server should in theory take 4-5
> hours.
>
> We are obviously comparing oranges and nectarines here, in the sense
> that there might be other aspects of our server and database that is
> different from yours (type of CPU, no of OUs, no of DEs/Indicators, 
> whether
> your instance have lots of tracker data, etc etc). I have not seen any
> scientific/quantified comparative performance values related to specific
> parameters like number of CPUs and/or number of cores, but 12 cores SHOULD
> improve analytics performance quite a bit - assuming around 30% then it
> means running analytics on your database/server should take around 3

Re: [Dhis2-devs] [Dhis2-users] 25 hours in completing Analytic

[Bob Jolliffe]

For the benefit of others on this thread, this is the tweaking I shared
with Neeraj -

He had 3200MB shared_buffers set (as per the online manual).  I suggested
with his 48G of RAM he tries:

shared_buffers = 12GB
effective_cache_size = 20GB

(remember the manual is based on an exampe machine with 12G)

I predicted that change would lead to the biggest speedup.

Also I suggested manually setting the jvm max heap size to 16G rather than
depending on java 8 to pick a good default.  Or at least verify that the
default was ok with

java -XX:+PrintFlagsFinal  -version | grep MaxHeapSize

These are still instinctive thumbsuck figures from me.  Neeraj you can
still try some further adjustment (try upping postgres shared_buffers to
20G for example) but I think you are only going to get marginal
improvements from here.  Maybe getting from 11 hours to 8.  You are still
faced with the hard limit of the underlying disk speed and also the
database size and structure.  I'd be curious to know whether people have
experience in slicing up the database to run smaller analytics jobs in
parallel - for example by time or by space.  And the effect of allocating
different numbers of worker threads depending on available cpu cores.

What really needs to be done is to run some sort of measurements while the
analytics is running to understand where the main bottlenecks lie.  To
answer the question of which processes are starved and are they starved for
RAM, CPU or waiting on disk.

On 24 October 2016 at 09:39, Neeraj Gupta  wrote:

> Calle,
>
> We have around 500 million record in database with 3666 data elements
> having 26 category combinations and 201 indicators and there are 14398
> organisation units.
>
> Thanks,
> Neeraj
>
> On Mon, Oct 24, 2016 at 1:50 PM, Calle Hedberg 
> wrote:
>
>> Neeraj,
>>
>> You never stated the number of records you have in the datavalue table -
>> what is it?
>>
>> In the same context: anybody have a rough idea of how many datavalue
>> records there are in the global DATIM database - which I think currently
>> might be the largest DHIS2 instance around?
>>
>> Given our own recent work on performance + what Neeraj has reported, I've
>> been thinking of creating one test instance with let us say 500 mill
>> datavalue records and another with let us say 1 billion, then use them to
>> identify key bottlenecks in various processes AND use them to ensure that
>> DHIS2 analytics performance is as linear as possible in terms of database
>> size. Postgresql has introduced a number of new indexing algorithms in
>> recent versions, and I'm not sure if DHIS2 is taking full advantage of them.
>>
>> Best regards
>> Calle
>>
>> On 24 October 2016 at 07:53, Brajesh Murari 
>> wrote:
>>
>>> Congratulation Neeraj and team ...it much appreciated
>>>
>>> On Mon, Oct 24, 2016 at 11:08 AM, Neeraj Gupta 
>>> wrote:
>>>
 Dear Team, Thanks for all your suggestions.

 Now the time of analytic is reduced to 10 hours 41 minutes.

 We tried to VACUUM as Sam suggested but it didn't help then we upgraded
 postgres from 9.4 to 9.5.4 and as Calle and Bob suggested we made some
 changes in configuration file of postgres and it reduced the time. But the
 database size is still same.

 Thanks for all your help!

 Thanks,
 Neeraj

 On Wed, Oct 19, 2016 at 6:46 PM, Calle Hedberg  wrote:

> Neeraj,
>
> It's always an element of uncertainty linked to database sizes - ref
> Sam's post over. So indicating the number of records you have in the
> datavalue table & key meta-data tables would be useful + indicating 
> whether
> you are running other instances on the same server. Some comments - I've
> been doing a lot of similar optimising work recently:
>
> 1. Upgrading to 9.5.4 is strongly recommended (and don't use 9.6
> before the worst bugs are fixed and it has stabilised).
>
> 2. Carefully check your postgres.conf against the recommended
> settings. The guide is a bit superficial in the sense that it has
> recommended "fixed" values only and no explanations around ranges below or
> above those, but you can experiment a bit yourself (e.g. the recommended
> "max_connections = 200" might not be sufficient for a really large system
> like what you have.
>
> 3. If your server is running that single instance only, then 48GB or
> RAM should be sufficient. Our servers are all having 128GB RAM so we
> experimented quite a bit earlier this year with giving a DHIS2 instance
> large amounts or RAM (up to 60-70gb), with negligible impact on
> performance. According to Lars, the DHIS2 cannot really utilize more than
> around 16gb RAM (at least that is how I understood his communication at 
> the
> time). So 48GB should be sufficient for a single instance.

Re: [Dhis2-devs] [Dhis2-users] 25 hours in completing Analytic

[Neeraj Gupta]

Calle,

We have around 500 million record in database with 3666 data elements
having 26 category combinations and 201 indicators and there are 14398
organisation units.

Thanks,
Neeraj

On Mon, Oct 24, 2016 at 1:50 PM, Calle Hedberg 
wrote:

> Neeraj,
>
> You never stated the number of records you have in the datavalue table -
> what is it?
>
> In the same context: anybody have a rough idea of how many datavalue
> records there are in the global DATIM database - which I think currently
> might be the largest DHIS2 instance around?
>
> Given our own recent work on performance + what Neeraj has reported, I've
> been thinking of creating one test instance with let us say 500 mill
> datavalue records and another with let us say 1 billion, then use them to
> identify key bottlenecks in various processes AND use them to ensure that
> DHIS2 analytics performance is as linear as possible in terms of database
> size. Postgresql has introduced a number of new indexing algorithms in
> recent versions, and I'm not sure if DHIS2 is taking full advantage of them.
>
> Best regards
> Calle
>
> On 24 October 2016 at 07:53, Brajesh Murari 
> wrote:
>
>> Congratulation Neeraj and team ...it much appreciated
>>
>> On Mon, Oct 24, 2016 at 11:08 AM, Neeraj Gupta 
>> wrote:
>>
>>> Dear Team, Thanks for all your suggestions.
>>>
>>> Now the time of analytic is reduced to 10 hours 41 minutes.
>>>
>>> We tried to VACUUM as Sam suggested but it didn't help then we upgraded
>>> postgres from 9.4 to 9.5.4 and as Calle and Bob suggested we made some
>>> changes in configuration file of postgres and it reduced the time. But the
>>> database size is still same.
>>>
>>> Thanks for all your help!
>>>
>>> Thanks,
>>> Neeraj
>>>
>>> On Wed, Oct 19, 2016 at 6:46 PM, Calle Hedberg 
>>> wrote:
>>>
 Neeraj,

 It's always an element of uncertainty linked to database sizes - ref
 Sam's post over. So indicating the number of records you have in the
 datavalue table & key meta-data tables would be useful + indicating whether
 you are running other instances on the same server. Some comments - I've
 been doing a lot of similar optimising work recently:

 1. Upgrading to 9.5.4 is strongly recommended (and don't use 9.6 before
 the worst bugs are fixed and it has stabilised).

 2. Carefully check your postgres.conf against the recommended settings.
 The guide is a bit superficial in the sense that it has recommended "fixed"
 values only and no explanations around ranges below or above those, but you
 can experiment a bit yourself (e.g. the recommended "max_connections = 200"
 might not be sufficient for a really large system like what you have.

 3. If your server is running that single instance only, then 48GB or
 RAM should be sufficient. Our servers are all having 128GB RAM so we
 experimented quite a bit earlier this year with giving a DHIS2 instance
 large amounts or RAM (up to 60-70gb), with negligible impact on
 performance. According to Lars, the DHIS2 cannot really utilize more than
 around 16gb RAM (at least that is how I understood his communication at the
 time). So 48GB should be sufficient for a single instance.

 4. I've been doing performance optimizing recently on an instance with
 - 4-core server with 2x 512gb ssd, 12gb allocated to DHIS2
 - 31,000 Orgunits
 - 420 data elements
 - 250 indicators
 - around 100 mill datavalue records
 - total size around 140gb with analytics tables.

 So the size is only 25% of your 500GB, but RUNNING ANALYTICS ON THAT
 DATABASE INSTANCE IS TAKING JUST OVER 1 HOUR. Fundamentally, if the
 analytics engine is designed well, I would expect a nearly linear
 relationship between database size and the time analytics takes to run. So
 running analytics on your database on our server should in theory take 4-5
 hours.

 We are obviously comparing oranges and nectarines here, in the sense
 that there might be other aspects of our server and database that is
 different from yours (type of CPU, no of OUs, no of DEs/Indicators, whether
 your instance have lots of tracker data, etc etc). I have not seen any
 scientific/quantified comparative performance values related to specific
 parameters like number of CPUs and/or number of cores, but 12 cores SHOULD
 improve analytics performance quite a bit - assuming around 30% then it
 means running analytics on your database/server should take around 3
 hours..

 I tried getting comparative, quantitative data on various
 configurations of hardware and software (e.g. some users prefer CentOS,
 others Ubuntu) during the academy in August, but did not get much - it
 seems most users/providers have found a setup that works for them for now
 and nobody is doing 

Re: [Dhis2-devs] Testing upgrade of DHIS2 from 2.21 to 2.24

[Vanya Seth]

HI Morten

Hope you are doing well.

Can you please have a look at these issues as well?

Regards
Vanya

On Fri, Oct 21, 2016 at 3:15 PM, Aamer Mohammed 
wrote:

> Hi Lars/Morten,
>
> Recently we did a DHIS upgrade test to upgrade our instances from 2.21 to
> 2.24. We started with 2.21 instance having good amount of metadata. We
> sequentially upgraded that instance to 2.24 by following the upgrade
> scripts for each DHIS version.
> From that upgraded instance, an export was taken using the new exporter
> (/api/24/metadata). It was then imported to another fresh instance on 2.24
> using the new importer. Below are our findings.
>
> 1) Most of the blockers were due to introducing of new constraints across
> DHIS versions. The upgraded instance did not have these constraints but
> the new fresh instance had them so it was preventing the import.
> ex: shortName in dataelements table became unique
> numerator is required for all indicators
> dataelement_operand_unique_key (it is removed now)
> We manually corrected our data so it abides as per the constraints. *The
> list of constraints introduced in a DHIS version would be part of upgrade
> notes. Is this a right assumption?*
>
> 2) Allow for many-to-many between Group and GroupSet. If we have groups
> which are part of multiple group sets, Hibernate will create duplicate
> GroupSets and it results in database integrity issue when importing.
> We manually corrected the data and ensured one to one mapping between them
> to proceed further. *When do we plan to have the many-to-many mapping
> between Group and GroupSet?*
>
> 3) Error importing Charts, Report Tables, Event charts, Event reports.
> We excluded these metadata types from the export.json to continue further.
> We are unable to import these in 2.25 also. *Please do inform us once the
> fix is done in 2.25 so that we can retest.*
>
> 4) When we update a new DHIS war, the instance is migrated to new DHIS
> version. But if existing data in our instance does not abide by the new
> constraints, the new constraints are not applied in our upgraded instance.
> So effectively it is not pure DHIS upgrade. So ideally, before applying the
> war file we should modify our data to abide the new constraints. *Is this
> a right assumption?*
>
> Please suggest.
>
> Thanks
> Aamer.
>
> ___
> Mailing list: https://launchpad.net/~dhis2-devs
> Post to : dhis2-devs@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~dhis2-devs
> More help   : https://help.launchpad.net/ListHelp
>
>


-- 
With Regards
ThoughtWorks Technologies
Hyderabad

--Stay Hungry Stay Foolish!!
___
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help   : https://help.launchpad.net/ListHelp

Re: [Dhis2-devs] GIS 2.25

[Morten Olav Hansen]

Hi

I have seen this issue also. I have reported it at [1], hopefully it will
get fixed

[1] https://github.com/dhis2/gis-app/issues/164

-- 
Morten Olav Hansen
Senior Engineer, DHIS 2
University of Oslo
http://www.dhis2.org

On Mon, Oct 24, 2016 at 3:27 PM, Absolom MURAMIRA 
wrote:

> Dear Calle,
>
> Sorry that was a typo error about the postgresql version.
>
> I have attached a screenshot of the error from the browser console. I have
> no idea on how to resolve it.
> I tested in different browsers and it doesn't work either. it displays a
> similar error in Firefox.
>
> Kindly advise.
>
> Thanks.
>
> Absolom
>
>
> On Monday, October 24, 2016 9:58 AM, Calle Hedberg <
> calle.hedb...@gmail.com> wrote:
>
>
>
> Absalom,
>
> Postgis 9.5.3 don't exist... The latest version of postgis to use with
> PostgreSQL 9.5.2+ is Postgis 2.3.0
>
> just note this from the download instructions:
> *Make sure you are using PostgreSQL 9.5.2+ or PostgreSQL 9.4.7+ or
> PostgreSQL 9.6.0+.*
>
> Otherwise follow Morten's advice - press CTRL-SHIFT-J to open the Chrome
> console (then check the Console tab), and look for errors which might give
> a clue.
>
> Regards
> Calle
>
> On 24 October 2016 at 06:54, Morten Olav Hansen  wrote:
>
> Hi
>
> Are you using chrome? could you open up the developer tools, and see if
> there is anything in the console. Sounds like it can't load because of some
> issues with your configuration.
>
> --
> Morten Olav Hansen
> Senior Engineer, DHIS 2
> University of Oslo
> http://www.dhis2.org
>
> On Mon, Oct 24, 2016 at 11:51 AM, Absolom MURAMIRA 
> wrote:
>
> Dear Calle,
>
> I installed postgis 9.5.3. Would that be the reason?
>
> Sent from Yahoo Mail on Android
> 
>
> On Mon, 24 Oct, 2016 at 0:35, Calle Hedberg
>  wrote:
> Absalom,
>
> You have installed postgis with pg 9.5.4?
>
> Regards
> Calle
>
> On 22 October 2016 at 10:36, Absolom MURAMIRA  wrote:
>
> Dear All,
>
> I am testing release 2.25 and appreciate the developers' contribution to
> the good improvements to the DHIS2 application.
> Most of the apps are running well apart from the GIS app. It loads forever
> and cannot load its interface to enable me generate maps. Can someone
> please advise me if there is some configuration I need to enable first? I
> checked the demo site which is okay. the pivot tables and data visualizers
> are loading normally.
>
> The testing environment in Linux server 16.04 LTS and postgresql 9.5
>
> Thank you.
>
> Absolom
> Kigali
>
> __ _
> Mailing list: https://launchpad.net/~dhis2- devs
> 
> Post to : dhis2-devs@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~dhis2- devs
> 
> More help   : https://help.launchpad.net/ ListHelp
> 
>
>
>
>
> --
> ** *
> Calle Hedberg
> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
> Tel/fax (home): +27-21-685-6472
> Cell: +27-82-853-5352
> Iridium SatPhone: +8816-315-19119
> Email: calle.hedb...@gmail.com
> Skype: calle_hedberg
> ** *
>
>
> __ _
> Mailing list: https://launchpad.net/~dhis2-d evs
> 
> Post to : dhis2-devs@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~dhis2-d evs
> 
> More help   : https://help.launchpad.net/Lis tHelp
> 
>
>
>
>
>
> --
> ***
> Calle Hedberg
> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
> Tel/fax (home): +27-21-685-6472
> Cell: +27-82-853-5352
> Iridium SatPhone: +8816-315-19119
> Email: calle.hedb...@gmail.com
> Skype: calle_hedberg
> ***
>
>
>
>
___
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help   : https://help.launchpad.net/ListHelp

Re: [Dhis2-devs] DHIS2 API feedback

[Morten Olav Hansen]

On Mon, Oct 24, 2016 at 3:21 PM, Martin Van Aken 
wrote:

> Regarding my question re: user rights through the API, maybe I was
> mistaken. Just to be sure: if I connect with a specific user using the API,
> I should get exactly the same access as I get through the UI with the same
> user, correct?
>

Yes, today almost all our apps are using the web-api directly.. so if
anything is blocked there, it will be blocked by the UI also (we still have
some old UI which is not using the web-api, but they should be secured also)

-- 
Morten Olav Hansen
Senior Engineer, DHIS 2
University of Oslo
http://www.dhis2.org
___
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help   : https://help.launchpad.net/ListHelp

Re: [Dhis2-devs] [Dhis2-users] 25 hours in completing Analytic

[Calle Hedberg]

Neeraj,

You never stated the number of records you have in the datavalue table -
what is it?

In the same context: anybody have a rough idea of how many datavalue
records there are in the global DATIM database - which I think currently
might be the largest DHIS2 instance around?

Given our own recent work on performance + what Neeraj has reported, I've
been thinking of creating one test instance with let us say 500 mill
datavalue records and another with let us say 1 billion, then use them to
identify key bottlenecks in various processes AND use them to ensure that
DHIS2 analytics performance is as linear as possible in terms of database
size. Postgresql has introduced a number of new indexing algorithms in
recent versions, and I'm not sure if DHIS2 is taking full advantage of them.

Best regards
Calle

On 24 October 2016 at 07:53, Brajesh Murari 
wrote:

> Congratulation Neeraj and team ...it much appreciated
>
> On Mon, Oct 24, 2016 at 11:08 AM, Neeraj Gupta 
> wrote:
>
>> Dear Team, Thanks for all your suggestions.
>>
>> Now the time of analytic is reduced to 10 hours 41 minutes.
>>
>> We tried to VACUUM as Sam suggested but it didn't help then we upgraded
>> postgres from 9.4 to 9.5.4 and as Calle and Bob suggested we made some
>> changes in configuration file of postgres and it reduced the time. But the
>> database size is still same.
>>
>> Thanks for all your help!
>>
>> Thanks,
>> Neeraj
>>
>> On Wed, Oct 19, 2016 at 6:46 PM, Calle Hedberg 
>> wrote:
>>
>>> Neeraj,
>>>
>>> It's always an element of uncertainty linked to database sizes - ref
>>> Sam's post over. So indicating the number of records you have in the
>>> datavalue table & key meta-data tables would be useful + indicating whether
>>> you are running other instances on the same server. Some comments - I've
>>> been doing a lot of similar optimising work recently:
>>>
>>> 1. Upgrading to 9.5.4 is strongly recommended (and don't use 9.6 before
>>> the worst bugs are fixed and it has stabilised).
>>>
>>> 2. Carefully check your postgres.conf against the recommended settings.
>>> The guide is a bit superficial in the sense that it has recommended "fixed"
>>> values only and no explanations around ranges below or above those, but you
>>> can experiment a bit yourself (e.g. the recommended "max_connections = 200"
>>> might not be sufficient for a really large system like what you have.
>>>
>>> 3. If your server is running that single instance only, then 48GB or RAM
>>> should be sufficient. Our servers are all having 128GB RAM so we
>>> experimented quite a bit earlier this year with giving a DHIS2 instance
>>> large amounts or RAM (up to 60-70gb), with negligible impact on
>>> performance. According to Lars, the DHIS2 cannot really utilize more than
>>> around 16gb RAM (at least that is how I understood his communication at the
>>> time). So 48GB should be sufficient for a single instance.
>>>
>>> 4. I've been doing performance optimizing recently on an instance with
>>> - 4-core server with 2x 512gb ssd, 12gb allocated to DHIS2
>>> - 31,000 Orgunits
>>> - 420 data elements
>>> - 250 indicators
>>> - around 100 mill datavalue records
>>> - total size around 140gb with analytics tables.
>>>
>>> So the size is only 25% of your 500GB, but RUNNING ANALYTICS ON THAT
>>> DATABASE INSTANCE IS TAKING JUST OVER 1 HOUR. Fundamentally, if the
>>> analytics engine is designed well, I would expect a nearly linear
>>> relationship between database size and the time analytics takes to run. So
>>> running analytics on your database on our server should in theory take 4-5
>>> hours.
>>>
>>> We are obviously comparing oranges and nectarines here, in the sense
>>> that there might be other aspects of our server and database that is
>>> different from yours (type of CPU, no of OUs, no of DEs/Indicators, whether
>>> your instance have lots of tracker data, etc etc). I have not seen any
>>> scientific/quantified comparative performance values related to specific
>>> parameters like number of CPUs and/or number of cores, but 12 cores SHOULD
>>> improve analytics performance quite a bit - assuming around 30% then it
>>> means running analytics on your database/server should take around 3
>>> hours..
>>>
>>> I tried getting comparative, quantitative data on various configurations
>>> of hardware and software (e.g. some users prefer CentOS, others Ubuntu)
>>> during the academy in August, but did not get much - it seems most
>>> users/providers have found a setup that works for them for now and nobody
>>> is doing any systematic performance testing (some of the international
>>> NGOs/companies using DHIS2 might have, but as with internally developed
>>> apps they are not that keen on sharing). So it would be highly appreciated
>>> if you would post the results on analytics time with every upgrade / tweak
>>> you do - starting with the upgrade to Pg 9.5.4
>>>
>>> Best regards
>>> Calle
>>>

Re: [Dhis2-devs] DHIS2 API feedback

[Martin Van Aken]

Hi,
Thanks for the "report mode", I'll try that out.
Regarding my question re: user rights through the API, maybe I was
mistaken. Just to be sure: if I connect with a specific user using the API,
I should get exactly the same access as I get through the UI with the same
user, correct?

Martin

On Mon, Oct 24, 2016 at 4:44 AM, Morten Olav Hansen 
wrote:

>
>>- *Return IDs of created items:* when we create a meta data (org
>>unit, data element, etc), the API should return the newly created object 
>> id
>>(or ids if multiple ones). Failed to do so require us to make a second 
>> call
>>using a field to fetch the object back, this being a risky thing when
>>fields are not always unique. Same thing when a batch process occurs,
>>having something such "2 updated, 2 ignored" does not help me - I need to
>>know which have been updated and which not so that my app can react
>>accordingly.
>>
>>
> Yes, this can be done using the new `importReportMode` in 225 (you
> probably want to set it to full)
>
> https://dhis2.github.io/dhis2-docs/master/en/developer/html/
> webapi_metadata_import.html
>
>
>>
>>- *Respect user rights management:* if I can only access some object
>>using the app with a given user/password, I should have exactly the same
>>rights throught the API - the way I access the app should not impact what 
>> I
>>can see/do
>>
>>
> Not exactly sure what you mean here. Where do you see these issues?
>
>
>>
>>-
>>
>> Cf my previous mail, the additionnal point is to have a lot of thought
>> before altering the API in a backward incompatible change - this can break
>> a lot of software, and at minimum to document those closely.
>>
>> I'll be happy to discuss more on those topics, and to contribute any help
>> that can be useful.
>>
>> Martin
>>
>>
>> ___
>> Mailing list: https://launchpad.net/~dhis2-devs
>> Post to : dhis2-devs@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~dhis2-devs
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>


-- 
*Martin Van Aken - **Freelance Enthusiast Developer*

Mobile : +32 486 899 652

Follow me on Twitter : @martinvanaken 
Call me on Skype : vanakenm
Hang out with me : mar...@joyouscoding.com
Contact me on LinkedIn : http://www.linkedin.com/in/martinvanaken
Company website : www.joyouscoding.com
___
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help   : https://help.launchpad.net/ListHelp

Re: [Dhis2-devs] [Dhis2-users] GIS failing to load in 2.25

[Calle Hedberg]

Morten,

Thanks - noted,

Regards
Calle

On 24 October 2016 at 10:08, Morten Olav Hansen  wrote:

> Hi Calle
>
> No, this is just general Chrome Developer Tools stuff.. should be easy to
> google for a guide :)
>
> I see that there are plenty videos on youtube also (search for "chrome
> developer tools"), this can be very useful in figuring these kinds of
> issues.
>
> --
> Morten Olav Hansen
> Senior Engineer, DHIS 2
> University of Oslo
> http://www.dhis2.org
>
> On Mon, Oct 24, 2016 at 2:49 PM, Calle Hedberg 
> wrote:
>
>> Morten,
>>
>> i was not aware of the "log XHR requests" feature.
>>
>> Is there a DHIS2-related guide to using advanced features of the Chrome
>> console somewhere?
>>
>> Regards
>> Calle
>>
>> On 24 October 2016 at 04:31, Morten Olav Hansen  wrote:
>>
>>> Hi Sam
>>>
>>> This is caused by the metadata having some issues I'm guessing... I
>>> would try and have a look at the network panel (in the web inspector) to
>>> see exactly which request it stops on.. since you are using Chrome, you
>>> should be able to right click on the console and select "log XHR requests",
>>> that will hopefully give you a clue to exactly which request is failing.
>>>
>>> --
>>> Morten Olav Hansen
>>> Senior Engineer, DHIS 2
>>> University of Oslo
>>> http://www.dhis2.org
>>>
>>> On Mon, Oct 17, 2016 at 9:37 PM, Sam Kasozi  wrote:
>>>
 Hi Lars, this is the original error message from an upgraded instance
 from 2.25.

 The developer console, gives the error below:

 ext-all.js:15 Uncaught You're trying to decode an invalid JSON String:
 Also in the attached./ Screen shot attached as well.


 P.s With the new instance, I could not access the orgunits; nor were
 they displaying as part of the maintenance, GIS module wasn't loading
 either - It would be good to try out from your end.

 regards,


 Sam Kasozi
 +256 788-993-565
 Skype: sam.kasoziug

 On Mon, Oct 17, 2016 at 4:44 PM, Lars Helge Øverland 
 wrote:

>
>
> On Mon, Oct 17, 2016 at 3:42 PM, Sam Kasozi  wrote:
>
>> GIS is failing to load in a new instance of 2.25; not sure if this is
>> a bug or whether some pre-configuration needs to happen.
>>
>> regards,
>>
>>
>
> Hi Sam,
>
> we need some more info, can you check in the browser developer console
> and post output? If nothing there, tomcat log is helpful.
>
> regards,
>
> Lars
>
>
>
>
>
>
>> Sam Kasozi
>> +256 788-993-565
>> Skype: sam.kasoziug
>>
>> ___
>> Mailing list: https://launchpad.net/~dhis2-users
>> Post to : dhis2-us...@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~dhis2-users
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>
>
> --
> Lars Helge Øverland
> Lead developer, DHIS 2
> University of Oslo
> Skype: larshelgeoverland
> l...@dhis2.org
> http://www.dhis2.org 
>
>

 ___
 Mailing list: https://launchpad.net/~dhis2-devs
 Post to : dhis2-devs@lists.launchpad.net
 Unsubscribe : https://launchpad.net/~dhis2-devs
 More help   : https://help.launchpad.net/ListHelp


>>>
>>> ___
>>> Mailing list: https://launchpad.net/~dhis2-devs
>>> Post to : dhis2-devs@lists.launchpad.net
>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>> More help   : https://help.launchpad.net/ListHelp
>>>
>>>
>>
>>
>> --
>>
>> ***
>>
>> Calle Hedberg
>>
>> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>>
>> Tel/fax (home): +27-21-685-6472
>>
>> Cell: +27-82-853-5352
>>
>> Iridium SatPhone: +8816-315-19119
>>
>> Email: calle.hedb...@gmail.com
>>
>> Skype: calle_hedberg
>>
>> ***
>>
>>
>


-- 

***

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedb...@gmail.com

Skype: calle_hedberg

***
___
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help   : https://help.launchpad.net/ListHelp

Re: [Dhis2-devs] [Dhis2-users] GIS failing to load in 2.25

[Morten Olav Hansen]

Hi Calle

No, this is just general Chrome Developer Tools stuff.. should be easy to
google for a guide :)

I see that there are plenty videos on youtube also (search for "chrome
developer tools"), this can be very useful in figuring these kinds of
issues.

-- 
Morten Olav Hansen
Senior Engineer, DHIS 2
University of Oslo
http://www.dhis2.org

On Mon, Oct 24, 2016 at 2:49 PM, Calle Hedberg 
wrote:

> Morten,
>
> i was not aware of the "log XHR requests" feature.
>
> Is there a DHIS2-related guide to using advanced features of the Chrome
> console somewhere?
>
> Regards
> Calle
>
> On 24 October 2016 at 04:31, Morten Olav Hansen  wrote:
>
>> Hi Sam
>>
>> This is caused by the metadata having some issues I'm guessing... I would
>> try and have a look at the network panel (in the web inspector) to see
>> exactly which request it stops on.. since you are using Chrome, you should
>> be able to right click on the console and select "log XHR requests", that
>> will hopefully give you a clue to exactly which request is failing.
>>
>> --
>> Morten Olav Hansen
>> Senior Engineer, DHIS 2
>> University of Oslo
>> http://www.dhis2.org
>>
>> On Mon, Oct 17, 2016 at 9:37 PM, Sam Kasozi  wrote:
>>
>>> Hi Lars, this is the original error message from an upgraded instance
>>> from 2.25.
>>>
>>> The developer console, gives the error below:
>>>
>>> ext-all.js:15 Uncaught You're trying to decode an invalid JSON String:
>>> Also in the attached./ Screen shot attached as well.
>>>
>>>
>>> P.s With the new instance, I could not access the orgunits; nor were
>>> they displaying as part of the maintenance, GIS module wasn't loading
>>> either - It would be good to try out from your end.
>>>
>>> regards,
>>>
>>>
>>> Sam Kasozi
>>> +256 788-993-565
>>> Skype: sam.kasoziug
>>>
>>> On Mon, Oct 17, 2016 at 4:44 PM, Lars Helge Øverland 
>>> wrote:
>>>


 On Mon, Oct 17, 2016 at 3:42 PM, Sam Kasozi  wrote:

> GIS is failing to load in a new instance of 2.25; not sure if this is
> a bug or whether some pre-configuration needs to happen.
>
> regards,
>
>

 Hi Sam,

 we need some more info, can you check in the browser developer console
 and post output? If nothing there, tomcat log is helpful.

 regards,

 Lars






> Sam Kasozi
> +256 788-993-565
> Skype: sam.kasoziug
>
> ___
> Mailing list: https://launchpad.net/~dhis2-users
> Post to : dhis2-us...@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~dhis2-users
> More help   : https://help.launchpad.net/ListHelp
>
>


 --
 Lars Helge Øverland
 Lead developer, DHIS 2
 University of Oslo
 Skype: larshelgeoverland
 l...@dhis2.org
 http://www.dhis2.org 


>>>
>>> ___
>>> Mailing list: https://launchpad.net/~dhis2-devs
>>> Post to : dhis2-devs@lists.launchpad.net
>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>> More help   : https://help.launchpad.net/ListHelp
>>>
>>>
>>
>> ___
>> Mailing list: https://launchpad.net/~dhis2-devs
>> Post to : dhis2-devs@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~dhis2-devs
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>
>
> --
>
> ***
>
> Calle Hedberg
>
> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>
> Tel/fax (home): +27-21-685-6472
>
> Cell: +27-82-853-5352
>
> Iridium SatPhone: +8816-315-19119
>
> Email: calle.hedb...@gmail.com
>
> Skype: calle_hedberg
>
> ***
>
>
___
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help   : https://help.launchpad.net/ListHelp

Re: [Dhis2-devs] Removing the limit on the number of users to send a message

[Calle Hedberg]

Hi,

Just to add one piece of info here: Having the ability to email or whatsapp
more than 200-250 users is going to be critical for disease surveillance
systems too. In case of a major disease outbreak, these systems will have
to broadcast messages via multiple protocols to far more than 200-250
people.

Regards
Calle

On 24 October 2016 at 08:29, Sean Broomhead  wrote:

> Hello Lars and team - thanks for having a look at this for us.
>
> The requirement has emerged out of the success of the system Mike built to
> manage placement allocations for all Interns and Community Service Doctors
> in South Africa. Newly qualified doctors apply centrally, expressing their
> priority choices for hospital placements for their two years of internship
> and one year of community service. We run an algorithm to determine which
> hospital each doctor gets sent to.
>
> Both doctors and government love the system. It's email tool now provides
> the formal mechanism for communicating with these doctors about the
> placement process. Unfortunately, with the batch size limits, we're having
> to setup the small batches manually. The current cohort of doctors is about
> 4,000 and the next cohort of nurses could reach 10,000.
>
> Please advise if this needs to go into Launchpad or GitHub, or if you deal
> with this sort of request differently.
>
> Your assistance will be very helpful.
>
> Best regards,
> Sean
>
> On Mon, Oct 24, 2016 at 8:03 AM, Mike Nelushi 
> wrote:
>
>> Hi Lars/Morten,
>>
>> We want to remove the limit on the number of users to send a DHIS2(2.23)
>> message. At the moment the limit is 200 users per group.
>>
>> I would like to implement this change since we have a system that need to
>> send 10 000+ emails to our users for notifications.
>>
>> Was the limitation created to resolve server health issues when sending
>> bulk emails?
>>
>> Regards,
>> Mike
>>
>>
>
>
>
> --
> Dr Sean Broomhead
> Director ICT
> HISP
> +27 79 496 1993
> Skype: getsean
>
> *This message and any attachments are subject to a disclaimer published at
> http://www.hisp.org/policies.html#comms_disclaimer
> .  Please read the
> disclaimer before opening any attachment or taking any other action in
> terms of this electronic transmission.  If you cannot access the
> disclaimer, kindly send an email to disclai...@hisp.org
>  and a copy will be provided to you. By replying to
> this e-mail or opening any attachment you agree to be bound by the
> provisions of the disclaimer.*
>



-- 

***

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedb...@gmail.com

Skype: calle_hedberg

***
___
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help   : https://help.launchpad.net/ListHelp

Re: [Dhis2-devs] GIS 2.25

[Calle Hedberg]

Absalom,

Postgis 9.5.3 don't exist... The latest version of postgis to use with
PostgreSQL 9.5.2+ is Postgis 2.3.0

just note this from the download instructions:
*Make sure you are using PostgreSQL 9.5.2+ or PostgreSQL 9.4.7+ or
PostgreSQL 9.6.0+.*

Otherwise follow Morten's advice - press CTRL-SHIFT-J to open the Chrome
console (then check the Console tab), and look for errors which might give
a clue.

Regards
Calle

On 24 October 2016 at 06:54, Morten Olav Hansen  wrote:

> Hi
>
> Are you using chrome? could you open up the developer tools, and see if
> there is anything in the console. Sounds like it can't load because of some
> issues with your configuration.
>
> --
> Morten Olav Hansen
> Senior Engineer, DHIS 2
> University of Oslo
> http://www.dhis2.org
>
> On Mon, Oct 24, 2016 at 11:51 AM, Absolom MURAMIRA 
> wrote:
>
>> Dear Calle,
>>
>> I installed postgis 9.5.3. Would that be the reason?
>>
>> Sent from Yahoo Mail on Android
>> 
>>
>> On Mon, 24 Oct, 2016 at 0:35, Calle Hedberg
>>  wrote:
>> Absalom,
>>
>> You have installed postgis with pg 9.5.4?
>>
>> Regards
>> Calle
>>
>> On 22 October 2016 at 10:36, Absolom MURAMIRA 
>> wrote:
>>
>>> Dear All,
>>>
>>> I am testing release 2.25 and appreciate the developers' contribution to
>>> the good improvements to the DHIS2 application.
>>> Most of the apps are running well apart from the GIS app. It loads
>>> forever and cannot load its interface to enable me generate maps. Can
>>> someone please advise me if there is some configuration I need to enable
>>> first? I checked the demo site which is okay. the pivot tables and data
>>> visualizers are loading normally.
>>>
>>> The testing environment in Linux server 16.04 LTS and postgresql 9.5
>>>
>>> Thank you.
>>>
>>> Absolom
>>> Kigali
>>>
>>> __ _
>>> Mailing list: https://launchpad.net/~dhis2- devs
>>> 
>>> Post to : dhis2-devs@lists.launchpad.net
>>> Unsubscribe : https://launchpad.net/~dhis2- devs
>>> 
>>> More help   : https://help.launchpad.net/ ListHelp
>>> 
>>>
>>>
>>
>>
>> --
>>
>> ***
>>
>> Calle Hedberg
>>
>> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>>
>> Tel/fax (home): +27-21-685-6472
>>
>> Cell: +27-82-853-5352
>>
>> Iridium SatPhone: +8816-315-19119
>>
>> Email: calle.hedb...@gmail.com
>>
>> Skype: calle_hedberg
>>
>> ***
>>
>>
>> ___
>> Mailing list: https://launchpad.net/~dhis2-devs
>> Post to : dhis2-devs@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~dhis2-devs
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>


-- 

***

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedb...@gmail.com

Skype: calle_hedberg

***
___
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help   : https://help.launchpad.net/ListHelp

Re: [Dhis2-devs] [Dhis2-users] GIS failing to load in 2.25

[Calle Hedberg]

Morten,

i was not aware of the "log XHR requests" feature.

Is there a DHIS2-related guide to using advanced features of the Chrome
console somewhere?

Regards
Calle

On 24 October 2016 at 04:31, Morten Olav Hansen  wrote:

> Hi Sam
>
> This is caused by the metadata having some issues I'm guessing... I would
> try and have a look at the network panel (in the web inspector) to see
> exactly which request it stops on.. since you are using Chrome, you should
> be able to right click on the console and select "log XHR requests", that
> will hopefully give you a clue to exactly which request is failing.
>
> --
> Morten Olav Hansen
> Senior Engineer, DHIS 2
> University of Oslo
> http://www.dhis2.org
>
> On Mon, Oct 17, 2016 at 9:37 PM, Sam Kasozi  wrote:
>
>> Hi Lars, this is the original error message from an upgraded instance
>> from 2.25.
>>
>> The developer console, gives the error below:
>>
>> ext-all.js:15 Uncaught You're trying to decode an invalid JSON String:
>> Also in the attached./ Screen shot attached as well.
>>
>>
>> P.s With the new instance, I could not access the orgunits; nor were they
>> displaying as part of the maintenance, GIS module wasn't loading either -
>> It would be good to try out from your end.
>>
>> regards,
>>
>>
>> Sam Kasozi
>> +256 788-993-565
>> Skype: sam.kasoziug
>>
>> On Mon, Oct 17, 2016 at 4:44 PM, Lars Helge Øverland 
>> wrote:
>>
>>>
>>>
>>> On Mon, Oct 17, 2016 at 3:42 PM, Sam Kasozi  wrote:
>>>
 GIS is failing to load in a new instance of 2.25; not sure if this is a
 bug or whether some pre-configuration needs to happen.

 regards,


>>>
>>> Hi Sam,
>>>
>>> we need some more info, can you check in the browser developer console
>>> and post output? If nothing there, tomcat log is helpful.
>>>
>>> regards,
>>>
>>> Lars
>>>
>>>
>>>
>>>
>>>
>>>
 Sam Kasozi
 +256 788-993-565
 Skype: sam.kasoziug

 ___
 Mailing list: https://launchpad.net/~dhis2-users
 Post to : dhis2-us...@lists.launchpad.net
 Unsubscribe : https://launchpad.net/~dhis2-users
 More help   : https://help.launchpad.net/ListHelp


>>>
>>>
>>> --
>>> Lars Helge Øverland
>>> Lead developer, DHIS 2
>>> University of Oslo
>>> Skype: larshelgeoverland
>>> l...@dhis2.org
>>> http://www.dhis2.org 
>>>
>>>
>>
>> ___
>> Mailing list: https://launchpad.net/~dhis2-devs
>> Post to : dhis2-devs@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~dhis2-devs
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>
> ___
> Mailing list: https://launchpad.net/~dhis2-devs
> Post to : dhis2-devs@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~dhis2-devs
> More help   : https://help.launchpad.net/ListHelp
>
>


-- 

***

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedb...@gmail.com

Skype: calle_hedberg

***
___
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help   : https://help.launchpad.net/ListHelp

[Dhis2-devs] Fwd: Secundary geographical data needed for analysis

[Marc Garnica]

Any ideas on that?

Thanks a lot,
Marc
-- Forwarded message --
From: Marc Garnica 
Date: 2016-10-21 17:03 GMT+02:00
Subject: Secundary geographical data needed for analysis
To: dhis2-us...@lists.launchpad.net


Hi all,
First of all, thank you for the support and the new release, it really has
some nice new features that will help us a lot!. We want to share with you
a discussion we have been developing this week to see if you can give us
some advice.

For every single data entry (individual with or without registration and
aggregated data) we always need at least one geographical data which refers
to where the data comes from. This is information can be selected in the
interface through the Org unit tree in the left.


But we sometimes need and additional geographical data referring to where
the patient comes from. In this case we add a data element to add this
data. This information is so important to create great analysis and reports
so we need to think well how we want to collect this geographical data.


The first option we did was just create a typed TEXT data element where the
user could enter manually the village where the patient came from. But
obviously, this was not correct enough because the user can write the same
village in more than one spelling way and also there was no feasible way to
use this data element as a dimension in the analysis so we were not able to
map the data in a map of villages or create a table with the cases by
village. So we need to formulate a new treatment


*POSSIBILITIES*

*1.   **CAPTURE THE VILLAGE AS A SET OF COORDINATES*

This options consists in create a data element with type COORDINATES and
add it to the form. With this options we can add a the geographical
information of where the patient comes from by providing the coordinates of
the village or even better (in new versions of DHIS2) we can click on the
map, search for the village in the search bar and capture the coordinates
of it visually.

Nice feature but then the village information is only stored as a pair of
coordinates, there is no auxiliary information as for instance its name. In
the analytical point of view we can now show the cases in a map distributed
by village (which is nice) but we cannot create the desired table with the
cases by village.

Another issue in this option is that for the end-user sometimes it will be
difficult to interact with the map and be able to capture the correct
village.

*2.   **CAPTURE THE VILLAGE AS A ORGANISATION  UNIT*

This option appeared once 2.25 DHIS2 version was released with a new
Organisation Unit Type for data elements. Using that, we can create a new
data element with type = Organisation unit to capture the information of
where the patient comes from.

So in this option we can have a user with data capture permission only on
the Sierra Leona hierarchy of organisation units but able to select a
village for example in Benin hierarchy of organisation units.

Even though the Organisation unit type data element is the more natural way
to specify where the patient comes from, we experienced some problems.
Firstly, we are not able to use this data as a dimension for aggregate
events information (for instance knowing how many events are registered
with a patient coming from X village). And secondly, this option will mean
to add all the villages included in our area of research which may lead to
memory problems in our server.

*3.   **CAPTURE THE VILLAGE AS A RELATIONSHIP WITH THE PATIENT*

The 3rd option is to use the “Relationship” implemented in Tracker Capture
App. This app can create links between different entities registered in the
system; originally it was created for tracking a pregnant patient and then
registers to the system her new child. But we could use this relationship
in another way just registering villages in the system and then linking
these registrations with the patient. This is a more complex way to
implement our needs and it needs a lot of research to know if we will be
able to show the relationship in a map and to creates tables from them.



As I said, thank you very much. We look forward to your answer.


Marc Garnica
___
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help   : https://help.launchpad.net/ListHelp

Re: [Dhis2-devs] Removing the limit on the number of users to send a message

[Sean Broomhead]

Hello Lars and team - thanks for having a look at this for us.

The requirement has emerged out of the success of the system Mike built to
manage placement allocations for all Interns and Community Service Doctors
in South Africa. Newly qualified doctors apply centrally, expressing their
priority choices for hospital placements for their two years of internship
and one year of community service. We run an algorithm to determine which
hospital each doctor gets sent to.

Both doctors and government love the system. It's email tool now provides
the formal mechanism for communicating with these doctors about the
placement process. Unfortunately, with the batch size limits, we're having
to setup the small batches manually. The current cohort of doctors is about
4,000 and the next cohort of nurses could reach 10,000.

Please advise if this needs to go into Launchpad or GitHub, or if you deal
with this sort of request differently.

Your assistance will be very helpful.

Best regards,
Sean

On Mon, Oct 24, 2016 at 8:03 AM, Mike Nelushi 
wrote:

> Hi Lars/Morten,
>
> We want to remove the limit on the number of users to send a DHIS2(2.23)
> message. At the moment the limit is 200 users per group.
>
> I would like to implement this change since we have a system that need to
> send 10 000+ emails to our users for notifications.
>
> Was the limitation created to resolve server health issues when sending
> bulk emails?
>
> Regards,
> Mike
>
>



-- 
Dr Sean Broomhead
Director ICT
HISP
+27 79 496 1993
Skype: getsean

-- 


*This message and any attachments are subject to a disclaimer published at 
http://www.hisp.org/policies.html#comms_disclaimer 
.  Please read the 
disclaimer before opening any attachment or taking any other action in 
terms of this electronic transmission.  If you cannot access the 
disclaimer, kindly send an email to disclai...@hisp.org 
 and a copy will be provided to you. By replying to 
this e-mail or opening any attachment you agree to be bound by the 
provisions of the disclaimer.*
___
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help   : https://help.launchpad.net/ListHelp

[Dhis2-devs] Removing the limit on the number of users to send a message

[Mike Nelushi]

Hi Lars/Morten,

We want to remove the limit on the number of users to send a DHIS2(2.23)
message. At the moment the limit is 200 users per group.

I would like to implement this change since we have a system that need to
send 10 000+ emails to our users for notifications.

Was the limitation created to resolve server health issues when sending
bulk emails?

Regards,
Mike
___
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help   : https://help.launchpad.net/ListHelp