Hi Uwe, Are you importing this as aggregate data or as events?
Regards, Jason On Wed, Jun 8, 2016 at 2:27 AM, Morten Olav Hansen <[email protected]> wrote: > Just to make sure, we are talking about the same thing: the problem does >> not >> appear during import, but when generating of all possible combinations >> (when >> saving the CategoryCombination or when manually evoking the update of >> categoryOptionCombinations) >> > > Ah, sorry.. I was thinking it was the import that was slow.. so that part > is ok? > > >> so I can still use /api/metadata without version to call the current >> api-version? >> > > That will give you the legacy importer, so going forward you would need to > use /api/{version}/{endpoint}, we will have more > info about it in the release notes. > > And no, the UI is not switched to new importer yet (in 2.24), not 100% it > will... > > >> >> Thanks for your replies at this time of the day :-) >> >> Regards, Uwe >> >> --- >> >> >> > Morten Olav Hansen <[email protected]> hat am 7. Juni 2016 um 19:28 >> > geschrieben: >> > >> > >> > Hi Uwe >> > >> > The improvements are mainly for speed and validation. Yes, we are now >> (in >> > 2.24) introducing versioned web-api, so that endpoint importer will be >> > available until 2.26 (we will support 3 versions). In 2.24, the same >> > endpoint is available at /api/24/metadata. >> > >> > If you are using cURL, or another utility.. the import part would be the >> > same, but the UI in 2.23 can not be used, as it's hardcoded to legacy >> > importer. >> > >> > -- >> > Morten Olav Hansen >> > Senior Engineer, DHIS 2 >> > University of Oslo >> > http://www.dhis2.org >> > >> > On Tue, Jun 7, 2016 at 11:25 PM, Uwe Wahser <[email protected]> wrote: >> > >> > > Hi Morten, >> > > >> > > no, i didn't. What would be the procedure for that? Importing >> Categories, >> > > Options and CategoryCombinations via api and having DHIS2 generate the >> > > CategoryOptionCombinations? Would that bring about any change at all >> or >> > > does the >> > > importer use different libs for generating the COCs? >> > > >> > > btw. is the 23 in the api link valid for future dhis2 versions? I >> noticed >> > > it in >> > > a few api descriptions recently ... >> > > >> > > Regards, Uwe >> > > >> > > > Morten Olav Hansen <[email protected]> hat am 7. Juni 2016 um 18:50 >> > > > geschrieben: >> > > > >> > > > >> > > > Hi Uwe >> > > > >> > > > Did you try out new importer? Available as /api/23/metadata in 2.23 >> > > > >> > > > On Tuesday, 7 June 2016, Uwe Wahser <[email protected]> wrote: >> > > > >> > > > > Dear devs, >> > > > > >> > > > > I am experiencing problems when handling category combinations. >> Our >> > > > > protoype >> > > > > with 5 dimensions went through the process of generating >> > > > > categoryOptionCombinations (~20.000 records) quite well. 7 >> dimensions >> > > > > (~400.000) >> > > > > worked as well, although it took a very long time. >> > > > > >> > > > > Now we defined the next datamodel with 10 dimensions (expecting >> ~5Mio >> > > > > categoryOptionCombinations) and the process dies without further >> > > notice. >> > > > > Last >> > > > > words in catalina.out: >> > > > > * INFO 2016-06-07 13:29:33,783 Building object-bridge maps >> > > (preheatCache: >> > > > > true, >> > > > > 3 classes). (DefaultObjectBridge.java [http-bio-8180-exec-15]) >> > > > > * INFO 2016-06-07 13:29:36,779 Building object-bridge maps took >> 2.99 >> > > > > seconds. >> > > > > (DefaultObjectBridge.java [http-bio-8180-exec-15]) >> > > > > * INFO 2016-06-07 13:29:36,896 'admin' update >> > > > > org.hisp.dhis.dataelement.DataElementCategoryCombo, name: >> Membership, >> > > uid: >> > > > > SCgLXYHqVzz (AuditLogUtil.java [http-bio-8180-exec-15]) >> > > > > >> > > > > Ten dimensions with not extraordinarily big option sets is >> actually not >> > > > > unusual >> > > > > and rather slim for multi-dimensional data-models in data >> warehouses, >> > > so >> > > > > I'd >> > > > > expect DHIS2 to be able to handle this easily. >> > > > > >> > > > > Could of course be a memory problem (tried up to 14g for tomcat >> on a >> > > 4-core >> > > > > Ubuntu 14.04 server, DHIS 2.23) Before I'll start experimenting >> with >> > > other >> > > > > parameters, I am hoping to get some hints on known limitations or >> > > > > workarounds >> > > > > from you (not allowed: reducing the number of options or >> categories, >> > > > > sql-hacks >> > > > > :-) ). Is there any info on whether optimizations on this process >> are >> > > being >> > > > > planned in the kernel? >> > > > > >> > > > > Some observations on the process: >> > > > > >> > > > > * during generation (either when saving the categoryCombination >> or in >> > > the >> > > > > data >> > > > > maintenance menu): >> > > > > - long names - cOCs are generated with generated names that are >> getting >> > > > > extremely long as they are mere concats of the involved >> > > categoryOptions. >> > > > > Could >> > > > > there be an option to just use the codes as basis or to leave >> away the >> > > > > names >> > > > > completely? Could be one reason for a memory problem and >> performance >> > > > > issues. >> > > > > - long log entries - every single entry is logged in catalina.out >> with >> > > > > several >> > > > > lines of text, causing catalina to become extremely big. >> > > > > - during execution lots of Java-memory are being used and no >> DB-memory, >> > > > > which >> > > > > looks to me as if all the logic is happening in the java machine. >> It >> > > might >> > > > > be >> > > > > more usefull to transfer more logic into SQLs to the DB (e.g. use >> DB >> > > > > cross-joins >> > > > > for combining options) as the DB will be more efficient. >> > > > > - because of the log entries I assume that every single >> combination is >> > > > > being >> > > > > persisted into the DB with a single SQL statement, causing >> millions of >> > > > > single >> > > > > SQL requests. Prefer batch SQL instead of single record >> processing. >> > > > > >> > > > > * during import/export of categoryOptionCombinations: >> > > > > - prefer batch SQL instead of single record processing >> > > > > - huge log entries in catalina.out due to several lines of text >> per >> > > > > combination >> > > > > >> > > > > I'd be very happy about comments. >> > > > > >> > > > > Thanks in advance, >> > > > > >> > > > > Uwe >> > > > > >> > > > > _______________________________________________ >> > > > > Mailing list: https://launchpad.net/~dhis2-users >> > > > > Post to : [email protected] <javascript:;> >> > > > > Unsubscribe : https://launchpad.net/~dhis2-users >> > > > > More help : https://help.launchpad.net/ListHelp >> > > > > >> > > > >> > > > >> > > > -- >> > > > -- >> > > > Morten Olav Hansen >> > > > Senior Engineer, DHIS 2 >> > > > University of Oslo >> > > > http://www.dhis2.org >> > > >> > > > _______________________________________________ > Mailing list: https://launchpad.net/~dhis2-devs > Post to : [email protected] > Unsubscribe : https://launchpad.net/~dhis2-devs > More help : https://help.launchpad.net/ListHelp > > -- Jason P. Pickering email: [email protected] tel:+46764147049
_______________________________________________ Mailing list: https://launchpad.net/~dhis2-users Post to : [email protected] Unsubscribe : https://launchpad.net/~dhis2-users More help : https://help.launchpad.net/ListHelp
