On 21 May 2010 13:58, Knut Staring <[email protected]> wrote: > On Fri, May 21, 2010 at 2:35 PM, Bob Jolliffe <[email protected]> wrote: >> On 21 May 2010 13:30, Knut Staring <[email protected]> wrote: >>> On Fri, May 21, 2010 at 2:25 PM, Bob Jolliffe <[email protected]> wrote: >>>> Generating an orgunit hierarchy in dxf is relatively straightforward. >>>> As long as you have some kind of parent id reference to work with. >>>> >>>> I am not sure of the algorithm to use for the simplification part but >>>> I guess it must be pretty standard? You are just reducing the number >>>> of points on a polygon right? My rusty maths could probably figure >>>> out an algorithm but this has got to already exist. Does anyone have >>>> any pointers? >>> >>> Yes - there is Douglas-Peucker, Visvalingam and Special Visvalingam, >>> you can test them at the following link. But some of these create >>> cracks between polygons. >>> http://mapshaper.com/test/demo.html >> >> Ok. This an online shaper. Can't see any code to download :-) I >> guess the three names above are the algorithms? Any idea what the >> cracks are about? Faulty algorithm, faulty implementation of >> algorithm or faulty shapefiles to start with. Might be the shapefiles >> need to be pre-processed (cleaned) before transforming. > > Not quite sure, maybe it is linked to polygons not really sharing > borders, just overlapping, and then Switzerland's border with Germany > gets simplified differently from Germany's border with Switzerland. > Though I could be completely wrong - and it may also depend a lot on > the parameters one chooses. But it seems that the commercial tools > like FME and ArcToolbox may handle it better, or maybe the data was > just clearner, or that the tools do some automatic precleaning, as > opposed to the default versions of the algorithms as implemented in > the online Mapshaper and also in PostGIS (see below link) don't quite > preserve common borders between polygons. > > http://bostongis.org/PrinterFriendly.aspx?content_name=postgis_simplify > >>> >>>> If there is some reasonable java code to do this, then >>>> I would create a java class to do it somewhere in dhis and make that >>>> class available as a an extension to the xalan xslt processor. That >>>> sounds complicated but its not really. I did something similar with >>>> calculating dates off excel's (dodgy) date representation. > > I think this is interesting, as it would potentially allow people to > upload their own shapefiles without worrying about simplification and > conversion to GeoJSON. What will also be needed is a matching > algorithm - though Jan Henrik has already implemented identical > matching and manual visual matching in the client. > > By the way, in addition to simplification, there is "smoothing". Most > of this is available as part of the Java Topology Suite: > http://www.vividsolutions.com/jts/jtshome.htm > http://lists.refractions.net/pipermail/jump-users/2005-July/002564.html >
Yes well if someone else can provide the simplification/smoothing or what have you I can make sure its available to the input transform. Other than that I'm a bit out of my depth with these processes at the moment. Cheers Bob > Knut > >>>> Can you dump your data source (or some of it) into some kind of xml >>>> and I can take a quick look at it. >>> >>> Will send you something soon, just let me get to know Talend SDI a bit >>> first. >>> >>>> Regarding importing orgunits from excel templates I have some >>>> unfinished work which has been lying on the backburner while I have >>>> been negotiating message formats with openmrs/ihris folk this week. >>>> Basically still need to tidy up importing from zip containers. I hope >>>> to get back to that soon. But that should not prevent this scenario >>>> from working. Regarding ids I think its right to avoid getting >>>> complicated with the database stuff. In the dxf we can just number >>>> them 1,2,3,4.. etc. The DHIS convertor will worry about what the >>>> actual database ids are. >>> >>> True. We may want to keep the 21 character LVLIDs in the CODE field. >>> >>> Knut >>> >>>> Regards >>>> Bob >>>> >>>> On 21 May 2010 11:09, Knut Staring <[email protected]> wrote: >>>>> You are quite right that we need to coordinate this with bulk orgunit >>>>> import - to some extent the hope is that if the admin boundaries we >>>>> have access to are reasonably current, the map import would obviate >>>>> the need for bulk orgunit import. >>>>> >>>>> It does seem best to route everything through the import functionality >>>>> more than the ad-hoc student code (if we have it), though perhaps we >>>>> can use good pieces of it. However, because of the particular issues >>>>> of simplification and conversion to geojson, I'm not sure it makes >>>>> sense to go via Excel. What is the current status, Bob? Does it make >>>>> sense to try and splice these processes now, or should we defer this? >>>>> >>>>> Knut >>>>> >>>>> On Fri, May 21, 2010 at 11:48 AM, Ola Hodne Titlestad >>>>> <[email protected]> wrote: >>>>>> We have already a blueprint on importing orgunit hierarchies off >>>>>> structured >>>>>> excel files and this seems very related. >>>>>> I am sure Bob can do some transformation magic to the shapefile >>>>>> information >>>>>> using level ids and orgunit names to create orgunit hierarchies off >>>>>> these, >>>>>> either directly or we find a way to convert the shapefile info to the >>>>>> format >>>>>> of the structured excel format we recommend for bulk orgunit import. >>>>>> >>>>>> Ola Hodne Titlestad |Technical Officer| >>>>>> Health Metrics Network (HMN) | World Health Organization >>>>>> Avenue Appia 20 |1211 Geneva 27, Switzerland | Email: >>>>>> [email protected]|Tel: +41 788216897 >>>>>> Website: www.healthmetricsnetwork.org >>>>>> >>>>>> Better Information. Better Decisions. Better Health. >>>>>> >>>>>> >>>>>> On 21 May 2010 11:42, Jason Pickering <[email protected]> >>>>>> wrote: >>>>>>> >>>>>>> I have been looking for this code for years, but apparently it was >>>>>>> "lost" when the server crashed a few years back. >>>>>>> >>>>>>> Would agree though doing it in Java would be a better idea. >>>>>>> Manipulating the DB is tricky and you need to be very careful, (for >>>>>>> instance, using the hibernate_sequence values for primary keys). But >>>>>>> in the absence of this code, we may have to document situations where >>>>>>> bulk importation /conversion of external hierarchies is going to be >>>>>>> required. >>>>>>> >>>>>>> Regards, >>>>>>> JPP >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 5/21/10, Ola Hodne Titlestad <[email protected]> wrote: >>>>>>> > Knut, >>>>>>> > >>>>>>> > Remember that we some years back had a student group working on >>>>>>> > converting >>>>>>> > shapefiles (basically a country set of level ids) to a DHIS orgunit >>>>>>> > hierarchy. >>>>>>> > Not sure where that code is now, but they certainly went through a >>>>>>> > similar >>>>>>> > process like the one you are describing. >>>>>>> > To me it seems it would be more robust to include this in the import >>>>>>> > module >>>>>>> > and take care of this hierarchy generation in the java code rather >>>>>>> > than >>>>>>> > manually manipulating the database. >>>>>>> > >>>>>>> > Ola >>>>>>> > -------- >>>>>>> > >>>>>>> > On 21 May 2010 11:17, Jason Pickering <[email protected]> >>>>>>> > wrote: >>>>>>> > >>>>>>> >> OK, you are all over the place on this one. Lets take it one at a >>>>>>> >> time. >>>>>>> >> >>>>>>> >> Use of the hibernate_sequence is a very good idea (as I found out the >>>>>>> >> hard way) as it is easy to use external tools to generate ids, but >>>>>>> >> you >>>>>>> >> have no guarentee that this will not clash with something that DHIS2 >>>>>>> >> inserts into the DB. >>>>>>> >> >>>>>>> >> I think you will need a staged approach. Dump everything into a >>>>>>> >> temporary table, and use the hibernate sequence to get a new primary >>>>>>> >> key. Use the parentID (mapped to the code field) to then update the >>>>>>> >> parentID field of the child. It should be pretty easily done, as the >>>>>>> >> parent/child relationship is implicit in the WHO levelid. If you >>>>>>> >> already have a parentID in the WHO information, then it should be >>>>>>> >> even >>>>>>> >> easier. >>>>>>> >> >>>>>>> >> You do not need to use FME to generate the ID. Just use something >>>>>>> >> like.. >>>>>>> >> >>>>>>> >> INSERT INTO organisationunit_temp (organisationunitid....) >>>>>>> >> VALUES (nextval('hibernate_sequence',....) >>>>>>> >> >>>>>>> >> You may need to remove the parentid primary key constraint during the >>>>>>> >> initial insert and then reconstruct them using an update statement. I >>>>>>> >> do not know exactly what the statement would be, but I was almost >>>>>>> >> certain I had written this before at some point in time . >>>>>>> >> >>>>>>> >> I do not think that this is enough of a justification to increase the >>>>>>> >> size of the organisationunitid field, as it should be big enough to >>>>>>> >> accommodate any realistic orgunit hierarchy. >>>>>>> >> >>>>>>> >> In general, I would suggest the use of a view to present to PostGIS >>>>>>> >> instead of directly linking to the table itself. Of course, there are >>>>>>> >> other problems with persisting views in DHIS2 which we are aware of, >>>>>>> >> but I do not anticipate that this table would ever be deleted, so it >>>>>>> >> should be pretty safe. >>>>>>> >> >>>>>>> >> Also, you may want to consider GeoKettle or Talend, as something that >>>>>>> >> could be integrated into DHIS2 for processing of the Geodata. >>>>>>> >> >>>>>>> >> Regards, >>>>>>> >> Jason >>>>>>> >> >>>>>>> >> >>>>>>> >> On 5/21/10, Knut Staring <[email protected]> wrote: >>>>>>> >> > Using the hibernate_sequence seems like a good idea in most cases, >>>>>>> >> > but >>>>>>> >> > for Orgunits it's really crucial to populate the parentid field >>>>>>> >> > (which >>>>>>> >> > of course would also have to change to bigint for this to make any >>>>>>> >> > sense). >>>>>>> >> > >>>>>>> >> > So while I agree that the original alphanumeric/string LVLID would >>>>>>> >> > fit >>>>>>> >> > well in the Code field, I need to be able to populate the hierarchy >>>>>>> >> > for the whole world from the database. I could conceivably come up >>>>>>> >> > with a script in FME to generate sequential IDs, but that seems >>>>>>> >> > quite >>>>>>> >> > complicated, and would also not use hibernate_sequence (possibly I >>>>>>> >> > just don't know enough about how to use that). I use FME mainly >>>>>>> >> > because I have not found a good alternative for simplifying >>>>>>> >> > polygons >>>>>>> >> > without causing cracks between them. It would in some ways be nice >>>>>>> >> > to >>>>>>> >> > be able to do everything in PostGIS which has functions like >>>>>>> >> > ST_AsGeoJSON and Simplify, but as you can see from the below link, >>>>>>> >> > the >>>>>>> >> > results are not quite satisfying: >>>>>>> >> > >>>>>>> >> > http://bostongis.org/PrinterFriendly.aspx?content_name=postgis_simplify >>>>>>> >> > >>>>>>> >> > mapshaper.org seem to have some of the same problems, which have >>>>>>> >> > been >>>>>>> >> > avoided by Bjørn Sandvik when he made these world datasets: >>>>>>> >> > http://thematicmapping.org/downloads/world_borders.php. The tool he >>>>>>> >> > used for simplfying is ArcToolbox Simplify Polygon tool (see page >>>>>>> >> > 19 >>>>>>> >> > of this master thesis: >>>>>>> >> > http://thematicmapping.org/downloads/Thematic_Mapping_Engine.pdf). >>>>>>> >> > Unfortunately, FME and ArcToolbox are not integratable to DHIS2. >>>>>>> >> > >>>>>>> >> > While on this topic, I do think we perhaps need to add a LEVEL >>>>>>> >> > field >>>>>>> >> > to the ORGANISATIONUNIT table. That would make it quite >>>>>>> >> > corresponding >>>>>>> >> > to a PostGIS table (separatable on the LEVEL field in order to >>>>>>> >> > generate layers for Provinces, Districts etc). This is sort of >>>>>>> >> > available in the generated ORGUNITSTRUCTURE table, but that a) >>>>>>> >> > needs >>>>>>> >> > to be generated and b) seems a bit inefficient to have to join to >>>>>>> >> > another big table just to get the level. And perhaps we might want >>>>>>> >> > to >>>>>>> >> > have a separate table in DHIS2 with the full precision technologies >>>>>>> >> > and a link to the orgunit table. >>>>>>> >> > >>>>>>> >> > Knut >>>>>>> >> > >>>>>>> >> > On Fri, May 21, 2010 at 10:28 AM, Jason Pickering >>>>>>> >> > <[email protected]> wrote: >>>>>>> >> >> I do not really have a problem with this, but shouldn't this >>>>>>> >> >> information go in the "code" field? Or is it a problem with the >>>>>>> >> >> number >>>>>>> >> >> of orgunits? It would seem unlikely that we would ever have more >>>>>>> >> >> than >>>>>>> >> >> 2,147,483,647 orgunits. >>>>>>> >> >> >>>>>>> >> >> Are you inserting the ID as the organisationunitid? This seems >>>>>>> >> >> this >>>>>>> >> >> might cause problems with possible clashes with the >>>>>>> >> >> hibernate_sequence >>>>>>> >> >> which is used to generate IDs? >>>>>>> >> >> >>>>>>> >> >> I have run into this issue only once, but since then, I always use >>>>>>> >> >> the >>>>>>> >> >> hibernate_sequence to generate IDs when I directly insert data >>>>>>> >> >> into >>>>>>> >> >> the DB. >>>>>>> >> >> >>>>>>> >> >> Regards, >>>>>>> >> >> Jason >>>>>>> >> >> >>>>>>> >> >> >>>>>>> >> >> On 5/20/10, Knut Staring <[email protected]> wrote: >>>>>>> >> >>> Hello, >>>>>>> >> >>> >>>>>>> >> >>> In the process of converting WHO identifiers for administrative >>>>>>> >> >>> boundaries (LVLID) to ids usable for DHIS2, I've run into the >>>>>>> >> >>> limits >>>>>>> >> >>> of the integer datatype we use in DHIS2. >>>>>>> >> >>> >>>>>>> >> >>> The LVLID is a three letter ISO code followed by 18 digits. We >>>>>>> >> >>> are >>>>>>> >> >>> converting the alphabetical ISO for the country to an ISO numeric >>>>>>> >> >>> code >>>>>>> >> >>> (preceeded by 1 to make it numeric). >>>>>>> >> >>> >>>>>>> >> >>> Would it be problematic to change the datatype for >>>>>>> >> >>> organisationunitid >>>>>>> >> >>> from integer to bigint? >>>>>>> >> >>> >>>>>>> >> >>> Knut >>>>>>> >> >>> >>>>>>> >> >>> _______________________________________________ >>>>>>> >> >>> Mailing list: >>>>>>> >> >>> >>>>>>> >> >>> https://launchpad.net/~dhis2-devs<https://launchpad.net/%7Edhis2-devs> >>>>>>> >> >>> Post to : [email protected] >>>>>>> >> >>> Unsubscribe : >>>>>>> >> >>> >>>>>>> >> >>> https://launchpad.net/~dhis2-devs<https://launchpad.net/%7Edhis2-devs> >>>>>>> >> >>> More help : https://help.launchpad.net/ListHelp >>>>>>> >> >>> >>>>>>> >> >> >>>>>>> >> >> >>>>>>> >> >> -- >>>>>>> >> >> -- >>>>>>> >> >> Jason P. Pickering >>>>>>> >> >> email: [email protected] >>>>>>> >> >> tel:+260968395190 >>>>>>> >> >> >>>>>>> >> > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> > -- >>>>>>> >> > Cheers, >>>>>>> >> > Knut Staring >>>>>>> >> > >>>>>>> >> >>>>>>> >> >>>>>>> >> -- >>>>>>> >> -- >>>>>>> >> Jason P. Pickering >>>>>>> >> email: [email protected] >>>>>>> >> tel:+260968395190 >>>>>>> >> >>>>>>> >> _______________________________________________ >>>>>>> >> Mailing list: >>>>>>> >> https://launchpad.net/~dhis2-devs<https://launchpad.net/%7Edhis2-devs> >>>>>>> >> Post to : [email protected] >>>>>>> >> Unsubscribe : >>>>>>> >> https://launchpad.net/~dhis2-devs<https://launchpad.net/%7Edhis2-devs> >>>>>>> >> More help : https://help.launchpad.net/ListHelp >>>>>>> >> >>>>>>> > >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> -- >>>>>>> Jason P. Pickering >>>>>>> email: [email protected] >>>>>>> tel:+260968395190 >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Cheers, >>>>> Knut Staring >>>>> >>>>> _______________________________________________ >>>>> Mailing list: https://launchpad.net/~dhis2-devs >>>>> Post to : [email protected] >>>>> Unsubscribe : https://launchpad.net/~dhis2-devs >>>>> More help : https://help.launchpad.net/ListHelp >>>>> >>>> >>> >>> >>> >>> -- >>> Cheers, >>> Knut Staring >>> >> > > > > -- > Cheers, > Knut Staring > _______________________________________________ Mailing list: https://launchpad.net/~dhis2-devs Post to : [email protected] Unsubscribe : https://launchpad.net/~dhis2-devs More help : https://help.launchpad.net/ListHelp
