There are open source implementations, embedded in tools like OpenJump and TalendSDI http://www.vividsolutions.com/jts/javadoc/com/vividsolutions/jts/simplify/TopologyPreservingSimplifier.html
Given the lincenses line up we could probably integrate parts (rather than the whole Talend SDI) On Fri, May 21, 2010 at 2:30 PM, Bob Jolliffe <[email protected]> wrote: > On 21 May 2010 13:25, Bob Jolliffe <[email protected]> wrote: >> Generating an orgunit hierarchy in dxf is relatively straightforward. >> As long as you have some kind of parent id reference to work with. >> >> I am not sure of the algorithm to use for the simplification part but >> I guess it must be pretty standard? You are just reducing the number >> of points on a polygon right? My rusty maths could probably figure >> out an algorithm but this has got to already exist. Does anyone have >> any pointers? > > Google seems to have knowledge of some such classes ... someone with > greater GIS knowledge would have to assess what is required. > Something like this: > http://edndoc.esri.com/arcobjects/9.0/Samples/Geodatabase/Creating_and_Converting_Data/Simplify_feature_geometry_for_a_shapefile/Simplify_feature_geometry_for_a_shapfile.htm > >>If there is some reasonable java code to do this, then >> I would create a java class to do it somewhere in dhis and make that >> class available as a an extension to the xalan xslt processor. That >> sounds complicated but its not really. I did something similar with >> calculating dates off excel's (dodgy) date representation. >> >> Can you dump your data source (or some of it) into some kind of xml >> and I can take a quick look at it. >> >> Regarding importing orgunits from excel templates I have some >> unfinished work which has been lying on the backburner while I have >> been negotiating message formats with openmrs/ihris folk this week. >> Basically still need to tidy up importing from zip containers. I hope >> to get back to that soon. But that should not prevent this scenario >> from working. Regarding ids I think its right to avoid getting >> complicated with the database stuff. In the dxf we can just number >> them 1,2,3,4.. etc. The DHIS convertor will worry about what the >> actual database ids are. >> >> Regards >> Bob >> >> On 21 May 2010 11:09, Knut Staring <[email protected]> wrote: >>> You are quite right that we need to coordinate this with bulk orgunit >>> import - to some extent the hope is that if the admin boundaries we >>> have access to are reasonably current, the map import would obviate >>> the need for bulk orgunit import. >>> >>> It does seem best to route everything through the import functionality >>> more than the ad-hoc student code (if we have it), though perhaps we >>> can use good pieces of it. However, because of the particular issues >>> of simplification and conversion to geojson, I'm not sure it makes >>> sense to go via Excel. What is the current status, Bob? Does it make >>> sense to try and splice these processes now, or should we defer this? >>> >>> Knut >>> >>> On Fri, May 21, 2010 at 11:48 AM, Ola Hodne Titlestad >>> <[email protected]> wrote: >>>> We have already a blueprint on importing orgunit hierarchies off structured >>>> excel files and this seems very related. >>>> I am sure Bob can do some transformation magic to the shapefile information >>>> using level ids and orgunit names to create orgunit hierarchies off these, >>>> either directly or we find a way to convert the shapefile info to the >>>> format >>>> of the structured excel format we recommend for bulk orgunit import. >>>> >>>> Ola Hodne Titlestad |Technical Officer| >>>> Health Metrics Network (HMN) | World Health Organization >>>> Avenue Appia 20 |1211 Geneva 27, Switzerland | Email: >>>> [email protected]|Tel: +41 788216897 >>>> Website: www.healthmetricsnetwork.org >>>> >>>> Better Information. Better Decisions. Better Health. >>>> >>>> >>>> On 21 May 2010 11:42, Jason Pickering <[email protected]> wrote: >>>>> >>>>> I have been looking for this code for years, but apparently it was >>>>> "lost" when the server crashed a few years back. >>>>> >>>>> Would agree though doing it in Java would be a better idea. >>>>> Manipulating the DB is tricky and you need to be very careful, (for >>>>> instance, using the hibernate_sequence values for primary keys). But >>>>> in the absence of this code, we may have to document situations where >>>>> bulk importation /conversion of external hierarchies is going to be >>>>> required. >>>>> >>>>> Regards, >>>>> JPP >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On 5/21/10, Ola Hodne Titlestad <[email protected]> wrote: >>>>> > Knut, >>>>> > >>>>> > Remember that we some years back had a student group working on >>>>> > converting >>>>> > shapefiles (basically a country set of level ids) to a DHIS orgunit >>>>> > hierarchy. >>>>> > Not sure where that code is now, but they certainly went through a >>>>> > similar >>>>> > process like the one you are describing. >>>>> > To me it seems it would be more robust to include this in the import >>>>> > module >>>>> > and take care of this hierarchy generation in the java code rather than >>>>> > manually manipulating the database. >>>>> > >>>>> > Ola >>>>> > -------- >>>>> > >>>>> > On 21 May 2010 11:17, Jason Pickering <[email protected]> >>>>> > wrote: >>>>> > >>>>> >> OK, you are all over the place on this one. Lets take it one at a time. >>>>> >> >>>>> >> Use of the hibernate_sequence is a very good idea (as I found out the >>>>> >> hard way) as it is easy to use external tools to generate ids, but you >>>>> >> have no guarentee that this will not clash with something that DHIS2 >>>>> >> inserts into the DB. >>>>> >> >>>>> >> I think you will need a staged approach. Dump everything into a >>>>> >> temporary table, and use the hibernate sequence to get a new primary >>>>> >> key. Use the parentID (mapped to the code field) to then update the >>>>> >> parentID field of the child. It should be pretty easily done, as the >>>>> >> parent/child relationship is implicit in the WHO levelid. If you >>>>> >> already have a parentID in the WHO information, then it should be even >>>>> >> easier. >>>>> >> >>>>> >> You do not need to use FME to generate the ID. Just use something >>>>> >> like.. >>>>> >> >>>>> >> INSERT INTO organisationunit_temp (organisationunitid....) >>>>> >> VALUES (nextval('hibernate_sequence',....) >>>>> >> >>>>> >> You may need to remove the parentid primary key constraint during the >>>>> >> initial insert and then reconstruct them using an update statement. I >>>>> >> do not know exactly what the statement would be, but I was almost >>>>> >> certain I had written this before at some point in time . >>>>> >> >>>>> >> I do not think that this is enough of a justification to increase the >>>>> >> size of the organisationunitid field, as it should be big enough to >>>>> >> accommodate any realistic orgunit hierarchy. >>>>> >> >>>>> >> In general, I would suggest the use of a view to present to PostGIS >>>>> >> instead of directly linking to the table itself. Of course, there are >>>>> >> other problems with persisting views in DHIS2 which we are aware of, >>>>> >> but I do not anticipate that this table would ever be deleted, so it >>>>> >> should be pretty safe. >>>>> >> >>>>> >> Also, you may want to consider GeoKettle or Talend, as something that >>>>> >> could be integrated into DHIS2 for processing of the Geodata. >>>>> >> >>>>> >> Regards, >>>>> >> Jason >>>>> >> >>>>> >> >>>>> >> On 5/21/10, Knut Staring <[email protected]> wrote: >>>>> >> > Using the hibernate_sequence seems like a good idea in most cases, >>>>> >> > but >>>>> >> > for Orgunits it's really crucial to populate the parentid field >>>>> >> > (which >>>>> >> > of course would also have to change to bigint for this to make any >>>>> >> > sense). >>>>> >> > >>>>> >> > So while I agree that the original alphanumeric/string LVLID would >>>>> >> > fit >>>>> >> > well in the Code field, I need to be able to populate the hierarchy >>>>> >> > for the whole world from the database. I could conceivably come up >>>>> >> > with a script in FME to generate sequential IDs, but that seems quite >>>>> >> > complicated, and would also not use hibernate_sequence (possibly I >>>>> >> > just don't know enough about how to use that). I use FME mainly >>>>> >> > because I have not found a good alternative for simplifying polygons >>>>> >> > without causing cracks between them. It would in some ways be nice to >>>>> >> > be able to do everything in PostGIS which has functions like >>>>> >> > ST_AsGeoJSON and Simplify, but as you can see from the below link, >>>>> >> > the >>>>> >> > results are not quite satisfying: >>>>> >> > >>>>> >> > http://bostongis.org/PrinterFriendly.aspx?content_name=postgis_simplify >>>>> >> > >>>>> >> > mapshaper.org seem to have some of the same problems, which have been >>>>> >> > avoided by Bjørn Sandvik when he made these world datasets: >>>>> >> > http://thematicmapping.org/downloads/world_borders.php. The tool he >>>>> >> > used for simplfying is ArcToolbox Simplify Polygon tool (see page 19 >>>>> >> > of this master thesis: >>>>> >> > http://thematicmapping.org/downloads/Thematic_Mapping_Engine.pdf). >>>>> >> > Unfortunately, FME and ArcToolbox are not integratable to DHIS2. >>>>> >> > >>>>> >> > While on this topic, I do think we perhaps need to add a LEVEL field >>>>> >> > to the ORGANISATIONUNIT table. That would make it quite corresponding >>>>> >> > to a PostGIS table (separatable on the LEVEL field in order to >>>>> >> > generate layers for Provinces, Districts etc). This is sort of >>>>> >> > available in the generated ORGUNITSTRUCTURE table, but that a) needs >>>>> >> > to be generated and b) seems a bit inefficient to have to join to >>>>> >> > another big table just to get the level. And perhaps we might want to >>>>> >> > have a separate table in DHIS2 with the full precision technologies >>>>> >> > and a link to the orgunit table. >>>>> >> > >>>>> >> > Knut >>>>> >> > >>>>> >> > On Fri, May 21, 2010 at 10:28 AM, Jason Pickering >>>>> >> > <[email protected]> wrote: >>>>> >> >> I do not really have a problem with this, but shouldn't this >>>>> >> >> information go in the "code" field? Or is it a problem with the >>>>> >> >> number >>>>> >> >> of orgunits? It would seem unlikely that we would ever have more >>>>> >> >> than >>>>> >> >> 2,147,483,647 orgunits. >>>>> >> >> >>>>> >> >> Are you inserting the ID as the organisationunitid? This seems this >>>>> >> >> might cause problems with possible clashes with the >>>>> >> >> hibernate_sequence >>>>> >> >> which is used to generate IDs? >>>>> >> >> >>>>> >> >> I have run into this issue only once, but since then, I always use >>>>> >> >> the >>>>> >> >> hibernate_sequence to generate IDs when I directly insert data into >>>>> >> >> the DB. >>>>> >> >> >>>>> >> >> Regards, >>>>> >> >> Jason >>>>> >> >> >>>>> >> >> >>>>> >> >> On 5/20/10, Knut Staring <[email protected]> wrote: >>>>> >> >>> Hello, >>>>> >> >>> >>>>> >> >>> In the process of converting WHO identifiers for administrative >>>>> >> >>> boundaries (LVLID) to ids usable for DHIS2, I've run into the >>>>> >> >>> limits >>>>> >> >>> of the integer datatype we use in DHIS2. >>>>> >> >>> >>>>> >> >>> The LVLID is a three letter ISO code followed by 18 digits. We are >>>>> >> >>> converting the alphabetical ISO for the country to an ISO numeric >>>>> >> >>> code >>>>> >> >>> (preceeded by 1 to make it numeric). >>>>> >> >>> >>>>> >> >>> Would it be problematic to change the datatype for >>>>> >> >>> organisationunitid >>>>> >> >>> from integer to bigint? >>>>> >> >>> >>>>> >> >>> Knut >>>>> >> >>> >>>>> >> >>> _______________________________________________ >>>>> >> >>> Mailing list: >>>>> >> >>> >>>>> >> >>> https://launchpad.net/~dhis2-devs<https://launchpad.net/%7Edhis2-devs> >>>>> >> >>> Post to : [email protected] >>>>> >> >>> Unsubscribe : >>>>> >> >>> >>>>> >> >>> https://launchpad.net/~dhis2-devs<https://launchpad.net/%7Edhis2-devs> >>>>> >> >>> More help : https://help.launchpad.net/ListHelp >>>>> >> >>> >>>>> >> >> >>>>> >> >> >>>>> >> >> -- >>>>> >> >> -- >>>>> >> >> Jason P. Pickering >>>>> >> >> email: [email protected] >>>>> >> >> tel:+260968395190 >>>>> >> >> >>>>> >> > >>>>> >> > >>>>> >> > >>>>> >> > -- >>>>> >> > Cheers, >>>>> >> > Knut Staring >>>>> >> > >>>>> >> >>>>> >> >>>>> >> -- >>>>> >> -- >>>>> >> Jason P. Pickering >>>>> >> email: [email protected] >>>>> >> tel:+260968395190 >>>>> >> >>>>> >> _______________________________________________ >>>>> >> Mailing list: >>>>> >> https://launchpad.net/~dhis2-devs<https://launchpad.net/%7Edhis2-devs> >>>>> >> Post to : [email protected] >>>>> >> Unsubscribe : >>>>> >> https://launchpad.net/~dhis2-devs<https://launchpad.net/%7Edhis2-devs> >>>>> >> More help : https://help.launchpad.net/ListHelp >>>>> >> >>>>> > >>>>> >>>>> >>>>> -- >>>>> -- >>>>> Jason P. Pickering >>>>> email: [email protected] >>>>> tel:+260968395190 >>>> >>>> >>> >>> >>> >>> -- >>> Cheers, >>> Knut Staring >>> >>> _______________________________________________ >>> Mailing list: https://launchpad.net/~dhis2-devs >>> Post to : [email protected] >>> Unsubscribe : https://launchpad.net/~dhis2-devs >>> More help : https://help.launchpad.net/ListHelp >>> >> > -- Cheers, Knut Staring _______________________________________________ Mailing list: https://launchpad.net/~dhis2-devs Post to : [email protected] Unsubscribe : https://launchpad.net/~dhis2-devs More help : https://help.launchpad.net/ListHelp
