Thanks Halvdan and Calle.
Elmarie
On 30 Jun 2016 22:05, "Calle Hedberg" <[email protected]> wrote:
> Elmarie,
>
> It is a weakness with ANY import process in DHIS2 that there is no
> differentiation between actual updates - i.e. where an imported value is
> different from the existing one - and all other imported data records that
> match existing data records with identical values. All imported records
> matched to existing records are regarded as "updates", which is only
> differentiated from valid records with no match ("new" records). (valid as
> in having a valid primary key etc).
>
> DHIS 1.4 provides a far more fine-grained control over any import process,
> because imported data records are automatically identified as
> - New data records
> - Existing data records with different value and where the lastupdated is
> newer (i.e. source data value has been updated more recently than the
> destination database value)
> - Existing data records with different value and where the lastupdated is
> older (i.e. the destination database value has been updated more recently
> than the source database value)
> - Existing data records with identical value (lastupdated value is in that
> case not relevant).
> During the import process, the user can review these different categories
> of imported data and decide whether to accept or reject them. That is not
> possible with the DHIS2 import model - everything in the imported file that
> fits existing primary keys will be imported/updated automatically.
>
> I'm not 100% sure why the core team opted for this very "basic" import
> methodology, but I suspect it's largely related to the assumption that most
> DHIS2 instances will be national single instances without much need for
> import or export of data. I suspect that changing the import methodology to
> give users the same high-granularity control as in 1.4 would require a
> major effort, but diversifying the treatment of the lastupdated field only
> should be a lot simpler - presumably.
>
> Halvdan's suggestion that "if you wish to avoid this you can give the
> importer a GML file which only contains the orgunits you actually wish to
> update" is not a very practical option, regrettably. If, as in our typical
> case, you have somewhere between 5,000 and 40,0000 coordinates with
> additions and corrections taking place regularly through various external
> spatial databases, your primary option will be to access the database
> instance directly and run queries that will identify any new or updated
> co-ordinates by comparing the current external data set with whatever is
> currently stored in the instance. After identifying the sub-set of new or
> updated co-ordinates that way, you can then generate a new GML file OR you
> might rather use the same queries to update the organisationunit table
> directly.
>
> (Another more theoretical option will be that somebody keep track of all
> such changes in those external spatial databases and GIS systems, but since
> spatial data development processes typically incorporate a multitude of
> public and private organisations, it's again not practical to keep track of
> all the updates and additions they generate).
>
> For now, I guess the most efficient method will be to drop using GML file
> imports and instead update our PostgreSQL database instances directly with
> ACTUAL updates. It's not a very attractive option because it reduces
> security and increase the possibility of somebody making crippling updates
> to the database. Alternatively we use the database queries to identify new
> and actually updated coordinates, and then go through the process of
> generating a GML file with only those records (the latter method is more
> cumbersome, but it will work with Production instances where we have
> disabled direct database access for security reasons).
>
> I'll make sure we raise this issue again during our meetings in Oslo in
> August. Halvdan might find himself voted down on this one, we'll see ;-)
>
> Regards
> Calle
>
> On 30 June 2016 at 17:05, Halvdan Hoem Grelland <[email protected]> wrote:
>
>> I've investigated and retract that this is a bug. I guess I
>> misunderstood at first, but reading your report again makes it clear
>> that you are actually supplying a GML file with coordinates for all
>> orgunits, including those which already have coordinates.
>>
>> As you are, in fact, importing coordinates for all of these orgunits,
>> it's technically not wrong that the lastUpdated is also set to reflect
>> this. We don't really discern between old vs. new value of metadata when
>> updating on import.
>>
>> If you wish to avoid this you can give the importer a GML file which
>> only contains the orgunits you actually wish to update.
>>
>> Setting this one to won't fix.
>>
>> ** Changed in: dhis2
>> Status: Confirmed => Invalid
>>
>> --
>> You received this bug notification because you are a member of DHIS 2
>> developers, which is subscribed to DHIS.
>> https://bugs.launchpad.net/bugs/1597724
>>
>> Title:
>> Import of GML file update lastupdated date of all objects with co-
>> ordinates
>>
>> Status in DHIS:
>> Invalid
>>
>> Bug description:
>> Importing of a GML file updates the "lastupdated" date of all
>> organisationunit with GIS co-ordinates to the date of the import.
>>
>> Not sure if this is by design but it creates a problem for facility
>> registries in that it appears that all organisationunits with co-
>> ordinates had some field updated every time a GML file is imported
>> when in fact only some of the objects had updates.
>>
>> Ideally the import should only update new/real updates and for those
>> update the GIS co-ordinates instead of overwriting all co-ordinates
>> and seeing them as updates.
>>
>> To manage notifications about this bug go to:
>> https://bugs.launchpad.net/dhis2/+bug/1597724/+subscriptions
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~dhis2-devs
>> Post to : [email protected]
>> Unsubscribe : https://launchpad.net/~dhis2-devs
>> More help : https://help.launchpad.net/ListHelp
>>
>
>
>
> --
>
> *******************************************
>
> Calle Hedberg
>
> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>
> Tel/fax (home): +27-21-685-6472
>
> Cell: +27-82-853-5352
>
> Iridium SatPhone: +8816-315-19119
>
> Email: [email protected]
>
> Skype: calle_hedberg
>
> *******************************************
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~dhis2-devs
> Post to : [email protected]
> Unsubscribe : https://launchpad.net/~dhis2-devs
> More help : https://help.launchpad.net/ListHelp
>
>
--
*This message and any attachments are subject to a disclaimer published at
http://www.hisp.org/policies.html#comms_disclaimer
<http://www.hisp.org/policies.html#comms_disclaimer>. Please read the
disclaimer before opening any attachment or taking any other action in
terms of this electronic transmission. If you cannot access the
disclaimer, kindly send an email to [email protected]
<[email protected]> and a copy will be provided to you. By replying to
this e-mail or opening any attachment you agree to be bound by the
provisions of the disclaimer.*
_______________________________________________
Mailing list: https://launchpad.net/~dhis2-devs
Post to : [email protected]
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp
