Re: [Dhis2-devs] Response differences between /api/metadata and /api/23/metadata

[Lorill Crees]

Hi Morten,

At long last I am revisiting using the enhancement you made for including
the displayName in the Error reports documented here, with the
importReportMode import parameter:

https://dhis2.github.io/dhis2-docs/master/en/developer/html/webapi_metadata_import.html

However in our case this doesn't end up reducing the amount of data that we
need to pass around. Basically, the only time we need to have the
displayName included in the response is if it is associated with an Error
that we need to pass back to the user. With the three options "ERRORS",
"FULL", and "DEBUG", we can only get the displayName with the option
"DEBUG", which also returns a separate object report for every object we
posted, regardless of whether or not it was an error.

Is it possible to augment this functionality with a fourth option, which is
"ERRORS" only, but with the displayName included? We don't want to have the
"FULL" functionality that is included in "DEBUG" as the point of us using
this functionality was to reduce the amount of data being passed around in
order to get at the displayName property of an error. In versions prior to
2.23, we got this information returned by default.

Please let me know what you think, and if a fourth option could be added.
It would be basically "ERRORS_DEBUG".

Thanks,

Lorill


On Sun, Oct 23, 2016 at 7:40 PM, Morten Olav Hansen 
wrote:

>
> On Tue, Oct 18, 2016 at 11:51 PM, Lorill Crees  wrote:
>

>1. The /api/23/metadata response with "errorReports" within
>"objectReports" appears to have replaced the /api/metadata
>"importConflicts". In /api/metadata, the "importConflicts" "object" 
> field
>contains the actual name of the object that has the error (eg: the 
> Program
>short name in our case). However, in /api/23/metadata there is only the
>index which makes it more work to debug. Is it possible to also 
> include the
>name of the object in the error report like the prior API version does?
>
> Hm, right. I understand that it's useful, but that can easily add up
 to a lot of data returned. Maybe we should add some kind of "debug" mode
 where more output is provided.

>>>
>>> Although I understand that this might add more data to the payload, it
>>> would decrease the amount of data that we need to pass around in our app,
>>> and also potentially decrease the number of API calls we need to do. Not
>>> having this information in the response now means we need to pass around
>>> our original post data (which can be quite large, and which we aren't
>>> currently doing) and do a lookup by ID to send any kind of useful
>>> information back to the user. Additionally, we may have even posted these
>>> objects by ID only, which means we will now need to do another API call to
>>> look up the object to get useful information to send back to the user. Keep
>>> in mind that in our app, the user doesn't actually even have a knowledge of
>>> IDs - we are doing those lookups for them by name or code.
>>> Would it be possible to add a flag to the api call where we can request
>>> the name or shortname or code to be returned in addition to just the ID?
>>> Perhaps similar to the metadata field filter?  Where we could say supply
>>> importConflictsFields=:identifiable?
>>>
>>
>> Yes, let me see what I can do for 225 at least (I might be able to
>> backport to 224 if it doesn't disrupt things too much)
>>
>> https://blueprints.launchpad.net/dhis2/+spec/importer-parame
>> ter-to-adjust-importreport-output
>>
>
> Thanks Morten, that's great!
>

>> I see this blueprint was completed in 2.25 - great! Was this backported
>> to 2.24? Is there documentation for this?
>>
>
> Yes, this is also in 224, docs are here:
>
> https://dhis2.github.io/dhis2-docs/master/en/developer/html/
> webapi_metadata_import.html
>
> --
> Morten Olav Hansen
> Senior Engineer, DHIS 2
> University of Oslo
> http://www.dhis2.org
>



-- 
Lorill Crees
Project Leader / Senior Developer
2Paths Solutions Ltd. 

lcr...@2paths.com
skype: lorill2paths
(604) 689-4123 x 15
___
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help   : https://help.launchpad.net/ListHelp

Re: [Dhis2-devs] Response differences between /api/metadata and /api/23/metadata

[Morten Olav Hansen]

On Tue, Oct 18, 2016 at 11:51 PM, Lorill Crees  wrote:

> I see this blueprint was completed in 2.25 - great! Was this backported to
> 2.24? Is there documentation for this?
>

Yes, this is also in 224, docs are here:

https://dhis2.github.io/dhis2-docs/master/en/developer/html/webapi_metadata_import.html

-- 
Morten Olav Hansen
Senior Engineer, DHIS 2
University of Oslo
http://www.dhis2.org
___
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help   : https://help.launchpad.net/ListHelp

Re: [Dhis2-devs] Response differences between /api/metadata and /api/23/metadata

[Lorill Crees]

Hi Morten,

I'm just following up on the two items below (inline).

Thanks!

Lorill


On Fri, Sep 16, 2016 at 8:24 AM, Lorill Crees  wrote:

> Thanks for getting back to me so quickly Morten! I've answered inline
> below.
>
> On Thu, Sep 15, 2016 at 10:19 PM, Morten Olav Hansen 
> wrote:
>
>1. When submitting an identical post containing
>{"programTrackedEntityAttributes" [...], "program" [...]}, where there
>is an error in the Program being posted, the call to /api/metadata responds
>with success for "programTrackedEntityAttributes" with the correct
>"importCount" counts, but the call to /api/23/metadata responds for
>"programTrackedEntityAttributes" with "stats" counts containing all
>zeros, and no "objectReports". It is like the first item being posted is
>completely ignored if the second item posted has errors. Is this the
>intended behaviour? And should there be "objectReports" regardless? Please
>see attached examples.
>
> Yes, what kind of error are you getting? a reference error, or a
 validation error? in the old importer, it would always try to just import
 what it could.. which was not always ideal (missing references etc was just
 ignored). If it's just reference error, and you want to ignore them..
 please try with `atomicMode=NONE`, regarding the object report..

>>>
>>> We need to write our response parser to be able to parse any kind of
>>> error that may happen. Our program ingests CSV files from the user and we
>>> attempt to do as much error checking in advance as possible, but there may
>>> be other errors that happen when posting to the API such as validation
>>> errors, reference errors, or even 500 errors if there is an error within
>>> DHIS 2 itself. Given that we need to cater to every possible scenario, I'm
>>> not sure if we should be using "atomicMode=NONE" or not. Also, should there
>>> be "objectReports" regardless?
>>>
>>
>> I'm not 100% sure what is the best approach here, I think maybe we want
>> to add some new parameters to control the output a bit. If you are posting
>> 10 new data elements, it is very useful to get the object report back as
>> they will contain the new UID of the object, on the flipside.. if you are
>> importing 5 objects.. the payload sent back might just too much in most
>> cases, and not very useful (in that case, probably you are more interested
>> in errors)
>>
>
>
1. Adding new parameters to control the output is a great idea, is this
something that you plan to implement as part of the blueprint you mention
below?
2. I'm also wondering if it would make more sense to mark the ignored
objects as "ignored" in the stats, instead of having zeros for all the
stats? I've re-attached our output, and you can see that I posted 4
programTrackedEntityAttributes, and the stats show 0 total / 0 ignored
which is not entirely accurate. I would have expected 4 total / 4 ignored.
Thoughts?

>
>>
>
>>
>1. The /api/23/metadata response with "errorReports" within
>"objectReports" appears to have replaced the /api/metadata
>"importConflicts". In /api/metadata, the "importConflicts" "object" 
> field
>contains the actual name of the object that has the error (eg: the 
> Program
>short name in our case). However, in /api/23/metadata there is only the
>index which makes it more work to debug. Is it possible to also 
> include the
>name of the object in the error report like the prior API version does?
>
> Hm, right. I understand that it's useful, but that can easily add up
 to a lot of data returned. Maybe we should add some kind of "debug" mode
 where more output is provided.

>>>
>>> Although I understand that this might add more data to the payload, it
>>> would decrease the amount of data that we need to pass around in our app,
>>> and also potentially decrease the number of API calls we need to do. Not
>>> having this information in the response now means we need to pass around
>>> our original post data (which can be quite large, and which we aren't
>>> currently doing) and do a lookup by ID to send any kind of useful
>>> information back to the user. Additionally, we may have even posted these
>>> objects by ID only, which means we will now need to do another API call to
>>> look up the object to get useful information to send back to the user. Keep
>>> in mind that in our app, the user doesn't actually even have a knowledge of
>>> IDs - we are doing those lookups for them by name or code.
>>> Would it be possible to add a flag to the api call where we can request
>>> the name or shortname or code to be returned in addition to just the ID?
>>> Perhaps similar to the metadata field filter?  Where we could say supply
>>> importConflictsFields=:identifiable?
>>>
>>
>> Yes, let me see what I can do for 225 at least (I might be able to
>> backport to 224 if it doesn't disrupt things 

Re: [Dhis2-devs] Response differences between /api/metadata and /api/23/metadata

[Lorill Crees]

Thanks for getting back to me so quickly Morten! I've answered inline below.

On Thu, Sep 15, 2016 at 10:19 PM, Morten Olav Hansen 
wrote:

   1. When submitting an identical post containing
   {"programTrackedEntityAttributes" [...], "program" [...]}, where there
   is an error in the Program being posted, the call to /api/metadata responds
   with success for "programTrackedEntityAttributes" with the correct
   "importCount" counts, but the call to /api/23/metadata responds for
   "programTrackedEntityAttributes" with "stats" counts containing all
   zeros, and no "objectReports". It is like the first item being posted is
   completely ignored if the second item posted has errors. Is this the
   intended behaviour? And should there be "objectReports" regardless? Please
   see attached examples.

Yes, what kind of error are you getting? a reference error, or a validation
>>> error? in the old importer, it would always try to just import what it
>>> could.. which was not always ideal (missing references etc was just
>>> ignored). If it's just reference error, and you want to ignore them..
>>> please try with `atomicMode=NONE`, regarding the object report..
>>>
>>
>> We need to write our response parser to be able to parse any kind of
>> error that may happen. Our program ingests CSV files from the user and we
>> attempt to do as much error checking in advance as possible, but there may
>> be other errors that happen when posting to the API such as validation
>> errors, reference errors, or even 500 errors if there is an error within
>> DHIS 2 itself. Given that we need to cater to every possible scenario, I'm
>> not sure if we should be using "atomicMode=NONE" or not. Also, should there
>> be "objectReports" regardless?
>>
>
> I'm not 100% sure what is the best approach here, I think maybe we want to
> add some new parameters to control the output a bit. If you are posting 10
> new data elements, it is very useful to get the object report back as they
> will contain the new UID of the object, on the flipside.. if you are
> importing 5 objects.. the payload sent back might just too much in most
> cases, and not very useful (in that case, probably you are more interested
> in errors)
>

1. Adding new parameters to control the output is a great idea, is this
something that you plan to implement as part of the blueprint you mention
below?
2. I'm also wondering if it would make more sense to mark the ignored
objects as "ignored" in the stats, instead of having zeros for all the
stats? I've re-attached our output, and you can see that I posted
4 programTrackedEntityAttributes, and the stats show 0 total / 0 ignored
which is not entirely accurate. I would have expected 4 total / 4 ignored.
Thoughts?

>
>
>>
1. There is no longer a "lastImported" field. Is this now
deprecated?

 There should now be a id exported in the object report.. I think there
>>> might be some inconsistencies there though.. I will go through it, in
>>> general this field was mostly used for our /api/type/id endpoints.. were
>>> you also using it for /api/metadata? (it's not very useful unless you are
>>> only sending 1 single object)
>>>
>>
>> Agreed that using "lastImported" for /api/metadata is not very useful
>> unless sending one single object. However, we are using our generic
>> response parser for all DHIS 2 API endpoints and are trying to avoid the
>> need for special case logic dependent upon which specific endpoint was
>> called. Additionally, we do still use this field for many of the API calls
>> where we do send one single object. Is the plan to deprecate the
>> "lastImported" field across the API, or only in the /api/metadata call, so
>> we can adjust our parser accordingly?
>>
>
> The lastImported field has been removed all over, what is happening now,
> is that if you POST a new data element to /api/dataElements, you will get a
> single object report back.. and the uid is contained within there
>

Thanks.


>
1. The /api/23/metadata response with "errorReports" within
"objectReports" appears to have replaced the /api/metadata
"importConflicts". In /api/metadata, the "importConflicts" "object" 
 field
contains the actual name of the object that has the error (eg: the 
 Program
short name in our case). However, in /api/23/metadata there is only the
index which makes it more work to debug. Is it possible to also include 
 the
name of the object in the error report like the prior API version does?

 Hm, right. I understand that it's useful, but that can easily add up to
>>> a lot of data returned. Maybe we should add some kind of "debug" mode where
>>> more output is provided.
>>>
>>
>> Although I understand that this might add more data to the payload, it
>> would decrease the amount of data that we need to pass around in our app,
>> and also potentially decrease the number of API calls we need to do. Not
>> 

Re: [Dhis2-devs] Response differences between /api/metadata and /api/23/metadata

[Morten Olav Hansen]

>
> Yes, what kind of error are you getting? a reference error, or a
>> validation error? in the old importer, it would always try to just import
>> what it could.. which was not always ideal (missing references etc was just
>> ignored). If it's just reference error, and you want to ignore them..
>> please try with `atomicMode=NONE`, regarding the object report..
>>
>
> We need to write our response parser to be able to parse any kind of error
> that may happen. Our program ingests CSV files from the user and we attempt
> to do as much error checking in advance as possible, but there may be other
> errors that happen when posting to the API such as validation errors,
> reference errors, or even 500 errors if there is an error within DHIS 2
> itself. Given that we need to cater to every possible scenario, I'm not
> sure if we should be using "atomicMode=NONE" or not. Also, should there be
> "objectReports" regardless?
>

I'm not 100% sure what is the best approach here, I think maybe we want to
add some new parameters to control the output a bit. If you are posting 10
new data elements, it is very useful to get the object report back as they
will contain the new UID of the object, on the flipside.. if you are
importing 5 objects.. the payload sent back might just too much in most
cases, and not very useful (in that case, probably you are more interested
in errors)


>
>>>1. There is no longer a "lastImported" field. Is this now deprecated?
>>>
>>> There should now be a id exported in the object report.. I think there
>> might be some inconsistencies there though.. I will go through it, in
>> general this field was mostly used for our /api/type/id endpoints.. were
>> you also using it for /api/metadata? (it's not very useful unless you are
>> only sending 1 single object)
>>
>
> Agreed that using "lastImported" for /api/metadata is not very useful
> unless sending one single object. However, we are using our generic
> response parser for all DHIS 2 API endpoints and are trying to avoid the
> need for special case logic dependent upon which specific endpoint was
> called. Additionally, we do still use this field for many of the API calls
> where we do send one single object. Is the plan to deprecate the
> "lastImported" field across the API, or only in the /api/metadata call, so
> we can adjust our parser accordingly?
>

The lastImported field has been removed all over, what is happening now, is
that if you POST a new data element to /api/dataElements, you will get a
single object report back.. and the uid is contained within there


>>>1. The /api/23/metadata response with "errorReports" within
>>>"objectReports" appears to have replaced the /api/metadata
>>>"importConflicts". In /api/metadata, the "importConflicts" "object" field
>>>contains the actual name of the object that has the error (eg: the 
>>> Program
>>>short name in our case). However, in /api/23/metadata there is only the
>>>index which makes it more work to debug. Is it possible to also include 
>>> the
>>>name of the object in the error report like the prior API version does?
>>>
>>> Hm, right. I understand that it's useful, but that can easily add up to
>> a lot of data returned. Maybe we should add some kind of "debug" mode where
>> more output is provided.
>>
>
> Although I understand that this might add more data to the payload, it
> would decrease the amount of data that we need to pass around in our app,
> and also potentially decrease the number of API calls we need to do. Not
> having this information in the response now means we need to pass around
> our original post data (which can be quite large, and which we aren't
> currently doing) and do a lookup by ID to send any kind of useful
> information back to the user. Additionally, we may have even posted these
> objects by ID only, which means we will now need to do another API call to
> look up the object to get useful information to send back to the user. Keep
> in mind that in our app, the user doesn't actually even have a knowledge of
> IDs - we are doing those lookups for them by name or code.
> Would it be possible to add a flag to the api call where we can request
> the name or shortname or code to be returned in addition to just the ID?
> Perhaps similar to the metadata field filter?  Where we could say supply
> importConflictsFields=:identifiable?
>

Yes, let me see what I can do for 225 at least (I might be able to backport
to 224 if it doesn't disrupt things too much)

https://blueprints.launchpad.net/dhis2/+spec/importer-parameter-to-adjust-importreport-output


>
>
>>
>> --
>> Morten
>>
>>
>>> Thanks,
>>>
>>> Lorill
>>>
>>>
>>>
>>> ___
>>> Mailing list: https://launchpad.net/~dhis2-devs
>>> Post to : dhis2-devs@lists.launchpad.net
>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>> More help   : https://help.launchpad.net/ListHelp
>>>
>>>
>>
>
>
> --
> Lorill Crees
> Project Leader / 

Re: [Dhis2-devs] Response differences between /api/metadata and /api/23/metadata

[Lorill Crees]

Hi Morten,

Thanks for your response, and sorry for my very delayed response. I've
answered inline below.

Your assistance is appreciated!

Thanks,

Lorill

On Sun, Sep 4, 2016 at 10:10 PM, Morten Olav Hansen 
wrote:

> Hi Lorill
>
> Sorry for the delay in answering.
>
>
>>1. Is there any documentation outlining the response structure from
>>all API calls, including when there are errors or any other scenarios? We
>>have been upgrading our response parser on a trial and error basis
>>attempting to mimic all possible scenarios which is not ideal.
>>
>> Right now there is no documentation for this, but it should be easy
> enough to deal with I hope.
>
> The basic structure is like this:
>
> {
>status: OK, ERROR, WARNING,
>typeReports: [
> {
>klass: "Klass type",
>stats: {created, updated, deleted, ignored, total},
>objectReports: [
>  {
> klass: "Klass type",
> index: 123 (index into array you were sending),
> uid: "uid of object",
> errorReports: [
>   {
> errorCode: (see attached image),
> message: "general message about what went wrong",
> mainKlass: (basically same klass as object report),
> errorKlass: (class where it went wrong.. if missing data
> element ref, this is DataElement),
> value: (can be anything.. something useful for error
> report to give more info)
>   }
> ]
>  }
>]
> }
>   ]
> }
>
> Thanks, that is helpful.

>
>>1. When submitting an identical post containing
>>{"programTrackedEntityAttributes" [...], "program" [...]}, where
>>there is an error in the Program being posted, the call to /api/metadata
>>responds with success for "programTrackedEntityAttributes" with the
>>correct "importCount" counts, but the call to /api/23/metadata responds 
>> for
>>"programTrackedEntityAttributes" with "stats" counts containing all
>>zeros, and no "objectReports". It is like the first item being posted is
>>completely ignored if the second item posted has errors. Is this the
>>intended behaviour? And should there be "objectReports" regardless? Please
>>see attached examples.
>>
>> Yes, what kind of error are you getting? a reference error, or a
> validation error? in the old importer, it would always try to just import
> what it could.. which was not always ideal (missing references etc was just
> ignored). If it's just reference error, and you want to ignore them..
> please try with `atomicMode=NONE`, regarding the object report..
>

We need to write our response parser to be able to parse any kind of error
that may happen. Our program ingests CSV files from the user and we attempt
to do as much error checking in advance as possible, but there may be other
errors that happen when posting to the API such as validation errors,
reference errors, or even 500 errors if there is an error within DHIS 2
itself. Given that we need to cater to every possible scenario, I'm not
sure if we should be using "atomicMode=NONE" or not. Also, should there be
"objectReports" regardless?

>
>
>>1. There is no longer a "lastImported" field. Is this now deprecated?
>>
>> There should now be a id exported in the object report.. I think there
> might be some inconsistencies there though.. I will go through it, in
> general this field was mostly used for our /api/type/id endpoints.. were
> you also using it for /api/metadata? (it's not very useful unless you are
> only sending 1 single object)
>

Agreed that using "lastImported" for /api/metadata is not very useful
unless sending one single object. However, we are using our generic
response parser for all DHIS 2 API endpoints and are trying to avoid the
need for special case logic dependent upon which specific endpoint was
called. Additionally, we do still use this field for many of the API calls
where we do send one single object. Is the plan to deprecate the
"lastImported" field across the API, or only in the /api/metadata call, so
we can adjust our parser accordingly?

>
>
>>
>>1. The /api/23/metadata response with "errorReports" within
>>"objectReports" appears to have replaced the /api/metadata
>>"importConflicts". In /api/metadata, the "importConflicts" "object" field
>>contains the actual name of the object that has the error (eg: the Program
>>short name in our case). However, in /api/23/metadata there is only the
>>index which makes it more work to debug. Is it possible to also include 
>> the
>>name of the object in the error report like the prior API version does?
>>
>> Hm, right. I understand that it's useful, but that can easily add up to a
> lot of data returned. Maybe we should add some kind of "debug" mode where
> more output is provided.
>

Although I understand that this might add more data to the payload, it
would decrease the 

Re: [Dhis2-devs] Response differences between /api/metadata and /api/23/metadata

[Lorill Crees]

Hi - I'm just following up on these questions. Your help is appreciated.

Thanks,

Lorill

On Mon, Aug 22, 2016 at 2:10 PM, Lorill Crees  wrote:

> Hi,
>
> I am working on updating our applications to DHIS 2 v 2.23 specifically in
> regards to upgrading to using the /api/23/metadata api call. I am testing
> against revision 23053.
>
> We have a few issues:
>
>1. Is there any documentation outlining the response structure from
>all API calls, including when there are errors or any other scenarios? We
>have been upgrading our response parser on a trial and error basis
>attempting to mimic all possible scenarios which is not ideal.
>2. When submitting an identical post containing {"
>programTrackedEntityAttributes" [...], "program" [...]}, where there
>is an error in the Program being posted, the call to /api/metadata responds
>with success for "programTrackedEntityAttributes" with the correct
>"importCount" counts, but the call to /api/23/metadata responds for "
>programTrackedEntityAttributes" with "stats" counts containing all
>zeros, and no "objectReports". It is like the first item being posted is
>completely ignored if the second item posted has errors. Is this the
>intended behaviour? And should there be "objectReports" regardless? Please
>see attached examples.
>3. There is no longer a "lastImported" field. Is this now deprecated?
>4. The /api/23/metadata response with "errorReports" within
>"objectReports" appears to have replaced the /api/metadata
>"importConflicts". In /api/metadata, the "importConflicts" "object" field
>contains the actual name of the object that has the error (eg: the Program
>short name in our case). However, in /api/23/metadata there is only the
>index which makes it more work to debug. Is it possible to also include the
>name of the object in the error report like the prior API version does?
>
> Thanks,
>
> Lorill
>
>
>
___
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help   : https://help.launchpad.net/ListHelp