Re: [OSM-dev] Structured error messages from API
On Fri, Dec 11, 2009 at 9:15 PM, Ian Dees ian.d...@gmail.com wrote: On Fri, Dec 11, 2009 at 12:39 PM, Matt Amos zerebub...@gmail.com wrote: On Fri, Dec 11, 2009 at 6:24 PM, Ian Dees ian.d...@gmail.com wrote: Actually, I was just now creating a stub page for API 0.7 brainstorming: http://wiki.openstreetmap.org/wiki/API_v0.7 Remember, it's a brainstorm: all ideas are good ideas at this point... ;-). cool! although i'd consider verified users / locked tags to be an anti-feature ;-) Yea, I suppose that requires some discussion. The idea arose from a meeting I recently attended with some US city government GIS people that have expressed interest in maintaining (at least part of) their official GIS database in OpenStreetMap. Their number one fear/concern is an OSM editor changing the official boundary of a state forest, pulling that change back to do cartography for a hunting season (for example), and then having a land owner call them up asking why people are hunting on their land. Migurski posted a blog about signing openstreetmap edits. So that you can be sure that the edit is alright. (But this is ass yout UUID mostly a client thing) http://mike.teczno.com/notes/gosm.html The question is do these bulk imports of legal boundaries really belong in Openstreetmap? Since they are not really editable, you can only change them by doing bulk import again. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Structured error messages from API
On Mon, Dec 14, 2009 at 7:00 AM, Erik Johansson e...@kth.se wrote: On Fri, Dec 11, 2009 at 9:15 PM, Ian Dees ian.d...@gmail.com wrote: On Fri, Dec 11, 2009 at 12:39 PM, Matt Amos zerebub...@gmail.com wrote: On Fri, Dec 11, 2009 at 6:24 PM, Ian Dees ian.d...@gmail.com wrote: Actually, I was just now creating a stub page for API 0.7 brainstorming: http://wiki.openstreetmap.org/wiki/API_v0.7 Remember, it's a brainstorm: all ideas are good ideas at this point... ;-). cool! although i'd consider verified users / locked tags to be an anti-feature ;-) Yea, I suppose that requires some discussion. The idea arose from a meeting I recently attended with some US city government GIS people that have expressed interest in maintaining (at least part of) their official GIS database in OpenStreetMap. Their number one fear/concern is an OSM editor changing the official boundary of a state forest, pulling that change back to do cartography for a hunting season (for example), and then having a land owner call them up asking why people are hunting on their land. Migurski posted a blog about signing openstreetmap edits. So that you can be sure that the edit is alright. (But this is ass yout UUID mostly a client thing) http://mike.teczno.com/notes/gosm.html The question is do these bulk imports of legal boundaries really belong in Openstreetmap? Since they are not really editable, you can only change them by doing bulk import again. I agree, and pointed that out to the government people asking. OSM should probably have some boundary data to make pretty maps, but since a regular old citizen can't walk up to the border, stick their GPS in the ground and say ah hah! The border was off by 0.12 degrees, I shall now go update OSM! it probably should not be in the official OSM database to begin with. However, someone else at the meeting countered that in Europe, where borders aren't so easy to come by in a licensable format, they seem to survive just fine with having border data be crowd-sourced. I tried to redirect the conversation by pointing out that other data, such as schools or water fountain POI, would be a much better option for insertion into OSM, at least to start off. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Structured error messages from API
Tom Hughes wrote: Basically, even if sending a streaming response from rails was possible (which I'm not sure it is - it's certainly hard) You can do it using render :text=proc, as amf_controller does. But I suspect this would be non-trivial to work into the existing XML API. cheers Richard -- View this message in context: http://old.nabble.com/Structured-error-messages-from-API-tp26730115p26747645.html Sent from the OpenStreetMap - Dev mailing list archive at Nabble.com. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Structured error messages from API
Karl Guggisberg karl.guggisb...@guggis.ch writes: For JOSM, the structured data currently embedded in the error message is important. Examples are object ids of already deleted objects (410 Gone) or a date (the close date of a changeset in a 409 Conflict). I'd prefer a parseable error document in case of http error codes, preferably in XML. This would not be much of a change because the content of the 'Error' http header is already replied as error document too (sometimes for some error cases). It would be nice if the OSM API replied a message in english *and* and in the language supplied in the Accept-Language http header. Example: osm-api-error error-id code=1232 / !-- unique error code? would be nice too -- message lang=enUpload of an object failed./message message lang=deHochladen eines Objekts ist fehlgeschlagen./message property name=object-id value=1223/ property name=closed-date value=/ /osm-api-error XML is probably the best solution because it is flexible and anybody communicating with the API knows how to pars XML anyway. Any objections against changing the body of the error document from the current text string (which is a duplicate of the Error header, AFAICT) to a XML document? Anybody interested in helping to come up with a specification for this? Matthias ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Structured error messages from API
Richard Fairhurst rich...@systemed.net writes: Tom Hughes wrote: Basically, even if sending a streaming response from rails was possible (which I'm not sure it is - it's certainly hard) You can do it using render :text=proc, as amf_controller does. But I suspect this would be non-trivial to work into the existing XML API. The API could also return an URL which the client can poll to get the status for an upload. This would also avoid issues with client timeouts. Currently, when the client hits a timeout and aborts the connection it has no way of knowing whether the upload succeeded or not. For new data this creates duplicates when the user tries to upload again. Matthias ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Structured error messages from API
On Fri, Dec 11, 2009 at 5:05 PM, Matthias Julius li...@julius-net.net wrote: Richard Fairhurst rich...@systemed.net writes: Tom Hughes wrote: Basically, even if sending a streaming response from rails was possible (which I'm not sure it is - it's certainly hard) You can do it using render :text=proc, as amf_controller does. But I suspect this would be non-trivial to work into the existing XML API. The API could also return an URL which the client can poll to get the status for an upload. This would also avoid issues with client timeouts. Currently, when the client hits a timeout and aborts the connection it has no way of knowing whether the upload succeeded or not. For new data this creates duplicates when the user tries to upload again. i have been thinking about something similar, which would work along the following lines. the changeset is a container and POSTing a diff to it returns a 202 Accepted result if diff processing takes more than a few seconds. the result is a Location redirect to the individual resource for the diff upload, which could be polled for status. unfortunately, that's where it gets complicated. because the diff upload occurs in a transaction, none of its outputs are visible until it commits. therefore any status would need to be posted on a different connection, in a different transaction. this makes things annoying a messy. if anyone's got any ideas then i'd be very interested to talk. this is something that we can put into API 0.7 - which it might be a good idea to start thinking about now, since the last API took about a year to develop ;-) cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Structured error messages from API
On Fri, Dec 11, 2009 at 12:13 PM, Matt Amos zerebub...@gmail.com wrote: On Fri, Dec 11, 2009 at 5:05 PM, Matthias Julius li...@julius-net.net wrote: Richard Fairhurst rich...@systemed.net writes: Tom Hughes wrote: Basically, even if sending a streaming response from rails was possible (which I'm not sure it is - it's certainly hard) You can do it using render :text=proc, as amf_controller does. But I suspect this would be non-trivial to work into the existing XML API. The API could also return an URL which the client can poll to get the status for an upload. This would also avoid issues with client timeouts. Currently, when the client hits a timeout and aborts the connection it has no way of knowing whether the upload succeeded or not. For new data this creates duplicates when the user tries to upload again. i have been thinking about something similar, which would work along the following lines. the changeset is a container and POSTing a diff to it returns a 202 Accepted result if diff processing takes more than a few seconds. the result is a Location redirect to the individual resource for the diff upload, which could be polled for status. unfortunately, that's where it gets complicated. because the diff upload occurs in a transaction, none of its outputs are visible until it commits. therefore any status would need to be posted on a different connection, in a different transaction. this makes things annoying a messy. if anyone's got any ideas then i'd be very interested to talk. this is something that we can put into API 0.7 - which it might be a good idea to start thinking about now, since the last API took about a year to develop ;-) Actually, I was just now creating a stub page for API 0.7 brainstorming: http://wiki.openstreetmap.org/wiki/API_v0.7 Remember, it's a brainstorm: all ideas are good ideas at this point... ;-). ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Structured error messages from API
On Fri, Dec 11, 2009 at 6:24 PM, Ian Dees ian.d...@gmail.com wrote: Actually, I was just now creating a stub page for API 0.7 brainstorming: http://wiki.openstreetmap.org/wiki/API_v0.7 Remember, it's a brainstorm: all ideas are good ideas at this point... ;-). cool! although i'd consider verified users / locked tags to be an anti-feature ;-) cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Structured error messages from API
Karl Guggisberg karl.guggisb...@guggis.ch writes: Anybody interested in helping to come up with a specification for this? I am. How can I help? Hmmm, where to start? I guess we first need to make a list of things we want to be included in the error document. If I find the time tonight, I will start browsing the rails code for all the API errors it might emit. Matthias ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Structured error messages from API
On Fri, Dec 11, 2009 at 12:39 PM, Matt Amos zerebub...@gmail.com wrote: On Fri, Dec 11, 2009 at 6:24 PM, Ian Dees ian.d...@gmail.com wrote: Actually, I was just now creating a stub page for API 0.7 brainstorming: http://wiki.openstreetmap.org/wiki/API_v0.7 Remember, it's a brainstorm: all ideas are good ideas at this point... ;-). cool! although i'd consider verified users / locked tags to be an anti-feature ;-) Yea, I suppose that requires some discussion. The idea arose from a meeting I recently attended with some US city government GIS people that have expressed interest in maintaining (at least part of) their official GIS database in OpenStreetMap. Their number one fear/concern is an OSM editor changing the official boundary of a state forest, pulling that change back to do cartography for a hunting season (for example), and then having a land owner call them up asking why people are hunting on their land. They suggested the ability to lock certain layers or ways with certain tags so that they can exist in the OSM database but not be changed. I pointed out that for this specific example, OSM probably doesn't really want official boundary data anyway, since it's not on the ground. The second part of that suggestion probably should be a separate line: locked tags is slang for a way to maintain a globally unique ID for a certain map feature. They want to be able to pull their data back out of OSM based on certain keys, and when I pointed out that anyone could add or remove the tags they apply to their data, they suggested that we maintain a UUID for every map feature. ...as I said, brainstorming :). ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Structured error messages from API
Hi, Ian Dees wrote: Yea, I suppose that requires some discussion. The idea arose from a meeting I recently attended with some US city government GIS people that have expressed interest in maintaining (at least part of) their official GIS database in OpenStreetMap. Their number one fear/concern is an OSM editor changing the official boundary of a state forest, pulling that change back to do cartography for a hunting season (for example), and then having a land owner call them up asking why people are hunting on their land. The proper way to deal with this is to make it easier for OSM-related tools to access third party databases, then tell the gov't guys to maintain their data in such a certified OSM compatible third party database. Anything that goes into OSM must be changeable by OSM mappers. We're not a hosting service for 3rd party data ,-) The second part of that suggestion probably should be a separate line: locked tags is slang for a way to maintain a globally unique ID for a certain map feature. They want to be able to pull their data back out of OSM based on certain keys, and when I pointed out that anyone could add or remove the tags they apply to their data, they suggested that we maintain a UUID for every map feature. I agree that some method of permalinking would benefit many. Today we have people explicitly linking to relation IDs which is not very helpful. It will be difficult to make such permalinks unbreakable without restricting editors but worth a thought. Bye Frederik ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Structured error messages from API
There are requests in JOSM's trac to improve the handling of API errors. To do that JOSM needs to get a better understanding on what is wrong with the data. Currently, JOSM is parsing the error strings the API is returning. This is far from ideal because they are not structured, not documented and might change without warning. To improve things I would like to see the API extended to return meta data about errors (error type, id of offending element, ...) in a structured way. There are a couple of ways to that (that came to my mind): - change the Error header - add home-made HTTP headers (X-Error-Type ...) - add pseudo headers to the body - return a XML document containing all the info (see also http://trac.openstreetmap.org/ticket/2544) What do you think? How important is that to people on the receiving end (application developers)? Any suggestions how the format should be? Matthias ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Structured error messages from API
On Thu, Dec 10, 2009 at 10:16 AM, Matthias Julius li...@julius-net.netwrote: There are requests in JOSM's trac to improve the handling of API errors. To do that JOSM needs to get a better understanding on what is wrong with the data. Currently, JOSM is parsing the error strings the API is returning. This is far from ideal because they are not structured, not documented and might change without warning. To improve things I would like to see the API extended to return meta data about errors (error type, id of offending element, ...) in a structured way. There are a couple of ways to that (that came to my mind): - change the Error header - add home-made HTTP headers (X-Error-Type ...) - add pseudo headers to the body - return a XML document containing all the info (see also http://trac.openstreetmap.org/ticket/2544) What do you think? How important is that to people on the receiving end (application developers)? On a related note, I would be very interested in seeing a progress of some kind being returned when doing a osmChange upload. I realize that it is difficult (because it could fail after spitting out lots of seemingly valid IDs), but if it was documented it would be doable. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Structured error messages from API
On a related note, I would be very interested in seeing a progress of some kind being returned when doing a osmChange upload. I realize that it is difficult (because it could fail after spitting out lots of seemingly valid IDs), but if it was documented it would be doable. Are you talking about getting a completed xx% message in the result of the open http connection? Peter ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Structured error messages from API
On Thu, Dec 10, 2009 at 10:52 AM, Peter Körner osm-li...@mazdermind.dewrote: On a related note, I would be very interested in seeing a progress of some kind being returned when doing a osmChange upload. I realize that it is difficult (because it could fail after spitting out lots of seemingly valid IDs), but if it was documented it would be doable. Are you talking about getting a completed xx% message in the result of the open http connection? Yes. To implement that, the HTTP server would need to respond with streaming XML of some sort over the open HTTP response channel. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Structured error messages from API
On Thu, Dec 10, 2009 at 16:20, Ian Dees ian.d...@gmail.com wrote: On Thu, Dec 10, 2009 at 10:16 AM, Matthias Julius li...@julius-net.net wrote: There are requests in JOSM's trac to improve the handling of API errors. To do that JOSM needs to get a better understanding on what is wrong with the data. Currently, JOSM is parsing the error strings the API is returning. This is far from ideal because they are not structured, not documented and might change without warning. To improve things I would like to see the API extended to return meta data about errors (error type, id of offending element, ...) in a structured way. There are a couple of ways to that (that came to my mind): - change the Error header - add home-made HTTP headers (X-Error-Type ...) - add pseudo headers to the body - return a XML document containing all the info (see also http://trac.openstreetmap.org/ticket/2544) What do you think? How important is that to people on the receiving end (application developers)? On a related note, I would be very interested in seeing a progress of some kind being returned when doing a osmChange upload. I realize that it is difficult (because it could fail after spitting out lots of seemingly valid IDs), but if it was documented it would be doable. You mean showing upload progress in JOSM as opposed to the current cylon impression? That could be implemented by counting the number of bytes of the osmChange request that have been successfully sent over the wire. That's how upload progress bars are usually implemented. Obviously the upload could fail but that's another issue. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Structured error messages from API
You mean showing upload progress in JOSM as opposed to the current cylon impression? That could be implemented by counting the number of bytes of the osmChange request that have been successfully sent over the wire. That's how upload progress bars are usually implemented. Obviously the upload could fail but that's another issue. That looks somehow more intelligent than streaming the progress from the server back to the client. Peter ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Structured error messages from API
On Thu, Dec 10, 2009 at 17:01, Peter Körner osm-li...@mazdermind.de wrote:
You mean showing upload progress in JOSM as opposed to the current
cylon impression? That could be implemented by counting the number of
bytes of the osmChange request that have been successfully sent over
the wire. That's how upload progress bars are usually implemented.
Obviously the upload could fail but that's another issue.
That looks somehow more intelligent than streaming the progress from the
server back to the client.
It is:) When I upload a file with Google Chrome it always shows the
progress of the upload. This isn't done with some ad-hoc streaming XML
from the server, the client is just counting how many bytes it has
sent over the wire and how large a percentage that is of the total.
I looked at the relevant code in JOSM once but I couldn't find a way
to do it (but I'm not familiar with Java). In Perl with
HTTP::Request::Common you can do e.g.:
$HTTP::Request::Common::DYNAMIC_FILE_UPLOAD = 1
my $req = HTTP::Request::Common-new( ... );
$req-content( sub {
my $chunk = $gen();
my $length = length $chunk;
warn I'm now uploading a chunk of length $length, of a total
of $total bytes;
return $chunk;
} );
my $result = LWP::UserAgent-request( $req );
(See this code for a practical example of this that I wrote:
http://cpansearch.perl.org/src/CPB/Flickr-Upload-1.32/Upload.pm)
I presume Java has some library to do this as well but I couldn't find it.
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Structured error messages from API
On Thu, Dec 10, 2009 at 10:55 AM, Ævar Arnfjörð Bjarmason ava...@gmail.comwrote: On Thu, Dec 10, 2009 at 16:20, Ian Dees ian.d...@gmail.com wrote: On Thu, Dec 10, 2009 at 10:16 AM, Matthias Julius li...@julius-net.net wrote: There are requests in JOSM's trac to improve the handling of API errors. To do that JOSM needs to get a better understanding on what is wrong with the data. Currently, JOSM is parsing the error strings the API is returning. This is far from ideal because they are not structured, not documented and might change without warning. To improve things I would like to see the API extended to return meta data about errors (error type, id of offending element, ...) in a structured way. There are a couple of ways to that (that came to my mind): - change the Error header - add home-made HTTP headers (X-Error-Type ...) - add pseudo headers to the body - return a XML document containing all the info (see also http://trac.openstreetmap.org/ticket/2544) What do you think? How important is that to people on the receiving end (application developers)? On a related note, I would be very interested in seeing a progress of some kind being returned when doing a osmChange upload. I realize that it is difficult (because it could fail after spitting out lots of seemingly valid IDs), but if it was documented it would be doable. You mean showing upload progress in JOSM as opposed to the current cylon impression? That could be implemented by counting the number of bytes of the osmChange request that have been successfully sent over the wire. That's how upload progress bars are usually implemented. Obviously the upload could fail but that's another issue. The number of bytes uploaded != the progress of the changeset upload process (although it's better than what's there now). I'm assuming the server reads the entire file to memory of some sort before parsing the XML and applying changes to the database? If so, the upload of the osmc file could happen in a few seconds but it could then take lots more time for the server to apply the changeset. At that point, the tool will only be able to block until it times out or receives a response from the server. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Structured error messages from API
For JOSM, the structured data currently embedded in the error message is important. Examples are object ids of already deleted objects (410 Gone) or a date (the close date of a changeset in a 409 Conflict). I'd prefer a parseable error document in case of http error codes, preferably in XML. This would not be much of a change because the content of the 'Error' http header is already replied as error document too (sometimes for some error cases). It would be nice if the OSM API replied a message in english *and* and in the language supplied in the Accept-Language http header. Example: osm-api-error error-id code=1232 / !-- unique error code? would be nice too -- message lang=enUpload of an object failed./message message lang=deHochladen eines Objekts ist fehlgeschlagen./message property name=object-id value=1223/ property name=closed-date value=/ /osm-api-error Regards Karl -Ursprüngliche Nachricht- Von: dev-boun...@openstreetmap.org [mailto:dev-boun...@openstreetmap.org] Im Auftrag von Matthias Julius Gesendet: Donnerstag, 10. Dezember 2009 17:16 An: dev@openstreetmap.org Betreff: [OSM-dev] Structured error messages from API There are requests in JOSM's trac to improve the handling of API errors. To do that JOSM needs to get a better understanding on what is wrong with the data. Currently, JOSM is parsing the error strings the API is returning. This is far from ideal because they are not structured, not documented and might change without warning. To improve things I would like to see the API extended to return meta data about errors (error type, id of offending element, ...) in a structured way. There are a couple of ways to that (that came to my mind): - change the Error header - add home-made HTTP headers (X-Error-Type ...) - add pseudo headers to the body - return a XML document containing all the info (see also http://trac.openstreetmap.org/ticket/2544) What do you think? How important is that to people on the receiving end (application developers)? Any suggestions how the format should be? Matthias ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Structured error messages from API
You mean showing upload progress in JOSM as opposed to the current cylon impression? That could be implemented by counting the number of bytes of the osmChange request that have been successfully sent over the wire. That's how upload progress bars are usually implemented. It's not the upload or download which takes most of the time but the processing on the server. Uploading an osmChange has three phases: 1. upload the osmChange document - can take some time, we could give feedback information based on the bytes sent, but can be neglected compared to phase 2 2. process on the server - this takes most of the time, no feedback here 3. download the diffResult - can be neglected compared to phase 2. Again, we could give feedback based on the bytes read from the server, but it's not worth the effort Actually, I implemented something based on the approach Avar proposes but I wasn't satisfied and didn't check it in. Regards Karl -Ursprüngliche Nachricht- Von: dev-boun...@openstreetmap.org [mailto:dev-boun...@openstreetmap.org] Im Auftrag von Ævar Arnfjörð Bjarmason Gesendet: Donnerstag, 10. Dezember 2009 17:55 An: Ian Dees Cc: dev@openstreetmap.org Betreff: Re: [OSM-dev] Structured error messages from API On Thu, Dec 10, 2009 at 16:20, Ian Dees ian.d...@gmail.com wrote: On Thu, Dec 10, 2009 at 10:16 AM, Matthias Julius li...@julius-net.net wrote: There are requests in JOSM's trac to improve the handling of API errors. To do that JOSM needs to get a better understanding on what is wrong with the data. Currently, JOSM is parsing the error strings the API is returning. This is far from ideal because they are not structured, not documented and might change without warning. To improve things I would like to see the API extended to return meta data about errors (error type, id of offending element, ...) in a structured way. There are a couple of ways to that (that came to my mind): - change the Error header - add home-made HTTP headers (X-Error-Type ...) - add pseudo headers to the body - return a XML document containing all the info (see also http://trac.openstreetmap.org/ticket/2544) What do you think? How important is that to people on the receiving end (application developers)? On a related note, I would be very interested in seeing a progress of some kind being returned when doing a osmChange upload. I realize that it is difficult (because it could fail after spitting out lots of seemingly valid IDs), but if it was documented it would be doable. You mean showing upload progress in JOSM as opposed to the current cylon impression? That could be implemented by counting the number of bytes of the osmChange request that have been successfully sent over the wire. That's how upload progress bars are usually implemented. Obviously the upload could fail but that's another issue. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Structured error messages from API
There is alreay ProgressInputStream
http://josm.openstreetmap.de/browser/trunk/src/org/openstreetmap/josm/io/ProgressInputStream.java
in the JOSM code base and a ProgressOutputStream could be built easily.
But again, I don't think this would help much. If we really wanted to display
progress information, we would need a kind of streaming reply from the server,
as proposed by others on this thread.
Regards
Karl
-Ursprüngliche Nachricht-
Von: dev-boun...@openstreetmap.org [mailto:dev-boun...@openstreetmap.org] Im
Auftrag von Ævar Arnfjörð Bjarmason
Gesendet: Donnerstag, 10. Dezember 2009 18:09
An: Peter Körner
Cc: dev@openstreetmap.org
Betreff: Re: [OSM-dev] Structured error messages from API
On Thu, Dec 10, 2009 at 17:01, Peter Körner osm-li...@mazdermind.de wrote:
You mean showing upload progress in JOSM as opposed to the current
cylon impression? That could be implemented by counting the number of
bytes of the osmChange request that have been successfully sent over
the wire. That's how upload progress bars are usually implemented.
Obviously the upload could fail but that's another issue.
That looks somehow more intelligent than streaming the progress from
the server back to the client.
It is:) When I upload a file with Google Chrome it always shows the progress of
the upload. This isn't done with some ad-hoc streaming XML from the server, the
client is just counting how many bytes it has sent over the wire and how large
a percentage that is of the total.
I looked at the relevant code in JOSM once but I couldn't find a way to do it
(but I'm not familiar with Java). In Perl with HTTP::Request::Common you can do
e.g.:
$HTTP::Request::Common::DYNAMIC_FILE_UPLOAD = 1
my $req = HTTP::Request::Common-new( ... );
$req-content( sub {
my $chunk = $gen();
my $length = length $chunk;
warn I'm now uploading a chunk of length $length, of a total of $total
bytes;
return $chunk;
} );
my $result = LWP::UserAgent-request( $req );
(See this code for a practical example of this that I wrote:
http://cpansearch.perl.org/src/CPB/Flickr-Upload-1.32/Upload.pm)
I presume Java has some library to do this as well but I couldn't find it.
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Structured error messages from API
On Thu, Dec 10, 2009 at 17:25, Karl Guggisberg karl.guggisb...@guggis.ch wrote: You mean showing upload progress in JOSM as opposed to the current cylon impression? That could be implemented by counting the number of bytes of the osmChange request that have been successfully sent over the wire. That's how upload progress bars are usually implemented. It's not the upload or download which takes most of the time but the processing on the server. Uploading an osmChange has three phases: Indeed, but I understood it from Ian's post when doing a[sic] osmChange upload that he was only interested in step #1. 1. upload the osmChange document - can take some time, we could give feedback information based on the bytes sent, but can be neglected compared to phase 2 Actually when I do uploads uploading the file itself and processing on the server seems to take around the same time, judging from network sniffing I've done. Results may vary though. 2. process on the server - this takes most of the time, no feedback here Yup, PostgreSQL can't tell you where it's at in the query: http://wiki.postgresql.org/wiki/Query_progress_indication 3. download the diffResult - can be neglected compared to phase 2. Again, we could give feedback based on the bytes read from the server, but it's not worth the effort Actually, I implemented something based on the approach Avar proposes but I wasn't satisfied and didn't check it in. I've filed a bug to track this, for what it's worth: http://josm.openstreetmap.de/ticket/4145 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Structured error messages from API
On Thu, Dec 10, 2009 at 11:39 AM, Ævar Arnfjörð Bjarmason ava...@gmail.comwrote: On Thu, Dec 10, 2009 at 17:25, Karl Guggisberg karl.guggisb...@guggis.ch wrote: You mean showing upload progress in JOSM as opposed to the current cylon impression? That could be implemented by counting the number of bytes of the osmChange request that have been successfully sent over the wire. That's how upload progress bars are usually implemented. It's not the upload or download which takes most of the time but the processing on the server. Uploading an osmChange has three phases: Indeed, but I understood it from Ian's post when doing a[sic] osmChange upload that he was only interested in step #1. My interest is in getting feedback from the API as it applies large changesets, so I'm mostly interested in #2 as the actual HTTP POST of the data file is a relatively small part of the transaction. 1. upload the osmChange document - can take some time, we could give feedback information based on the bytes sent, but can be neglected compared to phase 2 Actually when I do uploads uploading the file itself and processing on the server seems to take around the same time, judging from network sniffing I've done. Results may vary though. When uploading large diffs (changeset reverts, new data imports), time spent sending the data over the network is insignificant compared to the time spent applying to the server. 2. process on the server - this takes most of the time, no feedback here Yup, PostgreSQL can't tell you where it's at in the query: http://wiki.postgresql.org/wiki/Query_progress_indication Presumably the API server performs many SQL queries to update the database, so the API application could provide feedback as it goes through the steps of updating based on the changeset. Someone (sorry my shortterm memory is pretty crummy) pointed out that this would probably break the API's current error system as the server would no longer be able to send error headers once it starts sending streaming response information. 3. download the diffResult - can be neglected compared to phase 2. Again, we could give feedback based on the bytes read from the server, but it's not worth the effort Actually, I implemented something based on the approach Avar proposes but I wasn't satisfied and didn't check it in. I've filed a bug to track this, for what it's worth: http://josm.openstreetmap.de/ticket/4145 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Structured error messages from API
On 10/12/09 17:39, Ævar Arnfjörð Bjarmason wrote: 2. process on the server - this takes most of the time, no feedback here Yup, PostgreSQL can't tell you where it's at in the query: http://wiki.postgresql.org/wiki/Query_progress_indication Um... It's not like it's a single query. It will be thousands of individual queries each of which will, on their own, complete virtually instantaneously. On top of all that there is time parsing the XML, time running other ruby code to validate things, etc, etc. Basically, even if sending a streaming response from rails was possible (which I'm not sure it is - it's certainly hard) there is no simple way to measure progress. Tom -- Tom Hughes (t...@compton.nu) http://www.compton.nu/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
