Re: [osmosis-dev] [PATCH] --log-progress --buffer results in OsmosisRuntimeException: initialize has not been called
On 15 January 2012 08:11, Richard Hansen rhan...@bbn.com wrote: Hi all, With the latest Git revision (43089ed), I get an OsmosisRuntimeException when I use --log-progress followed by --buffer. For example: osmosis -v \ --rb massachusetts.osm.pbf --sort \ --rb connecticut.osm.pbf --sort \ --merge --log-progress --buffer --wb _combined.osm.pbf ... org.openstreetmap.osmosis.**core.OsmosisRuntimeException: initialize has not been called at org.openstreetmap.osmosis.**core.store.DataPostbox.put(** DataPostbox.java:288) at org.openstreetmap.osmosis.**core.buffer.v0_6.EntityBuffer.** process(EntityBuffer.java:48) at org.openstreetmap.osmosis.**core.progress.v0_6.** EntityProgressLogger.process(**EntityProgressLogger.java:70) at org.openstreetmap.osmosis.**core.sort.v0_6.EntitySorter.** complete(EntitySorter.java:71) at crosby.binary.osmosis.**OsmosisBinaryParser.complete(** OsmosisBinaryParser.java:35) at crosby.binary.file.**BlockInputStream.process(** BlockInputStream.java:37) at crosby.binary.osmosis.**OsmosisReader.run(** OsmosisReader.java:45) at java.lang.Thread.run(Thread.**java:679) If I remove the --log-progress argument, it works as expected. I chased this down to a missing call to sink.initialize() in EntityProgressLogger.**initialize(). Attached is a patch that I believe fixes this bug. Hi Richard, I've applied the patch. Thanks. I've also updated the ChangeProgressLogger class which had the same issue. Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] calling DataPostbox.release() multiple times
On 15 January 2012 14:09, Richard Hansen rhan...@bbn.com wrote: Hi all, I believe I have found a bug in DataPostbox.release(). (I'm using git master, currently 43089ed.) According to the javadoc comments, Releaseable.release() can be called multiple times. However, DataPostbox.release() can only be called once. If it is called a second time, it blocks forever waiting for the output thread (which no longer exists) to release. Good find, thanks for the report. You're correct in saying that Releasable classes can be released multiple times, however that is not typically how they're used in the Osmosis pipeline. DataPostbox is a bit tricky because it is the main synchronisation point between multiple threads. Two threads communicating via a DataPostbox perform handshakes during the initialize, complete, and release methods (the process methods run mostly independently). It is the release method where each threads tells the other thread that they are about to exit. DataPostbox does technically support multiple calls to release, but it would require each thread to restart after the release calls and this isn't supported (or even desirable) by the Osmosis pipeline. I noticed this because the pbf reader can call release() twice, triggering the lockup. To recreate, you can run the following: osmosis --read-pbf /dev/null --buffer --write-null The two calls to DataPostbox.release() have the following stack traces: java.lang.Exception: Stack trace at java.lang.Thread.dumpStack(**Thread.java:1266) at org.openstreetmap.osmosis.**core.store.DataPostbox.** release(DataPostbox.java:335) at org.openstreetmap.osmosis.**core.buffer.v0_6.EntityBuffer.** release(EntityBuffer.java:64) at crosby.binary.osmosis.**OsmosisBinaryParser.complete(** OsmosisBinaryParser.java:36) at crosby.binary.file.**BlockInputStream.process(** BlockInputStream.java:37) at crosby.binary.osmosis.**OsmosisReader.run(** OsmosisReader.java:45) at java.lang.Thread.run(Thread.**java:679) java.lang.Exception: Stack trace at java.lang.Thread.dumpStack(**Thread.java:1266) at org.openstreetmap.osmosis.**core.store.DataPostbox.** release(DataPostbox.java:335) at org.openstreetmap.osmosis.**core.buffer.v0_6.EntityBuffer.** release(EntityBuffer.java:64) at crosby.binary.osmosis.**OsmosisReader.run(** OsmosisReader.java:50) at java.lang.Thread.run(Thread.**java:679) Deleting line 36 of pbf/src/main/java/crosby/**binary/osmosis/**OsmosisBinaryParser.java will cause DataPostbox.release() to be called only once, but that just avoids the bug. I've removed line 36 as you've suggested because it shouldn't be there. Release should only be called in a finally block that gets called regardless of success or failure, the complete method only gets called on success. I think it was me that added the other release in OsmosisReader recently when I added initialize, but apparently I didn't make a corresponding change in OsmosisBinaryParser. Please let me know if you see any further issues. I believe a proper fix to DataPostbox.release() will require a minor redesign of the DataPostbox lifetime management code. I'm not familiar enough with the code to feel comfortable making such a change. I can take a crack at it, but I'm hoping an expert has the time to take a look. What changes would you suggest? I believe it's working as I originally intended, but that's not to say there isn't a better way :-) Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
[osmosis-dev] planet files appear truncated to osmosis, was: [OSM-talk] old planet files with 'nolt' in the name
Hi, I had the following issue with processing a planet file: On Sat, Jan 14, 2012 at 11:32 AM, Martijn van Exel m...@rtijn.org wrote: [..] Also, I'm running into problems with the Jan 6, 2010 planet: org.openstreetmap.osmosis.core.OsmosisRuntimeException: Unable to parse xml file /osm/planet/historical/planet-100106.osm.bz2. publicId=(null), syst emId=(null), lineNumber=4312, columnNumber=75. The MD5 hash check was OK. Anyone had trouble with this particular file before? It seems that the problem is not with the particular file. I tried the same process with a few other files, all give the same error around line 4000. I'm officially confused. I used osmosis a lot before on the same machine. This is what I am doing: mvexel@lima$ /osm/software/osmosis-0.39/bin/osmosis --rx /osm/planet/historical/planet-090701.osm.bz2 --bp file=/osm/poly/us.poly --wb /osm/planet/historical/us/planet-090701-us.osm.pbf I tried osmosis 0.39 and 0.40.1 with the same result. -- martijn van exel geospatial omnivore 1109 1st ave #2 salt lake city, ut 84103 801-550-5815 http://oegeo.wordpress.com ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] calling DataPostbox.release() multiple times
On 2012-01-15 07:34, Brett Henderson wrote:
You're correct in saying that Releasable classes can be released
multiple times, however that is not typically how they're used in the
Osmosis pipeline.
Does it make sense to change the Releasable API definition to say that
release can only be called once? Or might that break multiple-sink
stages like merge?
DataPostbox is a bit tricky because it is the main synchronisation point
between multiple threads. Two threads communicating via a DataPostbox
perform handshakes during the initialize, complete, and release methods
(the process methods run mostly independently). It is the release
method where each threads tells the other thread that they are about to
exit. DataPostbox does technically support multiple calls to release,
but it would require each thread to restart after the release calls and
this isn't supported (or even desirable) by the Osmosis pipeline.
...
What changes would you suggest? I believe it's working as I originally
intended, but that's not to say there isn't a better way :-)
Looking at the code a bit more I think it might only need minor changes.
I had thought that release() and outputRelease() changed the state to
equal that of a freshly constructed DataPostbox (to support reuse?), but
that's not true: inputExit and outputExit are false after construction
but true after release(). So simply checking inputExit at the beginning
of release() should take care of it. See the attached (untested) patch.
An alternative is to increase the burden on the users of DataPostbox.
They could be required to either ensure that the output thread restarts
or avoid calling DataPostbox.release() once the output thread exits.
For example, EntityBuffer.run() could set an 'outputThreadIsRunning'
boolean at the top and clear it at the end. Then EntityBuffer.release()
would be modified to only call buffer.release() if outputThreadIsRunning
is true.
-Richard
From 86c93d88e011ab76b79737c6185bd138f2eced76 Mon Sep 17 00:00:00 2001
From: Richard Hansen rhan...@bbn.com
Date: Sun, 15 Jan 2012 15:13:01 -0500
Subject: [PATCH] Do nothing in DataPostbox.release() if already released
The Releasable interface specification says that release() may be
called multiple times. Before, if DataPostbox.release() was called a
second time, it would block forever waiting for the output thread
(which would no longer exist) to release. Now subsequent calls to
release() are a no-op, preventing the lockup.
---
.../osmosis/core/store/DataPostbox.java|5 +
1 files changed, 5 insertions(+), 0 deletions(-)
diff --git a/core/src/main/java/org/openstreetmap/osmosis/core/store/DataPostbox.java b/core/src/main/java/org/openstreetmap/osmosis/core/store/DataPostbox.java
index 372de7f..415ee40 100644
--- a/core/src/main/java/org/openstreetmap/osmosis/core/store/DataPostbox.java
+++ b/core/src/main/java/org/openstreetmap/osmosis/core/store/DataPostbox.java
@@ -335,6 +335,11 @@ public class DataPostboxT implements Initializable {
lock.lock();
try {
+ // support release() being called multiple
+ // times (do nothing if already released)
+ if (inputExit)
+return;
+
// If release is being called without having completed successfully,
// it is an error condition.
if (!inputComplete) {
--
1.7.8.2
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [OSM-dev] 120111 Planet Node Order?
Lynn, current versions of osm2pgsql can import .pbf, so you are on the right track there. Richards text is good, but it was written when planet files were a third of the current size and is just a bit dated. If osm2pgsql can't hold the nodes in it's cache it will have to retrieve them from the database and that is in my and other peoples experience at least an order of magnitude slower. To give you current numbers, I just did (10 days ago) an import on a i7 2600, 16GB box that took 30 hours with the initial import phase running at Processing: Node(1322468k 340.2k/s) Way(120291k 35.64k/s) Relation(1243830 58.30/s) parse time: 28598s Node stats: total(1322468982), max(1576326287) in 3887s Way stats: total(120291564), max(144049709) in 3375s Relation stats: total(1243833), max(1951174) in 21335s Because the box has limited memory I did an earlier attempt with -C 8000 because I knew that it would swap a lot with -C 12000. However that ran at roughly 4k/s ways and after a couple of hours I aborted it. The good news is that you -can- import on a machine with less memory (good luck keeping up with the updates on a VM though), for example lonvia has imported on a machine with 6GB total (but with -C 12000). The error message you got is weird and may point to an issue with the export process, but it just states that the node cache is going to be less efficient space wise, so IMHO you can simply ignore that. So add some swap and try again :-) Simon Am 15.01.2012 02:57, schrieb Lynn W. Deffenbaugh (Mr): I'm a rank amateur at this, can you provide a link on how to use (or what to use instead of) osm2pgsql to import from a pbf instead of planet-120111.osm.bz2? .pbfs are not mentioned at http://weait.com/content/build-your-own-openstreetmap-server which is the best reference I've found on getting a tile server running for the planet. Lynn (D) On 1/14/2012 8:06 PM, Simon Poole wrote: Further tip: use the .pbf Files. It won't help a lot with your current issue, but is quite a bit faster. Simon -- Sent from my Galaxy Tab with Kaiten Mail. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] 120111 Planet Node Order?
On 1/15/2012 5:09 AM, Simon Poole wrote: current versions of osm2pgsql can import .pbf, so you are on the right track there. Richards text is good, but it was written when planet files were a third of the current size and is just a bit dated. If osm2pgsql can't hold the nodes in it's cache it will have to retrieve them from the database and that is in my and other peoples experience at least an order of magnitude slower. Any pointers on the osm2pgsql command line differences for .pbf vs .bz2-compressed OSM data? With the call for people to quit accessing the OSM tile servers, this really does need to be better, or more currently, documented. I managed to get one running, but apparently there's better ways to do it now. To give you current numbers, I just did (10 days ago) an import on a i7 2600, 16GB box that took 30 hours with the initial import phase running at Processing: Node(1322468k 340.2k/s) Way(120291k 35.64k/s) Relation(1243830 58.30/s) parse time: 28598s Node stats: total(1322468982), max(1576326287) in 3887s Way stats: total(120291564), max(144049709) in 3375s Relation stats: total(1243833), max(1951174) in 21335s THAT is the kind of data I was looking for and haven't noticed. So your Ways are about 10% of the rate of the Nodes, and Relations are about twice as fast as the Ways. and there are very few Relations by comparison. There is definitely something different in my nodes going at 45K/s and my ways going at 0.12k/s. I've since aborted that run and restarted with some VM changes, but it looks like I've got a ways to go yet. Because the box has limited memory I did an earlier attempt with -C 8000 because I knew that it would swap a lot with -C 12000. However that ran at roughly 4k/s ways and after a couple of hours I aborted it. The good news is that you -can- import on a machine with less memory (good luck keeping up with the updates on a VM though), for example lonvia has imported on a machine with 6GB total (but with -C 12000). The error message you got is weird and may point to an issue with the export process, but it just states that the node cache is going to be less efficient space wise, so IMHO you can simply ignore that. I'm also downloading the previous week's planet file just to see if my configuration goes faster on the Ways without the error, assuming I don't get the Node order error on the previous week's planet file. So add some swap and try again :-) I'll give that a go when the current run gets to the Ways and I see what the rate turns out to be. I'm at 705000k Nodes right now, so not too much longer. But I'm only getting 30.9k/s rate instead of my previous 45 so I think some of my change in the VM had less-than-desirable effects. Lynn (D) Simon Am 15.01.2012 02:57, schrieb Lynn W. Deffenbaugh (Mr): I'm a rank amateur at this, can you provide a link on how to use (or what to use instead of) osm2pgsql to import from a pbf instead of planet-120111.osm.bz2? .pbfs are not mentioned at http://weait.com/content/build-your-own-openstreetmap-server which is the best reference I've found on getting a tile server running for the planet. Lynn (D) On 1/14/2012 8:06 PM, Simon Poole wrote: Further tip: use the .pbf Files. It won't help a lot with your current issue, but is quite a bit faster. Simon -- Sent from my Galaxy Tab with Kaiten Mail. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] 120111 Planet Node Order?
Hi Simon, On 15.01.2012 11:09, Simon Poole wrote: To give you current numbers, I just did (10 days ago) an import on a i7 2600, 16GB box that took 30 hours with the initial import phase running at Can you add your specs to the benchmarks page? http://wiki.openstreetmap.org/wiki/Osm2pgsql/benchmarks Your machine look similar to the Hetzner EX4 listed there but your import was seven times faster. Stephan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] helping #switch2osm
Kai Krueger kakrue...@gmail.com wrote: We should try and get more of the rendering tool stack into the standard repositories of Ubuntu and Feodora. If you convince just one Debian Maintainer to package them they will end up beeing added to Ubuntu sooner or later anyway. Sven -- Ich fürchte mich nicht vor der Rückkehr der Faschisten in der Maske der Faschisten, sondern vor der Rückkehr der Faschisten in der Maske der Demokraten (Theodor W. Adorno) /me is giggls@ircnet, http://sven.gegg.us/ on the Web ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] 120111 Planet Node Order?
The machine -is- a Hetzner EX-4. The one in use for cleanmap has some additional HW, but the planet import can be done on a stock machine in 33 hours. Simon PS: there is no big secret, the large gain is from -C 12000 Stephan Knauss o...@stephans-server.de schrieb: Hi Simon, On 15.01.2012 11:09, Simon Poole wrote: To give you current numbers, I just did (10 days ago) an import on a i7 2600, 16GB box that took 30 hours with the initial import phase running at Can you add your specs to the benchmarks page? http://wiki.openstreetmap.org/wiki/Osm2pgsql/benchmarks Your machine look similar to the Hetzner EX4 listed there but your import was seven times faster. Stephan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev -- Diese Nachricht wurde von meinem Android-Mobiltelefon mit Kaiten Mail gesendet.___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] oAuth PUT not working
Hi, I had a working PHP script to edit tags. It uses the pecl oauth library. Now it stopped working. I suppose it is broken since the last changes on the OSM servers end of November. The script is rarely used so I just noticed the problem. My token is valid. It is possible to issue a GET request to read the preferences. But it fails with a result 401 when trying to issue a PUT request for creating a changeset. Is there anything I need to update on my side to make it work again or is the server side broken? Wiki mentions I have to use version 1.0 which the script seams to do. http://wiki.openstreetmap.org/wiki/Oauth I replaced some data by XXX: This is working: http://api.openstreetmap.org/api/0.6/user/preferences?oauth_consumer_key=XXXoauth_signature_method=HMAC-SHA1oauth_nonce=19951763504f130b4716dd67.25809464oauth_timestamp=1326648135oauth_version=1.0oauth_token=XXXoauth_signature=XXX This fails: [headers_sent] = PUT /api/0.6/changeset/create HTTP/1.1 User-Agent: PECL-OAuth/1.2.2 Host: api.openstreetmap.org Accept: */* Authorization: OAuth oauth_consumer_key=XXX,oauth_signature_method=HMAC-SHA1,oauth_nonce=20733262254f130b476499b0.46798374,oauth_timestamp=1326648135,oauth_version=1.0,oauth_token=XXX,oauth_signature=XXX Content-Length: 161 Content-Type: application/x-www-form-urlencoded [headers_recv] = HTTP/1.1 401 Authorization Required Date: Sun, 15 Jan 2012 17:22:15 GMT Server: Apache/2.2.14 (Ubuntu) X-Powered-By: Phusion Passenger (mod_rails/mod_rack) 3.0.11 WWW-Authenticate: Basic realm=Web Password X-UA-Compatible: IE=Edge,chrome=1 X-Runtime: 1.003074 Cache-Control: no-cache Status: 401 Vary: Accept-Encoding Content-Length: 25 Content-Type: text/html; charset=utf-8 Stephan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] 120111 Planet Node Order?
On 15 January 2012 11:26, Lynn W. Deffenbaugh (Mr) ldeff...@homeside.to wrote: On 1/15/2012 5:09 AM, Simon Poole wrote: To give you current numbers, I just did (10 days ago) an import on a i7 2600, 16GB box that took 30 hours with the initial import phase running at Processing: Node(1322468k 340.2k/s) Way(120291k 35.64k/s) Relation(1243830 58.30/s) parse time: 28598s Node stats: total(1322468982), max(1576326287) in 3887s Way stats: total(120291564), max(144049709) in 3375s Relation stats: total(1243833), max(1951174) in 21335s THAT is the kind of data I was looking for and haven't noticed. So your Ways are about 10% of the rate of the Nodes, and Relations are about twice as fast as the Ways. and there are very few Relations by comparison. There is definitely something different in my nodes going at 45K/s and my ways going at 0.12k/s. I've since aborted that run and restarted with some VM changes, but it looks like I've got a ways to go yet. The smaller cache will affect the ways import rate much more than nodes so this might be the result. Because the box has limited memory I did an earlier attempt with -C 8000 because I knew that it would swap a lot with -C 12000. However that ran at roughly 4k/s ways and after a couple of hours I aborted it. The good news is that you -can- import on a machine with less memory (good luck keeping up with the updates on a VM though), for example lonvia has imported on a machine with 6GB total (but with -C 12000). The error message you got is weird and may point to an issue with the export process, but it just states that the node cache is going to be less efficient space wise, so IMHO you can simply ignore that. If Lynn is running a 32-bit system or a 32-bit version of osm2pgsql, it might be impossible to allocate more than there is physical memory in the system if it's close to 2-3GB. Cheers ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] oAuth PUT not working
Hi, for the archive... On 15.01.2012 18:39, Stephan Knauss wrote: I sniffed JOSM communication to spot any difference in the oauth setup. To my surprise there was no difference, still my requests failed with 401. So I compared other headers as well. It happened that I did not declare any content type, so it ended up as application/x-www-form-urlencoded This fails: PUT /api/0.6/changeset/create HTTP/1.1 Content-Type: application/x-www-form-urlencoded [headers_recv] = HTTP/1.1 401 Authorization Required As the data is actually xml I changed it to Content-Type: text/xml And surprise: The server now accepts my edits again. But why did it respond with 401 Authorization Required? I guess there are other http status codes more appropriate. Why not 400 Bad Request? Stephan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] oAuth PUT not working
On 15/01/12 19:26, Stephan Knauss wrote: But why did it respond with 401 Authorization Required? I guess there are other http status codes more appropriate. Why not 400 Bad Request? Were you perhaps using an XML content type when computing the signature? That would lead to a signature mismatch if you then sent a different content type to the server. Tom -- Tom Hughes (t...@compton.nu) http://compton.nu/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] 120111 Planet Node Order?
On 1/15/2012 5:09 AM, Simon Poole wrote: Lynn, current versions of osm2pgsql can import .pbf, so you are on the right track there. Richards text is good, but it was written when planet files were a third of the current size and is just a bit dated. If osm2pgsql can't hold the nodes in it's cache it will have to retrieve them from the database and that is in my and other peoples experience at least an order of magnitude slower. Ok, any hints on where I can find a planet-level .pbf file? The only files I can find when linking from the Planet page (http://wiki.openstreetmap.org/wiki/Planet) download section are for the more traditional bz2 files (planet-date-osm.bz2). I found local area extracts in pbf format, but no planet pbf. I'm still trying to grok the options and appreciate the continuing assistance. Lynn (D) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] 120111 Planet Node Order?
On Sun, Jan 15, 2012 at 4:24 PM, Lynn W. Deffenbaugh (Mr) ldeff...@homeside.to wrote: On 1/15/2012 5:09 AM, Simon Poole wrote: Lynn, current versions of osm2pgsql can import .pbf, so you are on the right track there. Richards text is good, but it was written when planet files were a third of the current size and is just a bit dated. If osm2pgsql can't hold the nodes in it's cache it will have to retrieve them from the database and that is in my and other peoples experience at least an order of magnitude slower. Ok, any hints on where I can find a planet-level .pbf file? The only files I can find when linking from the Planet page ( http://wiki.openstreetmap.**org/wiki/Planethttp://wiki.openstreetmap.org/wiki/Planet) download section are for the more traditional bz2 files (planet-date-osm.bz2). I found local area extracts in pbf format, but no planet pbf. I'm still trying to grok the options and appreciate the continuing assistance. http://planet.openstreetmap.org/pbf/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] 120111 Planet Node Order?
http://planet.openstreetmap.org/pbf/ Simon PS: could you fix the wiki, as it really doesn't seem to point out where the planet pbf files are. Am 15.01.2012 23:24, schrieb Lynn W. Deffenbaugh (Mr): On 1/15/2012 5:09 AM, Simon Poole wrote: Lynn, current versions of osm2pgsql can import .pbf, so you are on the right track there. Richards text is good, but it was written when planet files were a third of the current size and is just a bit dated. If osm2pgsql can't hold the nodes in it's cache it will have to retrieve them from the database and that is in my and other peoples experience at least an order of magnitude slower. Ok, any hints on where I can find a planet-level .pbf file? The only files I can find when linking from the Planet page (http://wiki.openstreetmap.org/wiki/Planet) download section are for the more traditional bz2 files (planet-date-osm.bz2). I found local area extracts in pbf format, but no planet pbf. I'm still trying to grok the options and appreciate the continuing assistance. Lynn (D) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Planet pbf (was: 120111 Planet Node Order?)
Thank you both Ian and Andrew for the link. That's what I was looking for. Did I miss it somewhere in the Planet page (http://wiki.openstreetmap.org/wiki/Planet.osm) or should it be added there? Not even the PBF page (http://wiki.openstreetmap.org/wiki/PBF) seems to mention this link to the planet pbf, but only the extracts from Geofabrik.de. Lynn (D) - Making suggestions to make it easier for the next person following my path On 1/15/2012 5:28 PM, Andrew Ayre wrote: On 1/15/2012 5:28 PM, Ian Dees wrote: http://planet.openstreetmap.org/pbf/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] apmon's libapache2-mod-tile (was: 120111 Planet Node Order?)
On 1/15/2012 5:46 AM, Simon Poole wrote: Apmon has created a package that installs all the necessary bits and pieces see here http://wiki.openstreetmap.org/wiki/Ubuntu_tile_server since you've already done he install it wont help you a lot right now, but there are a couple of further tips on the page. I had to add python-software-properties in order to use add-apt-repository. I edited the wiki page to include this information as a note. When I executed sudo apt-get install libapache2-mod-tile gives Setting up openstreetmap-mapnik-stylesheet-data (0.1-r26689) ... unzip is not installed in /usr/bin/unzip, it is needed by this script I'm going to roll back to the original VM (http://www.thoughtpolice.co.uk/vmware/) and start over clean to see if it works after ensuring that unzip is installed before doing libapache2-mod-tile. Simply removing and re-installing libapache2-mod-tile doesn't seem to have re-executed setting up the stylesheet, so I have no real idea if the re-install actually did the unzip. Bottom line for apmon is that there seems to need to be a missing unzip dependency in one of the packages on which libapache2-mod-tile depends. Lynn (D) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Error sending mail
I have an error replying to an email, anyone seen this? thanks miek This is a permanent error. The following address(es) failed: m-239477-5b6...@messages.openstreetmap.org local delivery failed The following text was generated during the delivery attempt: -- m-239477-5b6...@messages.openstreetmap.org -- /var/lib/gems/1.8/gems/activerecord-3.1.3/lib/active_record/connection_adapters/postgresql_adapter.rb:1034:in `get_last_result': PGError: ERROR: invalid byte sequence for encoding UTF8: 0xa0 (ActiveRecord::StatementInvalid) HINT: This error can also happen if the byte sequence does not match the encoding expected by the server, which is controlled by client_encoding. : INSERT INTO messages (body, from_user_id, from_user_visible, message_read, sent_on, title, to_user_id, to_user_visible) VALUES ($1, $2, $3, $4, $5, $6, $7, $8) RETURNING id from /var/lib/gems/1.8/gems/activerecord-3.1.3/lib/active_record/connection_adapters/postgresql_adapter.rb:1034:in `exec_cache' from /var/lib/gems/1.8/gems/activerecord-3.1.3/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `exec_query' from /var/lib/gems/1.8/gems/activerecord-3.1.3/lib/active_record/connection_adapters/abstract_adapter.rb:244:in `log' from /var/lib/gems/1.8/gems/activesupport-3.1.3/lib/active_support/notifications/instrumenter.rb:21:in `instrument' from /var/lib/gems/1.8/gems/activerecord-3.1.3/lib/active_record/connection_adapters/abstract_adapter.rb:239:in `log' from /var/lib/gems/1.8/gems/activerecord-3.1.3/lib/active_record/connection_adapters/postgresql_adapter.rb:601:in `exec_query' from /var/lib/gems/1.8/gems/activerecord-3.1.3/lib/active_record/connection_adapters/abstract/database_statements.rb:63:in `exec_insert' from /var/lib/gems/1.8/gems/activerecord-3.1.3/lib/active_record/connection_adapters/abstract/database_statements.rb:90:in `insert' from /var/lib/gems/1.8/gems/activerecord-3.1.3/lib/active_record/connection_adapters/abstract/query_cache.rb:14:in `insert' from /var/lib/gems/1.8/gems/activerecord-3.1.3/lib/active_record/relation.rb:70:in `insert' from /var/lib/gems/1.8/gems/activerecord-3.1.3/lib/active_record/persistence.rb:313:in `create' from /var/lib/gems/1.8/gems/activerecord-3.1.3/lib/active_record/timestamp.rb:51:in `create' from /var/lib/gems/1.8/gems/activerecord-3.1.3/lib/active_record/callbacks.rb:268:in `create' from /var/lib/gems/1.8/gems/activesupport-3.1.3/lib/active_support/callbacks.rb:390:in `_run_create_callbacks' from /var/lib/gems/1.8/gems/activesupport-3.1.3/lib/active_support/callbacks.rb:81:in `send' from /var/lib/gems/1.8/gems/activesupport-3.1.3/lib/active_support/callbacks.rb:81:in `run_callbacks' from /var/lib/gems/1.8/gems/activerecord-3.1.3/lib/active_record/callbacks.rb:268:in `create' from /var/lib/gems/1.8/gems/activerecord-3.1.3/lib/active_record/persistence.rb:294:in `create_or_update' from /var/lib/gems/1.8/gems/activerecord-3.1.3/lib/active_record/callbacks.rb:264:in `create_or_update' from /var/lib/gems/1.8/gems/activesupport-3.1.3/lib/active_support/callbacks.rb:408:in `_run_save_callbacks' from /var/lib/gems/1.8/gems/activesupport-3.1.3/lib/active_support/callbacks.rb:81:in `send' from /var/lib/gems/1.8/gems/activesupport-3.1.3/lib/active_support/callbacks.rb:81:in `run_callbacks' from /var/lib/gems/1.8/gems/activerecord-3.1.3/lib/active_record/callbacks.rb:264:in `create_or_update' from /var/lib/gems/1.8/gems/activerecord-3.1.3/lib/active_record/persistence.rb:57:in `save!' from /var/lib/gems/1.8/gems/activerecord-3.1.3/lib/active_record/validations.rb:56:in `save!' from /var/lib/gems/1.8/gems/activerecord-3.1.3/lib/active_record/attribute_methods/dirty.rb:33:in `save!' from /var/lib/gems/1.8/gems/activerecord-3.1.3/lib/active_record/transactions.rb:246:in `save!' from /var/lib/gems/1.8/gems/activerecord-3.1.3/lib/active_record/transactions.rb:295:in `with_transaction_returning_status' from /var/lib/gems/1.8/gems/activerecord-3.1.3/lib/active_record/connection_adapters/abstract/database_statements.rb:192:in `transaction' from /var/lib/gems/1.8/gems/activerecord-3.1.3/lib/active_record/transactions.rb:208:in `transaction_without_deadlock_handling' from /srv/ www.openstreetmap.org/rails/vendor/plugins/deadlock_retry/lib/deadlock_retry.rb:46:in`transaction' from /var/lib/gems/1.8/gems/activerecord-3.1.3/lib/active_record/transactions.rb:293:in `with_transaction_returning_status' from /var/lib/gems/1.8/gems/activerecord-3.1.3/lib/active_record/transactions.rb:246:in `save!' from /srv/www.openstreetmap.org/rails/script/deliver-message:33 -- This is a copy of the message, including all the headers. -- -- James Michael DuPont Member of Free Libre Open Source Software Kosova http://flossk.org
Re: [OSM-dev] oAuth PUT not working
Hello Tom,
thank you for looking into this issue.
On 15.01.2012 20:42, Tom Hughes wrote:
On 15/01/12 19:26, Stephan Knauss wrote:
But why did it respond with 401 Authorization Required? I guess there
are other http status codes more appropriate. Why not 400 Bad Request?
Were you perhaps using an XML content type when computing the signature?
That would lead to a signature mismatch if you then sent a different
content type to the server.
The signature is computed inside the oauth library. It does all the
magic internally. I just pass my data in. the $changesetRequest contains
the XML of the changeset to create.
It worked in the past. Now I added the Content-Type header. I can't tell
what it is using internally to compute the signature. I saw the
content-type in the header and changed into text/xml as this was working
for JOSM and sounds a lot better than what was sent before.
$ret = $oauth-fetch($api_url/changeset/create, $changesetRequest,
OAUTH_HTTP_METHOD_PUT, array('Content-Type'='text/xml'));
Could it happen that the signature verification on the server is now
more strict so a thing that worked in the past is now failing?
Stephan
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] oAuth PUT not working
On 16/01/12 00:21, Stephan Knauss wrote: Could it happen that the signature verification on the server is now more strict so a thing that worked in the past is now failing? Simply put, no. The set of things included in the signature is fixed and if either end fails to include the right things then it would fail. Tom -- Tom Hughes (t...@compton.nu) http://compton.nu/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
