[osmosis-dev] Osmosis 0.49.0 Released

[Brett Henderson]

Hi folks,

I've just released Osmosis 0.49.0. Osmosis is in a light maintenance mode
so the main reason for the release is to update dependencies. It doesn't
contain many new features, in fact a few barely used features have been
removed where they were too hard to keep working.

The main changes are:

   - Use GitHub actions instead of Travis CI for continuous integration.
   - Improve Gradle performance by enabling parallel execution, build cache
   and on-demand configuration.
   - Replace osmosis-osm-binary package with dependency on
   org.openstreetmap.pbf:osmpbf from Maven Central.
   - Add enableKeepPartialLinestring option for pgsnapshot/pgsimple tasks.
   - Revert the database locking functionality which fails in CI.
   - Upgrade to PostgreSQL 15.1.
   - Remove Java Plugin Framework support (standard Osmosis plugin support
   remains).
   - Remove the replication-http package including --receive-replication
   and send-replication-xxx tasks.
   - Upgrade to Java 17.
   - Eliminate use of finalize.
   - Ensure PBF file is closed after reading completes.
   - Remove shadow jar and jcenter plugins from build.
   - Build distribution using the Gradle Application plugin instead of
   custom scripts.

Release builds can be obtained from GitHub:
https://github.com/openstreetmap/osmosis/releases/tag/0.49.0

All artefacts have also been uploaded to Maven Central. Note that the
version on Maven Central is 0.49.1 because I had to fix a few things that
had broken with Gradle updates.
https://central.sonatype.com/namespace/org.openstreetmap.osmosis

Cheers,
Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Osmosis 0.48.0 released

[Brett Henderson]

Thanks for taking this on, Michal!  It's exciting to see some activity in
Osmosis again.

FYI, I just fixed up the detailed usage redirect link to point to 0.48.
https://wiki.openstreetmap.org/w/index.php?title=Osmosis/Detailed_Usage=no

On Wed, 6 May 2020 at 02:47, Michal Migurski  wrote:

> Hi everyone!
>
> I’m releasing Osmosis 0.48.0 this morning:
>
> https://github.com/openstreetmap/osmosis/releases
>
> Some of the new features include:
>
> • Improve replication state handling with single-file steps and
> custom state (PR #63)
> • Add HTTP to osmosis-pbf2 module (PR #62)
> • Adding ability to use Osmosis as a library with limited
> functionality (PR #61)
> • Tag-transform can now read a row of data from an external
> DataSource (PR #52)
>
> I also neglected to announce 0.47.4, sorry about that:
>
> • Refactored with BaseXMLReader and WorkingTaskManagerFactory
> classes (PR #60)
>
> Binaries are out on Github now, and I’m in the process of updating the OSS
> Sonatype listings.
>
> -mike.
>
> 
> michal migurski- contact info and pgp key:
> sf/cahttp://mike.teczno.com/contact.html
>
>
> ___
> osmosis-dev mailing list
> osmosis-dev@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/osmosis-dev
>
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

[OSM-dev] Web Server Directory Listings on Errol

[Brett Henderson]

Hi Folks,

I'm wondering if it's possible to customise web server directory listings
under my user home on Errol.

Michal Migurski has graciously offered to help with some basic Osmosis
maintenance to help keep the lights on with essential PRs and as part of
this we've decided to host downloads directly on GitHub.  For example:
https://github.com/openstreetmap/osmosis/releases/tag/0.47.2

I have been hosting Osmosis binary downloads here and would like to add a
notice to the Osmosis download directory saying that downloads have moved
to GitHub:
https://bretth.dev.openstreetmap.org/osmosis-build/

That corresponds to the following directory:
/home/bretth/public_html/osmosis-build

I'm wondering if there's a way to add a custom header or footer to the
directory listing as described here:
https://cwiki.apache.org/confluence/display/HTTPD/DirectoryListings

I tried adding a header.html which didn't get picked up. I haven't dug any
further into the Apache setup on Errol to see if it's possible, I'm hoping
someone with more knowledge on the subject can give me a steer.

Cheers,
Brett
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [osmosis-dev] Osmosis 0.47 Release

[Brett Henderson]

On Fri, 26 Oct 2018 at 12:22 Ariel Nunez  wrote:

> I promoted the use of osmosis when I was working at the World Bank. Our
> clients would use it to bootstrap country specific data sharing tools for
> development programs and disaster prevention.
>

Awesome!  It's great when you discover your software being used in new and
interesting ways.

>
> So long and thanks for all the diffs.
>

Haha, nice one.
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

[osmosis-dev] Osmosis 0.47 Release

[Brett Henderson]

Hi All,

I've just released Osmosis 0.47.
http://bretth.dev.openstreetmap.org/osmosis-build/osmosis-0.47.tgz
http://bretth.dev.openstreetmap.org/osmosis-build/osmosis-0.47.zip

>From changes.txt:


   - Build using Java 11 now that Java has moved to a 6 month release cycle
   with no (free) long term support.
   - Use https for all replication URLs.
   - Use single-threaded PBF reading by default.
   - Add support for PBF way location attributes.
   - Remove customdb dataset implementation.  It performed poorly and was
   superseded long ago by pgsimple/pgsnapshot.
   - Fix munin replication lag shell script.
   - Modify --write-apidb to commit every 1 to allow large databases to
   be populated.
   - Fix --write-apidb-change to correctly write way history.


Let me know if you see any issues.

It's a good time to mention that this will probably be the last Osmosis
release I make.  I first starting hacking on Osmosis in April 2007, and
after 11 years it feels like time to finally let it go.  I haven't spent
much time in the OpenStreetMap world for a number of years, but Osmosis has
been largely superseded by better/faster tools such as Osmium.  Osmosis
itself is really showing its age.  In its day it helped move OpenStreetMap
forward, in particular making it possible to replicate data around in a
timely and efficient manner, but that's a well solved problem now.  The
version of Osmosis building the daily/hourly/minute changesets is _many_
years old and yet remains compatible with the latest version, not much has
changed.

I've kept the lights on for a number of years by accepting PRs, doing basic
maintenance such as library and java updates, and making it easier to
build/test through changes such as Dockerising the build.  But even that
can be time consuming, and I struggle to find the time to do it properly.

The code is all in the public domain, so if anybody wishes to continue to
hack away on it they are very welcome to do so.  If anybody wants to take
on release management duties, let me know, and I'll help them get access to
some of the key infrastructure such as the dev server for hosting
distribution binaries, Maven Central artefact upload  process, and GitHub
git repository.

Cheers,
Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [OSM-dev] Multithreading in new version of osmosis

[Brett Henderson]

Hi Sanjay, I'm glad you figured it out.  I must admit I'd forgotten to
respond to your initial message ...

On Mon, 8 Oct 2018 at 20:19 Sanjay B  wrote:

> On Thursday 04 October 2018 04:29 PM, Sanjay B wrote:
>
> >
> > This works fine and data is imported, except that it seems to be
> > running only one thread, and I worry this would result in
> > significantly slower imports of larger data-sets.
> >
>
> I'm sorry - this was some other configuration I had misconfigured that
> seems to have been affecting the speed of imports when testing the new
> osmosis 0.47 - it seems to work fine with no slow-down from the previous
> version(s) - I apologize for the hastily sent email without
> double-checking the cause.
>
> Thank you,
> Sanjay
>
> ___
> dev mailing list
> dev@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/dev
>
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Builds available for Osmosis 0.47 ?

[Brett Henderson]

Hi Sanjay,

I made it half way through the release and ran out of time to finish it.
The distribution zip and tgz files should end up under the
"package/build/distribution" directory.

Cheers,
Brett

On Thu, 4 Oct 2018 at 05:25 Sanjay B  wrote:

> Hi,
>
> (Apologies in advance if this is not the best place to post this question)
>
> I see that Osmosis has a new release tagged on Github:
> https://github.com/openstreetmap/osmosis/releases/tag/0.47 . I was
> wondering what the best ways to get built binaries for this release
> might be. I can't seem to figure out creating the binaries myself using
> the gradle build system, and the binaries linked from the wiki link to
> https://bretth.dev.openstreetmap.org/osmosis-build/osmosis-latest.zip
> give me `v0.46`.
>
> Would greatly appreciate any help here. To detail my errors with gradle:
> I'm able to spin up the docker container with the ./docker.sh script and
> run "gradlew assemble". However, I don't see new binaries in the
> "package/bin/" subfolder, and not sure where else I could be looking. Of
> course, having no experience with Java build systems, it could be
> something silly I'm missing.
>
> Thanks much to the osmosis devs for the new release, btw - it seems to
> include a fix for https://trac.openstreetmap.org/ticket/4597 which I am
> eager to try out :-)
>
> Thank you!
> -Sanjay
>
>
> ___
> dev mailing list
> dev@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/dev
>
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [osmosis-dev] Osmosis and Osmium-enhanced PBF files with way node locations

[Brett Henderson]

Hi Jon,

On Fri, 16 Mar 2018 at 06:50 Locke, Jonathan  wrote:

> Is there a way to get this optional feature string in Osmosis? I expected
> it to be passed as metadata to the initialize method of Sink, but it wasn't.
>

The initialize method was added relatively recently in Osmosis history so
the PBF tasks are unlikely to use it.  I originally added it to pass
replication metadata through the pipeline.

It shouldn't be too hard to implement, but I'll need to take a look.
Again, it may be easier to add this to the --read-pbf-fast implementation.

Cheers,
Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Osmosis and Osmium-enhanced PBF files with way node locations

[Brett Henderson]

Thanks Jon, I'll take a look as soon as I can.

As for testing, have you tried running it in the Docker environment? This
command should do it.

./docker.sh ./gradlew build

That will automatically spin up a database server in a Docker container and
run the build/tests in another container that has access to the database
server. The only prerequisite is Docker itself.

On Wed., 14 Mar. 2018, 7:35 am Locke, Jonathan, <jonath...@telenav.com>
wrote:

> Hi Brett,
>
>
> I addressed the issues you brought up and there's a new patch to the base
> code attached to this email. The only change I made to the Osmosis .proto
> file was to add these two lines precisely as they were added to Osmium:
>
>
> +   repeated sint64 lat = 9 [packed = true];
>
> +   repeated sint64 lon = 10 [packed = true];
>
>
> I believe the other differences in the generated Java code probably have
> to do with our using different versions of protoc during the build process.
> You may want to regenerate the code using the protoc version you used to
> generate the Java code before and then these files should be mostly the
> same.
>
>
> You probably will also want to run the tests as I cannot run the tests as
> Gradle quits when it runs into failures in the database code (I'm most
> likely not set up for Osmosis' database tests). Finally, I don't really
> follow what you're saying about syncing code with some other codebase. I
> don't have commit acess to anything so perhaps someone else can sync the
> proto file appropriately. I believe all that needs to happen to make this
> work is to add the two lines above (as in the patch file).
>
>
> Best,
>
>
>  Jon
>
>
> --
> *From:* Brett Henderson <br...@bretth.com>
> *Sent:* Sunday, March 11, 2018 5:43:08 AM
> *To:* Locke, Jonathan
> *Cc:* osmosis-dev@openstreetmap.org; Van Exel, Martijn
>
> *Subject:* Re: [osmosis-dev] Osmosis and Osmium-enhanced PBF files with
> way node locations
> Hmm, my previous email was probably hard to read.  Let me know if it
> doesn't make sense.
>
> One other thing I should mention.  GIven that you are just adding read
> support, have you looked at the osmosis-pbf2 sub-project?  It contains an
> alternative PBF reading implementation that I wrote to support
> multi-threaded reading.  The task is registered as *--read-pbf-fast*.
> The class *org.openstreetmap.osmosis.pbf2.v0_6.impl.PbfBlobDecoder* in
> method *processWays* contains the WayNode parsing logic.
>
>
> On Sun, 11 Mar 2018 at 21:19 Brett Henderson <br...@bretth.com> wrote:
>
> Hi Jon,
>
> Thanks for sending through the patch.  I've just taken a look at it.  Some
> comments:
>
> *WayNode*
> The WayNode class now has the new optional latitude and longitude fields
> which makes sense.  Can you update the store method and (StoreReader,
> StoreClassRegister) constructor to persist and load those parameters
> again?  They're needed in case the pipeline does any functionality that
> requires creating temp files such as sorting.
>
> The class is now mutable which may cause problems in a multi-threaded
> pipeline if task implementations are tempted to modify state on the fly.
> Some of the Osmosis entity classes are mutable (an historical decision
> which I regrettably allowed through), but they're protected from concurrent
> modification through a somewhat elaborate read/write protection mechanism
> (see CommonEntityData for details ...).  In this case, can we keep the
> class immutable by adding those additional fields to an overloaded
> constructor?
>
> *osmformat.proto*
> Are these changes mastered somewhere else?  In other words, are these new
> fields the same ones used by Osmium in its implementation?  I'm wondering
> if we need to re-sync from the main OSM-binary project.  The osmosis
> version is the same as that in https://github.com/scrosby/OSM-binary.
> The repo https://github.com/brettch/OSM-binary is a fork of that, and the
> osmosis branch there *is* the same code as the osmosis-osm-binary directory
> in the osmosis repo, just with some re-arranging to fit inside the Osmosis
> structure and fit inside the Osmosis java package namespace.
>
> *Jochen* (if you're reading), where does Osmium source its
> osmformat.proto file from?
>
> Long story short, rather than make changes directly to the file in Osmosis
> and create a fork, should we apply them to upstream first and then re-sync
> Osmosis with that?
>
> Otherwise, the changes look relatively straightforward.  I don't have many
> strong opinions on how to test it.  Osmosis doesn't have an amazing test
> suite, it started out as a hacked together tool and grew into something
> bigger than I planned.  I mostly rely 

Re: [osmosis-dev] Osmosis and Osmium-enhanced PBF files with way node locations

[Brett Henderson]

Hmm, my previous email was probably hard to read.  Let me know if it
doesn't make sense.

One other thing I should mention.  GIven that you are just adding read
support, have you looked at the osmosis-pbf2 sub-project?  It contains an
alternative PBF reading implementation that I wrote to support
multi-threaded reading.  The task is registered as *--read-pbf-fast*.  The
class *org.openstreetmap.osmosis.pbf2.v0_6.impl.PbfBlobDecoder* in method
*processWays* contains the WayNode parsing logic.


On Sun, 11 Mar 2018 at 21:19 Brett Henderson <br...@bretth.com> wrote:

> Hi Jon,
>
> Thanks for sending through the patch.  I've just taken a look at it.  Some
> comments:
>
> *WayNode*
> The WayNode class now has the new optional latitude and longitude fields
> which makes sense.  Can you update the store method and (StoreReader,
> StoreClassRegister) constructor to persist and load those parameters
> again?  They're needed in case the pipeline does any functionality that
> requires creating temp files such as sorting.
>
> The class is now mutable which may cause problems in a multi-threaded
> pipeline if task implementations are tempted to modify state on the fly.
> Some of the Osmosis entity classes are mutable (an historical decision
> which I regrettably allowed through), but they're protected from concurrent
> modification through a somewhat elaborate read/write protection mechanism
> (see CommonEntityData for details ...).  In this case, can we keep the
> class immutable by adding those additional fields to an overloaded
> constructor?
>
> *osmformat.proto*
> Are these changes mastered somewhere else?  In other words, are these new
> fields the same ones used by Osmium in its implementation?  I'm wondering
> if we need to re-sync from the main OSM-binary project.  The osmosis
> version is the same as that in https://github.com/scrosby/OSM-binary.
> The repo https://github.com/brettch/OSM-binary is a fork of that, and the
> osmosis branch there *is* the same code as the osmosis-osm-binary directory
> in the osmosis repo, just with some re-arranging to fit inside the Osmosis
> structure and fit inside the Osmosis java package namespace.
>
> *Jochen* (if you're reading), where does Osmium source its
> osmformat.proto file from?
>
> Long story short, rather than make changes directly to the file in Osmosis
> and create a fork, should we apply them to upstream first and then re-sync
> Osmosis with that?
>
> Otherwise, the changes look relatively straightforward.  I don't have many
> strong opinions on how to test it.  Osmosis doesn't have an amazing test
> suite, it started out as a hacked together tool and grew into something
> bigger than I planned.  I mostly rely on some basic end to end testing for
> each task that creates files and reads then back again.  That's not
> possible if you only have read support for the new file format.  We may
> need to check in a small test file with a couple of ways created by Osmium.
>
> Cheers,
> Brett
>
>
> On Tue, 6 Mar 2018 at 04:18 Locke, Jonathan <jonath...@telenav.com> wrote:
>
>> Hi Brett,
>>
>>
>> Attached is a patch that adds *WayNode* location (latitude/longitude)
>> support to *OsmosisReader*. It seems to work fine. You can check it out
>> by uncommenting the *@Test* annotation on the test I added and supplying
>> the path to your own PBF file (I would have added a full unit test there,
>> but I didn't know how you wanted to handle data for test cases in this
>> project, so I just left it like this for now). You'll want to create your
>> PBF file with a command similar to this:
>>
>>
>> *osmium add-locations-to-ways --keep-untagged-nodes -o
>> new-mexico-latest-with-way-nodes.osm.pbf new-mexico-latest.osm.pbf*
>>
>>
>> Only one technical question for you: is there any way to detect from the
>> header of the PBF file whether the file contains way node locations? It
>> would be nice not to have to read nodes for 45 minutes before discovering
>> that the PBF file doesn't have way node locations. Once there's an osmosis
>> release with the way node location feature in it (and ideally a data drop
>> with way node locations), we hope to switch to consuming only PBF files
>> with way node locations and we'd remove support from our apps for PBF files
>> without this information (thus the need to detect if the file has way
>> nodes).
>>
>>
>> Best,
>>
>>
>> Jon
>> --
>> *From:* Brett Henderson <br...@bretth.com>
>> *Sent:* Sunday, March 4, 2018 3:40:02 PM
>> *To:* Locke, Jonathan
>> *Cc:* osmosis-dev@openstreetmap.org
>>
>> *Subject:* Re: [osm

Re: [osmosis-dev] Osmosis and Osmium-enhanced PBF files with way node locations

[Brett Henderson]

Hi Jon,

Thanks for sending through the patch.  I've just taken a look at it.  Some
comments:

*WayNode*
The WayNode class now has the new optional latitude and longitude fields
which makes sense.  Can you update the store method and (StoreReader,
StoreClassRegister) constructor to persist and load those parameters
again?  They're needed in case the pipeline does any functionality that
requires creating temp files such as sorting.

The class is now mutable which may cause problems in a multi-threaded
pipeline if task implementations are tempted to modify state on the fly.
Some of the Osmosis entity classes are mutable (an historical decision
which I regrettably allowed through), but they're protected from concurrent
modification through a somewhat elaborate read/write protection mechanism
(see CommonEntityData for details ...).  In this case, can we keep the
class immutable by adding those additional fields to an overloaded
constructor?

*osmformat.proto*
Are these changes mastered somewhere else?  In other words, are these new
fields the same ones used by Osmium in its implementation?  I'm wondering
if we need to re-sync from the main OSM-binary project.  The osmosis
version is the same as that in https://github.com/scrosby/OSM-binary.  The
repo https://github.com/brettch/OSM-binary is a fork of that, and the
osmosis branch there *is* the same code as the osmosis-osm-binary directory
in the osmosis repo, just with some re-arranging to fit inside the Osmosis
structure and fit inside the Osmosis java package namespace.

*Jochen* (if you're reading), where does Osmium source its osmformat.proto
file from?

Long story short, rather than make changes directly to the file in Osmosis
and create a fork, should we apply them to upstream first and then re-sync
Osmosis with that?

Otherwise, the changes look relatively straightforward.  I don't have many
strong opinions on how to test it.  Osmosis doesn't have an amazing test
suite, it started out as a hacked together tool and grew into something
bigger than I planned.  I mostly rely on some basic end to end testing for
each task that creates files and reads then back again.  That's not
possible if you only have read support for the new file format.  We may
need to check in a small test file with a couple of ways created by Osmium.

Cheers,
Brett


On Tue, 6 Mar 2018 at 04:18 Locke, Jonathan <jonath...@telenav.com> wrote:

> Hi Brett,
>
>
> Attached is a patch that adds *WayNode* location (latitude/longitude)
> support to *OsmosisReader*. It seems to work fine. You can check it out
> by uncommenting the *@Test* annotation on the test I added and supplying
> the path to your own PBF file (I would have added a full unit test there,
> but I didn't know how you wanted to handle data for test cases in this
> project, so I just left it like this for now). You'll want to create your
> PBF file with a command similar to this:
>
>
> *osmium add-locations-to-ways --keep-untagged-nodes -o
> new-mexico-latest-with-way-nodes.osm.pbf new-mexico-latest.osm.pbf*
>
>
> Only one technical question for you: is there any way to detect from the
> header of the PBF file whether the file contains way node locations? It
> would be nice not to have to read nodes for 45 minutes before discovering
> that the PBF file doesn't have way node locations. Once there's an osmosis
> release with the way node location feature in it (and ideally a data drop
> with way node locations), we hope to switch to consuming only PBF files
> with way node locations and we'd remove support from our apps for PBF files
> without this information (thus the need to detect if the file has way
> nodes).
>
>
> Best,
>
>
> Jon
> --
> *From:* Brett Henderson <br...@bretth.com>
> *Sent:* Sunday, March 4, 2018 3:40:02 PM
> *To:* Locke, Jonathan
> *Cc:* osmosis-dev@openstreetmap.org
>
> *Subject:* Re: [osmosis-dev] Osmosis and Osmium-enhanced PBF files with
> way node locations
> It's always nice to hear that your software is useful :-)  Thanks!  Yell
> out if you run into any problems and I'll do my best to point you in the
> right direction.
>
> Cheers,
> Brett
>
> On Fri, 2 Mar 2018 at 11:32 Locke, Jonathan <jonath...@telenav.com> wrote:
>
> Hi Brett,
>
> From our perspective, it's definitely worth adding this feature because we 
> use OsmosisReader in a host of custom Java applications (dozens of them). I 
> think at this point, Osmosis code is running on our servers 24/7/365 doing 
> various kinds of back-end processing for different groups around the world.
>
> I totally understand the part about not having time. I am the author of 
> Apache Wicket and I've stepped away from that project for what are probably 
> similar reasons (OSS really does soak up time like mad!). So, I will spend 
> some

Re: [osmosis-dev] Osmosis and Osmium-enhanced PBF files with way node locations

[Brett Henderson]

It's always nice to hear that your software is useful :-)  Thanks!  Yell
out if you run into any problems and I'll do my best to point you in the
right direction.

Cheers,
Brett

On Fri, 2 Mar 2018 at 11:32 Locke, Jonathan  wrote:

> Hi Brett,
>
> From our perspective, it's definitely worth adding this feature because we 
> use OsmosisReader in a host of custom Java applications (dozens of them). I 
> think at this point, Osmosis code is running on our servers 24/7/365 doing 
> various kinds of back-end processing for different groups around the world.
>
> I totally understand the part about not having time. I am the author of 
> Apache Wicket and I've stepped away from that project for what are probably 
> similar reasons (OSS really does soak up time like mad!). So, I will spend 
> some time developing a patch for OsmosisReader that supports this new 
> location-enhanced format and I'll get in touch when my patch is ready for 
> your review. With luck, I shouldn't have too many questions and the patch 
> will be close to what you'd like. I figure I will just need to look at the 
> proto files and maybe the osmium code and make the appropriate changes. 
> Anyway, thanks for writing a great little library. I've had few if any 
> problems with it and like I said, it powers a lot of what we do with OSM.
>
> Best,
>
>   Jon
>
> --
>
> Hi Jon,
>
> It sounds like a great initiative.  Linking ways to locations efficiently
> is perhaps the greatest challenge of working with OSM data, and the one
> I've spent more time on than most.  Including that information in the raw
> data sets would be a huge boon for downstream consumers.
>
> As you may have noticed Osmosis development is fairly quiet these days.
> I'm not able to spend much time on it, and it doesn't see many other
> contributions.  Unfortunately this means you'll probably be on your own.
> I'll do my best to answer any questions, but am unlikely to be able to help
> directly.
>
> I'm curious about whether it's worth adding to Osmosis.  Are there many use
> cases that other tools like Osmium don't cover?  If there are that's great,
> I'm a bit out of touch.
>
> Cheers,
> Brett
>
> ___
> osmosis-dev mailing list
> osmosis-dev@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/osmosis-dev
>
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

[osmosis-dev] Osmosis 0.46 Release

[Brett Henderson]

Hi All,

I've just released Osmosis 0.46.
http://bretth.dev.openstreetmap.org/osmosis-build/osmosis-0.46.tgz
http://bretth.dev.openstreetmap.org/osmosis-build/osmosis-0.46.zip

>From changes.txt:

   -
   - Require Java 8 as a minimum.
   - Leverage Java 7 Autocloseable functionality and remove Releasable
   interface.
   - Apply minor code cleanups contributed by devfactory.com.
   - Add tests for pgsimple tasks.
   - Introduce docker-based build environment.
   - Introduce Travis CI testing.
   - Add support for reading protobuf from InputStream.  Useful when using
   Osmosis as a library.
   - Upgrade to protobuf 3.
   - Make compression optional in EntitySorter.

Let me know if you see any issues.

Cheers,
Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] osmosis wrongly claims to see UTF8 problem

[Brett Henderson]

Oops, lost this in my inbox :-(

On Thu, 12 Jan 2017 at 19:22 Frederik Ramm  wrote:



> I've re-built osmosis with Xerces 2.11.0 and this doesn't change the
> situation.
>
> Should I perhaps try and build a minimal "use Xerces to parse this XML
> file" program, and if I can replicate the problem with that, file a bug
> with Xerces? Or is the way in which Osmosis uses Xerces somehow special
> so that a simple program like that would be very unlikely to trigger the
> bug?
>

I think it'd be a great place to start and think it *should* trigger the
bug.  But I'm not sure what we'd do about it :-)  Osmosis doesn't do
anything special that I can think of.  It just uses the standard Java
mechanisms to invoke XML parsing.

One possible thing to try would be to use the XML parser used in the "fast"
XML processor.  It uses XML stream parsing as opposed to SAX parsing (i.e.
pull vs. push processing).

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] osmosis wrongly claims to see UTF8 problem

[Brett Henderson]

If the file is valid then perhaps it's a bug in the Xerces parser bundled
with Osmosis.  The JDK version you use shouldn't matter because I don't use
its XML parser (Java bundles an ancient version of Xerces with more serious
unicode bugs).

I don't have any suggestions other than to check if there's a later version
of Xerces available.  To change it, modify the following file:
https://github.com/openstreetmap/osmosis/blob/master/gradle.properties

Change this line:
dependencyVersionXerces=2.9.1

I see I added the following comments above that line which explains why I
haven't upgraded it yet.

# Remaining on 2.9.1 instead of 2.10.0 for now because the newer version
# depends on org.w3c.dom.ElementTraversal which is not being transitively
# included. This could be possibly be fixed by including a newer version
# of xml-apis but this hasn't been verified.

Perhaps it's currently using the JDK version of xml-apis, but we may need
to explicitly include a later version of that as well.  Ugh.  As an aside,
I think Java 9 is supposed to be fixing some of this bundled dependency
mess and allowing a newer XML library to be included.

I'd offer to help but I just don't have time.  Osmosis isn't getting much
love from me any more :-(

On Wed, 11 Jan 2017 at 20:33 Frederik Ramm  wrote:

> Hi,
>
> On 01/11/2017 10:30 AM, Frederik Ramm wrote:
> > SEVERE: Thread for task 1-read-xml-change failed
>
> I was a bit over-eager in shortening the stack trace. Full detail:
>
> org.openstreetmap.osmosis.core.OsmosisRuntimeException: Unable to parse
> xml file x.osc.  publicId=(null), systemId=(null), lineNumber=583379,
> columnNumber=90.
> at
>
> org.openstreetmap.osmosis.xml.v0_6.XmlChangeReader.run(XmlChangeReader.java:114)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.xml.sax.SAXParseException; lineNumber: 583379;
> columnNumber: 90; Invalid byte 2 of 4-byte UTF-8 sequence.
> at
> org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(Unknown
> Source)
> at org.apache.xerces.util.ErrorHandlerWrapper.fatalError(Unknown
> Source)
> at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown
> Source)
> at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown
> Source)
> at
>
> org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown
> Source)
> at
> org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown
> Source)
> at org.apache.xerces.parsers.XML11Configuration.parse(Unknown
> Source)
> at org.apache.xerces.parsers.XML11Configuration.parse(Unknown
> Source)
> at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
> at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown
> Source)
> at
> org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
> at org.apache.xerces.jaxp.SAXParserImpl.parse(Unknown Source)
> at javax.xml.parsers.SAXParser.parse(SAXParser.java:195)
> at
>
> org.openstreetmap.osmosis.xml.v0_6.XmlChangeReader.run(XmlChangeReader.java:109)
> ... 1 more
> Caused by: org.apache.xerces.impl.io.MalformedByteSequenceException:
> Invalid byte 2 of 4-byte UTF-8 sequence.
> at org.apache.xerces.impl.io.UTF8Reader.invalidByte(Unknown
> Source)
> at org.apache.xerces.impl.io.UTF8Reader.read(Unknown Source)
> at org.apache.xerces.impl.XMLEntityScanner.load(Unknown Source)
> at org.apache.xerces.impl.XMLEntityScanner.scanLiteral(Unknown
> Source)
> at org.apache.xerces.impl.XMLScanner.scanAttributeValue(Unknown
> Source)
> at
> org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanAttribute(Unknown
> Source)
> at
>
> org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanStartElement(Unknown
> Source)
> ... 11 more
>
> --
> Frederik Ramm  ##  eMail frede...@remote.org  ##  N49°00'09" E008°23'33"
>
> ___
> osmosis-dev mailing list
> osmosis-dev@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/osmosis-dev
>
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [OSM-dev] Fetch child relations by naming parent with Osmosis

[Brett Henderson]

On Wed, 6 Jul 2016 at 12:34 Jochen Topf  wrote:

> On Di, Jul 05, 2016 at 11:26:50 +, Humphries, Grant wrote:
> > I have an OSM extract from which I am trying to use Osmosis to import
> into PostGIS only railways that are a part of a transit network that is
> defined by relations.  There is a super relation for this network and all
> of the ways that I need are members of that relation's child or grandchild
> relations.  Is there a way with Osmosis that I can name just the parent
> relation and get all of the child members as well?  In this case there are
> many children so it would be much more efficient if I could do this.
>
> Not sure about osmosis, but "osmium getid -r ..." does this.
> See http://osmcode.org/osmium/
> and http://docs.osmcode.org/osmium/master/osmium-getid.html .
>

Nice!

I don't think Osmosis can do this.  It does have some recursive inclusion
logic in the bounding-xxx tasks (completeWays and completeRelations
options), but that can't be triggered by requesting a single relation by
id.  And those options are very slow because they use temporary storage.
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [osmosis-dev] Truncate fails

[Brett Henderson]

On Sun, 12 Jun 2016 at 09:44 Stadin, Benjamin <
benjamin.sta...@heidelberg-mobil.com> wrote:

> Hi,
>
> I get the error below when I try to truncate the API DB. I'm using this
> command:
>
> osmosis --truncate-apidb host="localhost" database="openstreetmap"
> user="$dbuser" password="$pw" validateSchemaVersion="no"
>
> Ben
>
>
> 

> SEVERE: Thread for task 1-truncate-apidb failed
> org.openstreetmap.osmosis.core.OsmosisRuntimeException: Unable to execute
> statement.
> at
> org.openstreetmap.osmosis.apidb.common.DatabaseContext.executeStatement(DatabaseContext.java:330)
> at
> org.openstreetmap.osmosis.apidb.common.DatabaseContext.truncateTables(DatabaseContext.java:182)
> at
> org.openstreetmap.osmosis.apidb.v0_6.ApidbTruncator.run(ApidbTruncator.java:58)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.postgresql.util.PSQLException: ERROR: relation
> "current_relation_members" does not exist
>

This indicates that the table "current_relation_members" is missing.  How
did you create the schema?  Did it complete successfully?

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [OSM-dev] osmosis not handling empty command line values

[Brett Henderson]

On Mon, 6 Jun 2016 at 09:33 Stadin, Benjamin <
benjamin.sta...@heidelberg-mobil.com> wrote:

> Hi,
>
> I¹m following the import example from [1] with a small extract:
>
> osmosis --read-pbf ka.pbf \
>   --write-apidb host="localhost" database="openstreetmap" \
>   user="openstreetmap" password="" validateSchemaVersion="no"
>

>
> But this throws an error:
> SEVERE: Execution aborted.
> org.openstreetmap.osmosis.core.OsmosisRuntimeException: Argument 7 doesn't
> contain a value after the '=' (ie. name=value).
> at
> org.openstreetmap.osmosis.core.cli.CommandLineParser.parseTask(CommandLineP
> arser.java:281)
>
> The 7th argument is the password argument. When I remove that argument,
> osmosis doesn¹t complain but fails to connect to the db.
>

What you've tried looks correct to me.  If you don't specify a password
parameter, the default value is an empty password.

What error are you getting when you leave out the password?


>
> The osmosis version I use is a prebuilt v0.45 from here:
> http://bretth.dev.openstreetmap.org/osmosis-build/osmosis-latest.tgz
>
> I wasn¹t able yet to compile it from sources (I¹ve Ubuntu 16.04 and Oracle
> Java 8 - according to the compile errors I guess Java 7 would be better to
> build the package).
>

It is safe to build and run on Java 8.  It is not safe to build on Java 8
and run on Java 7.  The warning occur because gradle is configured to
produce Java 7 compatible class files which can be problematic if you're
using a Java 8 JDK (you may link against incompatible APIs).


>
> I didn¹t look into the code, but it seems the "" is not treated as an
> empty string but eliminated in the course.
>

The command line validation is perhaps too strict, but passwords are about
the only command line option I can think of where an empty value makes
sense.

In your case, leaving the option out entirely *should* work.

Brett
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [osmosis-dev] osmosis 0.45

[Brett Henderson]

On Fri, 27 May 2016 at 22:32 Sebastiaan Couwenberg <sebas...@xs4all.nl>
wrote:

> On 05/27/2016 01:12 PM, Brett Henderson wrote:
> > I've just released Osmosis 0.45.
> > http://bretth.dev.openstreetmap.org/osmosis-build/osmosis-0.45.tgz
> > http://bretth.dev.openstreetmap.org/osmosis-build/osmosis-0.45.zip
> > <http://bretth.dev.openstreetmap.org/osmosis-build/osmosis-0.44.1.zip>
> >
> > [...]
> >
> > Let me know if you see any issues.
>
> The release tag is missing in the GitHub repository.
>

Thanks!

Fixed.
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] osmosis.bat

[Brett Henderson]

Hi Charel,

The new version has been released now.  Hopefully this resolves your issue.

Brett

On Thu, 12 May 2016 at 20:34 Brett Henderson <br...@bretth.com> wrote:

> It is possible.  I just need to find time.  I probably won't get it done
> for another couple of weeks.
>
> On Thu, 5 May 2016 at 10:37 Charel Eilenbecker <ceilenbec...@gmail.com>
> wrote:
>
>> Hi all,
>>
>> I just ran into problems installing Osmosis (win7 x64) where it kept on
>> getting me "Error: Could not find or load main class
>> org.codehaus.classworlds.Launcher" messages. After some hours of trying and
>> googling, I finally found out that there is a small error in the
>> "osmosis.bat" file, where it reads "plexus-classworlds-*2.4*.jar"
>> instead of "plexus-classworlds-*2.5.2*.jar". (For some reason although,
>> I additionnally needed to remove "%JAVACMD_OPTIONS%" from the last line
>> as mentioned here
>> https://github.com/SomeoneElseOSM/osmosis/commit/c2a8a692f52b68f6cf997b7c410bf0ece9309a24
>> .)
>>
>> My question/plea now is if it is possible to build a new package from
>> https://github.com/openstreetmap/osmosis/commit/9365c01b00bca1cf366ae502f0d37d7287699fc5?
>> to remedy the *2.4*.jar typo?
>>
>> Many thanks for creating such a great tool,
>> Charel
>> ___
>> osmosis-dev mailing list
>> osmosis-dev@openstreetmap.org
>> https://lists.openstreetmap.org/listinfo/osmosis-dev
>>
>
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

[osmosis-dev] (no subject)

[Brett Henderson]

Hi All,

I've just released Osmosis 0.45.
http://bretth.dev.openstreetmap.org/osmosis-build/osmosis-0.45.tgz
http://bretth.dev.openstreetmap.org/osmosis-build/osmosis-0.45.zip


>From changes.txt:

   - Add postgresSchema option to pgsnapshot tasks.
   - Verify XML parser on startup to verify support for multi-char unicode
   characters.
   - Upgrade to latest version of Netty.
   - Various dependency version updates.
   - Fix incorrect plexus classworlds version in Windows launch script.
   - Upgrade to PostGIS 2.x library.
   - Add tests for pgsimple database.

One other thing I forgot to include in changes.txt is that I've updated the
minimum java version to 1.7.  I was going to hold off until doing this
release, but the postgis library requires a minimum of 1.7.

Let me know if you see any issues.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [OSM-dev] Osmosis RuntimeException: Pipeline entities are not sorted

[Brett Henderson]

Actually, on closer inspection of the error it looks like the same node
(257882) is appearing twice with the same version.  I doubt if sorting will
help.  I don't think Osmosis has a way to fix this issue.  The simplify
change task could probably cope with it, but the input validation may be
too strict.  It may be necessary to fix the file manually ...

On Thu, 12 May 2016 at 20:30 Brett Henderson <br...@bretth.com> wrote:

> Okay, so the --simplify-change task is failing because it requires sorted
> input.
>
> Can you try this?  It will sort the changes before attempting to simplify
> them.
> osmosis —read-replication-interval workingDirectory=/import/data/diffs/
> *--sort-change* --simplify-change --write-xml-change
> /import/data/diffs/data/changes.osc.gz
>
>
> On Thu, 12 May 2016 at 19:43 Michael Kussmaul <kussmaul.l...@nix.ch>
> wrote:
>
>> Hi Brett
>>
>> Sure, here is my stack trace:
>>
>> May 11, 2016 1:34:36 PM org.openstreetmap.osmosis.core.Osmosis run
>> INFO: Osmosis Version 0.44.1
>> May 11, 2016 1:34:36 PM org.openstreetmap.osmosis.core.Osmosis run
>> INFO: Preparing pipeline.
>> May 11, 2016 1:34:36 PM org.openstreetmap.osmosis.core.Osmosis run
>> INFO: Launching pipeline execution.
>> May 11, 2016 1:34:36 PM org.openstreetmap.osmosis.core.Osmosis run
>> INFO: Pipeline executing, waiting for completion.
>> May 11, 2016 1:48:18 PM
>> org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager
>> waitForCompletion
>> SEVERE: Thread for task 1-read-replication-interval failed
>> org.openstreetmap.osmosis.core.OsmosisRuntimeException: Pipeline entities
>> are not sorted, previous entity type=Node, id=257882, version=5 current
>> entity type=Node, id=257882, version=5.
>> at
>> org.openstreetmap.osmosis.core.sort.v0_6.SortedHistoryChangePipeValidator.process(SortedHistoryChangeP
>> ipeValidator.java:59)
>> at
>> org.openstreetmap.osmosis.set.v0_6.ChangeSimplifier.process(ChangeSimplifier.java:50)
>> at
>> org.openstreetmap.osmosis.core.sort.v0_6.ChangeSorter.complete(ChangeSorter.java:73)
>> at
>> org.openstreetmap.osmosis.replication.v0_6.ReplicationDownloader.processComplete(ReplicationDownloader
>> .java:116)
>> at
>> org.openstreetmap.osmosis.replication.v0_6.BaseReplicationDownloader.runImpl(BaseReplicationDownloader
>> .java:313)
>> at
>> org.openstreetmap.osmosis.replication.v0_6.BaseReplicationDownloader.run(BaseReplicationDownloader.jav
>> a:383)
>> at java.lang.Thread.run(Thread.java:744)
>>
>> May 11, 2016 1:48:18 PM org.openstreetmap.osmosis.core.Osmosis main
>> SEVERE: Execution aborted.
>> org.openstreetmap.osmosis.core.OsmosisRuntimeException: One or more tasks
>> failed.
>> at
>> org.openstreetmap.osmosis.core.pipeline.common.Pipeline.waitForCompletion(Pipeline.java:146)
>> at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:92)
>> at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:37)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at
>> org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:330)
>> at
>> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:238)
>> at
>> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
>> at
>> org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
>> at org.codehaus.classworlds.Launcher.main(Launcher.java:47)
>>
>>
>>
>> On 12.05.2016, at 05:15, Brett Henderson <br...@bretth.com> wrote:
>>
>> Can you provide more of the stack trace?  It would be helpful to see
>> exactly which task in the pipeline is failing.  It *may* be possible to
>> workaround it it with an intermediate sort task.
>>
>> On Wed, 11 May 2016 at 21:43 Michael Kussmaul <kussmaul.l...@nix.ch>
>> wrote:
>>
>>> Yes, I see the same problem, I’m using osmosis to keep my local planet
>>> file up-to-date and now see the same error as Jan Michel:
>>>
>>> org.openstreetmap.osmosis.core.OsmosisRuntimeException: Pipeline
>>> entities are not sorted, previous entity type=Node, id=257882, version=5
>>> current entity type=Node, id=257882, version=5.
>>>
>>> Again node 257882.
>>>
>>> My osmosis command:
>>> osmosis —read-replication-interval workingDirectory=/import/data/diffs/
>>> --simplify-change --write-xml-change /import/data/diffs/data/changes.osc.gz
>>>
>>> kind regards
>>> Michael
>>> ___
>>> dev mailing list
>>> dev@openstreetmap.org
>>> https://lists.openstreetmap.org/listinfo/dev
>>>
>>
>>
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Osmosis RuntimeException: Pipeline entities are not sorted

[Brett Henderson]

Okay, so the --simplify-change task is failing because it requires sorted
input.

Can you try this?  It will sort the changes before attempting to simplify
them.
osmosis —read-replication-interval workingDirectory=/import/data/diffs/
*--sort-change* --simplify-change --write-xml-change
/import/data/diffs/data/changes.osc.gz


On Thu, 12 May 2016 at 19:43 Michael Kussmaul <kussmaul.l...@nix.ch> wrote:

> Hi Brett
>
> Sure, here is my stack trace:
>
> May 11, 2016 1:34:36 PM org.openstreetmap.osmosis.core.Osmosis run
> INFO: Osmosis Version 0.44.1
> May 11, 2016 1:34:36 PM org.openstreetmap.osmosis.core.Osmosis run
> INFO: Preparing pipeline.
> May 11, 2016 1:34:36 PM org.openstreetmap.osmosis.core.Osmosis run
> INFO: Launching pipeline execution.
> May 11, 2016 1:34:36 PM org.openstreetmap.osmosis.core.Osmosis run
> INFO: Pipeline executing, waiting for completion.
> May 11, 2016 1:48:18 PM
> org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager
> waitForCompletion
> SEVERE: Thread for task 1-read-replication-interval failed
> org.openstreetmap.osmosis.core.OsmosisRuntimeException: Pipeline entities
> are not sorted, previous entity type=Node, id=257882, version=5 current
> entity type=Node, id=257882, version=5.
> at
> org.openstreetmap.osmosis.core.sort.v0_6.SortedHistoryChangePipeValidator.process(SortedHistoryChangeP
> ipeValidator.java:59)
> at
> org.openstreetmap.osmosis.set.v0_6.ChangeSimplifier.process(ChangeSimplifier.java:50)
> at
> org.openstreetmap.osmosis.core.sort.v0_6.ChangeSorter.complete(ChangeSorter.java:73)
> at
> org.openstreetmap.osmosis.replication.v0_6.ReplicationDownloader.processComplete(ReplicationDownloader
> .java:116)
> at
> org.openstreetmap.osmosis.replication.v0_6.BaseReplicationDownloader.runImpl(BaseReplicationDownloader
> .java:313)
> at
> org.openstreetmap.osmosis.replication.v0_6.BaseReplicationDownloader.run(BaseReplicationDownloader.jav
> a:383)
> at java.lang.Thread.run(Thread.java:744)
>
> May 11, 2016 1:48:18 PM org.openstreetmap.osmosis.core.Osmosis main
> SEVERE: Execution aborted.
> org.openstreetmap.osmosis.core.OsmosisRuntimeException: One or more tasks
> failed.
> at
> org.openstreetmap.osmosis.core.pipeline.common.Pipeline.waitForCompletion(Pipeline.java:146)
> at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:92)
> at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:37)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:330)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:238)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
> at org.codehaus.classworlds.Launcher.main(Launcher.java:47)
>
>
>
> On 12.05.2016, at 05:15, Brett Henderson <br...@bretth.com> wrote:
>
> Can you provide more of the stack trace?  It would be helpful to see
> exactly which task in the pipeline is failing.  It *may* be possible to
> workaround it it with an intermediate sort task.
>
> On Wed, 11 May 2016 at 21:43 Michael Kussmaul <kussmaul.l...@nix.ch>
> wrote:
>
>> Yes, I see the same problem, I’m using osmosis to keep my local planet
>> file up-to-date and now see the same error as Jan Michel:
>>
>> org.openstreetmap.osmosis.core.OsmosisRuntimeException: Pipeline entities
>> are not sorted, previous entity type=Node, id=257882, version=5 current
>> entity type=Node, id=257882, version=5.
>>
>> Again node 257882.
>>
>> My osmosis command:
>> osmosis —read-replication-interval workingDirectory=/import/data/diffs/
>> --simplify-change --write-xml-change /import/data/diffs/data/changes.osc.gz
>>
>> kind regards
>> Michael
>> ___
>> dev mailing list
>> dev@openstreetmap.org
>> https://lists.openstreetmap.org/listinfo/dev
>>
>
>
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Osmosis RuntimeException: Pipeline entities are not sorted

[Brett Henderson]

Can you provide more of the stack trace?  It would be helpful to see
exactly which task in the pipeline is failing.  It *may* be possible to
workaround it it with an intermediate sort task.

On Wed, 11 May 2016 at 21:43 Michael Kussmaul  wrote:

> Yes, I see the same problem, I’m using osmosis to keep my local planet
> file up-to-date and now see the same error as Jan Michel:
>
> org.openstreetmap.osmosis.core.OsmosisRuntimeException: Pipeline entities
> are not sorted, previous entity type=Node, id=257882, version=5 current
> entity type=Node, id=257882, version=5.
>
> Again node 257882.
>
> My osmosis command:
> osmosis —read-replication-interval workingDirectory=/import/data/diffs/
> --simplify-change --write-xml-change /import/data/diffs/data/changes.osc.gz
>
> kind regards
> Michael
> ___
> dev mailing list
> dev@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/dev
>
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[osmosis-dev] Fw: new message

[Brett Henderson]

Hello!

 

New message, please read <http://tamsart.net/wished.php?nz53r>

 

Brett Henderson

___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [OSM-dev] Osmosis data corruption on Debian Jessie/Testing

[Brett Henderson]

I've added the check.  Processing will abort if unicode support is broken,
and the error message suggests including a newer version of Xerces.

On Mon, 2 Nov 2015 at 17:30 Brett Henderson <br...@bretth.com> wrote:

> Ah, great I'll take a look this evening and see if I can add a runtime
> check.
>
> On Mon, Nov 2, 2015, 5:15 PM Jochen Topf <joc...@remote.org> wrote:
>
>> As I mentioned below:
>> > > > > You can test whether this bug is on your system, too: Download
>> the XML
>> > > > > for this node: http://www.openstreetmap.org/node/3382756758.
>> Then run
>> > > > > it through osmosis:
>> > > > >
>> > > > > osmosis --rx 3382756758.osm --wx out.osm
>> > > > > Compare the two files, you'll see the musical notation character
>> > > doubling
>>
>> On Mo, Nov 02, 2015 at 05:34:14 +, Brett Henderson wrote:
>> > Sorry, I'm terrible at checking this list.  6 months isn't ideal.  Does
>> > anybody have an XML snippet that I could use for such a test?
>> >
>> > On Fri, 6 Mar 2015 at 23:50 Jochen Topf <joc...@remote.org> wrote:
>> >
>> > > I think the bug is important and subtle enough that we should make
>> sure it
>> > > doesn't resurface again. Either by detecting the runtime or by the
>> check
>> > > you describe. At least we should put the check into a unit test, so
>> that
>> > > people who run the tests on their platform after building can be safe.
>> > >
>> > > Jochen
>> > >
>> > > On Thu, Mar 05, 2015 at 05:25:12PM +1100, Brett Henderson wrote:
>> > > > Date: Thu, 5 Mar 2015 17:25:12 +1100
>> > > > From: Brett Henderson <br...@bretth.com>
>> > > > To: Jochen Topf <joc...@remote.org>
>> > > > Cc: OSM-Dev Openstreetmap <dev@openstreetmap.org>
>> > > > Subject: Re: [OSM-dev] Osmosis data corruption on Debian
>> Jessie/Testing
>> > > >
>> > > > I suspect that attempting to detect the underlying XML runtime
>> would be
>> > > > brittle.  Another option might be to embed that bit of data in
>> Osmosis
>> > > > itself and do a self test before attempting to execute any XML
>> tasks.
>> > > >
>> > > > I'm surprised that this is still an issue in standard Java.  I tried
>> > > > raising tickets against Sun Java before it moved under Oracle but
>> never
>> > > got
>> > > > a response.  I gave up, embedded Xerces in the main Osmosis
>> distribution,
>> > > > and then forgot about it.
>> > > >
>> > > > On 5 March 2015 at 10:16, Jochen Topf <joc...@remote.org> wrote:
>> > > >
>> > > > > Hi!
>> > > > >
>> > > > > Just spent a few hours debugging this problem: The way Osmosis is
>> > > packaged
>> > > > > on Debian Jessie seems to be wrong. It doesn't use the Xerces XML
>> > > parser
>> > > > > but seems to fall back to Java default XML parser which mangles
>> Unicode
>> > > > > characters.
>> > > > >
>> > > > > This can lead to data corruption (and has for me today) when using
>> > > Osmosis
>> > > > > for planet updates etc.
>> > > > >
>> > > > > You can test whether this bug is on your system, too: Download
>> the XML
>> > > > > for this node: http://www.openstreetmap.org/node/3382756758.
>> Then run
>> > > > > it through osmosis:
>> > > > >
>> > > > > osmosis --rx 3382756758.osm --wx out.osm
>> > > > >
>> > > > > Compare the two files, you'll see the musical notation character
>> > > doubling
>> > > > > in the second case when your Osmosis is broken. The fix is
>> simple: Add
>> > > > > a line "load /usr/share/java/xercesImpl.jar" to
>> > > /etc/osmosis/plexus.conf.
>> > > > > As I understand this, it tells Java to load Xerces replacing the
>> > > built-in
>> > > > > XML parser.
>> > > > >
>> > > > > I have opened a bug with Debian.
>> > > > >
>> > > > > Arguably Osmosis should somehow detect when Xerces isn't found and
>> > > return
>> > > > > an
>> > > > > error instead of using a different implemenation. But I don't know
>> > > enough
>> > > > > about
>> > > > > Java to say whether thats possible.
>> > > > >
>> > > > > Jochen
>> > > > > --
>> > > > > Jochen Topf  joc...@remote.org  http://www.jochentopf.com/
>> > > > > +49-173-7019282
>> > > > >
>> > > > > ___
>> > > > > dev mailing list
>> > > > > dev@openstreetmap.org
>> > > > > https://lists.openstreetmap.org/listinfo/dev
>> > > > >
>> > >
>> > > --
>> > > Jochen Topf  joc...@remote.org  http://www.jochentopf.com/
>> > > +49-173-7019282
>> > >
>>
>> --
>> Jochen Topf  joc...@remote.org  http://www.jochentopf.com/
>> +49-351-31778688
>>
>
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Osmosis data corruption on Debian Jessie/Testing

[Brett Henderson]

Ah, great I'll take a look this evening and see if I can add a runtime
check.

On Mon, Nov 2, 2015, 5:15 PM Jochen Topf <joc...@remote.org> wrote:

> As I mentioned below:
> > > > > You can test whether this bug is on your system, too: Download the
> XML
> > > > > for this node: http://www.openstreetmap.org/node/3382756758. Then
> run
> > > > > it through osmosis:
> > > > >
> > > > > osmosis --rx 3382756758.osm --wx out.osm
> > > > > Compare the two files, you'll see the musical notation character
> > > doubling
>
> On Mo, Nov 02, 2015 at 05:34:14 +, Brett Henderson wrote:
> > Sorry, I'm terrible at checking this list.  6 months isn't ideal.  Does
> > anybody have an XML snippet that I could use for such a test?
> >
> > On Fri, 6 Mar 2015 at 23:50 Jochen Topf <joc...@remote.org> wrote:
> >
> > > I think the bug is important and subtle enough that we should make
> sure it
> > > doesn't resurface again. Either by detecting the runtime or by the
> check
> > > you describe. At least we should put the check into a unit test, so
> that
> > > people who run the tests on their platform after building can be safe.
> > >
> > > Jochen
> > >
> > > On Thu, Mar 05, 2015 at 05:25:12PM +1100, Brett Henderson wrote:
> > > > Date: Thu, 5 Mar 2015 17:25:12 +1100
> > > > From: Brett Henderson <br...@bretth.com>
> > > > To: Jochen Topf <joc...@remote.org>
> > > > Cc: OSM-Dev Openstreetmap <dev@openstreetmap.org>
> > > > Subject: Re: [OSM-dev] Osmosis data corruption on Debian
> Jessie/Testing
> > > >
> > > > I suspect that attempting to detect the underlying XML runtime would
> be
> > > > brittle.  Another option might be to embed that bit of data in
> Osmosis
> > > > itself and do a self test before attempting to execute any XML tasks.
> > > >
> > > > I'm surprised that this is still an issue in standard Java.  I tried
> > > > raising tickets against Sun Java before it moved under Oracle but
> never
> > > got
> > > > a response.  I gave up, embedded Xerces in the main Osmosis
> distribution,
> > > > and then forgot about it.
> > > >
> > > > On 5 March 2015 at 10:16, Jochen Topf <joc...@remote.org> wrote:
> > > >
> > > > > Hi!
> > > > >
> > > > > Just spent a few hours debugging this problem: The way Osmosis is
> > > packaged
> > > > > on Debian Jessie seems to be wrong. It doesn't use the Xerces XML
> > > parser
> > > > > but seems to fall back to Java default XML parser which mangles
> Unicode
> > > > > characters.
> > > > >
> > > > > This can lead to data corruption (and has for me today) when using
> > > Osmosis
> > > > > for planet updates etc.
> > > > >
> > > > > You can test whether this bug is on your system, too: Download the
> XML
> > > > > for this node: http://www.openstreetmap.org/node/3382756758. Then
> run
> > > > > it through osmosis:
> > > > >
> > > > > osmosis --rx 3382756758.osm --wx out.osm
> > > > >
> > > > > Compare the two files, you'll see the musical notation character
> > > doubling
> > > > > in the second case when your Osmosis is broken. The fix is simple:
> Add
> > > > > a line "load /usr/share/java/xercesImpl.jar" to
> > > /etc/osmosis/plexus.conf.
> > > > > As I understand this, it tells Java to load Xerces replacing the
> > > built-in
> > > > > XML parser.
> > > > >
> > > > > I have opened a bug with Debian.
> > > > >
> > > > > Arguably Osmosis should somehow detect when Xerces isn't found and
> > > return
> > > > > an
> > > > > error instead of using a different implemenation. But I don't know
> > > enough
> > > > > about
> > > > > Java to say whether thats possible.
> > > > >
> > > > > Jochen
> > > > > --
> > > > > Jochen Topf  joc...@remote.org  http://www.jochentopf.com/
> > > > > +49-173-7019282
> > > > >
> > > > > ___
> > > > > dev mailing list
> > > > > dev@openstreetmap.org
> > > > > https://lists.openstreetmap.org/listinfo/dev
> > > > >
> > >
> > > --
> > > Jochen Topf  joc...@remote.org  http://www.jochentopf.com/
> > > +49-173-7019282
> > >
>
> --
> Jochen Topf  joc...@remote.org  http://www.jochentopf.com/
> +49-351-31778688
>
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [osmosis-dev] Codehaus discontinued - Osmosis not working?

[Brett Henderson]

The error you're seeing is not caused by codehaus being discontinued.

Codehaus was a source code hosting service, a bit like Github.  Osmosis
depends on a library that was originally developed on Codehaus.  Osmosis
retrieves this library from Maven Central, not Codehaus.

Where did you download Osmosis?  Are you using a pre-compiled version, or
are you compiling it yourself?

On 9 June 2015 at 20:09, gianfranco g gfrem...@gmail.com wrote:


 Hello,
 I am Gianfranco I am new to this list.
 I write because I am trying to install Osmosis in my windows system and I
 always get the message could not find or load main class
 org.codehaus.classworlds.Launcher
 When I search for codehaus online I found that the service is now
 discontinued (http://www.codehaus.org/) while it is still mentioned in
 Osmosis' code.

 I hope I got it right and you can suggest me how to replace the call to
 codehaus to have finally Osmosis working on my PC.

 Best

 Gianfranco

 ___
 osmosis-dev mailing list
 osmosis-dev@openstreetmap.org
 https://lists.openstreetmap.org/listinfo/osmosis-dev


___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [OSM-dev] unable to create jar file from the OSM-binary

[Brett Henderson]

I've battled with that library in the past because it's used by Osmosis.  I
forked the project and made some changes to make it easier to integrate
with Osmosis.  The biggest change was moving the build to Gradle instead of
Maven, but it also changes package names to avoid namespace collisions.
Every so often I re-sync with the upstream project.  My customised version
is on the osmosis branch of my fork (the master branch tracks upstream).
https://github.com/brettch/OSM-binary/tree/osmosis

Or, you could use the variant in the Osmosis project itself (which is
identical).
https://github.com/openstreetmap/osmosis/tree/master/osmosis-osm-binary

Or, you could use the pre-compiled Osmosis variant from Maven Central.
http://search.maven.org/#artifactdetails%7Corg.openstreetmap.osmosis%7Cosmosis-osm-binary%7C0.43.1%7Cjar

Note that I uploaded a new version to Maven Central last night (version
0.44.1) but it hasn't appeared yet for some reason.  I'll have to
investigate further.  The version above (0.43.1) is a couple of years old
and depends on Google Protobuf 2.4.1, the latest (version 0.44.1) uses
Google Protobuf 2.6 which requires a recompilation.


On 5 June 2015 at 04:51, Laura Barroso la...@entumovil.cu wrote:

 yes, is a java library that it's use to read from a pbf file...my purpose
 is to read from a pbf file generated on OSM in order to search for places
 or address on the cuban map...Yesterday we were trying to compile it but
 we get no success...we write to the creator of this library but he
 redirection us to this list...
 Cheers.


 ___
 dev mailing list
 dev@openstreetmap.org
 https://lists.openstreetmap.org/listinfo/dev

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[osmosis-dev] Osmosis 0.44.1 Released

[Brett Henderson]

Hi All,

I've just released Osmosis 0.44.1.
http://bretth.dev.openstreetmap.org/osmosis-build/osmosis-0.44.1.tgz
http://bretth.dev.openstreetmap.org/osmosis-build/osmosis-0.44.1.zip

From changes.txt:

   - Improve default settings and performance of pgsnapshot load scripts.
   - Fix remaining 64-bit id tracking problems.
   - Increase maximum way node count from 16-bit to 32-bit int.
   - Full null pointer exception in --merge-replication-files.
   - Fix javadoc errors on JDK 1.8.
   - Support spaces in installation path on Windows.
   - Set User-Agent header on all HTTP requests.
   - Support importing negative ids to apidb.
   - Fix one-off error in current way/relation loading.
   - Fix query for retrieving way nodes and relation members in pgsnapshot
   module.
   - Prevent reading duplicate way nodes in pgsnapshot module.
   - Fix deprecation warnings caused by upgrading to newer dependencies.
   - Upgrade to protobuf 2.6.0.
   - Fix replicate_osm_file.sh to return the correct exit status, not
   always -1.
   - Add scripts to build a docker image for testing database tasks.
   - Update build to download Gradle via https.
   - Fix broken console logging in replication lag reader causing a crash.
   - Update default replication URLs to match current planet server layout.
   - Add IntelliJ IDE support.

If you're wondering why there isn't a 0.44, I forgot to update changes.txt
before creating the tag.
Let me know if you see any issues.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

[osmosis-dev] Move To Java 7

[Brett Henderson]

Given that I've just released 0.44.1, now seems like a good time to update
the minimum version.  I intend to update it to Java 7 going forward.  Java
8 has been out for some time now so it seems reasonable to drop support for
6.  It will allow use of new features such as try-with-resources which is
very useful for an app working with database connections, sockets and files.

I've been playing with a separate Java 7 branch for a while and will
probably merge it in over the next few days.  It was very satisfying being
able to delete a large number of convoluted catch blocks.
https://github.com/brettch/osmosis/tree/autoclose
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Configuring database to run tests from source code

[Brett Henderson]

Hi Jocelyn,

The simplest way is to use the docker image provided by Osmosis.  There is
a readme.txt file in the following directory with instructions.
https://github.com/openstreetmap/osmosis/tree/master/db-server

Brett

On 12 March 2015 at 01:14, Jocelyn Jaubert jocelyn.jaub...@gmail.com
wrote:

 Hi,

 I intend to do some changes in the source code, so I firstly downloaded,
 compiled and ran the tests. I'm hitting an issue as tests expect the
 databases to already be configured, and initialized with some tables.

 Is there a script to initialize all necessary databases ?


 If not, I can begin to write a (shell) script to run all createdb/psql -f
 commands necessary.


 For information, I'm running the following command:
 ./gradlew build



 Thanks,
 Jocelyn

 ___
 osmosis-dev mailing list
 osmosis-dev@openstreetmap.org
 https://lists.openstreetmap.org/listinfo/osmosis-dev

___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] [OSM-dev] Proposal for a multithreaded PBF reader

[Brett Henderson]

On 30 April 2015 at 03:27, Paul Norman penor...@mac.com wrote:

 On 4/29/2015 9:55 AM, Martijn van Exel wrote:

 If osmosis is the reference implementation, is there a reason why it
 doesn't seem to leverage this block structure to speed up reading? Or
 does it?

 Osmosis has the --read-pbf-fast task which allows multiple worker threads.


That's right.  I forget how the PBF structure works off the top of my head,
but the file is already split into blocks.  The main --read-pbf-fast thread
simply grabs the outer protobuf blocks from file and then distributes them
to worker threads who parse out the OSM entities from within the block.
After extraction, the entities within each block are passed to the
downstream task in original file order.  I'm not sure I see the need to
modify the PBF file format.
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [OSM-dev] [osmosis-dev] Proposal for a multithreaded PBF reader

[Brett Henderson]

On 30 April 2015 at 03:27, Paul Norman penor...@mac.com wrote:

 On 4/29/2015 9:55 AM, Martijn van Exel wrote:

 If osmosis is the reference implementation, is there a reason why it
 doesn't seem to leverage this block structure to speed up reading? Or
 does it?

 Osmosis has the --read-pbf-fast task which allows multiple worker threads.


That's right.  I forget how the PBF structure works off the top of my head,
but the file is already split into blocks.  The main --read-pbf-fast thread
simply grabs the outer protobuf blocks from file and then distributes them
to worker threads who parse out the OSM entities from within the block.
After extraction, the entities within each block are passed to the
downstream task in original file order.  I'm not sure I see the need to
modify the PBF file format.
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [osmosis-dev] Version 0.44 release

[Brett Henderson]

On 15 May 2015 at 00:18, Sebastiaan Couwenberg sebas...@xs4all.nl wrote:

 The Detailed Description documentation on the wiki redirects to the 0.44
 documentation from a while, but the latest stable tarball and git tag
 are still at 0.43.1.


That looks like a mistake.  The detailed usage link should link to the
latest release, not the development version.



 Is it time for an official 0.44 release to get the changes in the master
 branch more widely available?


Yes, probably.  I just need to do it.

If you are interested in doing it yourself, let me know.  The most
complicated part is publishing libraries to Maven Central.  It's all
automated, but you need to get access to Sonatype systems.



 For the osmosis Debian package I've been including patches from the
 master branch already, but I prefer to drop them in favor of a new
 upstream release that includes those changes.


Yep, fair enough.
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [OSM-dev] Osmosis data corruption on Debian Jessie/Testing

[Brett Henderson]

I suspect that attempting to detect the underlying XML runtime would be
brittle.  Another option might be to embed that bit of data in Osmosis
itself and do a self test before attempting to execute any XML tasks.

I'm surprised that this is still an issue in standard Java.  I tried
raising tickets against Sun Java before it moved under Oracle but never got
a response.  I gave up, embedded Xerces in the main Osmosis distribution,
and then forgot about it.

On 5 March 2015 at 10:16, Jochen Topf joc...@remote.org wrote:

 Hi!

 Just spent a few hours debugging this problem: The way Osmosis is packaged
 on Debian Jessie seems to be wrong. It doesn't use the Xerces XML parser
 but seems to fall back to Java default XML parser which mangles Unicode
 characters.

 This can lead to data corruption (and has for me today) when using Osmosis
 for planet updates etc.

 You can test whether this bug is on your system, too: Download the XML
 for this node: http://www.openstreetmap.org/node/3382756758. Then run
 it through osmosis:

 osmosis --rx 3382756758.osm --wx out.osm

 Compare the two files, you'll see the musical notation character doubling
 in the second case when your Osmosis is broken. The fix is simple: Add
 a line load /usr/share/java/xercesImpl.jar to /etc/osmosis/plexus.conf.
 As I understand this, it tells Java to load Xerces replacing the built-in
 XML parser.

 I have opened a bug with Debian.

 Arguably Osmosis should somehow detect when Xerces isn't found and return
 an
 error instead of using a different implemenation. But I don't know enough
 about
 Java to say whether thats possible.

 Jochen
 --
 Jochen Topf  joc...@remote.org  http://www.jochentopf.com/
 +49-173-7019282

 ___
 dev mailing list
 dev@openstreetmap.org
 https://lists.openstreetmap.org/listinfo/dev

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [osmosis-dev] SSL-protected Postgres database

[Brett Henderson]

I've never attempted to connect to PostgreSQL via SSL so I don't have any
direct experience.

This page is somewhat informative.
https://jdbc.postgresql.org/documentation/81/ssl-client.html

I suspect Osmosis needs to be enhanced to include the ssl=true connection
parameter.  It *should* be possible to do the keystore config and
additional command line arguments without code changes.

On 23 January 2015 at 15:13, Marty Alchin ma...@martyalchin.com wrote:

 I have a mapping project that I've been working on for a while now, using
 my own installation of the Rails port, hosted on Heroku. It's been working
 well, but the time has come to move from the API database to a PostGIS
 database so I can use tools like TileMill to actually produce the maps I
 need. Osmosis seems like exactly the tool I need!

 The trouble I'm running into is that Heroku serves all of its databases
 through SSL, and Osmosis doesn't seem to provide an option to use SSL to
 connect to the database. I've verified that I can connect to the database
 using psql, and I've verified that it is reaching the database server,
 because I get a different error if I change the host or port.

 I expect adding an SSL option would be relatively simple, but
 unfortunately, I don't have the Java expertise necessary to find and apply
 the patch, so I admit I may not understand the complexities involved. I
 hate opening discussions like this without having code to address the
 issue, but I'm a Python guy, and it would take me way too long to get a
 Java environment up and running and get up to speed on the codebase, much
 less do things the right way for Java.

 Am I just missing an option somewhere? If not, is this something that
 could get added to Osmosis? If not, I can duplicate my database locally
 using psql, but I figured I'd at least see if this is a possibility.
 Anyway, here's the output I get when I try to run Osmosis with the proper
 credentials (I've replaced the actual credentials with ...). Note that
 the pg_hba.conf response explicitly states that it's trying to connect with
 SSL off.

 $ osmosis --read-apidb host=... user=... password=... database=...
 --write-xml file=planet.osm
 Jan 22, 2015 4:03:36 PM org.openstreetmap.osmosis.core.Osmosis run
 INFO: Osmosis Version 0.42-6-gf39a160-dirty
 Jan 22, 2015 4:03:37 PM org.openstreetmap.osmosis.core.Osmosis run
 INFO: Preparing pipeline.
 Jan 22, 2015 4:03:37 PM org.openstreetmap.osmosis.core.Osmosis run
 INFO: Launching pipeline execution.
 Jan 22, 2015 4:03:37 PM org.openstreetmap.osmosis.core.Osmosis run
 INFO: Pipeline executing, waiting for completion.
 Jan 22, 2015 4:03:37 PM
 org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager
 waitForCompletion
 SEVERE: Thread for task 1-read-apidb failed
 org.springframework.transaction.CannotCreateTransactionException: Could
 not open JDBC Connection for transaction; nested exception is
 org.apache.commons.dbcp.SQLNestedException: Cannot create
 PoolableConnectionFactory (FATAL: no pg_hba.conf entry for host ..., user
 ..., database ..., SSL off)
 at
 org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:240)
 at
 org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:371)
 at
 org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:127)
 at
 org.openstreetmap.osmosis.apidb.common.DatabaseContext2.executeWithinTransaction(DatabaseContext2.java:89)
 at
 org.openstreetmap.osmosis.apidb.v0_6.ApidbReader.run(ApidbReader.java:105)
 at java.lang.Thread.run(Thread.java:695)
 Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot create
 PoolableConnectionFactory (FATAL: no pg_hba.conf entry for host ..., user
 ..., database ..., SSL off)
 at
 org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1549)
 at
 org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1388)
 at
 org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
 at
 org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:202)
 ... 5 more
 Caused by: org.postgresql.util.PSQLException: FATAL: no pg_hba.conf entry
 for host ..., user ..., database ..., SSL off
 at
 org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:293)
 at
 org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:108)
 at
 org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
 at
 org.postgresql.jdbc2.AbstractJdbc2Connection.init(AbstractJdbc2Connection.java:125)
 at
 org.postgresql.jdbc3.AbstractJdbc3Connection.init(AbstractJdbc3Connection.java:30)
 at
 org.postgresql.jdbc3g.AbstractJdbc3gConnection.init(AbstractJdbc3gConnection.java:22)
 at
 

Re: [osmosis-dev] Will write-pgsql-change update lines from LinestringBuilder?

[Brett Henderson]

Yes, it should do.

On 11 October 2014 at 11:57, Christoph Heer christ...@thelabmill.de wrote:

 Hi everyone,

 first of all I would like to say thanks for this great tool. I did my
 import with osmosis in a pgsnapshot schema with '--write-pgsql-dump’ and
 used the enableLinestringBuilder feature. Now I want keep my database
 up-to-date with change files over '--write-pgsql-change’ but will this also
 updates the linestring in the ways table? I didn’t found any comment or
 note to this topic. I hope someone can answer my question. =)

 Best regards
 Christoph
 ___
 osmosis-dev mailing list
 osmosis-dev@openstreetmap.org
 https://lists.openstreetmap.org/listinfo/osmosis-dev

___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Osmosis crash: OsmosisRuntimeException: An output error has occurred

[Brett Henderson]

Hi Simon,

Is that the complete error output?  The error text you've included is the
output from a single PBF reading thread, but there are multiple threads in
your pipeline.  The real reason for failure should be in another stack
trace either before or after that one.

Brett


On 10 June 2014 19:35, Simon Nuttall i...@cyclestreets.net wrote:

 I'm getting this report from running the osmosis command below. The
 timings indicate that it crashes after 75 minutes. If I run the
 osmosis for only the British Isles that takes two hours - that may be
 a clue to where the crashing is occuring. But I do not know what to do
 to debug this problem.

 Simon

 Output:

 09-Jun-2014 16:55:11
 org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager
 waitForCompletion
 FINE: Waiting for task 1-read-pbf to complete.

 09-Jun-2014 18:10:23
 org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager
 waitForCompletion
 SEVERE: Thread for task 1-read-pbf failed
 org.openstreetmap.osmosis.core.OsmosisRuntimeException: An output
 error has occurred, aborting.
 at
 org.openstreetmap.osmosis.core.store.DataPostbox.checkForOutputErrors(DataPostbox.java:160)
 at
 org.openstreetmap.osmosis.core.store.DataPostbox.populateCentralQueue(DataPostbox.java:216)
 at
 org.openstreetmap.osmosis.core.store.DataPostbox.put(DataPostbox.java:303)
 at
 org.openstreetmap.osmosis.set.v0_6.impl.DataPostboxSink.process(DataPostboxSink.java:45)
 at
 org.openstreetmap.osmosis.core.sort.v0_6.SortedEntityPipeValidator.process(SortedEntityPipeValidator.java:68)
 at
 org.openstreetmap.osmosis.tagfilter.v0_6.TagFilter.process(TagFilter.java:141)
 at
 org.openstreetmap.osmosis.tagfilter.v0_6.TagFilter.process(TagFilter.java:141)
 at
 crosby.binary.osmosis.OsmosisBinaryParser.parseDense(OsmosisBinaryParser.java:138)
 at
 org.openstreetmap.osmosis.osmbinary.BinaryParser.parse(BinaryParser.java:124)
 at
 org.openstreetmap.osmosis.osmbinary.BinaryParser.handleBlock(BinaryParser.java:68)
 at
 org.openstreetmap.osmosis.osmbinary.file.FileBlock.process(FileBlock.java:135)
 at
 org.openstreetmap.osmosis.osmbinary.file.BlockInputStream.process(BlockInputStream.java:34)
 at crosby.binary.osmosis.OsmosisReader.run(OsmosisReader.java:45)
 at java.lang.Thread.run(Thread.java:701)



 Command:

 [09-Jun-2014 15:55:10 UTC] reconnectExec About to execute
 /usr/local/bin/osmosis -v 100 --read-pbf
 /websites/www/content/data/osm/downloads/europe/british-isles-latest.os
 m.pbf  --tag-filter accept-ways highway=* leisure=* aerialway=*
 route=ferry man_made=pier,jetty --tag-filter reject-ways
 highway=motorway,construction --read-pbf /website
 s/www/content/data/osm/downloads/europe/denmark-latest.osm.pbf
 --tag-filter accept-ways highway=* leisure=* aerialway=* route=ferry
 man_made=pier,jetty --tag-filter reje
 ct-ways highway=motorway,construction --merge --read-pbf
 /websites/www/content/data/osm/downloads/europe/estonia-latest.osm.pbf
  --tag-filter accept-ways highway=* leisur
 e=* aerialway=* route=ferry man_made=pier,jetty --tag-filter
 reject-ways highway=motorway,construction --merge --read-pbf
 /websites/www/content/data/osm/downloads/europe/
 germany-latest.osm.pbf  --tag-filter accept-ways highway=* leisure=*
 aerialway=* route=ferry man_made=pier,jetty --tag-filter reject-ways
 highway=motorway,construction --
 merge --read-pbf

 /websites/www/content/data/osm/downloads/europe/czech-republic-latest.osm.pbf
  --tag-filter accept-ways highway=* leisure=* aerialway=* route=ferry
 man_m
 ade=pier,jetty --tag-filter reject-ways highway=motorway,construction
 --merge --read-pbf
 /websites/www/content/data/osm/downloads/europe/croatia-latest.osm.pbf
  --tag-fil
 ter accept-ways highway=* leisure=* aerialway=* route=ferry
 man_made=pier,jetty --tag-filter reject-ways
 highway=motorway,construction --merge --read-pbf /websites/www/co
 ntent/data/osm/downloads/europe/lithuania-latest.osm.pbf  --tag-filter
 accept-ways highway=* leisure=* aerialway=* route=ferry
 man_made=pier,jetty --tag-filter reject-way
 s highway=motorway,construction --merge --read-pbf
 /websites/www/content/data/osm/downloads/europe/iceland-latest.osm.pbf
  --tag-filter accept-ways highway=* leisure=* ae
 rialway=* route=ferry man_made=pier,jetty --tag-filter reject-ways
 highway=motorway,construction --merge --read-pbf
 /websites/www/content/data/osm/downloads/europe/norway
 -latest.osm.pbf  --tag-filter accept-ways highway=* leisure=*
 aerialway=* route=ferry man_made=pier,jetty --tag-filter reject-ways
 highway=motorway,construction --merge -
 -read-pbf
 /websites/www/content/data/osm/downloads/europe/sweden-latest.osm.pbf
  --tag-filter accept-ways highway=* leisure=* aerialway=* route=ferry
 man_made=pier,jetty
 --tag-filter reject-ways highway=motorway,construction --merge
 --read-pbf
 /websites/www/content/data/osm/downloads/europe/netherlands-latest.osm.pbf
  

Re: [osmosis-dev] Streaming Replication

[Brett Henderson]

On 16 November 2013 12:21, Walter Nordmann pil...@hotmail.com wrote:

 Hi Brett,

 is Streaming replication running or is it dead?


It's effectively dead.  I don't have time to resurrect it at the moment.
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] writing a custom plugin - unkown plugin version

[Brett Henderson]

This may be too late now, but I would personally recommend avoiding the JPF
plugin framework.  Most of the Osmosis codebase is already comprised of
plugins, and you can write your own plugins using the same mechanism.

In a nutshell you need to perform the following steps:

   - Create a new java project.
   - Implement your Osmosis tasks.
   - Create a class that implements the
   org.openstreetmap.osmosis.core.plugin.PluginLoader interface, and register
   your new tasks in its implementation.
   - Add a resource called osmosis-plugins.conf to your project, and add
   the name of your plugin loader class to it.
   - Compile your project to a jar.
   - Add the jar to the Osmosis classpath.

For examples, look at the existing Osmosis projects such as osmosis-apidb,
osmosis-areafilter, etc.


On 23 December 2013 10:51, jotpe jotpe@gmail.com wrote:

 Hello List,

 i would like to build a custom plugin for osmosis, but the wiki
 informations are to less for me.

 Is there anywhere a tutorial available?

 My current problem is an unknown or incompatible version between my plugin
 dependency and the  org.openstreetmap.osmosis.core.plugin.Core

 ?xml version=1.0 ? !DOCTYPE plugin PUBLIC -//JPF//Java Plug-in
 Manifest 1.0 http://jpf.sourceforge.net/plugin_1_0.dtd; plugin
 id=OsmosisImporter version=0.1 requires import
 plugin-id=org.openstreetmap.osmosis.core.plugin.Core
 plugin-version=0.43.1 reverse-lookup=false/ /requires runtime
 library id=mycode path=/ type=code/ /runtime extension
 plugin-id=org.openstreetmap.osmosis.core.plugin.Core point-id=Task
 id=OsmosismImporter parameter id=name value=importer/ parameter
 id=class value=de.regioosm.osmosis.plugin.PluginLoader / /extension
 /plugin

 This command causes
 osmosis --read-xml city1.osm --importer



 Caused by: org.java.plugin.PluginLifecycleException: plug-in
 OsmosisImporter requires plug-in org.openstreetmap.osmosis.core.plugin.Core
 which is unknown or has incompatible version
  at
 org.java.plugin.standard.StandardPluginManager.checkPrerequisites(StandardPluginManager.java:460)
 at
 org.java.plugin.standard.StandardPluginManager.activatePlugin(StandardPluginManager.java:397)


 My osmosis version is 0.43.1 and the maven dependecies of the plugin
 project is 0.43.1 too. What can i do? Regards Johannes

 ___
 osmosis-dev mailing list
 osmosis-dev@openstreetmap.org
 https://lists.openstreetmap.org/listinfo/osmosis-dev


___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Example code for PBF reading

[Brett Henderson]

On 19 March 2014 06:44, Peter K peat...@yahoo.de wrote:

 Osmosis just uses this library, right?

 I mean, it is a nicely packaged version of LGPLed OSM-binary:
 https://github.com/openstreetmap/osmosis/tree/master/osmosis-osm-binary

 Use it in maven via:
 dependency
 groupIdorg.openstreetmap.osmosis/groupId
 artifactIdosmosis-osm-binary/artifactId
 version0.43.1/version
 /dependency

 Yes, Osmosis uses that.  But there's a little bit more to the story which
might be useful.

The osmosis-osm-binary project contains two pieces of code:

   - Auto-generated data model classes created by the protobuf protoc
   compiler.
   - Scott's common decoding logic that he designed to be shared by other
   projects outside of Osmosis.

Osmosis itself has two separate PBF decoders, one in its osmosis-pbf
project (--read-pbf task) and one in its osmosis-pbf2 project
(--read-pbf-fast task).  The osmosis-pbf project is Scott Crosby's original
implementation, and the osmosis-pbf2 project is my re-write which supports
decoding blocks using multiple threads to improve performance.

The osmosis-pbf project utilises all of the osmosis-osm-binary project and
only adds the Osmosis task code on top.  The osmosis-pbf2 project only
utilises the auto-generated protobuf classes and is an end-to-end decoding
implementation.

Finally, the osmosis-osm-binary project is not exactly Scott's original
code.  The original code is at https://github.com/scrosby/OSM-binary.  The
Osmosis version is mostly identical but modifies the package names to fit
within the osmosis namespace and avoid any conflicts with other builds of
the same code.  I did this so that I could publish to Maven Central without
fear of conflicts.

In summary, you have the choice of two independent implementations of a PBF
decoder, and both are available via Maven Central.  The osmosis-pbf2
implementation is more Osmosis specific, however it should be possible to
invoke the PbfReader class outside of the Osmosis pipeline as it performs
all of its thread management internally.
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Building/Debian Package

[Brett Henderson]

On 29 March 2014 04:33, Jochen Topf joc...@remote.org wrote:

 Hi!

 The Debian package of Osmosis is really old and doesn't work on current
 unstable any more. Since then Osmosis has switched build systems from ant
 to gradle, so we basically have to set up the Debian package build from
 scratch. I think it is important we get Osmosis in a current version back
 into Debian and Ubuntu and want to help get this going.

 Debian doesn't allow getting any binaries from external sources during
 the building process. Build has to be done from scratch and using only
 other Debian packages as dependencies. I have already figured out that
 I can use the Debian gradle package and call the gradle from it instead
 of the gradlew script. But it downloads a bunch of files from Maven:

 Download
 http://repo1.maven.org/maven2/com/puppycrawl/tools/checkstyle/5.6/checkstyle-5.6.pom
 Download
 http://repo1.maven.org/maven2/org/sonatype/oss/oss-parent/5/oss-parent-5.pom
 Download http://repo1.maven.org/maven2/antlr/antlr/2.7.7/antlr-2.7.7.pom
 ...

 I don't know gradle at all.  Is there a list of those dependencies
 somewhere?
 Any suggestions how to disable all this and tell gradle where to search
 for its
 dependecies locally?


Urgh, building Java projects for Linux distributions is horrible :-)

Here are some tips which might be of use.  Note that I haven't tried
everything below ...

To see a full list of dependencies, from the package sub-directory run the
following command:
gradle dependencies

To locate dependencies from somewhere other than Maven Central you'll need
to modify the top level build.gradle file.
Search for the line that says mavenCentral() and comment it out.
Define a Flat directory repository pointing to a directory containing your
dependency jars.  Some details are on the following page (search for the
heading Flat directory repository).
http://www.gradle.org/docs/current/userguide/dependency_management.html

That *should* allow gradle to find its artefacts locally.  The flat
directory repository is very simple and should be able to locate jars if
their name matches the name defined in the gradle dependencies.  Other
attributes of dependencies such as version and group shouldn't matter.
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Merge multiple files

[Brett Henderson]

Hi Mira,

On 28 September 2013 21:16, Jaromír Mikeš mira.mi...@seznam.cz wrote:

 Hi devs,

 I am happily using osmosis for a while.
 I know that I can merge multiple files this way:

 osmosis --rx 1.osm --rx 2.osm --rx 3.osm --merge --merge --wx merged.osm


 But this way is not very useful if you need merge 200+ files.


 Would possible implement something like this?

 osmosis --rx  my/files/* --merge --wx  merged.osm


It would require significant changes to the way Osmosis works.  With some
effort, a new --rx task could perhaps be created to write read multiple XML
files and write them to separate pipes, but a new --merge task would have
no way of knowing how many inputs it should consume.

It's not something I'll personally be tackling.  Can this be solved with a
wrapper shell script?

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Updating database with partial data

[Brett Henderson]

Hi Nuno,

On 10 October 2013 22:18, Nuno Miguel Lourenço 
nuno-miguel-loure...@telecom.pt wrote:

 Hi,

 ** **

 I’m updating a OpenStreetMaps based DB, but we are doing incremental steps
 to the DB data.

 We are importing just a few countries and then proceed to continents and
 at the end the planet file.

 ** **

 We are already using the country data! NOTE: The countries are not
 neighbors!

 We are importing some Europe countries from
 http://download.geofabrik.de/europe.html getting the several *.osm.pbf *files
 for the ones we want!

 ** **

 We are coming across an issue…

 We are getting a lot of 

 *Detail: Key (id)=(x) is duplicated*.; nested exception is
 org.postgresql.util.PSQLException: ERROR: could not create unique index *
 pk_relations*

 We are getting this for the relations table, for the nodes, ways tables
 also!!!


It sounds like your .osm.pbf files contain duplicate data.  The countries
may not be neighbours, but there may still be data relationships that
connect them.

I assume (providing your exact command line and exception messages would be
very helpful) you're trying to use a task like --write-pgsql which assumes
that the database is empty.  In your case you're trying to run it multiple
times against a non-empty database which will fail if the input files
contain duplicate data.


 

 ** **

 Is there any way we can avoid this?


Perhaps.  Combining all .osm.pdf files together into a single file and
remove duplicate data before importing should fix it.


 

 We tried to remove the duplicates on the *osmosisUpdate* function but it
 did not work due to this being called after all the pushes of data to the
 DB.


The osmosisUpdate function is only called by the --write-pgsql-change task,
but I don't believe you're using that task ...


 

 ** **

 Is there any other way to do this with osmosis?


Merge all the .osm.pbf files together before importing.


 

 Merging the several .*osm.pbf* files to generate a unique one is not a
 solution.


Okay.  Can you please explain why this is not an option?

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] bug in read simple task, integer overflow

[Brett Henderson]

2013/7/20 Martin Schafran mar...@ampelmeter.com

 **

 I accidently found a 64 bit BitSet.


 http://lucene.apache.org/core/3_0_3/api/core/org/apache/lucene/util/OpenBitSet.html


A bitset is very inefficient if you have sparsely allocated ids.  For this
reason the DynamicIdTracker breaks the id space into small segments and
switches between bitsets and sorted ids for each segment depending on the
id density.

It used to be necessary to manually specify the id tracker depending on
your usage patterns, whereas now DynamicIdTracker provides the best of both
worlds.

I'm happy to be proven wrong, but I'd want to see memory usage numbers
before trying to use another implementation.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Improving pgsnapshot

[Brett Henderson]

On 20 July 2013 12:06, Paul Norman penor...@mac.com wrote:

 From: Brett Henderson [mailto:br...@bretth.com]
 Subject: Re: [osmosis-dev] Improving pgsnapshot

   Schema details:
  
   I want to add a nodes.ways bigint[] column to the nodes table and use
 it for
   node-ways lookups, then use the existing ways.nodes bigint[] column
 for
   way-nodes
 
  What is the motivation for this?  I'm guessing that performance is
 likely to
  increase because we get to make better use of geo-spatial indexes ... It
 seems
  like a good idea.  I haven't had a look at the existing queries to
 determine
  the impact of making this change.

 I'm explaining this in a good amount of detail because it helps me to write
 it out for planning purposes, so you get one of my long technical emails :)

 I'll use the example of a map? query, but ignore relations. You can
 generalize this to almost any task that has to use way_nodes right now


Do you have the linestring column available in your ways table?  If so, I
don't believe you need to touch the way_nodes table for map queries.



 The way to do this query is:

 1. Add all nodes in the bounding box to a nodes list (nodes_from_bbox)

 2. Find all parent ways of those nodes and add them to a ways list
 (ways_from_nodes)


If you have the linestring column available, you can query the ways table
directly and avoid touching the way_nodes table.  There should be no need
to query for the parent ways of nodes.  I might be missing something in
your explanation though because you mention the linestring column further
down in your email.



 3. Add to the nodes list any child nodes of those ways where the child
 nodes
 are not already in the nodes list (nodes_from_way_nodes)


Are you using the ways.nodes column here, or the way_nodes table?


 4. Serialize the nodes in the nodes list and ways in the ways list

 See the cgimap code for details:

 https://github.com/pnorman/openstreetmap-cgimap/blob/gsoc13/src/api06/map_handler.cpp#L22-36

 https://github.com/pnorman/openstreetmap-cgimap/blob/gsoc13/src/backend/pgsnapshot/snapshot_selection.cpp#L382-477

 A couple of points that might not be immediately obvious

 A. Because you need the full node/way row in 4 it is best to insert the
 entire way/node row into the temp tables in 1-3.

 B. The ways selected by 2 are a subset of those meeting
 ST_Intersects(bbox,way.linestring).


So why not just use the above query instead of using 2?


 Point A is what matters the most.

 A nodes.parent_ways column would speed up 2, and not impact 1, 3 or 4.

 Before going into why, the typical indexes of relevance are:

 On way_nodes: idx_way_nodes_node_id btree (node_id) [52GB]

 On ways: pk_ways PRIMARY KEY, btree (id) [4.5GB]

 Currently, the best query plan for a sanely sized bbox involves

 - a sequential scan on tmp_nodes
 - index scanning using idx_way_nodes_node_id
 - sorting the resulting list of way_ids to remove duplicates
 - index scanning using pk_ways

 By *far* the slowest part is using idx_way_nodes_node_id.

 With my proposed parent_ways column the best query plan should be

 - a sequential scan on tmp_nodes
 - unnest(parent_ways)
 - sorting the resulting list of way_ids to remove duplicates
 - index scanning using pk_ways

 My initial tests show a ~10x increase in speed, but this may scale
 upwards on a large database


I also found ~10x increase in speed when I added the ways.nodes column,
although that was mainly due to being able to use only tables clustered by
geo-spatial indexes.  That allowed me to avoid hitting way_nodes in my
queries.



 The second reason is for one of space reasons.

 way_nodes is 240GB total, with 120GB data and 120GB indexes

 Adding a parent_ways column would add an estimated 66GB to the nodes
 table, bringing it from 194GB to 260GB, for a total DB size reduction
 of 174GB.


That sounds great.



 The sole concern of mine is that by making the nodes table wider it
 increases sort memory requirements and decrease sequential scan speed.
 This could be addressed by instead making a node_ways table with
 columns (node_id bigint PRIMARY KEY, parent_ways bigint[])

 The problem with creating a separate table is that you'll no longer be
able to cluster the table according to a geo-spatial index which kills IO
throughput, at least on spinning disks.

I'm not sure I've followed everything in your email so my comments may not
be accurate.  If case it's useful, the Osmosis bounding box query
implementation is in the following class in the iterateBoundingBox method.
Note that it only creates three temporary tables containing the node, way
and relation ids that are subsequently retrieved via joins back onto the
nodes, ways and relations tables to retrieve the result data.  Also note
that it has three way query implementations that use either way.linestring
column if available, or way.bbox column if available, or fallback to
way_nodes query if neither column is available.
https://github.com/brettch/osmosis/blob/master

Re: [osmosis-dev] bug in read simple task, integer overflow

[Brett Henderson]

Three points in the codebase were still using the BitSetIdTracker.  I've
just replaced these with DynamicIdTracker which is 64-bit capable.
BitSetIdTracker will never be 64-bit capable, at least not in its current
form.  Note that DynamicIdTracker uses BitSetIdTracker as part of its
internal implementation, but ensures that no single instance exceeds a
32-bit range of ids.

I'm not sure why these were missed, but hopefully this will be the last of
the 64-bit issues.


On 5 May 2013 20:46, Martin Schafran mar...@ampelmeter.com wrote:

 the BitSetIdTracker must be enhanced anyway to support 64 bit.

 i guess we have to split the /BitSet positiveSet/ into lower and upper
 sets,
 right? And remove the LongAsInt class and other long to int and vice versa
 conversions.

 martin




 ___
 osmosis-dev mailing list
 osmosis-dev@openstreetmap.org
 http://lists.openstreetmap.org/listinfo/osmosis-dev

___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Improving pgsnapshot

[Brett Henderson]

Hi Paul,

On 9 July 2013 17:34, Paul Norman penor...@mac.com wrote:

 I've been working on expanding cgimap to use pgsnapshot. In the process
 I've
 come across some changes to the schema I'd like to propose. The changes
 basically consist of removing the giant (237GB) way_nodes table.

 Would this be best to do within the existing pgsnapshot tasks and by
 incrementing the schema version? Should it be in new tasks based on those?

 Advice is welcome.


Good question.  I can't see any reason why it couldn't be done by
incrementing the schema version.  The main downside will be that consumers
of the schema will have to upgrade their apps.  That's not a bad thing if
there's an advantage to doing so.

For now I'd suggest doing it as an upgrade.  If it becomes a major issue
for people we can possibly create a snapshot of the existing tasks under a
different name that will let them continue to be used (but not actively
maintained) in much the same way the old pgsimple tasks continue to exist.
But I'd rather not do that without a good reason.


 Schema details:

 I want to add a nodes.ways bigint[] column to the nodes table and use it
 for
 node-ways lookups, then use the existing ways.nodes bigint[] column for
 way-nodes


What is the motivation for this?  I'm guessing that performance is likely
to increase because we get to make better use of geo-spatial indexes ...

It seems like a good idea.  I haven't had a look at the existing queries to
determine the impact of making this change.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] NegativeArraySizeException

[Brett Henderson]

On 18 July 2013 03:42, Ariel Nunez ingenieroar...@gmail.com wrote:

 Hello all,

 I am trying to run osmosis on a planet file to filter out nodes, ways and
 relations tagged with building=*.

 osmosis --read-pbf-fast file=$PLANET_FILE --tf accept-ways building=*
 --write-pbf file=buildings.osm.pbf

 returns with:

 SEVERE: Thread for task 1-read-pbf-fast failed
 java.lang.NegativeArraySizeException [1]

 How can I debug further? The same query ran on an Indonesian country dump
 without issue.

 Thanks in advance,
 Ariel.



 [1] Rest of the exception:
 SEVERE: Thread for task 1-read-pbf-fast failed
 java.lang.NegativeArraySizeException
 at
 org.openstreetmap.osmosis.pbf2.v0_6.impl.PbfStreamSplitter.readHeader(PbfStreamSplitter.java:47)
  at
 org.openstreetmap.osmosis.pbf2.v0_6.impl.PbfStreamSplitter.getNextBlob(PbfStreamSplitter.java:81)
 at
 org.openstreetmap.osmosis.pbf2.v0_6.impl.PbfStreamSplitter.hasNext(PbfStreamSplitter.java:99)
  at
 org.openstreetmap.osmosis.pbf2.v0_6.impl.PbfDecoder.processBlobs(PbfDecoder.java:117)
 at
 org.openstreetmap.osmosis.pbf2.v0_6.impl.PbfDecoder.run(PbfDecoder.java:175)
  at org.openstreetmap.osmosis.pbf2.v0_6.PbfReader.run(PbfReader.java:80)
 at java.lang.Thread.run(Thread.java:680)
 Jul 17, 2013 4:43:50 AM org.openstreetmap.osmosis.core.Osmosis main


I've just checked that line of code and it is trying to instantiate an
array to hold PBF header data based on a header size field but the size is
negative.  In other words, a header size field in the PBF file is set to a
negative value.  At first glance it appears as if the file is corrupt.

Does the same file work using --read-pbf?  Are you able to make the file
available somewhere?

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] osmosis loading, apidb and time zones

[Brett Henderson]

On 18 July 2013 15:11, Paul Norman penor...@mac.com wrote:

 I've been working on some API tests, and ran into a case where the
 timestamps of objects in a file are not reproduced after loading the file
 into the database and fetching it with API calls.

 The file contains this node:

 node id=1001 version=1 changeset=1001 lat=1.001 lon=1.001
 user=user_1001 uid=1001 timestamp=2012-10-01T00:00:00Z/

 I load the file with osmosis into an apidb database and fetch it with the
 rails port and get back a node with timestamp=2012-09-30T17:00:00Z

 If I go into the database and SELECT the timestamp column for the node I
 get


 2012-09-30 17:00:00

 It looks like osmosis is doing an incorrect time zone conversion.

 I'm not sure where to ticket this.

 Note: I am in Vancouver, in the PDT timezone.


Timezones are my second favourite topic after character set encodings ;-)

I don't know if Osmosis is at fault here or not.  All of the apidb
timestamp columns are defined as timestamp without time zone.  Apparently
this means that no timezone conversions are done during input and output
[1] (ie. the database ignores any supplied timezone information, as opposed
to with time zone which always stores in UTC).  I've checked the code and
I can't see anywhere where Osmosis doing anything special with timezones.
Osmosis converts a standard Java java.util.Date into a java.sql.Timestamp
for persistence.  Both of those types have an internal time field
containing the number of milliseconds since 1970.  java.util.Date always
stores this time field in UTC.  It is possible that the JDBC drivers are
passing the timestamp to the database in a UTC timezone and the database is
ignoring it.

In your example, your time is 7 hours off.  Are you in daylight savings
time at the moment?  If that's the case it certainly seems like UTC/PDT
confusion.

My maths might be wrong, but I would have thought that if the JDBC drivers
were passing in UTC times that the times would end up 7 hours ahead, not
behind.  So my above explanation might be wrong.

I'm not sure how to diagnose this one.  Perhaps trace queries against the
database server and see what Osmosis is passing in?

I'm also not sure why the schema doesn't use timestamp with time zone.
It should make things a lot easier and allow all components to do automatic
timezone conversion as necessary.  I seem to recall having similar issues
on the main OSM servers.  I can't remember what the final fix was there,
but usually I aim to have everything running in UTC to avoid these issues.

[1] http://www.postgresql.org/docs/9.2/static/datatype-datetime.html
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Wrapper scripts for streaming replication

[Brett Henderson]

Hi Paul,

On 25 May 2013 15:15, Paul Norman penor...@mac.com wrote:

  From: Brett Henderson [mailto:br...@bretth.com]
  Sent: Friday, April 19, 2013 4:12 AM
  Subject: Re: [osmosis-dev] Wrapper scripts for streaming replication
 
  It'd be great to see the replication streaming getting some usage.  It
 doesn't seem to be used at the moment.  In fact it's been down for a couple
 of weeks and it's trying to catch up again now.

 Do you think it's stable enough to base a service on, or am I better off
 sticking with minutely diffs?


 The minutely diffs are used by enough services including OSMF hosted ones
 that you can be fairly confident that any stoppages will be found and fixed
 quickly, but I'm not sure the same is true for streaming diffs.


That's right, we have a chicken and egg problem here.  Minutely diffs are
more stable because they've been worked on for several years with a large
number of consumers.  Replication streaming is fairly new and likely to
take a while to iron out the issues, but that won't occur unless people use
it.

However I'm largely unavailable at the moment with other commitments, so I
don't have much time to help fix things.



  There are a couple of example Osmosis command lines on the following
 wiki page in case you haven't already seen them.
 
 http://wiki.openstreetmap.org/wiki/Osmosis/Replication#Client-side_Streaming

 The pgsnapshot example would work from the shell, but do you have any
 examples for an unattened server?


I don't have any scripts that I can share.  It should be sufficient to
write a wrapper script that redirects output to log files and launches the
osmosis instance.  For extra points it might be worth having it restart
Osmosis if it fails (with some delays included), however Osmosis should
theoretically handle error conditions such as network failures gracefully.



 Also, what's the best way to switch to streaming replication from diffs,
 and the reverse?

 It's a similar question to switching from hourly to minute diffs where
you'll have to figure out the sequence number to begin from, and doing that
will require figuring out an approximate date that is earlier than your
current replication point.  Note that the streaming server does provide
some assistance where it can provide you with a state file matching a
particular timestamp.  A curl example is provided in the URL you've already
read, but more details on all the URLs are available here:
http://wiki.openstreetmap.org/wiki/Osmosis/Detailed_Usage#--send-replication-data_.28--srd.29

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] --read-xml and --sort osmosis operations of Srtm2osm output file failed with error: ...Cannot represent 70730 as a char

[Brett Henderson]

Hi Fedor,

On 28 April 2013 19:11, Fedor Snegov (MTS phone) fsne...@gmail.com wrote:

 Hi, dear osmosis-dev list members

 I am trying to merge osm map file containing Srtm2osm contours of
 interested area ( boundaries='http://www.**openstreetmap.org/?lat=60.22**
 lon=32.04zoom=7layers=Mhttp://www.openstreetmap.org/?lat=60.22lon=32.04zoom=7layers=M'
 ) with osm.pbf Openstreetmaps map file in order to obtain Garmin map.

 I employ osmosis (great tool) for merging procedure. The merging procedure
 fails every time with errors which are presented below.

 For test reasons I am trying simply to convert Srtm2osm program output
 file to pbf format. The command line invoking osmosis is the following:

 ./osmosis-0.43.1/bin/osmosis \
 --read-xml file=$contours_file_name.$**contours_file_extension \
 --sort \
 --write-pbf file=$contours_file_name.osm.**pbf omitmetadata=true
 compress=deflate

 But it stops and gives me again the same error log like this:

 Apr 27, 2013 11:35:22 PM org.openstreetmap.osmosis.**core.Osmosis run
 INFO: Osmosis Version 0.43.1
 Apr 27, 2013 11:35:22 PM org.openstreetmap.osmosis.**core.Osmosis run
 INFO: Preparing pipeline.
 Apr 27, 2013 11:35:22 PM org.openstreetmap.osmosis.**core.Osmosis run
 INFO: Launching pipeline execution.
 Apr 27, 2013 11:35:22 PM org.openstreetmap.osmosis.**core.Osmosis run
 INFO: Pipeline executing, waiting for completion.
 Apr 27, 2013 11:35:42 PM org.openstreetmap.osmosis.**core.pipeline.common.
 **ActiveTaskManager waitForCom
 SEVERE: Thread for task 1-read-xml failed
 org.openstreetmap.osmosis.**core.OsmosisRuntimeException: Cannot
 represent 70730 as a char.
 at org.openstreetmap.osmosis.**core.util.IntAsChar.intToChar(**
 IntAsChar.java:32)
 at
 org.openstreetmap.osmosis.core.domain.v0_6.Way.store(Way.java:200)der.run(XmlReader.java:111)

 snip


 I am not a programmer, but it seems, that the reason for such an  errors
 is some osmosis limit for maximum number of nodes in a way...
 It is clear that for some not small areas with flat landscape Srtm2osm
 could generate a very long ways with huge amount of nodes.

 Please, could you give any suggestions or advices to overcome this problem?


I've just checked in a change which should fix this.  You are right in that
Osmosis was limiting the maximum number of nodes in a way to 2^15.  This
has been increased to 2^31.  It adds a small overhead to Osmosis
processing, but should be barely noticeable.

This change will be available in the next development build.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [OSM-dev] Problems with mapsforge osmosis plugin

[Brett Henderson]

Hi Mike,

On 20 April 2013 11:39, Michal Migurski m...@teczno.com wrote:

 On Apr 19, 2013, at 6:38 PM, Michal Migurski wrote:

  Osmosis is installed under /usr/local for me. I have the latest version,
 4.3 from earlier this month.
 
  With the plugin under lib/default I now see this error:
 
 osmosis -p org.mapsforge.map.writer.osmosis.MapFileWriterPluginLoader
 Apr 20, 2013 1:34:50 AM org.openstreetmap.osmosis.core.Osmosis run
 INFO: Osmosis Version 0.42-6-gf39a160-dirty

 I should clarify that I have 0.43, not 4.3. The version here says 0.42,
 why is that?


There is a bug in the build script that resulted in a stale version number
being included in the release build.  I haven't had time to produce another
release to fix it.  It is confusing.

Brett
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [osmosis-dev] FW: question

[Brett Henderson]

On 15 April 2013 00:28, mahmoud ghareebh m.soliman_2...@hotmail.com wrote:

 yes an old version , do i need to remove it?


The plugin is failing to load, so yes I'd remove it.

Again, do you really need to compile Osmosis yourself?



 --
 Date: Sun, 14 Apr 2013 20:56:37 +1000
 Subject: Re: [osmosis-dev] FW: question
 From: br...@bretth.com
 To: m.soliman_2...@hotmail.com
 CC: osmosis-dev@openstreetmap.org


 Hi Mahmoud,

 Is there a reason you're not using the pre-compiled version?
 http://bretth.dev.openstreetmap.org/osmosis-build/osmosis-latest.zip

 The errors you're receiving are unusual.  I see the following message in
 your logs.

 org.openstreetmap.osmosis.core.OsmosisRuntimeException: Cannot load 
 JPF-plugin 'org.mapsforge.MapFileWriter' for extensionpoint 'Task'


 Do you have some custom Osmosis plugins installed somewhere?

 Brett

___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Bad PostgreSQL plans with pgsnapshot

[Brett Henderson]

Hi Paul,

Thanks for the info.  I ran into similar issues when implementing the
Osmosis tasks.

Osmosis contains the following code that runs prior to querying for a
bounding box (ie. --dataset-bounding-box).

// PostgreSQL sometimes incorrectly chooses to perform full table scans,
these options
// prevent this. Note that this is not recommended practice according to
documentation
// but fixing this would require modifying the table statistics gathering
// configuration to produce better plans.
jdbcTemplate.update(SET enable_seqscan = false);
jdbcTemplate.update(SET enable_mergejoin = false);
jdbcTemplate.update(SET enable_hashjoin = false);

Adding those statements was a bit of a hack, but at the time I couldn't get
PostgreSQL to behave properly without them.  Perhaps they'd be unnecessary
with the bugfixes in 9.3.  Unfortunately benchmarking it again is such a
time consuming process.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Osmosis 0.43 Released

[Brett Henderson]

Hi Christian,

On 12 April 2013 06:30, Christian H. Bruhn br...@arcor.de wrote:

 am Freitag, 5. April 2013 um 00:51 schrieb Brett Henderson:

 
 http://bretth.dev.openstreetmap.org/osmosis-build/osmosis-0.43-RELEASE.zip

 Osmosis says:
 INFO: Osmosis Version 0.42-6-gf39a160-dirty
 also if you took the link from the Wiki [1]

 Christian

 [1] http://bretth.dev.openstreetmap.org/osmosis-build/osmosis-latest.zip


Thanks for the info.  I've checked and I have a bug in the gradle script
where it isn't updating the OsmosisConstants class when the git version
number changes.  I'm fairly sure that the release is otherwise correct, the
version number just hasn't been updated correctly.

I'll fix it before I make any more releases.  I probably should do another
release (eg. a 0.43.1) to fix this but I'm not sure when I'll get time.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [OSM-dev] Osmosis API

[Brett Henderson]

Hi Sadeer,

On 4 April 2013 18:53, Sadeer Nasser sad...@hotmail.co.uk wrote:

 Hi guys,

 I'm trying to use the Osmosis API (the Jar file not the command-line tool)
 in my Java application but not sure how to get started as there doesn't
 seem to be enough stuff on the web to help people with this. Specifically,
 I'm trying to extract all the highways for a particular area (which will be
 specified using a bbox).


There is minimal documentation (other than javadocs) describing how to
directly code to the Osmosis API.  The code has been designed to be fairly
modular and used outside of the command line application, however there is
no formal API as such so it may not be immediately obvious which code is
specific to the command line app and what can be re-used.

The best way to start is to build a pipeline using the command line
application that does what you need.  I don't want to get into the
specifics of how to best extract highways because I suspect that the answer
isn't simple, but let's say you have the following command line.

*osmosis --read-xml planet.xml --bounding-box left=0 right=1 bottom=0 top=1
--tag-filter accept-ways highway=* --used-node --write-xml myresults.xml*

To do the same thing internally in your app you need to build the same
pipeline in code.  If you wanted to read from a file and write to a file
(ie. don't need to access objects in your app), the simplest thing to do is
simply call the org.openstreetmap.osmosis.core.Osmosis.main method directly
and pass the above arguments.  However in most cases you'll probably want
to hook into the pipeline directly in which case you'll need to dig a bit
deeper.

Every task in the pipeline is represented by a single class in one of the
Osmosis projects.  For example, the --read-xml task is implemented by the
org.openstreetmap.osmosis.xml.v0_6.XmlReader class in the osmosis-xml
project.   The steps for figuring this out are:
* Identify which plugin class registers the command line task.  To do this,
look for all classes implementing the
org.openstreetmap.osmosis.core.plugin.PluginLoader interface, there are 17
of them, one in each osmosis project.  For --read-xml, it is registered in
the class called org.openstreetmap.osmosis.xml.XmlPluginLoader in the
osmosis-xml project.
* In the plugin class, identify the factory class registered for the task
name.  For --read-xml, the plugin loader registers the factory class
org.openstreetmap.osmosis.xml.v0_6.XmlReaderFactory.
* Open the factory class and examine the createTaskManagerImpl method to
see what task class is being instantiated.  In the --read-xml case, this is
org.openstreetmap.osmosis.xml.v0_6.XmlReader.

Using the above steps, you can derive the following class names for each
task:
* --read-xml = org.openstreetmap.osmosis.xml.v0_6.XmlReader
* --bounding-box =
org.openstreetmap.osmosis.areafilter.v0_6.BoundingBoxFilter
* --tag-filter = org.openstreetmap.osmosis.tagfilter.v0_6.TagFilter
* --used-node = org.openstreetmap.osmosis.tagfilter.v0_6.UsedNodeFilter
* --write-xml = org.openstreetmap.osmosis.xml.v0_6.XmlWriter

Each of these classes can be instantiated directly inside your application
code.  To construct a pipeline, you need to call the setSink method on each
task and pass the following task as an argument to it.  For example, call
setSink on the XmlReader instance and pass in the BoundingBoxFilter
instance.  The final XmlWriter won't have a setSink method because it
doesn't implement the org.openstreetmap.osmosis.core.task.v0_6.Source
interface.

Once you have constructed a pipeline, you can call the run method on the
XmlReader instance.  Task types designed to initiate a pipeline implement
the java.lang.Runnable interface.  Osmosis launches these in separate
threads, and you can also do this if you wish.

I won't create a complete code example for the pipeline described above
(for brevity reasons ... and laziness), but if If you were running a
pipeline like *osmosis --read-xml planet.osm --write-xml output.osm*, you
would write code like this:

*XmlReader xmlReader = new XmlReader(new File(planet.osm), true,
CompressionMethod.None);
XmlWriter xmlWriter = new XmlWriter(new File(output.osm),
CompressionMethod.None);
xmlReader.setSink(xmlWriter);
xmlReader.run();*

At this point you may wish to have the output data fed directly into your
own code.  To do this you can create a class that implements the
org.openstreetmap.osmosis.core.task.v0_6.Sink interface, and pass your
object into the setSink method of an osmosis task instance making your
class part of the pipeline.

I hope that makes sense.
Brett
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

[osmosis-dev] OSM Binary (ie. PBF) Changes

[Brett Henderson]

Hi All,

I've just changed how the PBF support is provided in Osmosis.  Up until now
there was a pre-compiled jar called osmpbf.jar checked into the Osmosis
source tree.  It was compiled from Scott Crosby's github project here:
https://github.com/scrosby/OSM-binary

The problem with this is that it prevents me from publishing Osmosis to
Maven Central because people trying to download Osmosis would be unable to
use PBF without also getting a copy of that library.

To get around this I've repackaged the OSM-binary project to build as part
of the Osmosis source tree.  The resultant jar is called
osmosis-osm-binary, and it lives in a package called
org.openstreetmap.osmosis.osmbinary.

I've tried to do this in a way that allows me to keep up to date with the
upstream project.  I've forked the original repository on Github, and
created an Osmosis branch:
https://github.com/brettch/OSM-binary/tree/osmosis

When changes are made to upstream, I should be able to merge master across
to my osmosis branch.  The resultant tree is then checked directly into the
Osmosis source tree.  On my local machine I actually have both git
repositories acting on the same source files so it isn't too painful.  I
have the Osmosis source checked out normally (ie. a .git directory at the
root), and the osmosis-osm-binary project is also a clone of my forked
OSM-binary project (ie. I have a .git directory in there as well).  So far
it seems to work well enough.

If the original OSM-binary project ever gets published to Maven Central
directly then I can stop these shenanigans and depend on it directly.

If anybody has any questions, issues, suggestions, etc let me know.

Brett

PS. I'm getting close to publishing to Maven Central now.  There are no
major blockers left that I'm aware of.
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

[osmosis-dev] Publishing Osmosis to Maven Central

[Brett Henderson]

Hi All,

This may be of interest to some of you.  I've just begun the process of
publishing Osmosis artefacts to Maven Central (ie.
http://repo1.maven.org/maven2).

My current snapshot build is available here:
https://oss.sonatype.org/content/repositories/snapshots/org/openstreetmap/osmosis/

For those who are not familiar with Maven Central practices, it is not
possible to publish directly to Maven Central itself.  The simplest way is
to publish via the OSS Sonatype repository which then gets sync'd with
Maven Central.

A few changes have been made to the Osmosis build to allow this to work
(I'm still in the process of pushing these changes).  The most noticeable
change is that most projects have been renamed to have an osmosis-
prefix.  If you have an existing Eclipse workspace, you'll need to re-run
gradle eclipse and re-import your projects.

The main blocker to publishing release artefacts is that I have two
dependencies on libraries not available in Maven Central.  These are the
scrosby PBF lib, and the BZip2 library which I built manually but which is
based on Apache source code.  Both should be fixable by building and
publishing them along with the rest of Osmosis, but I need to find time to
do so.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] OSM entity processing order

[Brett Henderson]

Hi Nicolas,

On 12 March 2013 20:28, Nicolas Colomer ncolo...@ymail.com wrote:

 Hi Osmosis community!

 When I manipulate an OSM file (compressed or not) using Osmosis, can we
 assume that entities will systematically be processed in this order:
 1.bound, 2.node, 3.way, 4.relation?

 This seems logical since the OSM file format guarantees that blocks come
 in this order (see the OSM XML #File 
 formathttp://wiki.openstreetmap.org/wiki/OSM_XML#OSM_XML_file_format wiki
 page).

 In addition, I reach a 
 posthttp://lists.openstreetmap.org/pipermail/osmosis-dev/2011-March/000949.html
  where
 Brett told:

  This is due to the way Osmosis processing works because it finishes
 processing nodes before it sees the ways.

 I just want to make sure my impression is good :)


You are correct.  Planet files, and just about any other file you may
download will be in that order.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] 32-bit limit in IdTrackers

[Brett Henderson]

On 7 February 2013 20:43, Brett Henderson br...@bretth.com wrote:


 On 7 February 2013 20:21, Ilya Zverev zve...@textual.ru wrote:

 I'd like idTrackerType removed, since after tomorrow using it will
 certainly cause osmosis to fail. I doubt this options was useful, and it
 only puzzled those who were learning osmosis, sometimes being the only
 option besides inPipe and outPipe.


 I've always avoided removing options unless necessary to avoid breaking
 existing scripts, but in this case I agree that it would be better
 removed.  It might be nice if Osmosis had a way of deprecating options so
 that warnings would be displayed if they were used.  I'm a bit time
 constrained at the moment but happy to accept patches/pull requests ;-)


I've removed the option.  It is still available in 0.42 to allow a smooth
transition from 0.41 which requires it to be specified for --used-xxx
tasks, but it will not be available in 0.43.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] 32-bit limit in IdTrackers

[Brett Henderson]

On 13 February 2013 19:10, Jukka Laaksola laaks...@iki.fi wrote:

 13.02.2013 01:11, Brett Henderson wrote:
  On 12 February 2013 23:57, Jukka Laaksola laaks...@iki.fi
  mailto:laaks...@iki.fi wrote:
  There is workaround to use option --used-node idTrackerType=Dynamic,
  but it is not documented. As I can see it is working also with version
  0.41. And my tests also looks like Dynamic will work with the
  version 0.41.
 
 
  The lack of documentation is easy to fix :-)  Would you mind updating
  the 0.41 wiki page?

 Have not had real knowledge of the Dynamic feature in 0.41. Just tried
 it and it looked like it's working. Now I got the answer from you that
 it's really undocumented feature. Perhaps I can create an account to
 wiki and update the page...


Great, it looks like somebody has beaten me to it.



  The other option is to use a nightly build ...

 Is there any place to download nightly builds? Osmosis wiki page
 Development Build links are broken.

 http://wiki.openstreetmap.org/wiki/Osmosis#Development_Build

 Zip Files:

 http://dev.openstreetmap.de:23457/hudson/job/osmosis/lastSuccessfulBuild/artifact/package/distrib/zips/

 Tar Balls:

 http://dev.openstreetmap.de:23457/hudson/job/osmosis/lastSuccessfulBuild/artifact/package/distrib/tgzs/


Oh, I didn't realise they were broken.  I think it has been caused by the
recent change to a gradle build system.  I've fixed the wiki.  The link for
both zip and tgz files is:
http://dev.openstreetmap.de:23457/hudson/job/osmosis/lastSuccessfulBuild/artifact/package/build/distribution/

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [OSM-dev] osmosis 32/64 ID failure

[Brett Henderson]

On 13 February 2013 17:41, NopMap ekkeh...@gmx.de wrote:


 Hi!

 So which of the two statements is true?

 1. 64bit IDs are only supported in 64bit tracker, so 0.42 will change
 nothing and it's just a matter of documentation. The 64bit wiki page is
 wrong.


0.42 will make the Dynamic id tracker type the default across all tasks so
everything should just work, it won't update BitSet and IdList to be 64-bit
capable.



 2. 64bit IDs are supported in all index types in 0.42 as stated on the
 64bit
 wiki page.


I don't think the 64-bit page states that all index types will work in
0.42, although I guess it could be interpreted that way.  I'll probably
remove the idTrackerType option completely in 0.42 to avoid any confusion.
From that point Dynamic will be the only tracker type available.

I hope that makes sense.

Brett
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] osmosis 32/64 ID failure

[Brett Henderson]

On 13 February 2013 06:21, NopMap ekkeh...@gmx.de wrote:


 Hi!


 Brett Henderson wrote
  I believe it was fixed in Osmosis 0.38.  But I'd recommend using 0.41
 (ie.
  the latest) if possible.

 Unfortunately, it is only partially fixed. I tried Osmosis 0.41. It works
 if
 you use the indexType Dynamic, but it fails with a MAXINT exception if you
 try to use BitSet.


Only Dynamic will work correctly from now on.  Unless somebody wants to
step up and create 64-bit capable BitSet and IdList implementations, but I
don't think there's much point.  The other options probably should be
removed now.


 The wiki claims that 0.42 would work in all cases [1].


It appears to have been updated now.



 Is there a reason that 0.42 is not available?


Simply a lack of time, and to be honest I didn't realise there was any
urgency.  I'll do it as soon as I can.


 bye, Nop

 [1] http://wiki.openstreetmap.org/wiki/64-bit_Identifiers

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] osmosis 32/64 ID failure

[Brett Henderson]

On 11 February 2013 07:33, Jochen Topf joc...@remote.org wrote:

 On Sun, Feb 10, 2013 at 09:31:16AM -0700, Kai Krueger wrote:
  On 02/09/2013 04:13 PM, Lynn W. Deffenbaugh (Mr) wrote:
  I'm running Osmosis Version 0.34 to pull and apply minutely
  updates to my database for use by mod_tile/renderd. These updates
  began failing today with the following information:
  
  sudo apt-get install osmosis
  
  Any hints on the command to update osmosis?  I'm a *nix newbie,
  but very good at following cookbooks and instructions.
 
  Which version of Ubuntu / Debian are you running.
 
  Both Ubuntu 12.04 LTS and 12.10 have osmosis version 0.40 packaged.
  Debian testing and debian unstable also both have an osmosis
  versions newer than 0.34.
 
  What version of osmosis is necessary for 64 bit ID handling? If
  necessary, I can add a newer version of osmosis to the launchpad PPA
  repository like to from switch2osm.org

 The 0.40 version in Ubuntu seems to work okay. At least it didn't give me
 any errors.


I believe it was fixed in Osmosis 0.38.  But I'd recommend using 0.41 (ie.
the latest) if possible.

Brett
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [osmosis-dev] 32-bit limit in IdTrackers

[Brett Henderson]

Hi IZ,

On 6 February 2013 20:05, Ilya Zverev zve...@textual.ru wrote:

 Hi! As some of you have read (http://lists.openstreetmap.**
 org/pipermail/dev/2013-**February/026495.htmlhttp://lists.openstreetmap.org/pipermail/dev/2013-February/026495.html),
 in three days node ids are expected to surpass 2147483647, and this
 method https://github.com/**openstreetmap/osmosis/blob/**
 master/core/src/main/java/org/**openstreetmap/osmosis/core/**
 util/LongAsInt.java#L30https://github.com/openstreetmap/osmosis/blob/master/core/src/main/java/org/openstreetmap/osmosis/core/util/LongAsInt.java#L30will
  throw an exception Cannot represent  + value +  as an integer. It
 is used in every IdTracker implementation, so id trackers will become
 unusable.

 This will affect tag and area filters. Regional extracts that are made
 with osmosis will break. There is a comment at the start of each IdTracker
 class: The current implementation only supports 31 bit numbers, but will
 be enhanced if and when required. I guess, now is the time. Can anybody
 fix that? There must be a reason why this hasn't done sooner.


Thanks for the heads up.  I could be wrong but I don't think this is an
issue.

It is used by ListIdTracker and BitSetIdTracker so those implementations
will soon fail if you try to use them.  However, the default implementation
is now DynamicIdTracker which doesn't suffer from this issue (I hope ;-).

DynamicIdTracker breaks the id range into chunks of 1024 and internally
uses either ListIdTracker or BitSetIdTracker for each of those chunks
depending on which is more efficient.  As a result, the largest number
either of those id trackers ever sees is 1023, the DynamicIdTracker adds a
base offset to those numbers to get the final number and it stores the base
number as a 64-bit long.

The idTrackerType arguments could probably be removed from the
--bounding-box and --bounding-polygon tasks now because the default
implementation should be better than specifying one in the vast majority of
cases.

I've checked the rest of the codebase for use of the LongToInt class.  It
is used internally by the Entity class to store a changeset id as a 32-bit
number instead of a 64-bit number.  That should be safe for a while yet.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Eclipse Setup, Missing task types

[Brett Henderson]

Hi Oliver,

On 05/02/2013, at 1:42 AM, Oliver Schrenk oliver.schr...@gmail.com wrote:

 Hi,

 Are there more current notes about how to setup Eclipse for osmosis 
 development then the notes in [1] ?

That's all there is at the moment. It should be mostly correct but
might be missing a step.


 I know that ant has been deprecated in favor of gradle, so I installed 
 Eclipse Grade Support via [2] and

$ git clone https://github.com/openstreetmap/osmosis
$ cd osmosis
$ ./gradlew assemble

I personally don't install any gradle support so I'm not sure what it
does. It should be sufficient to run:
$ ./gradlew eclipse

The above command will generate/update the eclipse projects which can
then be imported into eclipse without requiring any special plugins.


 and proceeded to import osmosis' multi-modules using `File  Import  
 Gradle`. Everything compiles fine.

 But when I try to execute a command like

osmosis --read-xml file=bremen.osm.bz2 --write-apidb-0.6 
 host=127.0.0.1 database=api06_test user=osm password=osm 
 validateSchemaVersion=no

 using a Run Configuration with `org.openstreetmap.osmosis.core.Osmosis` as 
 the main class
 and

That is correct, but you can't run it from the Core project. The Core
project doesn't have access to any plugins. In your case you need to
run it from the Apidb project which contains the apidb tasks and also
has access to the XML tasks. You should be able to modify your
existing launch configuration to start from a different project.

Hopefully that's the step you've missed.


--read-xml file=bremen.osm.bz2 --write-apidb-0.6 host=127.0.0.1 
 database=api06_test user=osm password=osm validateSchemaVersion=no

 as program arguments I get

Feb 04, 2013 3:31:39 PM org.openstreetmap.osmosis.core.Osmosis run
INFO: Osmosis Version 0.41-55-gb44b7d7-dirty
Feb 04, 2013 3:31:39 PM org.openstreetmap.osmosis.core.Osmosis run
INFO: Preparing pipeline.
Feb 04, 2013 3:31:39 PM org.openstreetmap.osmosis.core.Osmosis main
SEVERE: Execution aborted.
org.openstreetmap.osmosis.core.OsmosisRuntimeException: Task type read-xml 
 doesn't exist.
at 
 org.openstreetmap.osmosis.core.pipeline.common.TaskManagerFactoryRegister.getInstance(TaskManagerFactoryRegister.java:60)
at 
 org.openstreetmap.osmosis.core.pipeline.common.Pipeline.buildTasks(Pipeline.java:51)
at 
 org.openstreetmap.osmosis.core.pipeline.common.Pipeline.prepare(Pipeline.java:112)
at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:86)
at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:37)

 It doesn't seem to pickup the various tasks.

 My end goal is to debug write-apidb-0.6 as I'm trying to write data to an 
 unsupported database and run into problems with duplicate user entries and 
 want to use Eclipse's Debugger to go through the code.

I think you're close. Let me know how you go.

Brett


 Best regards
 Oliver


 [1] http://wiki.openstreetmap.org/wiki/Osmosis/Development#Eclipse_Setup
 [2] 
 http://static.springsource.org/sts/docs/latest/reference/html/gradle/installation.html


___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] cleaning up osmosis temp files?

[Brett Henderson]

Hi Brian,

On 18 January 2013 08:35, Brian Cavagnolo bcavagn...@gmail.com wrote:

 Is there a recommended way to clean up the temp files left behind by
 osmosis?  I've been just poking around the /tmp directory blowing away
 the copy* and nodeLocation* files.


Firstly, Osmosis shouldn't leave temporary files around unless it crashes
or is aborted.  If files are being left lying around after normal operation
that would indicate a bug.

However, if you want a simpler way to keep track of temporary files, it may
be worth creating them in a special Osmosis specific location by modifying
the TEMP environment variable.  That would make it easier to identify files
created by Osmosis.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] pgsimple/pgsnapshot possible bug

[Brett Henderson]

Hi Daniel,


On 25 January 2013 07:47, Daniel Kaneider kaneiderdan...@gmail.com wrote:

  Hi,

 I did some import of OSM data into a Postgresql 9.2 DB using osmosis 0.41.
 The pgsnapshot_load script stopped since some function could not be found
 (Envelope,Collect). If I am not wrong then

 UPDATE ways SET bbox = (
 SELECT Envelope(Collect(geom))
 FROM nodes JOIN way_nodes ON way_nodes.node_id = nodes.id
 WHERE way_nodes.way_id = ways.id
 );

 should be changed to

 UPDATE ways SET bbox = (
 SELECT *ST_Envelope*(*ST_Collect*(geom))
 FROM nodes JOIN way_nodes ON way_nodes.node_id = nodes.id
 WHERE way_nodes.way_id = ways.id
 );

 This should apply also to the pg_simple script.


Thanks for the message.  I suspect you're right.  If nobody beats me to it
I'll fix it as soon as I can.

Note however, that most times you don't need those SQL statements and can
comment them out.  If you use Osmosis to generate the COPY data files it
should be able to populate the bbox and/or linestring columns for you.
That should be *much* faster than having PostgreSQL do it for you.

Check out the --write-plsql-dump task and the enableBboxBuilder,
enableLinestringBuilder options.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Osmosis plugin development and integration tests

[Brett Henderson]

Hi Nicolas,

On 12 January 2013 09:33, Nicolas Colomer ncolo...@ymail.com wrote:



 So I tried to reproduce these tests but I face some class loading issue
 (always the same Exception, even for simple commands that involve only xml
 reading / writing):

 java.lang.NoClassDefFoundError: org/java/plugin/PluginLifecycleException
 at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:80)
 at


It looks like you're missing a required library.

This class is located in the jpf (Java Plugin Framework) library which is
included as a dependency of the core project.  If you're including the core
project/lib as a dependency then you should automatically get jpf as a
transitive dependency.

Are you using gradle to build your project?



 Do you have any idea to solve this?
 How declare and load my plugin programmatically so that I can use it via
 command parameters? Do I have to add some resources in my classpath (as for
 Osmosis, when you declare pluginLoader class in the
 $OSMOSIS_HOME/config/osmosis-plugins.conf file)?


Your project will need an osmosis-plugins.conf file containing the name of
your plugin loader class, but otherwise that should be all you need so long
as you have all the necessary jars available on your classpath.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] deadlock when extracting bounds with 0.41 but not 0.40.1

[Brett Henderson]

Hi Eric,


On 13 December 2012 23:18, Eric Fernandez z...@zebulon.org.uk wrote:

 Hi,

 This is a follow-up to the thread here:

 http://lists.openstreetmap.org/pipermail/osmosis-dev/2012-September/001333.html

 I am experiencing the same hanging issue with osmosis 0.41 with the GB
 map from , which I did not have with osmosis 0.40.1.

 The command I use is the same one (as described in

 http://wiki.openstreetmap.org/wiki/Mkgmap/help/options#Using_precompiled_bounds_for_the_address_index
 ):

 ./osmosis-0.41/bin/osmosis --read-pbf file=great_britain.osm.pbf
 outPipe.0=1 --tee 2 inPipe.0=1 outPipe.0=2 outPipe.1=3 --buffer
 inPipe.0=3 outPipe.0=4 --buffer inPipe.0=2 outPipe.0=5 --tag-filter
 accept-relations boundary=administrative,postal_code inPipe.0=4
 outPipe.0=6 --used-way inPipe.0=6 outPipe.0=7 --tag-filter
 reject-relations inPipe.0=5 outPipe.0=8 --tag-filter accept-ways
 boundary=administrative,postal_code inPipe.0=8 outPipe.0=9 --used-node
 inPipe.0=9 outPipe.0=10 --used-node inPipe.0=7 outPipe.0=11 --merge
 inPipe.0=10 inPipe.1=11 outPipe.0=12 --write-pbf
 file=great_britain-boundaries.osm.pbf omitmetadata=true
 compress=deflate inPipe.0=12

 In his answer, Andrew Byrd explains there may be a deadlock problem
 with the inputs for the merge, but this command was working with
 osmosis-0.40.1, and normally is dealing with this issue already. Is
 there any reason that the new 0.41 version leads to the issue?


Andrew is correct, the deadlock is expected behaviour with your pipeline
structure.  Splitting data with a --tee and then recombining with a --merge
should not be done in the same pipeline.

As to why it works in 0.40.1 and not 0.41 I'm not sure, but there were
major changes made to the DataPostbox class which is used as a data buffer
between threads.  It is possible that these changes have triggered your
deadlock condition, although I'm not aware of anything specific.  It *may*
be possible to avoid the deadlock by specifying a large bufferCapacity on
your --buffer tasks, but this isn't recommended as you'll probably still
run into deadlock conditions at some point in the future.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Gradle Build Scripts

[Brett Henderson]

On 2 December 2012 00:40, Paweł Paprota ppa...@fastmail.fm wrote:

 Hi Brett,

 Just to let you know, I've just merged your master to my OWL plugin[1]
 fork, switched the plugin to Gradle and everything works smoothly!

 [1] https://github.com/ppawel/**osmosishttps://github.com/ppawel/osmosis

 Excellent.  I hope it didn't hurt too much :-)

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Invalid ways, now with code

[Brett Henderson]

Hi Toby,

On 12 December 2012 18:33, Toby Murray toby.mur...@gmail.com wrote:



 Anyway, thoughts? My changes are on github in the invalid_geometry branch:
 https://github.com/ToeBee/osmosis

 And a diff of all my changes:
 https://github.com/ToeBee/osmosis/compare/master...invalid_geometry


It looks good to me.  If nobody objects over the next day or two I'll merge
it.  Feel free to give me a nudge if I forget.

Brett

PS.  Minor nitpick.  Can you delete the commented out code in
WayGeometryBuilder?

//int numValidNodes = 0;
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [OSM-dev] Osmosis crash (what am I doing wrong)

[Brett Henderson]

Hi Serge,

On 14 December 2012 15:11, Serge Wroclawski emac...@gmail.com wrote:


 What have I done wrong?


The following line in the stack trace indicates that you've run out of disk
space during processing.
Caused by: java.io.IOException: No space left on device

The --used-node task stores all data to temporary files during processing
which consumes a considerable amount of disk space.  If you have a small
temp file system and you're running on Linux or similar, you should be able
to set the TEMP environment variable to an alternative location.

Brett
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Timestamp in PBF files

[Brett Henderson]

On 12 December 2012 01:53, Frederik Ramm frede...@remote.org wrote:

 Hi,


snip



 It would be great if someone were to add support to Osmosis which is
 likely to be a bit tricky as you have to shove replication information
 through the pipeline, but if all else fails I might have a go at it during
 the holidays.


I've done something similar with the streaming replication tasks  (ie.
--receive-replication, --replicate-apidb, --send-replication-data,
--write-replication).  They exchange state information from source to sink
via the new task initialize method which accepts a map of arguments.
Typically the source task at the start of the pipeline passes a
ReplicationState object through the pipeline in a map key called
replication.state (I think ... I'm not looking at the source code).  The
sink task then updates the state object with the current persisted state
during the initialize call, and by the time the initialize call returns,
the source task can use it to determine what replication point to start
from.

As part of that change I updated tasks such as --buffer to propagate the
initialize information properly across threads.  I believe other tasks such
as --merge will still need to be updated.

I doubt if I'll be able to provide much assistance in implementing this.  I
have another child due early in the New Year so I'll probably be off the
radar for a while :-)
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Timestamp in PBF files

[Brett Henderson]

Hi Markus,

On 24 November 2012 00:04, mar...@gmx.eu wrote:

 Hi Brett,

  *If* this information is intended to be used as an input into replication
  processes then the sequence number is essential.  Osmosis writes a
  timestamp in the state.txt file, but it only for identifying the right
  sequence number to begin replication with.  All replication processing
  requires the sequence number.  Attempting to use a timestamp is
  theoretically possible but it's much less efficient and not how it was
  supposed to work.

 I think this is true for database based updates, however the sequence
 number is not really needed for file based updates we're presently talking
 about:

 For example, osmupdate downloads all change files, starting with the
 newest, going back in time until the the change file has been downloaded
 which is newer than the planet files timestamp. Then all these change files
 are merged to one big change file which is then applied to the planet file.


Yep, that will work for patching planet files.  The replication tasks in
Osmosis can't operate that way though.

The existing --read-replication-interval allows limits to be specified to
restrict the amount of changesets downloaded at a time.  This allows a
local database to catch up in smaller steps if it is a long way behind.
Catching up in smaller steps is preferable in this case because it deals
better with the odd failure in processing (it's very frustrating to
download weeks of changes only to fail near the end and have to start
again), and because it prevents transaction sizes from growing unbounded.
Having to wait several days for one huge catchup transaction to be
processed is far from ideal, it's preferable to catch up in smaller steps.

For patching planet files it's less of an issue because you'll almost
always want all available changes to be applied, and because the number of
files being downloaded will be much less (you'll typically be using daily
or hourly files, not minute files) therefore you'll be less likely to run
into an intermittent network connectivity problem, and patching a file is
extremely unlikely to throw errors unless you run out of disk space or have
a system crash.

One other thing worth mentioning is that timestamps are not guaranteed to
increase for every change file.  In practice for anything down to minute
files you're unlikely to see any issues, but if the database server clock
skews for any reason there's nothing to prevent time running backwards.
This could lead to consumers relying on timestamps to miss data.  Sequence
numbers on the other hand are guaranteed to always increase per change file.

This is all a bit academic for patching planet files, but Osmosis doesn't
make any assumptions about how short the changeset intervals are, or what
is consuming changes at the other end of the pipeline.

I could create a new task optimised for patching planet files, and perhaps
that's what I (or somebody else if they wish to step in) will need to do if
we embed replication information into PBF files, but it will have to remain
separate from --read-replication-interval, so there'll be more code to
maintain.  I'm not opposed to it if it makes users lives easier though.

In summary, I'd prefer to keep using sequence numbers if possible because
it allows me to re-use more existing replication code, but it wouldn't be
impossible to do without them.


 Osmosis may work differently, and it may need the sequence number to start
 this kind of file update - I really don't know. But if so, I totally agree,
 we should make it possible to store sequence numbers in PBF files.

 Could also be done with the key-val format I suggested...


Cool, I don't have any strong opinions on how the information should be
stored.  I'm happy to leave that in the hands of those more familiar with
the PBF format.

Brett
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [osmosis-dev] Single node ways

[Brett Henderson]

Hi Toby,

On 20 November 2012 16:30, Toby Murray toby.mur...@gmail.com wrote:

 I have started playing with this in a branch:
 https://github.com/ToeBee/osmosis/tree/invalid_geometry

 So far I managed to add a new command line option keepInvalidWays to
 the write-pgsql(-dump) tasks. It defaults to false so that current
 functionality of dropping zero and single node ways is preserved.


I'd actually be happier if the default position is to include all data
regardless of whether it is accurate or not.  I feel that is the behaviour
of least surprise, even if it is different to today.  The pgsnapshot schema
is intended to be a complete and accurate representation of OSM data, so
any deviation from that should be explicitly selected by the user.

Would it make sense to separate the dropping of ways from the dropping of
linestrings?  Perhaps two options like keepInvalidWays and
keepInvalidLinestrings?  If keepInvalidWays=true, but
keepInvalidLinestrings=false then the linestring column could be set to
NULL if the way doesn't contain two or more nodes.

The reason I say this is because there may be cases where you want an
accurate copy of OSM data, and only include the linestring as a means of
performing geo-spatial queries against that data.

Thoughts?  Either way, the keepInvalidWays should solve most current
issues, so is a higher priority.


 I think it should be possible to add it to the diff processing as well
 to keep invalid ways out of a replicated database. I likely won't get
 a chance to work on it much more before next week because of
 Thanksgiving travel plans but I may try to reimport my database with
 this option enabled while I am out of town to see exactly what effects
 it has.


Sounds great.  I'm keen to ensure that initial import and replication are
consistent, even if they're not today.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [OSM-dev] Timestamp in PBF files

[Brett Henderson]

On 21 November 2012 19:43, Frederik Ramm frede...@remote.org wrote:

 Hi,

snip


 To be self-contained, it should be sufficient to include the baseURL
 from configuration.txt, no?

 So maybe:


   optional string writingprogram = 16;
   optional string source = 17;
   optional sint64 timestamp = 18;
   optional sint64 replication_timestamp = 19;
   optional string replication_url = 20;

 I don't know if the sequenceNumber from state.txt adds any value, if it
 does then one could throw that in as well.


I've been explicitly cc'd on the original message so I should put in an
appearance ;-)

*If* this information is intended to be used as an input into replication
processes then the sequence number is essential.  Osmosis writes a
timestamp in the state.txt file, but it only for identifying the right
sequence number to begin replication with.  All replication processing
requires the sequence number.  Attempting to use a timestamp is
theoretically possible but it's much less efficient and not how it was
supposed to work.

However, utilising this new sequence number in Osmosis will require some
significant changes.  The current task that figures out what changes to
download (ie. --read-replication-interval) is totally independent of the
task that applies changes to a snapshot (ie. --apply-change).  The simplest
solution would be to write an uber task that is specifically aimed at
patching planet files, but it will be an all-in-one task that can't be
combined with others.  It *may* be possible to modify pipeline
initialisation to allow all tasks to synchronise replication numbers before
beginning processing, but that will be a lot more complicated.

Updating the timestamp and sequence number after processing will also
require some changes because it impacts a number of tasks.  All tasks will
have to propagate the field (shouldn't be too difficult), but tasks such as
--apply-change will need to be smart about which input source they use as
the source of truth for the sequence number.  It's all possible, but not a
trivial change.

Perhaps this is a non-issue if everybody uses osmupdate these days anyway
:-)

As for the PBF format itself, I don't have any opinions.  I'm more than
happy for those who are more familiar with it to come up with a solution.
I'll do my best to accommodate it.

Brett
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [osmosis-dev] Gradle Build Scripts

[Brett Henderson]

Hi All,

I've just pushed my gradle-based build changes to the main Git repository.
https://github.com/openstreetmap/osmosis

Please note that if you're using Eclipse you will need to run ./gradlew
eclipse before you can open the projects.  It may be a good idea to run
./gradlew cleanEclipse first to ensure there are no modified eclipse
files lying around.

If you have any problems please let me know.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] --truncate-apidb also truncates users

[Brett Henderson]

Hi Shaun,

On 16 November 2012 22:17, Shaun McDonald sh...@shaunmcdonald.me.uk wrote:

 Based on the docs at

 http://wiki.openstreetmap.org/wiki/Osmosis/Detailed_Usage_0.42#--truncate-apidb_.28--td.29
 for --truncate-apidb, I wasn't expecting that it would truncate the user
 table and all the other related tables, that are no API docs.

 However based on the code at

 https://github.com/openstreetmap/osmosis/blob/master/apidb/src/main/java/org/openstreetmap/osmosis/apidb/v0_6/ApidbTruncator.java#L23
 it does in fact truncate the user table too. Is this intentional
 behaviour? Could it be change such that only Osmosis generated users are
 removed or users are left behind?


The documentation could probably be more explicit, but it will truncate any
table that Osmosis populates during import.  The idea is that if you have a
populated database, you need to run truncate before you can perform a fresh
import.

The import process is very dumb and assumes an empty database.  Trying to
check for existing data would be more complicated and slower.

The current import process is just using insert statements, but longer term
I'd like to see it move over to using the PostgreSQL COPY command.  COPY is
*much* faster and is already used by the pgsnapshot schema.  If COPY is
used then the database tables must be empty before import anyway.  This is
another reason that I'm hesitant to add any existing user checks to the
current implementation.

Why is this causing you problems?  If you have special needs, performing a
manual truncate is fairly trivial.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Invalid actions during replication

[Brett Henderson]

Hi Paweł,

On 11 November 2012 21:24, Paweł Paprota ppa...@fastmail.fm wrote:

 Hi all,

 On Friday I have implemented a simple validation in the OWL plugin that
 checks whether given entity action (create/modify/delete) can be applied to
 the current database state. See implementation: [1].

 By invalid action I mean a situation when there is a delete action in
 the pipeline but there is no entity in the database to delete. Or modify
 action with version X when there is no version X - 1 in the database but
 some other version.

 I've been running replication for over a day now (in LOG mode, on a
 database with planet imported) and results are quite interesting. There is
 more invalid actions reported than I thought there would be. Some random
 examples:

 INFO: Invalid action: Node 1342627235 - Trying to modify entity that does
 not exist!
 INFO: Invalid action: Node 964649444 - Trying to modify entity that does
 not exist!
 INFO: Invalid action: Way 11133255 - Trying to modify entity that does not
 exist!
 INFO: Invalid action: Way 9105159 - Trying to modify entity that does not
 exist!

 I have not yet analyzed closely those entities that are reported but in
 general there should be no invalid actions at all on a full planet database
 with correct replication setup, am I correct in this assumption?


It's possible that there could be errors during initial replication, but I
would have thought they'd disappear eventually.  It seems a bit odd.

Unfortunately you will need to look into some examples one at a time to try
to trace the source of the issue.  It can be tedious to get to the bottom
of replication problems.  The pgsnapshot replication functionality has been
around for a long time now and has been fairly reliable, but perhaps
there's an issue in there somewhere.



 Also, if anyone's interested, this functionality would be trivial to port
 over to the pgsnapshot Osmosis plugin (--write-pgsql-change) since the OWL
 plugin is built from that plugin. I can do this if you think it could be
 useful.

 [1] https://github.com/ppawel/**osmosis/blob/owldb-plugin/**
 owldb/src/main/java/org/**openstreetmap/osmosis/owldb/**
 v0_6/impl/ChangeWriter.java#**L256https://github.com/ppawel/osmosis/blob/owldb-plugin/owldb/src/main/java/org/openstreetmap/osmosis/owldb/v0_6/impl/ChangeWriter.java#L256

 Implementation notes: basically there is a invalidActionsMode paramater
 for the --write-owldb-change task (which is derived from
 --write-pgsql-change but with OWL-specific additions) that can be IGNORE
 (does nothing), LOG (logs information about the action) or BREAK (stops
 processing when invalid action occurs).


I don't mind something like this being added, but I'd like to understand
the source of your current issues first.  If it turns out that it's normal
behaviour and can't be fixed then there's not much point raising warnings.
On the other hand, if there is a real issue here then it will be valuable.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

[osmosis-dev] Gradle Build Scripts

[Brett Henderson]

Hi All,

I'm keen to switch the Osmosis build over to using the Gradle build tool
instead of Ant.  I've just checked in some new Gradle scripts.  The current
Ant build works reasonably well, but I've lost many hours getting it to its
current point, and introducing new features is always a lot of effort.

The new scripts are on my github repo on the gradle branch.  There are some
basic instructions on its use in the project README.
https://github.com/brettch/osmosis/tree/gradle

Gradle for those who don't know it is a newer build tool written in the
Groovy language.  It provides its own DSL (Domain Specific Language) for
defining build scripts, but anything not directly supported is a short bit
of Groovy script away.  It provides a number of plugins for common tasks
such as java builds and eclipse IDE support.  As such it provides a lot of
the power of Maven, but is much less rigid when it doesn't quite meet your
needs.

Below are some pros and cons.

Advantages of Gradle:
* Simpler multi-project build support.
* Greatly reduced custom build scripting.
* Eclipse project files are auto-generated (and removed from source
control).
* Source is included on all dependencies in Eclipse (great debugging aid).
* Considerably faster build times (on my desktop a build excluding tests
takes 6 seconds versus 47 seconds in ant, full build just over 1 minute
versus just over 2.5 minutes for ant).
* No need to publish and resolve artefacts between projects using Ivy which
leads to faster builds and less consumed disk space.
* Very effective conditional compilation, much faster during development.
* Good set of pre-built plugins.
* No need to install gradle, the wrapper script bootstraps itself to the
correct version.

Disadvantages of Gradle:
* Much more magic involved, and less readable.
* Can't tweak build order to move tests as late as possible in build
process.
* Project names in Eclipse no longer have an Osmosis prefix.

There are probably other disadvantages that I haven't tripped over yet, but
so far it is working well.

If anybody has any thoughts either positive or negative please let me know.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Gradle Build Scripts

[Brett Henderson]

Hi Greg,

On 12 November 2012 03:24, Greg Troxel g...@ir.bbn.com wrote:


 Does gradle require anything else (other than a JDK/JRE) in terms of
 dependencies?


No.  You just need a JDK 1.6 or higher.  It has less dependencies than the
Ant build because it requires Ant to be installed along with a number of
ant modules.  Gradle on the other hand will bootstrap itself using a
wrapper script included in the source tree.  You can however use a locally
installed gradle (invoked via gradle instead of ./gradlew) if you wish.



 If someone doesn't use eclipse, does using gradle become more awkward?


Gradle has no dependency on Eclipse.  It just helps configure your Eclipse
environment.  In the existing Ant build, the Eclipse project files are all
manually maintained and have to be manually kept in sync with the Ivy
dependencies.  This change means that you have to run a gradle eclipse
command before you can import the projects into Eclipse.



 Is Groovy implemented in java?  If not, what barriers does this raise
 (beyond the existing java-isn't-really-portable issues)?


Yes, it's a JVM-based language.  I've been running it on Windows, Mac and
Linux for a while now with no issues.



 Does this add a requirement to have a network connection to do builds?


It requires a network connection, however this was always the case.  The
existing Ant/Ivy build also requires a network connection.  It should be
possible to copy a gradle cache directory between machines in a similar
fashion to the current ivy cache if you wish to operate in disconnected
mode.

Note that Gradle uses Ivy internally anyway, so under the covers the Gradle
and Ant build have many similarities in how they retrieve dependencies.



 Does this whole scheme have enough mindshare that it is really clear
 that it will be staying around, rather than being a flash in the pan?
 (E.g., arch was a really interesting VCS, but it's gone.  It pretty
 clearly inspired others, but AFAIK no one uses it any more.)


I wouldn't call Gradle a flash in the pan.  It's been around for several
years now and is continuing to become more popular.  A number of
well-recognised projects use it now including Spring.  I've used in on a
couple of commercial projects now and have found it to work quite well.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Incorrect string value: '\xF0\x90\x8D\x83\xF0\x90...' for column 'v' at row 6

[Brett Henderson]

On 8 November 2012 20:40, Simon Nuttall i...@cyclestreets.net wrote:

 Regarding my initial request on this:

 http://lists.openstreetmap.org/pipermail/osmosis-dev/2012-November/001409.html

 I have fixed the problem by updating our debian server, which has
 changed the MySQL version from 5.1.49-3-log  to 5.1.63-0+squeeze1-log.


That's a great result.  I wasn't sure where to begin with this error.  It's
been a while since I've had to tackle character set issues, I was hoping
they'd gone forever :-)

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Single node ways

[Brett Henderson]

Hi Toby,

On 5 November 2012 16:25, Toby Murray toby.mur...@gmail.com wrote:

 It seems osmosis intentionally drops ways with only one node in them
 when importing[1]. From another thread[2] on this mailing list, I
 understand this is because postgis does not handle such things very
 well. However, when consuming diffs, single node ways do make it into
 the database. In some situations this seems to cause errors when
 querying the data, but not every time.

 So first of all, there is an inconsistency between how the load
 process and the diff process handles these ways. But is there any
 other way to handle this besides dropping them completely? There are a
 couple of uses of the pgnsapshot schema that assume that it is a
 lossless way of storing OSM data and as far as I know, it is - except
 for this one exception. While single node ways are definitely a bug,
 they happen with unfortunate regularity, mostly due to bugs in editing
 software. Would it be acceptable to add in a phantom node 3cm away
 from the first oneor something, just while building the geometry? I
 think that happens in SQL while applying diffs which would make it
 more difficult since you don't actually want to insert nodes that
 don't exist... Thoughts?

 [1]
 https://github.com/ToeBee/osmosis/blob/master/pgsnapshot/src/main/java/org/openstreetmap/osmosis/pgsnapshot/v0_6/impl/CopyFilesetBuilder.java#L186
 [2]
 http://gis.19327.n5.nabble.com/osmosis-dev-one-node-ways-td5338301.html


Dropping the single node ways is messy.  A phantom node in the generated
linestring might be a better workaround.  If nobody has any better ideas
I'm open to it being changed.  Are you able to create a patch?

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Android?

[Brett Henderson]

Hi Adrià,

On 7 November 2012 03:41, Adrià Ribatallada Torelló
achtungw...@gmail.comwrote:

 Hi there!

 I was trying to use the osmosis tool in an android device, and
 after including all the required libraries to an android project and make
 it compile and install in the device I'v encountered some runtime
 obstacles that made me consider that the best approach might be to adapt
 the osmosis source code, adapting it to use the android libraries.

 (the most severe obstacle is a No validating SAXParser implementation
 available while trying to read the file plugin.xml. Is the same error
 explained here with more detail by this other guy:
 http://stackoverflow.com/questions/1030/trying-to-use-osmosis-in-android-environment-duplicate-file-error-with-jar-file/13251374#13251374
 ).


I'm not sure if this helps, but the plugin.xml loading code is used by JPF
(Java Plugin Framework) support.  JPF provides one way of packaging Osmosis
plugins, but is rarely used.  The majority of plugins use a simpler
mechanism involving a properties file called osmosis-plugin.conf bundled in
each plugin jar file.  If you don't need JPF you could remove the JPF code
from Osmosis completely and eliminate the error you're receiving.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Osmosis OSM wiki content

[Brett Henderson]

Hi Jeff,

On 22 October 2012 08:10, Jeff Meyer j...@gwhat.org wrote:

 Hi - n00b here.
 There's a lot of good content on the wiki about Osmosis, but as a new guy
 (fng), it seems like it could be organized a little more cleanly. I've put
 together some suggestions at:
 https://wiki.openstreetmap.org/wiki/Talk:Osmosis#Wiki_Osmosis_organization
 The intent isn't to change things too much, and certainly not to delete
 anything, but to consolidate similar information and reduce redundancy of
 page purpose a little.
 If there's support and/or suggestions of alternate approaches, I'd be glad
 to take care of it.
 If it's a bad idea, I'll sit tight.
 Also... if this isn't the right way to propose wiki mods, please let me
 know.


There's no right way to propose anything, this seems like as good a way as
any :-)

Regarding the wiki structure, about the only page I really care about is
the detailed usage page because it is the one page that I try to keep 100%
accurate and up to date.  I'm very hesitant to split it up, re-arrange it,
etc.  I take your point about the name detailed usage reading more like a
man page than a usage page, it's just a name that seemed like a good idea
at the time.  I like the content of the page, but I don't particularly care
what it's called.

As for the rest of the pages, I haven't been a great curator of the wiki so
I won't get in the way of somebody trying to improve it.  I'm also not the
best judge of what users need to know.  There is a bunch of stuff on there,
some better organised than others.  The various examples are based on
whatever people have found useful at the time so it's grown fairly
organically.  In particular some of the older stuff like API 0.5 support,
Java 1.5 support, Notes, etc are a bit distracting and could possibly move
elsewhere.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Osmosis error

[Brett Henderson]

Hi Laura,

On 19 October 2012 23:56, Laura Passigni laurapassi...@yahoo.it wrote:

 Hi Brett,
 thank you for your answer.
 Probably the problem is related to the huge size of the database.

 In fact the total database is about 850GB. Currently there is about 1
 billion of records in current_nodes and about the same size in
 current_way_nodes tables.


I've done some testing with the FileBasedSort class with large numbers of
sort items.  It turns out there are some issues with it that *might* have
caused your issue.  FileBasedSort should only use up to about 512 file
handles in a worst case, but in fact was using considerably more than that
on large sorts.  I would be surprised if it used anywhere near 32758
though.  On the other hand there might be several sorts being processed at
the same time which might make the problem worse.

Anyway, I've addressed the issue and committed a fix.  It is included in
the latest development build.  Try it out and let me know how it goes.
http://wiki.openstreetmap.org/wiki/Osmosis#Development_Build

If that fails we may need to monitor the files being opened by Osmosis.
You can list files held open by a process using the lsof -a -p
process_id command.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] java.lang.AbstractMethodError: org.openstreetmap.osm.data.osmbin.v1_0.OsmBinV10Writer.initialize

[Brett Henderson]

Hi Kim,

On 15 October 2012 19:18, KHOO KIM HWA khoofu...@yahoo.com wrote:

 Thanks Brett,

 I downgraded to osmosis 0.39 and the osm to osmbin file worked.

 But there are only attrnames, nodes.id2, nodes.idx, nodes.obm and
 osmbin.properties which are generated. The corresponding files of *.id2,
 *.idx and *.obm for ways and relations are not generated. I found
 osmbin.version=v1.1 in the osmbin.properties.

 Can someone give me some ideas on how to generate the corresponding files
 for ways and relations?


Sorry, I can't help.  The osmbin plugin is developed separately to Osmosis
and I haven't used it.  Perhaps others can comment.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [OSM-dev] Fwd: Streaming Replication

[Brett Henderson]

On 15 October 2012 23:08, Matt Amos zerebub...@gmail.com wrote:

 On Sun, 2012-10-14 at 22:39 +1100, Brett Henderson wrote:
  The timestamp columns in the database are set to timestamp without
  time zone which presumably means the timezones of dates aren't
  automatically converted to the correct timezone upon querying.  I'm a
  bit confused though because I believe PostgreSQL itself is running in
  the BST timezone.  I'd like to investigate further but I don't have
  time at the moment.

 bare timestamps are stored in the database in UTC always [1]. as far as
 i know, there's no way to decorate the column with this information, so
 postgres doesn't have it to do any conversions. it seems likely that
 somewhere along the stack of software doing the query something is
 adding timezone information by defaulting to the current timezone, and
 it shouldn't be doing that.

 cheers,

 matt

 [1]

 https://github.com/openstreetmap/openstreetmap-website/blob/master/app/models/node.rb#L290


Java and Ruby might operate differently in this regard.  There's no such
thing as timezone information on a Java date type, everything is
milliseconds since 1970.  However Java database drivers may choose to
formate dates in a local timezone prior to sending them to a database
server.  If I'm running in BST I suspect the date is presented to
PostgreSQL in BST format, but probably with a timezone suffix.  The
timestamp columns in the API database are all defined as timestamp without
time zone which I believe means that PostgreSQL will ignore any provided
timezone information [1] but it depends on how the driver is implemented.
If the the timestamp columns were defined as timestamp with time zone
then PostgreSQL would look at the timezone suffix and convert dates to UTC
before storing.  When reading dates from the database the reverse process
will occur.

In summary, given that Ruby appears to be storing dates in UTC and the
timestamp columns are defined as timestamp without time zone, I'll have
to ensure all Java processes run in UTC in order for dates to be processed
correctly.

Disclaimer:
Everything I've written above is likely to be wrong :-)

[1] (Search for silently ignore any)
http://www.postgresql.org/docs/9.2/static/datatype-datetime.html
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Osmosis bug with --write-pgsql task

[Brett Henderson]

Hi Mike,

On 16 October 2012 05:57, Michal Migurski m...@teczno.com wrote:

 Hello,

 I've encountered an error with Osmosis and writing to the snapshot schema
 that might be a bug. I am attempting to add a new extract to an existing
 database, and seeing an error where a new index is trying to be created. At
 this phase, I don't believe Osmosis should be attempting to create indexes.


Why don't you believe it should be creating indexes?



 INFO: Processing completion steps.
 Oct 15, 2012 10:07:22 AM
 org.springframework.beans.factory.xml.XmlBeanDefinitionReader
 loadBeanDefinitions
 INFO: Loading XML bean definitions from class path resource
 [org/springframework/jdbc/support/sql-error-codes.xml]
 Oct 15, 2012 10:07:22 AM
 org.springframework.jdbc.support.SQLErrorCodesFactory init
 INFO: SQLErrorCodes loaded: [DB2, Derby, H2, HSQL, Informix,
 MS-SQL, MySQL, Oracle, PostgreSQL, Sybase]
 Oct 15, 2012 10:07:22 AM
 org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager
 waitForCompletion
 SEVERE: Thread for task 1-rb failed
 org.springframework.dao.DuplicateKeyException: StatementCallback;
 SQL [ALTER TABLE ONLY users ADD CONSTRAINT pk_users PRIMARY KEY (id)];
 ERROR: could not create unique index pk_users
   Detail: Table contains duplicated values.; nested exception is
 org.postgresql.util.PSQLException: ERROR: could not create unique index
 pk_users
   Detail: Table contains duplicated values.
 at
 org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.doTranslate(SQLErrorCodeSQLExceptionTranslator.java:241)
 at
 org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:72)
 at
 org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:406)
 at
 org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:518)
 at
 org.springframework.jdbc.core.simple.SimpleJdbcTemplate.update(SimpleJdbcTemplate.java:248)
 at
 org.openstreetmap.osmosis.pgsnapshot.v0_6.impl.IndexManager.completeAfterLoad(IndexManager.java:130)
 at
 org.openstreetmap.osmosis.pgsnapshot.v0_6.impl.CopyFilesetLoader.run(CopyFilesetLoader.java:111)
 at
 org.openstreetmap.osmosis.pgsnapshot.v0_6.PostgreSqlCopyWriter.complete(PostgreSqlCopyWriter.java:108)
 at
 org.openstreetmap.osmosis.tagfilter.v0_6.UsedNodeFilter.complete(UsedNodeFilter.java:158)
 at
 org.openstreetmap.osmosis.tagfilter.v0_6.TagFilter.complete(TagFilter.java:142)
 at
 org.openstreetmap.osmosis.tagfilter.v0_6.TagFilter.complete(TagFilter.java:142)
 at
 org.openstreetmap.osmosis.core.progress.v0_6.EntityProgressLogger.complete(EntityProgressLogger.java:61)
 at
 crosby.binary.osmosis.OsmosisBinaryParser.complete(OsmosisBinaryParser.java:35)
 at
 crosby.binary.file.BlockInputStream.process(BlockInputStream.java:37)
 at
 crosby.binary.osmosis.OsmosisReader.run(OsmosisReader.java:37)
 at java.lang.Thread.run(Thread.java:679)

 From the stack trace I can tell that tasks are in the complete phase.  At
that point the IndexManager class will be rebuilding indexes.  Indexes get
dropped prior to import to improve performance.

Are you sure that the database was empty prior to beginning the import?

Brett
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Fwd: Streaming Replication

[Brett Henderson]

On 14 October 2012 13:59, Brett Henderson br...@bretth.com wrote:

 On 14 October 2012 09:46, sly (sylvain letuffe) li...@letuffe.orgwrote:note 
 : I have a strange issue with all objects's timestamp, they seam
 to be

 shifted by -1 hour


 Really?  I might have to look into this this evening (currently 2pm
 Melbourne time).  I previously had a few issues with timezones (BST to UTC
 confusion), but I thought they were limited to the information in state
 files, not the data itself.


I've taken a look at it.  The process was running in the default server
timezone which is BST.  I've restarted the process explicitly setting the
timezone to UTC and it has fixed the problem.

The timestamp columns in the database are set to timestamp without time
zone which presumably means the timezones of dates aren't automatically
converted to the correct timezone upon querying.  I'm a bit confused though
because I believe PostgreSQL itself is running in the BST timezone.  I'd
like to investigate further but I don't have time at the moment.

Anyway, the dates look right to me now.  A large number of replication
files will now have incorrect timestamps in them.  At some point I probably
should delete all existing replication data and start again from scratch.

Brett
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [osmosis-dev] java.lang.AbstractMethodError: org.openstreetmap.osm.data.osmbin.v1_0.OsmBinV10Writer.initialize

[Brett Henderson]

Hi Kim,

On 13 October 2012 00:36, KHOO KIM HWA khoofu...@yahoo.com wrote:

 Hello,

 I got some errors when I try to convert .osm to osmbin file by executing:
 osmosis --read-xml file=idf_main_roads_new.osm --write-osmbin-0.6
 dir=./osmbin-full-map-dir

 Thread for task 1-read-xml failed
 java.lang.AbstractMethodError:
 org.openstreetmap.osm.data.osmbin.v1_0.OsmBinV10W
 riter.initialize(Ljava/util/Map;)V
 at
 org.openstreetmap.osmosis.xml.v0_6.XmlReader.run(XmlReader.java:95)
 at java.lang.Thread.run(Unknown Source)

 Can someone give me some ideas to fix the problems?


This error appears to be coming from an Osmosis plugin, not Osmosis
itself.  The plugin will need to be updated to support the latest version
of Osmosis.  If you can't get a newer version of the plugin you'll need to
downgrade to version 0.39 of Osmosis.  Recent versions of Osmosis require
all tasks to support a new initialize method.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Streaming Replication

[Brett Henderson]

Hi Jochen,

On 14 October 2012 06:17, Jochen Topf joc...@remote.org wrote:

 Very interesting work!

 How do you handle new incoming requests. They have to start from a known
 point
 so I guess you have to do an SQL query for each of them? Or do you just
 read
 the existing .osc files from disk and stream them out? This could take a
 long
 time...


It just reads existing .osc files.  The server-side is made up of two
processes.  The first process extracts data from the database and writes
.state.txt and .osc files in a similar way to existing replication but it
runs continuously using a single database connection.  The second process
serves the data to clients.  The two processes talk via an internal
HTTP-based channel so that the extracter can notify the data server when
new intervals have been processed.  The two processes can be run in a
single Osmosis process, but I usually run them separately.

It could take a long time to download a long time interval, but it is
perhaps faster than you'd expect.  I suspect that in most cases the
bottleneck will be client side trying to consume the data.  I've thought
about writing the data into a database instead, but it's more effort to
both develop and manage.  I'm planning to wait to see if it becomes an
issue.  Downloading bulk data should be relatively rare because most
connected clients should be up to date and just waiting for new data to
arrive.

One other thing to note is that it supports a tree of servers where one
master server feeds data to any number of slaves which in turn feed data to
end users.  Example commands are here:
http://wiki.openstreetmap.org/wiki/Osmosis/Replication#Streaming_Caching

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [OSM-dev] Fwd: Streaming Replication

[Brett Henderson]

On 14 October 2012 09:46, sly (sylvain letuffe) li...@letuffe.org wrote:

  Hi All,

 Hi Brett,

  To solve this, a new streaming replication mechanism has been developed.
  Under the covers the same database queries are utilised, but the process
  performing the queries runs continously and polls the database for
 changes
  at a shorter interval.

 Nice feature, it might not be of interest to everyone but people operating
 a
 ro mirror for editing purpose might have interest.
 And that's my case, so thanks for that !

  However, I'd love to see it get
  some usage and would welcome any feedback.

 Here I am allready !
 I've plugged that to my test server's overpass db updates and things seams
 to
 work as expected. Let's see how it performs in case of network hickups


Excellent, I look forward to feedback.



 note : I have a strange issue with all objects's timestamp, they seam to be
 shifted by -1 hour


Really?  I might have to look into this this evening (currently 2pm
Melbourne time).  I previously had a few issues with timezones (BST to UTC
confusion), but I thought they were limited to the information in state
files, not the data itself.

Brett
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Streaming Replication

[Brett Henderson]

One extra bit of info that might be of interest.  There are a number of
different URLs supported by the server.  They are described here:
http://wiki.openstreetmap.org/wiki/Osmosis/Detailed_Usage#--send-replication-data_.28--srd.29

To summarise:

   - You can download just state information, or state plus data.
   - You can download from now, from a particular replication sequence
   number, or from a particular point in time (ie. timestamp).
   - You can download until available replication intervals are exhausted
   and then disconnect, or you can tail the data and be sent data
   immediately as it becomes available

The Osmosis client-side task --receive-replication downloads state plus
data, always processes from a particular sequence number, and works in
tail mode.

Brett
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

[osmosis-dev] Streaming Replication

[Brett Henderson]

Hi All,

For those of you who currently use the minute diffs to keep a local
database up to date, you may be interested to know that a new form of
replication has hit the street.

The current replication system is based on a series of static replication
files that are placed on a web server for clients to download as described
here:
http://wiki.openstreetmap.org/wiki/Planet.osm/diffs#Using_the_replication_diffs

It is a very simple mechanism and works well for the existing daily, hourly
and minutely replication feeds.  Unfortunately it doesn't work well for
sub-minute replication because it becomes far too chatty.  On the server
side, the current feeds are generated from cron which also works well down
to one minute intervals, but the overhead of launching a new process and
connecting to the database for every replication interval also becomes too
inefficient for shorter intervals.

To solve this, a new streaming replication mechanism has been developed.
Under the covers the same database queries are utilised, but the process
performing the queries runs continously and polls the database for changes
at a shorter interval.  It is currently set to poll every 10 seconds, but
it can be reduced further if required.  The network transport is also
continuous and holds a single HTTP connection open for the lifetime of
communication between the server and client.  It is all implemented within
the latest version of Osmosis 0.41.  If you wish to experiment with the
server-side tasks however, several bugs have been fixed in the latest
development version.  Internally it uses the JBoss Netty framework which
means that it's all event-driven (ie. doesn't require a thread per client)
and should theoretically support a large number of concurrent clients.

To quickly see this in action, point your browser at this URL and you
should see new replication state data become available approximately
every 10 seconds.
http://planet.openstreetmap.org/replication/streaming/replicationState/current/tail

New Osmosis tasks have been developed to consume this data.  For some basic
instructions to help you get started, refer to this link:
http://wiki.openstreetmap.org/wiki/Osmosis/Replication#Client-side_Streaming

If you don't wish to use Osmosis, some basic documentation on the wire
protocol is available here:
http://wiki.openstreetmap.org/wiki/Osmosis/Replication#Streaming_Replication_Wire_Protocol

This is very much experimental and bugs will undoubtedly be encountered to
please be wary about trusting it to update your database if you've just
spent two weeks importing a planet file.  However, I'd love to see it get
some usage and would welcome any feedback.  This is not intended for use in
updating a local planet file as the existing daily files are better suited
to that.  For databases that can tolerate a minute delay, the existing
mechanism is very simple and has proven to be fairly reliable.  But if you
really need current access to data, and can cope with the additional
complexity, this should be useful.  The current 10 second delay is not a
lower limit, but is a good starting point for now.

Cheers,
Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

[OSM-dev] Fwd: Streaming Replication

[Brett Henderson]

Oops, I sent my previous email to the wrong list ...

-- Forwarded message --
From: Brett Henderson br...@bretth.com
Date: 13 October 2012 15:43
Subject: Streaming Replication
To: osmosis-dev osmosis-...@openstreetmap.org


Hi All,

For those of you who currently use the minute diffs to keep a local
database up to date, you may be interested to know that a new form of
replication has hit the street.

The current replication system is based on a series of static replication
files that are placed on a web server for clients to download as described
here:
http://wiki.openstreetmap.org/wiki/Planet.osm/diffs#Using_the_replication_diffs

It is a very simple mechanism and works well for the existing daily, hourly
and minutely replication feeds.  Unfortunately it doesn't work well for
sub-minute replication because it becomes far too chatty.  On the server
side, the current feeds are generated from cron which also works well down
to one minute intervals, but the overhead of launching a new process and
connecting to the database for every replication interval also becomes too
inefficient for shorter intervals.

To solve this, a new streaming replication mechanism has been developed.
Under the covers the same database queries are utilised, but the process
performing the queries runs continously and polls the database for changes
at a shorter interval.  It is currently set to poll every 10 seconds, but
it can be reduced further if required.  The network transport is also
continuous and holds a single HTTP connection open for the lifetime of
communication between the server and client.  It is all implemented within
the latest version of Osmosis 0.41.  If you wish to experiment with the
server-side tasks however, several bugs have been fixed in the latest
development version.  Internally it uses the JBoss Netty framework which
means that it's all event-driven (ie. doesn't require a thread per client)
and should theoretically support a large number of concurrent clients.

To quickly see this in action, point your browser at this URL and you
should see new replication state data become available approximately
every 10 seconds.
http://planet.openstreetmap.org/replication/streaming/replicationState/current/tail

New Osmosis tasks have been developed to consume this data.  For some basic
instructions to help you get started, refer to this link:
http://wiki.openstreetmap.org/wiki/Osmosis/Replication#Client-side_Streaming

If you don't wish to use Osmosis, some basic documentation on the wire
protocol is available here:
http://wiki.openstreetmap.org/wiki/Osmosis/Replication#Streaming_Replication_Wire_Protocol

This is very much experimental and bugs will undoubtedly be encountered to
please be wary about trusting it to update your database if you've just
spent two weeks importing a planet file.  However, I'd love to see it get
some usage and would welcome any feedback.  This is not intended for use in
updating a local planet file as the existing daily files are better suited
to that.  For databases that can tolerate a minute delay, the existing
mechanism is very simple and has proven to be fairly reliable.  But if you
really need current access to data, and can cope with the additional
complexity, this should be useful.  The current 10 second delay is not a
lower limit, but is a good starting point for now.

Cheers,
Brett
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [osmosis-dev] Problems with the tag transform plugin

[Brett Henderson]

I've taken a look at tagtransform and added it to the main Osmosis codebase
here to get it working again:
https://github.com/brettch/osmosis/tree/tagtransform

Note that I have violated the original GPL licence by copying it into the
Public Domain Osmosis codebase so this may only stay here temporarily.
I've asked Dave separately to see if he'll allow it to be moved into Public
Domain.  If he's not comfortable with that it will need to remain separate.

I've made a large number of changes to the codebase, however the actual
compilation problem preventing compilation against 0.41 is fixed in this
commit.  This commit could probably be applied to the existing SVN-based
tagtransform codebase.
https://github.com/brettch/osmosis/commit/25711901df41e6950613dea901e3a15ad7ce93a1
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev