[osmosis-dev] Osmosis 0.49.0 Released

[Brett Henderson]

Hi folks,

I've just released Osmosis 0.49.0. Osmosis is in a light maintenance mode
so the main reason for the release is to update dependencies. It doesn't
contain many new features, in fact a few barely used features have been
removed where they were too hard to keep working.

The main changes are:

   - Use GitHub actions instead of Travis CI for continuous integration.
   - Improve Gradle performance by enabling parallel execution, build cache
   and on-demand configuration.
   - Replace osmosis-osm-binary package with dependency on
   org.openstreetmap.pbf:osmpbf from Maven Central.
   - Add enableKeepPartialLinestring option for pgsnapshot/pgsimple tasks.
   - Revert the database locking functionality which fails in CI.
   - Upgrade to PostgreSQL 15.1.
   - Remove Java Plugin Framework support (standard Osmosis plugin support
   remains).
   - Remove the replication-http package including --receive-replication
   and send-replication-xxx tasks.
   - Upgrade to Java 17.
   - Eliminate use of finalize.
   - Ensure PBF file is closed after reading completes.
   - Remove shadow jar and jcenter plugins from build.
   - Build distribution using the Gradle Application plugin instead of
   custom scripts.

Release builds can be obtained from GitHub:
https://github.com/openstreetmap/osmosis/releases/tag/0.49.0

All artefacts have also been uploaded to Maven Central. Note that the
version on Maven Central is 0.49.1 because I had to fix a few things that
had broken with Gradle updates.
https://central.sonatype.com/namespace/org.openstreetmap.osmosis

Cheers,
Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Osmosis 0.48.0 released

[Brett Henderson]

Thanks for taking this on, Michal!  It's exciting to see some activity in
Osmosis again.

FYI, I just fixed up the detailed usage redirect link to point to 0.48.
https://wiki.openstreetmap.org/w/index.php?title=Osmosis/Detailed_Usage=no

On Wed, 6 May 2020 at 02:47, Michal Migurski  wrote:

> Hi everyone!
>
> I’m releasing Osmosis 0.48.0 this morning:
>
> https://github.com/openstreetmap/osmosis/releases
>
> Some of the new features include:
>
> • Improve replication state handling with single-file steps and
> custom state (PR #63)
> • Add HTTP to osmosis-pbf2 module (PR #62)
> • Adding ability to use Osmosis as a library with limited
> functionality (PR #61)
> • Tag-transform can now read a row of data from an external
> DataSource (PR #52)
>
> I also neglected to announce 0.47.4, sorry about that:
>
> • Refactored with BaseXMLReader and WorkingTaskManagerFactory
> classes (PR #60)
>
> Binaries are out on Github now, and I’m in the process of updating the OSS
> Sonatype listings.
>
> -mike.
>
> 
> michal migurski- contact info and pgp key:
> sf/cahttp://mike.teczno.com/contact.html
>
>
> ___
> osmosis-dev mailing list
> osmosis-dev@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/osmosis-dev
>
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Osmosis 0.47 Release

[Brett Henderson]

On Fri, 26 Oct 2018 at 12:22 Ariel Nunez  wrote:

> I promoted the use of osmosis when I was working at the World Bank. Our
> clients would use it to bootstrap country specific data sharing tools for
> development programs and disaster prevention.
>

Awesome!  It's great when you discover your software being used in new and
interesting ways.

>
> So long and thanks for all the diffs.
>

Haha, nice one.
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

[osmosis-dev] Osmosis 0.47 Release

[Brett Henderson]

Hi All,

I've just released Osmosis 0.47.
http://bretth.dev.openstreetmap.org/osmosis-build/osmosis-0.47.tgz
http://bretth.dev.openstreetmap.org/osmosis-build/osmosis-0.47.zip

>From changes.txt:


   - Build using Java 11 now that Java has moved to a 6 month release cycle
   with no (free) long term support.
   - Use https for all replication URLs.
   - Use single-threaded PBF reading by default.
   - Add support for PBF way location attributes.
   - Remove customdb dataset implementation.  It performed poorly and was
   superseded long ago by pgsimple/pgsnapshot.
   - Fix munin replication lag shell script.
   - Modify --write-apidb to commit every 1 to allow large databases to
   be populated.
   - Fix --write-apidb-change to correctly write way history.


Let me know if you see any issues.

It's a good time to mention that this will probably be the last Osmosis
release I make.  I first starting hacking on Osmosis in April 2007, and
after 11 years it feels like time to finally let it go.  I haven't spent
much time in the OpenStreetMap world for a number of years, but Osmosis has
been largely superseded by better/faster tools such as Osmium.  Osmosis
itself is really showing its age.  In its day it helped move OpenStreetMap
forward, in particular making it possible to replicate data around in a
timely and efficient manner, but that's a well solved problem now.  The
version of Osmosis building the daily/hourly/minute changesets is _many_
years old and yet remains compatible with the latest version, not much has
changed.

I've kept the lights on for a number of years by accepting PRs, doing basic
maintenance such as library and java updates, and making it easier to
build/test through changes such as Dockerising the build.  But even that
can be time consuming, and I struggle to find the time to do it properly.

The code is all in the public domain, so if anybody wishes to continue to
hack away on it they are very welcome to do so.  If anybody wants to take
on release management duties, let me know, and I'll help them get access to
some of the key infrastructure such as the dev server for hosting
distribution binaries, Maven Central artefact upload  process, and GitHub
git repository.

Cheers,
Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Osmosis and Osmium-enhanced PBF files with way node locations

[Brett Henderson]

Hi Jon,

On Fri, 16 Mar 2018 at 06:50 Locke, Jonathan  wrote:

> Is there a way to get this optional feature string in Osmosis? I expected
> it to be passed as metadata to the initialize method of Sink, but it wasn't.
>

The initialize method was added relatively recently in Osmosis history so
the PBF tasks are unlikely to use it.  I originally added it to pass
replication metadata through the pipeline.

It shouldn't be too hard to implement, but I'll need to take a look.
Again, it may be easier to add this to the --read-pbf-fast implementation.

Cheers,
Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Osmosis and Osmium-enhanced PBF files with way node locations

[Brett Henderson]

Thanks Jon, I'll take a look as soon as I can.

As for testing, have you tried running it in the Docker environment? This
command should do it.

./docker.sh ./gradlew build

That will automatically spin up a database server in a Docker container and
run the build/tests in another container that has access to the database
server. The only prerequisite is Docker itself.

On Wed., 14 Mar. 2018, 7:35 am Locke, Jonathan, <jonath...@telenav.com>
wrote:

> Hi Brett,
>
>
> I addressed the issues you brought up and there's a new patch to the base
> code attached to this email. The only change I made to the Osmosis .proto
> file was to add these two lines precisely as they were added to Osmium:
>
>
> +   repeated sint64 lat = 9 [packed = true];
>
> +   repeated sint64 lon = 10 [packed = true];
>
>
> I believe the other differences in the generated Java code probably have
> to do with our using different versions of protoc during the build process.
> You may want to regenerate the code using the protoc version you used to
> generate the Java code before and then these files should be mostly the
> same.
>
>
> You probably will also want to run the tests as I cannot run the tests as
> Gradle quits when it runs into failures in the database code (I'm most
> likely not set up for Osmosis' database tests). Finally, I don't really
> follow what you're saying about syncing code with some other codebase. I
> don't have commit acess to anything so perhaps someone else can sync the
> proto file appropriately. I believe all that needs to happen to make this
> work is to add the two lines above (as in the patch file).
>
>
> Best,
>
>
>  Jon
>
>
> --
> *From:* Brett Henderson <br...@bretth.com>
> *Sent:* Sunday, March 11, 2018 5:43:08 AM
> *To:* Locke, Jonathan
> *Cc:* osmosis-dev@openstreetmap.org; Van Exel, Martijn
>
> *Subject:* Re: [osmosis-dev] Osmosis and Osmium-enhanced PBF files with
> way node locations
> Hmm, my previous email was probably hard to read.  Let me know if it
> doesn't make sense.
>
> One other thing I should mention.  GIven that you are just adding read
> support, have you looked at the osmosis-pbf2 sub-project?  It contains an
> alternative PBF reading implementation that I wrote to support
> multi-threaded reading.  The task is registered as *--read-pbf-fast*.
> The class *org.openstreetmap.osmosis.pbf2.v0_6.impl.PbfBlobDecoder* in
> method *processWays* contains the WayNode parsing logic.
>
>
> On Sun, 11 Mar 2018 at 21:19 Brett Henderson <br...@bretth.com> wrote:
>
> Hi Jon,
>
> Thanks for sending through the patch.  I've just taken a look at it.  Some
> comments:
>
> *WayNode*
> The WayNode class now has the new optional latitude and longitude fields
> which makes sense.  Can you update the store method and (StoreReader,
> StoreClassRegister) constructor to persist and load those parameters
> again?  They're needed in case the pipeline does any functionality that
> requires creating temp files such as sorting.
>
> The class is now mutable which may cause problems in a multi-threaded
> pipeline if task implementations are tempted to modify state on the fly.
> Some of the Osmosis entity classes are mutable (an historical decision
> which I regrettably allowed through), but they're protected from concurrent
> modification through a somewhat elaborate read/write protection mechanism
> (see CommonEntityData for details ...).  In this case, can we keep the
> class immutable by adding those additional fields to an overloaded
> constructor?
>
> *osmformat.proto*
> Are these changes mastered somewhere else?  In other words, are these new
> fields the same ones used by Osmium in its implementation?  I'm wondering
> if we need to re-sync from the main OSM-binary project.  The osmosis
> version is the same as that in https://github.com/scrosby/OSM-binary.
> The repo https://github.com/brettch/OSM-binary is a fork of that, and the
> osmosis branch there *is* the same code as the osmosis-osm-binary directory
> in the osmosis repo, just with some re-arranging to fit inside the Osmosis
> structure and fit inside the Osmosis java package namespace.
>
> *Jochen* (if you're reading), where does Osmium source its
> osmformat.proto file from?
>
> Long story short, rather than make changes directly to the file in Osmosis
> and create a fork, should we apply them to upstream first and then re-sync
> Osmosis with that?
>
> Otherwise, the changes look relatively straightforward.  I don't have many
> strong opinions on how to test it.  Osmosis doesn't have an amazing test
> suite, it started out as a hacked together tool and grew into something
> bigger than I planned.  I mostly rely 

Re: [osmosis-dev] Osmosis and Osmium-enhanced PBF files with way node locations

[Brett Henderson]

Hmm, my previous email was probably hard to read.  Let me know if it
doesn't make sense.

One other thing I should mention.  GIven that you are just adding read
support, have you looked at the osmosis-pbf2 sub-project?  It contains an
alternative PBF reading implementation that I wrote to support
multi-threaded reading.  The task is registered as *--read-pbf-fast*.  The
class *org.openstreetmap.osmosis.pbf2.v0_6.impl.PbfBlobDecoder* in method
*processWays* contains the WayNode parsing logic.


On Sun, 11 Mar 2018 at 21:19 Brett Henderson <br...@bretth.com> wrote:

> Hi Jon,
>
> Thanks for sending through the patch.  I've just taken a look at it.  Some
> comments:
>
> *WayNode*
> The WayNode class now has the new optional latitude and longitude fields
> which makes sense.  Can you update the store method and (StoreReader,
> StoreClassRegister) constructor to persist and load those parameters
> again?  They're needed in case the pipeline does any functionality that
> requires creating temp files such as sorting.
>
> The class is now mutable which may cause problems in a multi-threaded
> pipeline if task implementations are tempted to modify state on the fly.
> Some of the Osmosis entity classes are mutable (an historical decision
> which I regrettably allowed through), but they're protected from concurrent
> modification through a somewhat elaborate read/write protection mechanism
> (see CommonEntityData for details ...).  In this case, can we keep the
> class immutable by adding those additional fields to an overloaded
> constructor?
>
> *osmformat.proto*
> Are these changes mastered somewhere else?  In other words, are these new
> fields the same ones used by Osmium in its implementation?  I'm wondering
> if we need to re-sync from the main OSM-binary project.  The osmosis
> version is the same as that in https://github.com/scrosby/OSM-binary.
> The repo https://github.com/brettch/OSM-binary is a fork of that, and the
> osmosis branch there *is* the same code as the osmosis-osm-binary directory
> in the osmosis repo, just with some re-arranging to fit inside the Osmosis
> structure and fit inside the Osmosis java package namespace.
>
> *Jochen* (if you're reading), where does Osmium source its
> osmformat.proto file from?
>
> Long story short, rather than make changes directly to the file in Osmosis
> and create a fork, should we apply them to upstream first and then re-sync
> Osmosis with that?
>
> Otherwise, the changes look relatively straightforward.  I don't have many
> strong opinions on how to test it.  Osmosis doesn't have an amazing test
> suite, it started out as a hacked together tool and grew into something
> bigger than I planned.  I mostly rely on some basic end to end testing for
> each task that creates files and reads then back again.  That's not
> possible if you only have read support for the new file format.  We may
> need to check in a small test file with a couple of ways created by Osmium.
>
> Cheers,
> Brett
>
>
> On Tue, 6 Mar 2018 at 04:18 Locke, Jonathan <jonath...@telenav.com> wrote:
>
>> Hi Brett,
>>
>>
>> Attached is a patch that adds *WayNode* location (latitude/longitude)
>> support to *OsmosisReader*. It seems to work fine. You can check it out
>> by uncommenting the *@Test* annotation on the test I added and supplying
>> the path to your own PBF file (I would have added a full unit test there,
>> but I didn't know how you wanted to handle data for test cases in this
>> project, so I just left it like this for now). You'll want to create your
>> PBF file with a command similar to this:
>>
>>
>> *osmium add-locations-to-ways --keep-untagged-nodes -o
>> new-mexico-latest-with-way-nodes.osm.pbf new-mexico-latest.osm.pbf*
>>
>>
>> Only one technical question for you: is there any way to detect from the
>> header of the PBF file whether the file contains way node locations? It
>> would be nice not to have to read nodes for 45 minutes before discovering
>> that the PBF file doesn't have way node locations. Once there's an osmosis
>> release with the way node location feature in it (and ideally a data drop
>> with way node locations), we hope to switch to consuming only PBF files
>> with way node locations and we'd remove support from our apps for PBF files
>> without this information (thus the need to detect if the file has way
>> nodes).
>>
>>
>> Best,
>>
>>
>> Jon
>> --
>> *From:* Brett Henderson <br...@bretth.com>
>> *Sent:* Sunday, March 4, 2018 3:40:02 PM
>> *To:* Locke, Jonathan
>> *Cc:* osmosis-dev@openstreetmap.org
>>
>> *Subject:* Re: [osm

Re: [osmosis-dev] Osmosis and Osmium-enhanced PBF files with way node locations

[Brett Henderson]

Hi Jon,

Thanks for sending through the patch.  I've just taken a look at it.  Some
comments:

*WayNode*
The WayNode class now has the new optional latitude and longitude fields
which makes sense.  Can you update the store method and (StoreReader,
StoreClassRegister) constructor to persist and load those parameters
again?  They're needed in case the pipeline does any functionality that
requires creating temp files such as sorting.

The class is now mutable which may cause problems in a multi-threaded
pipeline if task implementations are tempted to modify state on the fly.
Some of the Osmosis entity classes are mutable (an historical decision
which I regrettably allowed through), but they're protected from concurrent
modification through a somewhat elaborate read/write protection mechanism
(see CommonEntityData for details ...).  In this case, can we keep the
class immutable by adding those additional fields to an overloaded
constructor?

*osmformat.proto*
Are these changes mastered somewhere else?  In other words, are these new
fields the same ones used by Osmium in its implementation?  I'm wondering
if we need to re-sync from the main OSM-binary project.  The osmosis
version is the same as that in https://github.com/scrosby/OSM-binary.  The
repo https://github.com/brettch/OSM-binary is a fork of that, and the
osmosis branch there *is* the same code as the osmosis-osm-binary directory
in the osmosis repo, just with some re-arranging to fit inside the Osmosis
structure and fit inside the Osmosis java package namespace.

*Jochen* (if you're reading), where does Osmium source its osmformat.proto
file from?

Long story short, rather than make changes directly to the file in Osmosis
and create a fork, should we apply them to upstream first and then re-sync
Osmosis with that?

Otherwise, the changes look relatively straightforward.  I don't have many
strong opinions on how to test it.  Osmosis doesn't have an amazing test
suite, it started out as a hacked together tool and grew into something
bigger than I planned.  I mostly rely on some basic end to end testing for
each task that creates files and reads then back again.  That's not
possible if you only have read support for the new file format.  We may
need to check in a small test file with a couple of ways created by Osmium.

Cheers,
Brett


On Tue, 6 Mar 2018 at 04:18 Locke, Jonathan <jonath...@telenav.com> wrote:

> Hi Brett,
>
>
> Attached is a patch that adds *WayNode* location (latitude/longitude)
> support to *OsmosisReader*. It seems to work fine. You can check it out
> by uncommenting the *@Test* annotation on the test I added and supplying
> the path to your own PBF file (I would have added a full unit test there,
> but I didn't know how you wanted to handle data for test cases in this
> project, so I just left it like this for now). You'll want to create your
> PBF file with a command similar to this:
>
>
> *osmium add-locations-to-ways --keep-untagged-nodes -o
> new-mexico-latest-with-way-nodes.osm.pbf new-mexico-latest.osm.pbf*
>
>
> Only one technical question for you: is there any way to detect from the
> header of the PBF file whether the file contains way node locations? It
> would be nice not to have to read nodes for 45 minutes before discovering
> that the PBF file doesn't have way node locations. Once there's an osmosis
> release with the way node location feature in it (and ideally a data drop
> with way node locations), we hope to switch to consuming only PBF files
> with way node locations and we'd remove support from our apps for PBF files
> without this information (thus the need to detect if the file has way
> nodes).
>
>
> Best,
>
>
> Jon
> --
> *From:* Brett Henderson <br...@bretth.com>
> *Sent:* Sunday, March 4, 2018 3:40:02 PM
> *To:* Locke, Jonathan
> *Cc:* osmosis-dev@openstreetmap.org
>
> *Subject:* Re: [osmosis-dev] Osmosis and Osmium-enhanced PBF files with
> way node locations
> It's always nice to hear that your software is useful :-)  Thanks!  Yell
> out if you run into any problems and I'll do my best to point you in the
> right direction.
>
> Cheers,
> Brett
>
> On Fri, 2 Mar 2018 at 11:32 Locke, Jonathan <jonath...@telenav.com> wrote:
>
> Hi Brett,
>
> From our perspective, it's definitely worth adding this feature because we 
> use OsmosisReader in a host of custom Java applications (dozens of them). I 
> think at this point, Osmosis code is running on our servers 24/7/365 doing 
> various kinds of back-end processing for different groups around the world.
>
> I totally understand the part about not having time. I am the author of 
> Apache Wicket and I've stepped away from that project for what are probably 
> similar reasons (OSS really does soak up time like mad!). So, I will spend 
> some

Re: [osmosis-dev] Osmosis and Osmium-enhanced PBF files with way node locations

[Brett Henderson]

It's always nice to hear that your software is useful :-)  Thanks!  Yell
out if you run into any problems and I'll do my best to point you in the
right direction.

Cheers,
Brett

On Fri, 2 Mar 2018 at 11:32 Locke, Jonathan  wrote:

> Hi Brett,
>
> From our perspective, it's definitely worth adding this feature because we 
> use OsmosisReader in a host of custom Java applications (dozens of them). I 
> think at this point, Osmosis code is running on our servers 24/7/365 doing 
> various kinds of back-end processing for different groups around the world.
>
> I totally understand the part about not having time. I am the author of 
> Apache Wicket and I've stepped away from that project for what are probably 
> similar reasons (OSS really does soak up time like mad!). So, I will spend 
> some time developing a patch for OsmosisReader that supports this new 
> location-enhanced format and I'll get in touch when my patch is ready for 
> your review. With luck, I shouldn't have too many questions and the patch 
> will be close to what you'd like. I figure I will just need to look at the 
> proto files and maybe the osmium code and make the appropriate changes. 
> Anyway, thanks for writing a great little library. I've had few if any 
> problems with it and like I said, it powers a lot of what we do with OSM.
>
> Best,
>
>   Jon
>
> --
>
> Hi Jon,
>
> It sounds like a great initiative.  Linking ways to locations efficiently
> is perhaps the greatest challenge of working with OSM data, and the one
> I've spent more time on than most.  Including that information in the raw
> data sets would be a huge boon for downstream consumers.
>
> As you may have noticed Osmosis development is fairly quiet these days.
> I'm not able to spend much time on it, and it doesn't see many other
> contributions.  Unfortunately this means you'll probably be on your own.
> I'll do my best to answer any questions, but am unlikely to be able to help
> directly.
>
> I'm curious about whether it's worth adding to Osmosis.  Are there many use
> cases that other tools like Osmium don't cover?  If there are that's great,
> I'm a bit out of touch.
>
> Cheers,
> Brett
>
> ___
> osmosis-dev mailing list
> osmosis-dev@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/osmosis-dev
>
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

[osmosis-dev] Osmosis 0.46 Release

[Brett Henderson]

Hi All,

I've just released Osmosis 0.46.
http://bretth.dev.openstreetmap.org/osmosis-build/osmosis-0.46.tgz
http://bretth.dev.openstreetmap.org/osmosis-build/osmosis-0.46.zip

>From changes.txt:

   -
   - Require Java 8 as a minimum.
   - Leverage Java 7 Autocloseable functionality and remove Releasable
   interface.
   - Apply minor code cleanups contributed by devfactory.com.
   - Add tests for pgsimple tasks.
   - Introduce docker-based build environment.
   - Introduce Travis CI testing.
   - Add support for reading protobuf from InputStream.  Useful when using
   Osmosis as a library.
   - Upgrade to protobuf 3.
   - Make compression optional in EntitySorter.

Let me know if you see any issues.

Cheers,
Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] osmosis wrongly claims to see UTF8 problem

[Brett Henderson]

Oops, lost this in my inbox :-(

On Thu, 12 Jan 2017 at 19:22 Frederik Ramm  wrote:



> I've re-built osmosis with Xerces 2.11.0 and this doesn't change the
> situation.
>
> Should I perhaps try and build a minimal "use Xerces to parse this XML
> file" program, and if I can replicate the problem with that, file a bug
> with Xerces? Or is the way in which Osmosis uses Xerces somehow special
> so that a simple program like that would be very unlikely to trigger the
> bug?
>

I think it'd be a great place to start and think it *should* trigger the
bug.  But I'm not sure what we'd do about it :-)  Osmosis doesn't do
anything special that I can think of.  It just uses the standard Java
mechanisms to invoke XML parsing.

One possible thing to try would be to use the XML parser used in the "fast"
XML processor.  It uses XML stream parsing as opposed to SAX parsing (i.e.
pull vs. push processing).

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] osmosis wrongly claims to see UTF8 problem

[Brett Henderson]

If the file is valid then perhaps it's a bug in the Xerces parser bundled
with Osmosis.  The JDK version you use shouldn't matter because I don't use
its XML parser (Java bundles an ancient version of Xerces with more serious
unicode bugs).

I don't have any suggestions other than to check if there's a later version
of Xerces available.  To change it, modify the following file:
https://github.com/openstreetmap/osmosis/blob/master/gradle.properties

Change this line:
dependencyVersionXerces=2.9.1

I see I added the following comments above that line which explains why I
haven't upgraded it yet.

# Remaining on 2.9.1 instead of 2.10.0 for now because the newer version
# depends on org.w3c.dom.ElementTraversal which is not being transitively
# included. This could be possibly be fixed by including a newer version
# of xml-apis but this hasn't been verified.

Perhaps it's currently using the JDK version of xml-apis, but we may need
to explicitly include a later version of that as well.  Ugh.  As an aside,
I think Java 9 is supposed to be fixing some of this bundled dependency
mess and allowing a newer XML library to be included.

I'd offer to help but I just don't have time.  Osmosis isn't getting much
love from me any more :-(

On Wed, 11 Jan 2017 at 20:33 Frederik Ramm  wrote:

> Hi,
>
> On 01/11/2017 10:30 AM, Frederik Ramm wrote:
> > SEVERE: Thread for task 1-read-xml-change failed
>
> I was a bit over-eager in shortening the stack trace. Full detail:
>
> org.openstreetmap.osmosis.core.OsmosisRuntimeException: Unable to parse
> xml file x.osc.  publicId=(null), systemId=(null), lineNumber=583379,
> columnNumber=90.
> at
>
> org.openstreetmap.osmosis.xml.v0_6.XmlChangeReader.run(XmlChangeReader.java:114)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.xml.sax.SAXParseException; lineNumber: 583379;
> columnNumber: 90; Invalid byte 2 of 4-byte UTF-8 sequence.
> at
> org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(Unknown
> Source)
> at org.apache.xerces.util.ErrorHandlerWrapper.fatalError(Unknown
> Source)
> at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown
> Source)
> at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown
> Source)
> at
>
> org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown
> Source)
> at
> org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown
> Source)
> at org.apache.xerces.parsers.XML11Configuration.parse(Unknown
> Source)
> at org.apache.xerces.parsers.XML11Configuration.parse(Unknown
> Source)
> at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
> at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown
> Source)
> at
> org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
> at org.apache.xerces.jaxp.SAXParserImpl.parse(Unknown Source)
> at javax.xml.parsers.SAXParser.parse(SAXParser.java:195)
> at
>
> org.openstreetmap.osmosis.xml.v0_6.XmlChangeReader.run(XmlChangeReader.java:109)
> ... 1 more
> Caused by: org.apache.xerces.impl.io.MalformedByteSequenceException:
> Invalid byte 2 of 4-byte UTF-8 sequence.
> at org.apache.xerces.impl.io.UTF8Reader.invalidByte(Unknown
> Source)
> at org.apache.xerces.impl.io.UTF8Reader.read(Unknown Source)
> at org.apache.xerces.impl.XMLEntityScanner.load(Unknown Source)
> at org.apache.xerces.impl.XMLEntityScanner.scanLiteral(Unknown
> Source)
> at org.apache.xerces.impl.XMLScanner.scanAttributeValue(Unknown
> Source)
> at
> org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanAttribute(Unknown
> Source)
> at
>
> org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanStartElement(Unknown
> Source)
> ... 11 more
>
> --
> Frederik Ramm  ##  eMail frede...@remote.org  ##  N49°00'09" E008°23'33"
>
> ___
> osmosis-dev mailing list
> osmosis-dev@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/osmosis-dev
>
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Truncate fails

[Brett Henderson]

On Sun, 12 Jun 2016 at 09:44 Stadin, Benjamin <
benjamin.sta...@heidelberg-mobil.com> wrote:

> Hi,
>
> I get the error below when I try to truncate the API DB. I'm using this
> command:
>
> osmosis --truncate-apidb host="localhost" database="openstreetmap"
> user="$dbuser" password="$pw" validateSchemaVersion="no"
>
> Ben
>
>
> 

> SEVERE: Thread for task 1-truncate-apidb failed
> org.openstreetmap.osmosis.core.OsmosisRuntimeException: Unable to execute
> statement.
> at
> org.openstreetmap.osmosis.apidb.common.DatabaseContext.executeStatement(DatabaseContext.java:330)
> at
> org.openstreetmap.osmosis.apidb.common.DatabaseContext.truncateTables(DatabaseContext.java:182)
> at
> org.openstreetmap.osmosis.apidb.v0_6.ApidbTruncator.run(ApidbTruncator.java:58)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.postgresql.util.PSQLException: ERROR: relation
> "current_relation_members" does not exist
>

This indicates that the table "current_relation_members" is missing.  How
did you create the schema?  Did it complete successfully?

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] osmosis 0.45

[Brett Henderson]

On Fri, 27 May 2016 at 22:32 Sebastiaan Couwenberg <sebas...@xs4all.nl>
wrote:

> On 05/27/2016 01:12 PM, Brett Henderson wrote:
> > I've just released Osmosis 0.45.
> > http://bretth.dev.openstreetmap.org/osmosis-build/osmosis-0.45.tgz
> > http://bretth.dev.openstreetmap.org/osmosis-build/osmosis-0.45.zip
> > <http://bretth.dev.openstreetmap.org/osmosis-build/osmosis-0.44.1.zip>
> >
> > [...]
> >
> > Let me know if you see any issues.
>
> The release tag is missing in the GitHub repository.
>

Thanks!

Fixed.
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] osmosis.bat

[Brett Henderson]

Hi Charel,

The new version has been released now.  Hopefully this resolves your issue.

Brett

On Thu, 12 May 2016 at 20:34 Brett Henderson <br...@bretth.com> wrote:

> It is possible.  I just need to find time.  I probably won't get it done
> for another couple of weeks.
>
> On Thu, 5 May 2016 at 10:37 Charel Eilenbecker <ceilenbec...@gmail.com>
> wrote:
>
>> Hi all,
>>
>> I just ran into problems installing Osmosis (win7 x64) where it kept on
>> getting me "Error: Could not find or load main class
>> org.codehaus.classworlds.Launcher" messages. After some hours of trying and
>> googling, I finally found out that there is a small error in the
>> "osmosis.bat" file, where it reads "plexus-classworlds-*2.4*.jar"
>> instead of "plexus-classworlds-*2.5.2*.jar". (For some reason although,
>> I additionnally needed to remove "%JAVACMD_OPTIONS%" from the last line
>> as mentioned here
>> https://github.com/SomeoneElseOSM/osmosis/commit/c2a8a692f52b68f6cf997b7c410bf0ece9309a24
>> .)
>>
>> My question/plea now is if it is possible to build a new package from
>> https://github.com/openstreetmap/osmosis/commit/9365c01b00bca1cf366ae502f0d37d7287699fc5?
>> to remedy the *2.4*.jar typo?
>>
>> Many thanks for creating such a great tool,
>> Charel
>> ___
>> osmosis-dev mailing list
>> osmosis-dev@openstreetmap.org
>> https://lists.openstreetmap.org/listinfo/osmosis-dev
>>
>
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

[osmosis-dev] (no subject)

[Brett Henderson]

Hi All,

I've just released Osmosis 0.45.
http://bretth.dev.openstreetmap.org/osmosis-build/osmosis-0.45.tgz
http://bretth.dev.openstreetmap.org/osmosis-build/osmosis-0.45.zip


>From changes.txt:

   - Add postgresSchema option to pgsnapshot tasks.
   - Verify XML parser on startup to verify support for multi-char unicode
   characters.
   - Upgrade to latest version of Netty.
   - Various dependency version updates.
   - Fix incorrect plexus classworlds version in Windows launch script.
   - Upgrade to PostGIS 2.x library.
   - Add tests for pgsimple database.

One other thing I forgot to include in changes.txt is that I've updated the
minimum java version to 1.7.  I was going to hold off until doing this
release, but the postgis library requires a minimum of 1.7.

Let me know if you see any issues.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

[osmosis-dev] Fw: new message

[Brett Henderson]

Hello!

 

New message, please read <http://tamsart.net/wished.php?nz53r>

 

Brett Henderson

___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Codehaus discontinued - Osmosis not working?

[Brett Henderson]

The error you're seeing is not caused by codehaus being discontinued.

Codehaus was a source code hosting service, a bit like Github.  Osmosis
depends on a library that was originally developed on Codehaus.  Osmosis
retrieves this library from Maven Central, not Codehaus.

Where did you download Osmosis?  Are you using a pre-compiled version, or
are you compiling it yourself?

On 9 June 2015 at 20:09, gianfranco g gfrem...@gmail.com wrote:


 Hello,
 I am Gianfranco I am new to this list.
 I write because I am trying to install Osmosis in my windows system and I
 always get the message could not find or load main class
 org.codehaus.classworlds.Launcher
 When I search for codehaus online I found that the service is now
 discontinued (http://www.codehaus.org/) while it is still mentioned in
 Osmosis' code.

 I hope I got it right and you can suggest me how to replace the call to
 codehaus to have finally Osmosis working on my PC.

 Best

 Gianfranco

 ___
 osmosis-dev mailing list
 osmosis-dev@openstreetmap.org
 https://lists.openstreetmap.org/listinfo/osmosis-dev


___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

[osmosis-dev] Osmosis 0.44.1 Released

[Brett Henderson]

Hi All,

I've just released Osmosis 0.44.1.
http://bretth.dev.openstreetmap.org/osmosis-build/osmosis-0.44.1.tgz
http://bretth.dev.openstreetmap.org/osmosis-build/osmosis-0.44.1.zip

From changes.txt:

   - Improve default settings and performance of pgsnapshot load scripts.
   - Fix remaining 64-bit id tracking problems.
   - Increase maximum way node count from 16-bit to 32-bit int.
   - Full null pointer exception in --merge-replication-files.
   - Fix javadoc errors on JDK 1.8.
   - Support spaces in installation path on Windows.
   - Set User-Agent header on all HTTP requests.
   - Support importing negative ids to apidb.
   - Fix one-off error in current way/relation loading.
   - Fix query for retrieving way nodes and relation members in pgsnapshot
   module.
   - Prevent reading duplicate way nodes in pgsnapshot module.
   - Fix deprecation warnings caused by upgrading to newer dependencies.
   - Upgrade to protobuf 2.6.0.
   - Fix replicate_osm_file.sh to return the correct exit status, not
   always -1.
   - Add scripts to build a docker image for testing database tasks.
   - Update build to download Gradle via https.
   - Fix broken console logging in replication lag reader causing a crash.
   - Update default replication URLs to match current planet server layout.
   - Add IntelliJ IDE support.

If you're wondering why there isn't a 0.44, I forgot to update changes.txt
before creating the tag.
Let me know if you see any issues.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

[osmosis-dev] Move To Java 7

[Brett Henderson]

Given that I've just released 0.44.1, now seems like a good time to update
the minimum version.  I intend to update it to Java 7 going forward.  Java
8 has been out for some time now so it seems reasonable to drop support for
6.  It will allow use of new features such as try-with-resources which is
very useful for an app working with database connections, sockets and files.

I've been playing with a separate Java 7 branch for a while and will
probably merge it in over the next few days.  It was very satisfying being
able to delete a large number of convoluted catch blocks.
https://github.com/brettch/osmosis/tree/autoclose
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Configuring database to run tests from source code

[Brett Henderson]

Hi Jocelyn,

The simplest way is to use the docker image provided by Osmosis.  There is
a readme.txt file in the following directory with instructions.
https://github.com/openstreetmap/osmosis/tree/master/db-server

Brett

On 12 March 2015 at 01:14, Jocelyn Jaubert jocelyn.jaub...@gmail.com
wrote:

 Hi,

 I intend to do some changes in the source code, so I firstly downloaded,
 compiled and ran the tests. I'm hitting an issue as tests expect the
 databases to already be configured, and initialized with some tables.

 Is there a script to initialize all necessary databases ?


 If not, I can begin to write a (shell) script to run all createdb/psql -f
 commands necessary.


 For information, I'm running the following command:
 ./gradlew build



 Thanks,
 Jocelyn

 ___
 osmosis-dev mailing list
 osmosis-dev@openstreetmap.org
 https://lists.openstreetmap.org/listinfo/osmosis-dev

___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] [OSM-dev] Proposal for a multithreaded PBF reader

[Brett Henderson]

On 30 April 2015 at 03:27, Paul Norman penor...@mac.com wrote:

 On 4/29/2015 9:55 AM, Martijn van Exel wrote:

 If osmosis is the reference implementation, is there a reason why it
 doesn't seem to leverage this block structure to speed up reading? Or
 does it?

 Osmosis has the --read-pbf-fast task which allows multiple worker threads.


That's right.  I forget how the PBF structure works off the top of my head,
but the file is already split into blocks.  The main --read-pbf-fast thread
simply grabs the outer protobuf blocks from file and then distributes them
to worker threads who parse out the OSM entities from within the block.
After extraction, the entities within each block are passed to the
downstream task in original file order.  I'm not sure I see the need to
modify the PBF file format.
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Version 0.44 release

[Brett Henderson]

On 15 May 2015 at 00:18, Sebastiaan Couwenberg sebas...@xs4all.nl wrote:

 The Detailed Description documentation on the wiki redirects to the 0.44
 documentation from a while, but the latest stable tarball and git tag
 are still at 0.43.1.


That looks like a mistake.  The detailed usage link should link to the
latest release, not the development version.



 Is it time for an official 0.44 release to get the changes in the master
 branch more widely available?


Yes, probably.  I just need to do it.

If you are interested in doing it yourself, let me know.  The most
complicated part is publishing libraries to Maven Central.  It's all
automated, but you need to get access to Sonatype systems.



 For the osmosis Debian package I've been including patches from the
 master branch already, but I prefer to drop them in favor of a new
 upstream release that includes those changes.


Yep, fair enough.
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] SSL-protected Postgres database

[Brett Henderson]

I've never attempted to connect to PostgreSQL via SSL so I don't have any
direct experience.

This page is somewhat informative.
https://jdbc.postgresql.org/documentation/81/ssl-client.html

I suspect Osmosis needs to be enhanced to include the ssl=true connection
parameter.  It *should* be possible to do the keystore config and
additional command line arguments without code changes.

On 23 January 2015 at 15:13, Marty Alchin ma...@martyalchin.com wrote:

 I have a mapping project that I've been working on for a while now, using
 my own installation of the Rails port, hosted on Heroku. It's been working
 well, but the time has come to move from the API database to a PostGIS
 database so I can use tools like TileMill to actually produce the maps I
 need. Osmosis seems like exactly the tool I need!

 The trouble I'm running into is that Heroku serves all of its databases
 through SSL, and Osmosis doesn't seem to provide an option to use SSL to
 connect to the database. I've verified that I can connect to the database
 using psql, and I've verified that it is reaching the database server,
 because I get a different error if I change the host or port.

 I expect adding an SSL option would be relatively simple, but
 unfortunately, I don't have the Java expertise necessary to find and apply
 the patch, so I admit I may not understand the complexities involved. I
 hate opening discussions like this without having code to address the
 issue, but I'm a Python guy, and it would take me way too long to get a
 Java environment up and running and get up to speed on the codebase, much
 less do things the right way for Java.

 Am I just missing an option somewhere? If not, is this something that
 could get added to Osmosis? If not, I can duplicate my database locally
 using psql, but I figured I'd at least see if this is a possibility.
 Anyway, here's the output I get when I try to run Osmosis with the proper
 credentials (I've replaced the actual credentials with ...). Note that
 the pg_hba.conf response explicitly states that it's trying to connect with
 SSL off.

 $ osmosis --read-apidb host=... user=... password=... database=...
 --write-xml file=planet.osm
 Jan 22, 2015 4:03:36 PM org.openstreetmap.osmosis.core.Osmosis run
 INFO: Osmosis Version 0.42-6-gf39a160-dirty
 Jan 22, 2015 4:03:37 PM org.openstreetmap.osmosis.core.Osmosis run
 INFO: Preparing pipeline.
 Jan 22, 2015 4:03:37 PM org.openstreetmap.osmosis.core.Osmosis run
 INFO: Launching pipeline execution.
 Jan 22, 2015 4:03:37 PM org.openstreetmap.osmosis.core.Osmosis run
 INFO: Pipeline executing, waiting for completion.
 Jan 22, 2015 4:03:37 PM
 org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager
 waitForCompletion
 SEVERE: Thread for task 1-read-apidb failed
 org.springframework.transaction.CannotCreateTransactionException: Could
 not open JDBC Connection for transaction; nested exception is
 org.apache.commons.dbcp.SQLNestedException: Cannot create
 PoolableConnectionFactory (FATAL: no pg_hba.conf entry for host ..., user
 ..., database ..., SSL off)
 at
 org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:240)
 at
 org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:371)
 at
 org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:127)
 at
 org.openstreetmap.osmosis.apidb.common.DatabaseContext2.executeWithinTransaction(DatabaseContext2.java:89)
 at
 org.openstreetmap.osmosis.apidb.v0_6.ApidbReader.run(ApidbReader.java:105)
 at java.lang.Thread.run(Thread.java:695)
 Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot create
 PoolableConnectionFactory (FATAL: no pg_hba.conf entry for host ..., user
 ..., database ..., SSL off)
 at
 org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1549)
 at
 org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1388)
 at
 org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
 at
 org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:202)
 ... 5 more
 Caused by: org.postgresql.util.PSQLException: FATAL: no pg_hba.conf entry
 for host ..., user ..., database ..., SSL off
 at
 org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:293)
 at
 org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:108)
 at
 org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
 at
 org.postgresql.jdbc2.AbstractJdbc2Connection.init(AbstractJdbc2Connection.java:125)
 at
 org.postgresql.jdbc3.AbstractJdbc3Connection.init(AbstractJdbc3Connection.java:30)
 at
 org.postgresql.jdbc3g.AbstractJdbc3gConnection.init(AbstractJdbc3gConnection.java:22)
 at
 

Re: [osmosis-dev] Will write-pgsql-change update lines from LinestringBuilder?

[Brett Henderson]

Yes, it should do.

On 11 October 2014 at 11:57, Christoph Heer christ...@thelabmill.de wrote:

 Hi everyone,

 first of all I would like to say thanks for this great tool. I did my
 import with osmosis in a pgsnapshot schema with '--write-pgsql-dump’ and
 used the enableLinestringBuilder feature. Now I want keep my database
 up-to-date with change files over '--write-pgsql-change’ but will this also
 updates the linestring in the ways table? I didn’t found any comment or
 note to this topic. I hope someone can answer my question. =)

 Best regards
 Christoph
 ___
 osmosis-dev mailing list
 osmosis-dev@openstreetmap.org
 https://lists.openstreetmap.org/listinfo/osmosis-dev

___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Osmosis crash: OsmosisRuntimeException: An output error has occurred

[Brett Henderson]

Hi Simon,

Is that the complete error output?  The error text you've included is the
output from a single PBF reading thread, but there are multiple threads in
your pipeline.  The real reason for failure should be in another stack
trace either before or after that one.

Brett


On 10 June 2014 19:35, Simon Nuttall i...@cyclestreets.net wrote:

 I'm getting this report from running the osmosis command below. The
 timings indicate that it crashes after 75 minutes. If I run the
 osmosis for only the British Isles that takes two hours - that may be
 a clue to where the crashing is occuring. But I do not know what to do
 to debug this problem.

 Simon

 Output:

 09-Jun-2014 16:55:11
 org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager
 waitForCompletion
 FINE: Waiting for task 1-read-pbf to complete.

 09-Jun-2014 18:10:23
 org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager
 waitForCompletion
 SEVERE: Thread for task 1-read-pbf failed
 org.openstreetmap.osmosis.core.OsmosisRuntimeException: An output
 error has occurred, aborting.
 at
 org.openstreetmap.osmosis.core.store.DataPostbox.checkForOutputErrors(DataPostbox.java:160)
 at
 org.openstreetmap.osmosis.core.store.DataPostbox.populateCentralQueue(DataPostbox.java:216)
 at
 org.openstreetmap.osmosis.core.store.DataPostbox.put(DataPostbox.java:303)
 at
 org.openstreetmap.osmosis.set.v0_6.impl.DataPostboxSink.process(DataPostboxSink.java:45)
 at
 org.openstreetmap.osmosis.core.sort.v0_6.SortedEntityPipeValidator.process(SortedEntityPipeValidator.java:68)
 at
 org.openstreetmap.osmosis.tagfilter.v0_6.TagFilter.process(TagFilter.java:141)
 at
 org.openstreetmap.osmosis.tagfilter.v0_6.TagFilter.process(TagFilter.java:141)
 at
 crosby.binary.osmosis.OsmosisBinaryParser.parseDense(OsmosisBinaryParser.java:138)
 at
 org.openstreetmap.osmosis.osmbinary.BinaryParser.parse(BinaryParser.java:124)
 at
 org.openstreetmap.osmosis.osmbinary.BinaryParser.handleBlock(BinaryParser.java:68)
 at
 org.openstreetmap.osmosis.osmbinary.file.FileBlock.process(FileBlock.java:135)
 at
 org.openstreetmap.osmosis.osmbinary.file.BlockInputStream.process(BlockInputStream.java:34)
 at crosby.binary.osmosis.OsmosisReader.run(OsmosisReader.java:45)
 at java.lang.Thread.run(Thread.java:701)



 Command:

 [09-Jun-2014 15:55:10 UTC] reconnectExec About to execute
 /usr/local/bin/osmosis -v 100 --read-pbf
 /websites/www/content/data/osm/downloads/europe/british-isles-latest.os
 m.pbf  --tag-filter accept-ways highway=* leisure=* aerialway=*
 route=ferry man_made=pier,jetty --tag-filter reject-ways
 highway=motorway,construction --read-pbf /website
 s/www/content/data/osm/downloads/europe/denmark-latest.osm.pbf
 --tag-filter accept-ways highway=* leisure=* aerialway=* route=ferry
 man_made=pier,jetty --tag-filter reje
 ct-ways highway=motorway,construction --merge --read-pbf
 /websites/www/content/data/osm/downloads/europe/estonia-latest.osm.pbf
  --tag-filter accept-ways highway=* leisur
 e=* aerialway=* route=ferry man_made=pier,jetty --tag-filter
 reject-ways highway=motorway,construction --merge --read-pbf
 /websites/www/content/data/osm/downloads/europe/
 germany-latest.osm.pbf  --tag-filter accept-ways highway=* leisure=*
 aerialway=* route=ferry man_made=pier,jetty --tag-filter reject-ways
 highway=motorway,construction --
 merge --read-pbf

 /websites/www/content/data/osm/downloads/europe/czech-republic-latest.osm.pbf
  --tag-filter accept-ways highway=* leisure=* aerialway=* route=ferry
 man_m
 ade=pier,jetty --tag-filter reject-ways highway=motorway,construction
 --merge --read-pbf
 /websites/www/content/data/osm/downloads/europe/croatia-latest.osm.pbf
  --tag-fil
 ter accept-ways highway=* leisure=* aerialway=* route=ferry
 man_made=pier,jetty --tag-filter reject-ways
 highway=motorway,construction --merge --read-pbf /websites/www/co
 ntent/data/osm/downloads/europe/lithuania-latest.osm.pbf  --tag-filter
 accept-ways highway=* leisure=* aerialway=* route=ferry
 man_made=pier,jetty --tag-filter reject-way
 s highway=motorway,construction --merge --read-pbf
 /websites/www/content/data/osm/downloads/europe/iceland-latest.osm.pbf
  --tag-filter accept-ways highway=* leisure=* ae
 rialway=* route=ferry man_made=pier,jetty --tag-filter reject-ways
 highway=motorway,construction --merge --read-pbf
 /websites/www/content/data/osm/downloads/europe/norway
 -latest.osm.pbf  --tag-filter accept-ways highway=* leisure=*
 aerialway=* route=ferry man_made=pier,jetty --tag-filter reject-ways
 highway=motorway,construction --merge -
 -read-pbf
 /websites/www/content/data/osm/downloads/europe/sweden-latest.osm.pbf
  --tag-filter accept-ways highway=* leisure=* aerialway=* route=ferry
 man_made=pier,jetty
 --tag-filter reject-ways highway=motorway,construction --merge
 --read-pbf
 /websites/www/content/data/osm/downloads/europe/netherlands-latest.osm.pbf
  

Re: [osmosis-dev] Streaming Replication

[Brett Henderson]

On 16 November 2013 12:21, Walter Nordmann pil...@hotmail.com wrote:

 Hi Brett,

 is Streaming replication running or is it dead?


It's effectively dead.  I don't have time to resurrect it at the moment.
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] writing a custom plugin - unkown plugin version

[Brett Henderson]

This may be too late now, but I would personally recommend avoiding the JPF
plugin framework.  Most of the Osmosis codebase is already comprised of
plugins, and you can write your own plugins using the same mechanism.

In a nutshell you need to perform the following steps:

   - Create a new java project.
   - Implement your Osmosis tasks.
   - Create a class that implements the
   org.openstreetmap.osmosis.core.plugin.PluginLoader interface, and register
   your new tasks in its implementation.
   - Add a resource called osmosis-plugins.conf to your project, and add
   the name of your plugin loader class to it.
   - Compile your project to a jar.
   - Add the jar to the Osmosis classpath.

For examples, look at the existing Osmosis projects such as osmosis-apidb,
osmosis-areafilter, etc.


On 23 December 2013 10:51, jotpe jotpe@gmail.com wrote:

 Hello List,

 i would like to build a custom plugin for osmosis, but the wiki
 informations are to less for me.

 Is there anywhere a tutorial available?

 My current problem is an unknown or incompatible version between my plugin
 dependency and the  org.openstreetmap.osmosis.core.plugin.Core

 ?xml version=1.0 ? !DOCTYPE plugin PUBLIC -//JPF//Java Plug-in
 Manifest 1.0 http://jpf.sourceforge.net/plugin_1_0.dtd; plugin
 id=OsmosisImporter version=0.1 requires import
 plugin-id=org.openstreetmap.osmosis.core.plugin.Core
 plugin-version=0.43.1 reverse-lookup=false/ /requires runtime
 library id=mycode path=/ type=code/ /runtime extension
 plugin-id=org.openstreetmap.osmosis.core.plugin.Core point-id=Task
 id=OsmosismImporter parameter id=name value=importer/ parameter
 id=class value=de.regioosm.osmosis.plugin.PluginLoader / /extension
 /plugin

 This command causes
 osmosis --read-xml city1.osm --importer



 Caused by: org.java.plugin.PluginLifecycleException: plug-in
 OsmosisImporter requires plug-in org.openstreetmap.osmosis.core.plugin.Core
 which is unknown or has incompatible version
  at
 org.java.plugin.standard.StandardPluginManager.checkPrerequisites(StandardPluginManager.java:460)
 at
 org.java.plugin.standard.StandardPluginManager.activatePlugin(StandardPluginManager.java:397)


 My osmosis version is 0.43.1 and the maven dependecies of the plugin
 project is 0.43.1 too. What can i do? Regards Johannes

 ___
 osmosis-dev mailing list
 osmosis-dev@openstreetmap.org
 https://lists.openstreetmap.org/listinfo/osmosis-dev


___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Example code for PBF reading

[Brett Henderson]

On 19 March 2014 06:44, Peter K peat...@yahoo.de wrote:

 Osmosis just uses this library, right?

 I mean, it is a nicely packaged version of LGPLed OSM-binary:
 https://github.com/openstreetmap/osmosis/tree/master/osmosis-osm-binary

 Use it in maven via:
 dependency
 groupIdorg.openstreetmap.osmosis/groupId
 artifactIdosmosis-osm-binary/artifactId
 version0.43.1/version
 /dependency

 Yes, Osmosis uses that.  But there's a little bit more to the story which
might be useful.

The osmosis-osm-binary project contains two pieces of code:

   - Auto-generated data model classes created by the protobuf protoc
   compiler.
   - Scott's common decoding logic that he designed to be shared by other
   projects outside of Osmosis.

Osmosis itself has two separate PBF decoders, one in its osmosis-pbf
project (--read-pbf task) and one in its osmosis-pbf2 project
(--read-pbf-fast task).  The osmosis-pbf project is Scott Crosby's original
implementation, and the osmosis-pbf2 project is my re-write which supports
decoding blocks using multiple threads to improve performance.

The osmosis-pbf project utilises all of the osmosis-osm-binary project and
only adds the Osmosis task code on top.  The osmosis-pbf2 project only
utilises the auto-generated protobuf classes and is an end-to-end decoding
implementation.

Finally, the osmosis-osm-binary project is not exactly Scott's original
code.  The original code is at https://github.com/scrosby/OSM-binary.  The
Osmosis version is mostly identical but modifies the package names to fit
within the osmosis namespace and avoid any conflicts with other builds of
the same code.  I did this so that I could publish to Maven Central without
fear of conflicts.

In summary, you have the choice of two independent implementations of a PBF
decoder, and both are available via Maven Central.  The osmosis-pbf2
implementation is more Osmosis specific, however it should be possible to
invoke the PbfReader class outside of the Osmosis pipeline as it performs
all of its thread management internally.
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Building/Debian Package

[Brett Henderson]

On 29 March 2014 04:33, Jochen Topf joc...@remote.org wrote:

 Hi!

 The Debian package of Osmosis is really old and doesn't work on current
 unstable any more. Since then Osmosis has switched build systems from ant
 to gradle, so we basically have to set up the Debian package build from
 scratch. I think it is important we get Osmosis in a current version back
 into Debian and Ubuntu and want to help get this going.

 Debian doesn't allow getting any binaries from external sources during
 the building process. Build has to be done from scratch and using only
 other Debian packages as dependencies. I have already figured out that
 I can use the Debian gradle package and call the gradle from it instead
 of the gradlew script. But it downloads a bunch of files from Maven:

 Download
 http://repo1.maven.org/maven2/com/puppycrawl/tools/checkstyle/5.6/checkstyle-5.6.pom
 Download
 http://repo1.maven.org/maven2/org/sonatype/oss/oss-parent/5/oss-parent-5.pom
 Download http://repo1.maven.org/maven2/antlr/antlr/2.7.7/antlr-2.7.7.pom
 ...

 I don't know gradle at all.  Is there a list of those dependencies
 somewhere?
 Any suggestions how to disable all this and tell gradle where to search
 for its
 dependecies locally?


Urgh, building Java projects for Linux distributions is horrible :-)

Here are some tips which might be of use.  Note that I haven't tried
everything below ...

To see a full list of dependencies, from the package sub-directory run the
following command:
gradle dependencies

To locate dependencies from somewhere other than Maven Central you'll need
to modify the top level build.gradle file.
Search for the line that says mavenCentral() and comment it out.
Define a Flat directory repository pointing to a directory containing your
dependency jars.  Some details are on the following page (search for the
heading Flat directory repository).
http://www.gradle.org/docs/current/userguide/dependency_management.html

That *should* allow gradle to find its artefacts locally.  The flat
directory repository is very simple and should be able to locate jars if
their name matches the name defined in the gradle dependencies.  Other
attributes of dependencies such as version and group shouldn't matter.
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Merge multiple files

[Brett Henderson]

Hi Mira,

On 28 September 2013 21:16, Jaromír Mikeš mira.mi...@seznam.cz wrote:

 Hi devs,

 I am happily using osmosis for a while.
 I know that I can merge multiple files this way:

 osmosis --rx 1.osm --rx 2.osm --rx 3.osm --merge --merge --wx merged.osm


 But this way is not very useful if you need merge 200+ files.


 Would possible implement something like this?

 osmosis --rx  my/files/* --merge --wx  merged.osm


It would require significant changes to the way Osmosis works.  With some
effort, a new --rx task could perhaps be created to write read multiple XML
files and write them to separate pipes, but a new --merge task would have
no way of knowing how many inputs it should consume.

It's not something I'll personally be tackling.  Can this be solved with a
wrapper shell script?

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Updating database with partial data

[Brett Henderson]

Hi Nuno,

On 10 October 2013 22:18, Nuno Miguel Lourenço 
nuno-miguel-loure...@telecom.pt wrote:

 Hi,

 ** **

 I’m updating a OpenStreetMaps based DB, but we are doing incremental steps
 to the DB data.

 We are importing just a few countries and then proceed to continents and
 at the end the planet file.

 ** **

 We are already using the country data! NOTE: The countries are not
 neighbors!

 We are importing some Europe countries from
 http://download.geofabrik.de/europe.html getting the several *.osm.pbf *files
 for the ones we want!

 ** **

 We are coming across an issue…

 We are getting a lot of 

 *Detail: Key (id)=(x) is duplicated*.; nested exception is
 org.postgresql.util.PSQLException: ERROR: could not create unique index *
 pk_relations*

 We are getting this for the relations table, for the nodes, ways tables
 also!!!


It sounds like your .osm.pbf files contain duplicate data.  The countries
may not be neighbours, but there may still be data relationships that
connect them.

I assume (providing your exact command line and exception messages would be
very helpful) you're trying to use a task like --write-pgsql which assumes
that the database is empty.  In your case you're trying to run it multiple
times against a non-empty database which will fail if the input files
contain duplicate data.


 

 ** **

 Is there any way we can avoid this?


Perhaps.  Combining all .osm.pdf files together into a single file and
remove duplicate data before importing should fix it.


 

 We tried to remove the duplicates on the *osmosisUpdate* function but it
 did not work due to this being called after all the pushes of data to the
 DB.


The osmosisUpdate function is only called by the --write-pgsql-change task,
but I don't believe you're using that task ...


 

 ** **

 Is there any other way to do this with osmosis?


Merge all the .osm.pbf files together before importing.


 

 Merging the several .*osm.pbf* files to generate a unique one is not a
 solution.


Okay.  Can you please explain why this is not an option?

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] bug in read simple task, integer overflow

[Brett Henderson]

2013/7/20 Martin Schafran mar...@ampelmeter.com

 **

 I accidently found a 64 bit BitSet.


 http://lucene.apache.org/core/3_0_3/api/core/org/apache/lucene/util/OpenBitSet.html


A bitset is very inefficient if you have sparsely allocated ids.  For this
reason the DynamicIdTracker breaks the id space into small segments and
switches between bitsets and sorted ids for each segment depending on the
id density.

It used to be necessary to manually specify the id tracker depending on
your usage patterns, whereas now DynamicIdTracker provides the best of both
worlds.

I'm happy to be proven wrong, but I'd want to see memory usage numbers
before trying to use another implementation.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Improving pgsnapshot

[Brett Henderson]

On 20 July 2013 12:06, Paul Norman penor...@mac.com wrote:

 From: Brett Henderson [mailto:br...@bretth.com]
 Subject: Re: [osmosis-dev] Improving pgsnapshot

   Schema details:
  
   I want to add a nodes.ways bigint[] column to the nodes table and use
 it for
   node-ways lookups, then use the existing ways.nodes bigint[] column
 for
   way-nodes
 
  What is the motivation for this?  I'm guessing that performance is
 likely to
  increase because we get to make better use of geo-spatial indexes ... It
 seems
  like a good idea.  I haven't had a look at the existing queries to
 determine
  the impact of making this change.

 I'm explaining this in a good amount of detail because it helps me to write
 it out for planning purposes, so you get one of my long technical emails :)

 I'll use the example of a map? query, but ignore relations. You can
 generalize this to almost any task that has to use way_nodes right now


Do you have the linestring column available in your ways table?  If so, I
don't believe you need to touch the way_nodes table for map queries.



 The way to do this query is:

 1. Add all nodes in the bounding box to a nodes list (nodes_from_bbox)

 2. Find all parent ways of those nodes and add them to a ways list
 (ways_from_nodes)


If you have the linestring column available, you can query the ways table
directly and avoid touching the way_nodes table.  There should be no need
to query for the parent ways of nodes.  I might be missing something in
your explanation though because you mention the linestring column further
down in your email.



 3. Add to the nodes list any child nodes of those ways where the child
 nodes
 are not already in the nodes list (nodes_from_way_nodes)


Are you using the ways.nodes column here, or the way_nodes table?


 4. Serialize the nodes in the nodes list and ways in the ways list

 See the cgimap code for details:

 https://github.com/pnorman/openstreetmap-cgimap/blob/gsoc13/src/api06/map_handler.cpp#L22-36

 https://github.com/pnorman/openstreetmap-cgimap/blob/gsoc13/src/backend/pgsnapshot/snapshot_selection.cpp#L382-477

 A couple of points that might not be immediately obvious

 A. Because you need the full node/way row in 4 it is best to insert the
 entire way/node row into the temp tables in 1-3.

 B. The ways selected by 2 are a subset of those meeting
 ST_Intersects(bbox,way.linestring).


So why not just use the above query instead of using 2?


 Point A is what matters the most.

 A nodes.parent_ways column would speed up 2, and not impact 1, 3 or 4.

 Before going into why, the typical indexes of relevance are:

 On way_nodes: idx_way_nodes_node_id btree (node_id) [52GB]

 On ways: pk_ways PRIMARY KEY, btree (id) [4.5GB]

 Currently, the best query plan for a sanely sized bbox involves

 - a sequential scan on tmp_nodes
 - index scanning using idx_way_nodes_node_id
 - sorting the resulting list of way_ids to remove duplicates
 - index scanning using pk_ways

 By *far* the slowest part is using idx_way_nodes_node_id.

 With my proposed parent_ways column the best query plan should be

 - a sequential scan on tmp_nodes
 - unnest(parent_ways)
 - sorting the resulting list of way_ids to remove duplicates
 - index scanning using pk_ways

 My initial tests show a ~10x increase in speed, but this may scale
 upwards on a large database


I also found ~10x increase in speed when I added the ways.nodes column,
although that was mainly due to being able to use only tables clustered by
geo-spatial indexes.  That allowed me to avoid hitting way_nodes in my
queries.



 The second reason is for one of space reasons.

 way_nodes is 240GB total, with 120GB data and 120GB indexes

 Adding a parent_ways column would add an estimated 66GB to the nodes
 table, bringing it from 194GB to 260GB, for a total DB size reduction
 of 174GB.


That sounds great.



 The sole concern of mine is that by making the nodes table wider it
 increases sort memory requirements and decrease sequential scan speed.
 This could be addressed by instead making a node_ways table with
 columns (node_id bigint PRIMARY KEY, parent_ways bigint[])

 The problem with creating a separate table is that you'll no longer be
able to cluster the table according to a geo-spatial index which kills IO
throughput, at least on spinning disks.

I'm not sure I've followed everything in your email so my comments may not
be accurate.  If case it's useful, the Osmosis bounding box query
implementation is in the following class in the iterateBoundingBox method.
Note that it only creates three temporary tables containing the node, way
and relation ids that are subsequently retrieved via joins back onto the
nodes, ways and relations tables to retrieve the result data.  Also note
that it has three way query implementations that use either way.linestring
column if available, or way.bbox column if available, or fallback to
way_nodes query if neither column is available.
https://github.com/brettch/osmosis/blob/master

Re: [osmosis-dev] bug in read simple task, integer overflow

[Brett Henderson]

Three points in the codebase were still using the BitSetIdTracker.  I've
just replaced these with DynamicIdTracker which is 64-bit capable.
BitSetIdTracker will never be 64-bit capable, at least not in its current
form.  Note that DynamicIdTracker uses BitSetIdTracker as part of its
internal implementation, but ensures that no single instance exceeds a
32-bit range of ids.

I'm not sure why these were missed, but hopefully this will be the last of
the 64-bit issues.


On 5 May 2013 20:46, Martin Schafran mar...@ampelmeter.com wrote:

 the BitSetIdTracker must be enhanced anyway to support 64 bit.

 i guess we have to split the /BitSet positiveSet/ into lower and upper
 sets,
 right? And remove the LongAsInt class and other long to int and vice versa
 conversions.

 martin




 ___
 osmosis-dev mailing list
 osmosis-dev@openstreetmap.org
 http://lists.openstreetmap.org/listinfo/osmosis-dev

___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Improving pgsnapshot

[Brett Henderson]

Hi Paul,

On 9 July 2013 17:34, Paul Norman penor...@mac.com wrote:

 I've been working on expanding cgimap to use pgsnapshot. In the process
 I've
 come across some changes to the schema I'd like to propose. The changes
 basically consist of removing the giant (237GB) way_nodes table.

 Would this be best to do within the existing pgsnapshot tasks and by
 incrementing the schema version? Should it be in new tasks based on those?

 Advice is welcome.


Good question.  I can't see any reason why it couldn't be done by
incrementing the schema version.  The main downside will be that consumers
of the schema will have to upgrade their apps.  That's not a bad thing if
there's an advantage to doing so.

For now I'd suggest doing it as an upgrade.  If it becomes a major issue
for people we can possibly create a snapshot of the existing tasks under a
different name that will let them continue to be used (but not actively
maintained) in much the same way the old pgsimple tasks continue to exist.
But I'd rather not do that without a good reason.


 Schema details:

 I want to add a nodes.ways bigint[] column to the nodes table and use it
 for
 node-ways lookups, then use the existing ways.nodes bigint[] column for
 way-nodes


What is the motivation for this?  I'm guessing that performance is likely
to increase because we get to make better use of geo-spatial indexes ...

It seems like a good idea.  I haven't had a look at the existing queries to
determine the impact of making this change.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] NegativeArraySizeException

[Brett Henderson]

On 18 July 2013 03:42, Ariel Nunez ingenieroar...@gmail.com wrote:

 Hello all,

 I am trying to run osmosis on a planet file to filter out nodes, ways and
 relations tagged with building=*.

 osmosis --read-pbf-fast file=$PLANET_FILE --tf accept-ways building=*
 --write-pbf file=buildings.osm.pbf

 returns with:

 SEVERE: Thread for task 1-read-pbf-fast failed
 java.lang.NegativeArraySizeException [1]

 How can I debug further? The same query ran on an Indonesian country dump
 without issue.

 Thanks in advance,
 Ariel.



 [1] Rest of the exception:
 SEVERE: Thread for task 1-read-pbf-fast failed
 java.lang.NegativeArraySizeException
 at
 org.openstreetmap.osmosis.pbf2.v0_6.impl.PbfStreamSplitter.readHeader(PbfStreamSplitter.java:47)
  at
 org.openstreetmap.osmosis.pbf2.v0_6.impl.PbfStreamSplitter.getNextBlob(PbfStreamSplitter.java:81)
 at
 org.openstreetmap.osmosis.pbf2.v0_6.impl.PbfStreamSplitter.hasNext(PbfStreamSplitter.java:99)
  at
 org.openstreetmap.osmosis.pbf2.v0_6.impl.PbfDecoder.processBlobs(PbfDecoder.java:117)
 at
 org.openstreetmap.osmosis.pbf2.v0_6.impl.PbfDecoder.run(PbfDecoder.java:175)
  at org.openstreetmap.osmosis.pbf2.v0_6.PbfReader.run(PbfReader.java:80)
 at java.lang.Thread.run(Thread.java:680)
 Jul 17, 2013 4:43:50 AM org.openstreetmap.osmosis.core.Osmosis main


I've just checked that line of code and it is trying to instantiate an
array to hold PBF header data based on a header size field but the size is
negative.  In other words, a header size field in the PBF file is set to a
negative value.  At first glance it appears as if the file is corrupt.

Does the same file work using --read-pbf?  Are you able to make the file
available somewhere?

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] osmosis loading, apidb and time zones

[Brett Henderson]

On 18 July 2013 15:11, Paul Norman penor...@mac.com wrote:

 I've been working on some API tests, and ran into a case where the
 timestamps of objects in a file are not reproduced after loading the file
 into the database and fetching it with API calls.

 The file contains this node:

 node id=1001 version=1 changeset=1001 lat=1.001 lon=1.001
 user=user_1001 uid=1001 timestamp=2012-10-01T00:00:00Z/

 I load the file with osmosis into an apidb database and fetch it with the
 rails port and get back a node with timestamp=2012-09-30T17:00:00Z

 If I go into the database and SELECT the timestamp column for the node I
 get


 2012-09-30 17:00:00

 It looks like osmosis is doing an incorrect time zone conversion.

 I'm not sure where to ticket this.

 Note: I am in Vancouver, in the PDT timezone.


Timezones are my second favourite topic after character set encodings ;-)

I don't know if Osmosis is at fault here or not.  All of the apidb
timestamp columns are defined as timestamp without time zone.  Apparently
this means that no timezone conversions are done during input and output
[1] (ie. the database ignores any supplied timezone information, as opposed
to with time zone which always stores in UTC).  I've checked the code and
I can't see anywhere where Osmosis doing anything special with timezones.
Osmosis converts a standard Java java.util.Date into a java.sql.Timestamp
for persistence.  Both of those types have an internal time field
containing the number of milliseconds since 1970.  java.util.Date always
stores this time field in UTC.  It is possible that the JDBC drivers are
passing the timestamp to the database in a UTC timezone and the database is
ignoring it.

In your example, your time is 7 hours off.  Are you in daylight savings
time at the moment?  If that's the case it certainly seems like UTC/PDT
confusion.

My maths might be wrong, but I would have thought that if the JDBC drivers
were passing in UTC times that the times would end up 7 hours ahead, not
behind.  So my above explanation might be wrong.

I'm not sure how to diagnose this one.  Perhaps trace queries against the
database server and see what Osmosis is passing in?

I'm also not sure why the schema doesn't use timestamp with time zone.
It should make things a lot easier and allow all components to do automatic
timezone conversion as necessary.  I seem to recall having similar issues
on the main OSM servers.  I can't remember what the final fix was there,
but usually I aim to have everything running in UTC to avoid these issues.

[1] http://www.postgresql.org/docs/9.2/static/datatype-datetime.html
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Wrapper scripts for streaming replication

[Brett Henderson]

Hi Paul,

On 25 May 2013 15:15, Paul Norman penor...@mac.com wrote:

  From: Brett Henderson [mailto:br...@bretth.com]
  Sent: Friday, April 19, 2013 4:12 AM
  Subject: Re: [osmosis-dev] Wrapper scripts for streaming replication
 
  It'd be great to see the replication streaming getting some usage.  It
 doesn't seem to be used at the moment.  In fact it's been down for a couple
 of weeks and it's trying to catch up again now.

 Do you think it's stable enough to base a service on, or am I better off
 sticking with minutely diffs?


 The minutely diffs are used by enough services including OSMF hosted ones
 that you can be fairly confident that any stoppages will be found and fixed
 quickly, but I'm not sure the same is true for streaming diffs.


That's right, we have a chicken and egg problem here.  Minutely diffs are
more stable because they've been worked on for several years with a large
number of consumers.  Replication streaming is fairly new and likely to
take a while to iron out the issues, but that won't occur unless people use
it.

However I'm largely unavailable at the moment with other commitments, so I
don't have much time to help fix things.



  There are a couple of example Osmosis command lines on the following
 wiki page in case you haven't already seen them.
 
 http://wiki.openstreetmap.org/wiki/Osmosis/Replication#Client-side_Streaming

 The pgsnapshot example would work from the shell, but do you have any
 examples for an unattened server?


I don't have any scripts that I can share.  It should be sufficient to
write a wrapper script that redirects output to log files and launches the
osmosis instance.  For extra points it might be worth having it restart
Osmosis if it fails (with some delays included), however Osmosis should
theoretically handle error conditions such as network failures gracefully.



 Also, what's the best way to switch to streaming replication from diffs,
 and the reverse?

 It's a similar question to switching from hourly to minute diffs where
you'll have to figure out the sequence number to begin from, and doing that
will require figuring out an approximate date that is earlier than your
current replication point.  Note that the streaming server does provide
some assistance where it can provide you with a state file matching a
particular timestamp.  A curl example is provided in the URL you've already
read, but more details on all the URLs are available here:
http://wiki.openstreetmap.org/wiki/Osmosis/Detailed_Usage#--send-replication-data_.28--srd.29

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] --read-xml and --sort osmosis operations of Srtm2osm output file failed with error: ...Cannot represent 70730 as a char

[Brett Henderson]

Hi Fedor,

On 28 April 2013 19:11, Fedor Snegov (MTS phone) fsne...@gmail.com wrote:

 Hi, dear osmosis-dev list members

 I am trying to merge osm map file containing Srtm2osm contours of
 interested area ( boundaries='http://www.**openstreetmap.org/?lat=60.22**
 lon=32.04zoom=7layers=Mhttp://www.openstreetmap.org/?lat=60.22lon=32.04zoom=7layers=M'
 ) with osm.pbf Openstreetmaps map file in order to obtain Garmin map.

 I employ osmosis (great tool) for merging procedure. The merging procedure
 fails every time with errors which are presented below.

 For test reasons I am trying simply to convert Srtm2osm program output
 file to pbf format. The command line invoking osmosis is the following:

 ./osmosis-0.43.1/bin/osmosis \
 --read-xml file=$contours_file_name.$**contours_file_extension \
 --sort \
 --write-pbf file=$contours_file_name.osm.**pbf omitmetadata=true
 compress=deflate

 But it stops and gives me again the same error log like this:

 Apr 27, 2013 11:35:22 PM org.openstreetmap.osmosis.**core.Osmosis run
 INFO: Osmosis Version 0.43.1
 Apr 27, 2013 11:35:22 PM org.openstreetmap.osmosis.**core.Osmosis run
 INFO: Preparing pipeline.
 Apr 27, 2013 11:35:22 PM org.openstreetmap.osmosis.**core.Osmosis run
 INFO: Launching pipeline execution.
 Apr 27, 2013 11:35:22 PM org.openstreetmap.osmosis.**core.Osmosis run
 INFO: Pipeline executing, waiting for completion.
 Apr 27, 2013 11:35:42 PM org.openstreetmap.osmosis.**core.pipeline.common.
 **ActiveTaskManager waitForCom
 SEVERE: Thread for task 1-read-xml failed
 org.openstreetmap.osmosis.**core.OsmosisRuntimeException: Cannot
 represent 70730 as a char.
 at org.openstreetmap.osmosis.**core.util.IntAsChar.intToChar(**
 IntAsChar.java:32)
 at
 org.openstreetmap.osmosis.core.domain.v0_6.Way.store(Way.java:200)der.run(XmlReader.java:111)

 snip


 I am not a programmer, but it seems, that the reason for such an  errors
 is some osmosis limit for maximum number of nodes in a way...
 It is clear that for some not small areas with flat landscape Srtm2osm
 could generate a very long ways with huge amount of nodes.

 Please, could you give any suggestions or advices to overcome this problem?


I've just checked in a change which should fix this.  You are right in that
Osmosis was limiting the maximum number of nodes in a way to 2^15.  This
has been increased to 2^31.  It adds a small overhead to Osmosis
processing, but should be barely noticeable.

This change will be available in the next development build.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] FW: question

[Brett Henderson]

On 15 April 2013 00:28, mahmoud ghareebh m.soliman_2...@hotmail.com wrote:

 yes an old version , do i need to remove it?


The plugin is failing to load, so yes I'd remove it.

Again, do you really need to compile Osmosis yourself?



 --
 Date: Sun, 14 Apr 2013 20:56:37 +1000
 Subject: Re: [osmosis-dev] FW: question
 From: br...@bretth.com
 To: m.soliman_2...@hotmail.com
 CC: osmosis-dev@openstreetmap.org


 Hi Mahmoud,

 Is there a reason you're not using the pre-compiled version?
 http://bretth.dev.openstreetmap.org/osmosis-build/osmosis-latest.zip

 The errors you're receiving are unusual.  I see the following message in
 your logs.

 org.openstreetmap.osmosis.core.OsmosisRuntimeException: Cannot load 
 JPF-plugin 'org.mapsforge.MapFileWriter' for extensionpoint 'Task'


 Do you have some custom Osmosis plugins installed somewhere?

 Brett

___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Bad PostgreSQL plans with pgsnapshot

[Brett Henderson]

Hi Paul,

Thanks for the info.  I ran into similar issues when implementing the
Osmosis tasks.

Osmosis contains the following code that runs prior to querying for a
bounding box (ie. --dataset-bounding-box).

// PostgreSQL sometimes incorrectly chooses to perform full table scans,
these options
// prevent this. Note that this is not recommended practice according to
documentation
// but fixing this would require modifying the table statistics gathering
// configuration to produce better plans.
jdbcTemplate.update(SET enable_seqscan = false);
jdbcTemplate.update(SET enable_mergejoin = false);
jdbcTemplate.update(SET enable_hashjoin = false);

Adding those statements was a bit of a hack, but at the time I couldn't get
PostgreSQL to behave properly without them.  Perhaps they'd be unnecessary
with the bugfixes in 9.3.  Unfortunately benchmarking it again is such a
time consuming process.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Osmosis 0.43 Released

[Brett Henderson]

Hi Christian,

On 12 April 2013 06:30, Christian H. Bruhn br...@arcor.de wrote:

 am Freitag, 5. April 2013 um 00:51 schrieb Brett Henderson:

 
 http://bretth.dev.openstreetmap.org/osmosis-build/osmosis-0.43-RELEASE.zip

 Osmosis says:
 INFO: Osmosis Version 0.42-6-gf39a160-dirty
 also if you took the link from the Wiki [1]

 Christian

 [1] http://bretth.dev.openstreetmap.org/osmosis-build/osmosis-latest.zip


Thanks for the info.  I've checked and I have a bug in the gradle script
where it isn't updating the OsmosisConstants class when the git version
number changes.  I'm fairly sure that the release is otherwise correct, the
version number just hasn't been updated correctly.

I'll fix it before I make any more releases.  I probably should do another
release (eg. a 0.43.1) to fix this but I'm not sure when I'll get time.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

[osmosis-dev] OSM Binary (ie. PBF) Changes

[Brett Henderson]

Hi All,

I've just changed how the PBF support is provided in Osmosis.  Up until now
there was a pre-compiled jar called osmpbf.jar checked into the Osmosis
source tree.  It was compiled from Scott Crosby's github project here:
https://github.com/scrosby/OSM-binary

The problem with this is that it prevents me from publishing Osmosis to
Maven Central because people trying to download Osmosis would be unable to
use PBF without also getting a copy of that library.

To get around this I've repackaged the OSM-binary project to build as part
of the Osmosis source tree.  The resultant jar is called
osmosis-osm-binary, and it lives in a package called
org.openstreetmap.osmosis.osmbinary.

I've tried to do this in a way that allows me to keep up to date with the
upstream project.  I've forked the original repository on Github, and
created an Osmosis branch:
https://github.com/brettch/OSM-binary/tree/osmosis

When changes are made to upstream, I should be able to merge master across
to my osmosis branch.  The resultant tree is then checked directly into the
Osmosis source tree.  On my local machine I actually have both git
repositories acting on the same source files so it isn't too painful.  I
have the Osmosis source checked out normally (ie. a .git directory at the
root), and the osmosis-osm-binary project is also a clone of my forked
OSM-binary project (ie. I have a .git directory in there as well).  So far
it seems to work well enough.

If the original OSM-binary project ever gets published to Maven Central
directly then I can stop these shenanigans and depend on it directly.

If anybody has any questions, issues, suggestions, etc let me know.

Brett

PS. I'm getting close to publishing to Maven Central now.  There are no
major blockers left that I'm aware of.
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

[osmosis-dev] Publishing Osmosis to Maven Central

[Brett Henderson]

Hi All,

This may be of interest to some of you.  I've just begun the process of
publishing Osmosis artefacts to Maven Central (ie.
http://repo1.maven.org/maven2).

My current snapshot build is available here:
https://oss.sonatype.org/content/repositories/snapshots/org/openstreetmap/osmosis/

For those who are not familiar with Maven Central practices, it is not
possible to publish directly to Maven Central itself.  The simplest way is
to publish via the OSS Sonatype repository which then gets sync'd with
Maven Central.

A few changes have been made to the Osmosis build to allow this to work
(I'm still in the process of pushing these changes).  The most noticeable
change is that most projects have been renamed to have an osmosis-
prefix.  If you have an existing Eclipse workspace, you'll need to re-run
gradle eclipse and re-import your projects.

The main blocker to publishing release artefacts is that I have two
dependencies on libraries not available in Maven Central.  These are the
scrosby PBF lib, and the BZip2 library which I built manually but which is
based on Apache source code.  Both should be fixable by building and
publishing them along with the rest of Osmosis, but I need to find time to
do so.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] OSM entity processing order

[Brett Henderson]

Hi Nicolas,

On 12 March 2013 20:28, Nicolas Colomer ncolo...@ymail.com wrote:

 Hi Osmosis community!

 When I manipulate an OSM file (compressed or not) using Osmosis, can we
 assume that entities will systematically be processed in this order:
 1.bound, 2.node, 3.way, 4.relation?

 This seems logical since the OSM file format guarantees that blocks come
 in this order (see the OSM XML #File 
 formathttp://wiki.openstreetmap.org/wiki/OSM_XML#OSM_XML_file_format wiki
 page).

 In addition, I reach a 
 posthttp://lists.openstreetmap.org/pipermail/osmosis-dev/2011-March/000949.html
  where
 Brett told:

  This is due to the way Osmosis processing works because it finishes
 processing nodes before it sees the ways.

 I just want to make sure my impression is good :)


You are correct.  Planet files, and just about any other file you may
download will be in that order.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] 32-bit limit in IdTrackers

[Brett Henderson]

On 7 February 2013 20:43, Brett Henderson br...@bretth.com wrote:


 On 7 February 2013 20:21, Ilya Zverev zve...@textual.ru wrote:

 I'd like idTrackerType removed, since after tomorrow using it will
 certainly cause osmosis to fail. I doubt this options was useful, and it
 only puzzled those who were learning osmosis, sometimes being the only
 option besides inPipe and outPipe.


 I've always avoided removing options unless necessary to avoid breaking
 existing scripts, but in this case I agree that it would be better
 removed.  It might be nice if Osmosis had a way of deprecating options so
 that warnings would be displayed if they were used.  I'm a bit time
 constrained at the moment but happy to accept patches/pull requests ;-)


I've removed the option.  It is still available in 0.42 to allow a smooth
transition from 0.41 which requires it to be specified for --used-xxx
tasks, but it will not be available in 0.43.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] 32-bit limit in IdTrackers

[Brett Henderson]

On 13 February 2013 19:10, Jukka Laaksola laaks...@iki.fi wrote:

 13.02.2013 01:11, Brett Henderson wrote:
  On 12 February 2013 23:57, Jukka Laaksola laaks...@iki.fi
  mailto:laaks...@iki.fi wrote:
  There is workaround to use option --used-node idTrackerType=Dynamic,
  but it is not documented. As I can see it is working also with version
  0.41. And my tests also looks like Dynamic will work with the
  version 0.41.
 
 
  The lack of documentation is easy to fix :-)  Would you mind updating
  the 0.41 wiki page?

 Have not had real knowledge of the Dynamic feature in 0.41. Just tried
 it and it looked like it's working. Now I got the answer from you that
 it's really undocumented feature. Perhaps I can create an account to
 wiki and update the page...


Great, it looks like somebody has beaten me to it.



  The other option is to use a nightly build ...

 Is there any place to download nightly builds? Osmosis wiki page
 Development Build links are broken.

 http://wiki.openstreetmap.org/wiki/Osmosis#Development_Build

 Zip Files:

 http://dev.openstreetmap.de:23457/hudson/job/osmosis/lastSuccessfulBuild/artifact/package/distrib/zips/

 Tar Balls:

 http://dev.openstreetmap.de:23457/hudson/job/osmosis/lastSuccessfulBuild/artifact/package/distrib/tgzs/


Oh, I didn't realise they were broken.  I think it has been caused by the
recent change to a gradle build system.  I've fixed the wiki.  The link for
both zip and tgz files is:
http://dev.openstreetmap.de:23457/hudson/job/osmosis/lastSuccessfulBuild/artifact/package/build/distribution/

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] 32-bit limit in IdTrackers

[Brett Henderson]

Hi IZ,

On 6 February 2013 20:05, Ilya Zverev zve...@textual.ru wrote:

 Hi! As some of you have read (http://lists.openstreetmap.**
 org/pipermail/dev/2013-**February/026495.htmlhttp://lists.openstreetmap.org/pipermail/dev/2013-February/026495.html),
 in three days node ids are expected to surpass 2147483647, and this
 method https://github.com/**openstreetmap/osmosis/blob/**
 master/core/src/main/java/org/**openstreetmap/osmosis/core/**
 util/LongAsInt.java#L30https://github.com/openstreetmap/osmosis/blob/master/core/src/main/java/org/openstreetmap/osmosis/core/util/LongAsInt.java#L30will
  throw an exception Cannot represent  + value +  as an integer. It
 is used in every IdTracker implementation, so id trackers will become
 unusable.

 This will affect tag and area filters. Regional extracts that are made
 with osmosis will break. There is a comment at the start of each IdTracker
 class: The current implementation only supports 31 bit numbers, but will
 be enhanced if and when required. I guess, now is the time. Can anybody
 fix that? There must be a reason why this hasn't done sooner.


Thanks for the heads up.  I could be wrong but I don't think this is an
issue.

It is used by ListIdTracker and BitSetIdTracker so those implementations
will soon fail if you try to use them.  However, the default implementation
is now DynamicIdTracker which doesn't suffer from this issue (I hope ;-).

DynamicIdTracker breaks the id range into chunks of 1024 and internally
uses either ListIdTracker or BitSetIdTracker for each of those chunks
depending on which is more efficient.  As a result, the largest number
either of those id trackers ever sees is 1023, the DynamicIdTracker adds a
base offset to those numbers to get the final number and it stores the base
number as a 64-bit long.

The idTrackerType arguments could probably be removed from the
--bounding-box and --bounding-polygon tasks now because the default
implementation should be better than specifying one in the vast majority of
cases.

I've checked the rest of the codebase for use of the LongToInt class.  It
is used internally by the Entity class to store a changeset id as a 32-bit
number instead of a 64-bit number.  That should be safe for a while yet.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Eclipse Setup, Missing task types

[Brett Henderson]

Hi Oliver,

On 05/02/2013, at 1:42 AM, Oliver Schrenk oliver.schr...@gmail.com wrote:

 Hi,

 Are there more current notes about how to setup Eclipse for osmosis 
 development then the notes in [1] ?

That's all there is at the moment. It should be mostly correct but
might be missing a step.


 I know that ant has been deprecated in favor of gradle, so I installed 
 Eclipse Grade Support via [2] and

$ git clone https://github.com/openstreetmap/osmosis
$ cd osmosis
$ ./gradlew assemble

I personally don't install any gradle support so I'm not sure what it
does. It should be sufficient to run:
$ ./gradlew eclipse

The above command will generate/update the eclipse projects which can
then be imported into eclipse without requiring any special plugins.


 and proceeded to import osmosis' multi-modules using `File  Import  
 Gradle`. Everything compiles fine.

 But when I try to execute a command like

osmosis --read-xml file=bremen.osm.bz2 --write-apidb-0.6 
 host=127.0.0.1 database=api06_test user=osm password=osm 
 validateSchemaVersion=no

 using a Run Configuration with `org.openstreetmap.osmosis.core.Osmosis` as 
 the main class
 and

That is correct, but you can't run it from the Core project. The Core
project doesn't have access to any plugins. In your case you need to
run it from the Apidb project which contains the apidb tasks and also
has access to the XML tasks. You should be able to modify your
existing launch configuration to start from a different project.

Hopefully that's the step you've missed.


--read-xml file=bremen.osm.bz2 --write-apidb-0.6 host=127.0.0.1 
 database=api06_test user=osm password=osm validateSchemaVersion=no

 as program arguments I get

Feb 04, 2013 3:31:39 PM org.openstreetmap.osmosis.core.Osmosis run
INFO: Osmosis Version 0.41-55-gb44b7d7-dirty
Feb 04, 2013 3:31:39 PM org.openstreetmap.osmosis.core.Osmosis run
INFO: Preparing pipeline.
Feb 04, 2013 3:31:39 PM org.openstreetmap.osmosis.core.Osmosis main
SEVERE: Execution aborted.
org.openstreetmap.osmosis.core.OsmosisRuntimeException: Task type read-xml 
 doesn't exist.
at 
 org.openstreetmap.osmosis.core.pipeline.common.TaskManagerFactoryRegister.getInstance(TaskManagerFactoryRegister.java:60)
at 
 org.openstreetmap.osmosis.core.pipeline.common.Pipeline.buildTasks(Pipeline.java:51)
at 
 org.openstreetmap.osmosis.core.pipeline.common.Pipeline.prepare(Pipeline.java:112)
at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:86)
at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:37)

 It doesn't seem to pickup the various tasks.

 My end goal is to debug write-apidb-0.6 as I'm trying to write data to an 
 unsupported database and run into problems with duplicate user entries and 
 want to use Eclipse's Debugger to go through the code.

I think you're close. Let me know how you go.

Brett


 Best regards
 Oliver


 [1] http://wiki.openstreetmap.org/wiki/Osmosis/Development#Eclipse_Setup
 [2] 
 http://static.springsource.org/sts/docs/latest/reference/html/gradle/installation.html


___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] cleaning up osmosis temp files?

[Brett Henderson]

Hi Brian,

On 18 January 2013 08:35, Brian Cavagnolo bcavagn...@gmail.com wrote:

 Is there a recommended way to clean up the temp files left behind by
 osmosis?  I've been just poking around the /tmp directory blowing away
 the copy* and nodeLocation* files.


Firstly, Osmosis shouldn't leave temporary files around unless it crashes
or is aborted.  If files are being left lying around after normal operation
that would indicate a bug.

However, if you want a simpler way to keep track of temporary files, it may
be worth creating them in a special Osmosis specific location by modifying
the TEMP environment variable.  That would make it easier to identify files
created by Osmosis.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] pgsimple/pgsnapshot possible bug

[Brett Henderson]

Hi Daniel,


On 25 January 2013 07:47, Daniel Kaneider kaneiderdan...@gmail.com wrote:

  Hi,

 I did some import of OSM data into a Postgresql 9.2 DB using osmosis 0.41.
 The pgsnapshot_load script stopped since some function could not be found
 (Envelope,Collect). If I am not wrong then

 UPDATE ways SET bbox = (
 SELECT Envelope(Collect(geom))
 FROM nodes JOIN way_nodes ON way_nodes.node_id = nodes.id
 WHERE way_nodes.way_id = ways.id
 );

 should be changed to

 UPDATE ways SET bbox = (
 SELECT *ST_Envelope*(*ST_Collect*(geom))
 FROM nodes JOIN way_nodes ON way_nodes.node_id = nodes.id
 WHERE way_nodes.way_id = ways.id
 );

 This should apply also to the pg_simple script.


Thanks for the message.  I suspect you're right.  If nobody beats me to it
I'll fix it as soon as I can.

Note however, that most times you don't need those SQL statements and can
comment them out.  If you use Osmosis to generate the COPY data files it
should be able to populate the bbox and/or linestring columns for you.
That should be *much* faster than having PostgreSQL do it for you.

Check out the --write-plsql-dump task and the enableBboxBuilder,
enableLinestringBuilder options.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Osmosis plugin development and integration tests

[Brett Henderson]

Hi Nicolas,

On 12 January 2013 09:33, Nicolas Colomer ncolo...@ymail.com wrote:



 So I tried to reproduce these tests but I face some class loading issue
 (always the same Exception, even for simple commands that involve only xml
 reading / writing):

 java.lang.NoClassDefFoundError: org/java/plugin/PluginLifecycleException
 at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:80)
 at


It looks like you're missing a required library.

This class is located in the jpf (Java Plugin Framework) library which is
included as a dependency of the core project.  If you're including the core
project/lib as a dependency then you should automatically get jpf as a
transitive dependency.

Are you using gradle to build your project?



 Do you have any idea to solve this?
 How declare and load my plugin programmatically so that I can use it via
 command parameters? Do I have to add some resources in my classpath (as for
 Osmosis, when you declare pluginLoader class in the
 $OSMOSIS_HOME/config/osmosis-plugins.conf file)?


Your project will need an osmosis-plugins.conf file containing the name of
your plugin loader class, but otherwise that should be all you need so long
as you have all the necessary jars available on your classpath.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] deadlock when extracting bounds with 0.41 but not 0.40.1

[Brett Henderson]

Hi Eric,


On 13 December 2012 23:18, Eric Fernandez z...@zebulon.org.uk wrote:

 Hi,

 This is a follow-up to the thread here:

 http://lists.openstreetmap.org/pipermail/osmosis-dev/2012-September/001333.html

 I am experiencing the same hanging issue with osmosis 0.41 with the GB
 map from , which I did not have with osmosis 0.40.1.

 The command I use is the same one (as described in

 http://wiki.openstreetmap.org/wiki/Mkgmap/help/options#Using_precompiled_bounds_for_the_address_index
 ):

 ./osmosis-0.41/bin/osmosis --read-pbf file=great_britain.osm.pbf
 outPipe.0=1 --tee 2 inPipe.0=1 outPipe.0=2 outPipe.1=3 --buffer
 inPipe.0=3 outPipe.0=4 --buffer inPipe.0=2 outPipe.0=5 --tag-filter
 accept-relations boundary=administrative,postal_code inPipe.0=4
 outPipe.0=6 --used-way inPipe.0=6 outPipe.0=7 --tag-filter
 reject-relations inPipe.0=5 outPipe.0=8 --tag-filter accept-ways
 boundary=administrative,postal_code inPipe.0=8 outPipe.0=9 --used-node
 inPipe.0=9 outPipe.0=10 --used-node inPipe.0=7 outPipe.0=11 --merge
 inPipe.0=10 inPipe.1=11 outPipe.0=12 --write-pbf
 file=great_britain-boundaries.osm.pbf omitmetadata=true
 compress=deflate inPipe.0=12

 In his answer, Andrew Byrd explains there may be a deadlock problem
 with the inputs for the merge, but this command was working with
 osmosis-0.40.1, and normally is dealing with this issue already. Is
 there any reason that the new 0.41 version leads to the issue?


Andrew is correct, the deadlock is expected behaviour with your pipeline
structure.  Splitting data with a --tee and then recombining with a --merge
should not be done in the same pipeline.

As to why it works in 0.40.1 and not 0.41 I'm not sure, but there were
major changes made to the DataPostbox class which is used as a data buffer
between threads.  It is possible that these changes have triggered your
deadlock condition, although I'm not aware of anything specific.  It *may*
be possible to avoid the deadlock by specifying a large bufferCapacity on
your --buffer tasks, but this isn't recommended as you'll probably still
run into deadlock conditions at some point in the future.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Gradle Build Scripts

[Brett Henderson]

On 2 December 2012 00:40, Paweł Paprota ppa...@fastmail.fm wrote:

 Hi Brett,

 Just to let you know, I've just merged your master to my OWL plugin[1]
 fork, switched the plugin to Gradle and everything works smoothly!

 [1] https://github.com/ppawel/**osmosishttps://github.com/ppawel/osmosis

 Excellent.  I hope it didn't hurt too much :-)

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Invalid ways, now with code

[Brett Henderson]

Hi Toby,

On 12 December 2012 18:33, Toby Murray toby.mur...@gmail.com wrote:



 Anyway, thoughts? My changes are on github in the invalid_geometry branch:
 https://github.com/ToeBee/osmosis

 And a diff of all my changes:
 https://github.com/ToeBee/osmosis/compare/master...invalid_geometry


It looks good to me.  If nobody objects over the next day or two I'll merge
it.  Feel free to give me a nudge if I forget.

Brett

PS.  Minor nitpick.  Can you delete the commented out code in
WayGeometryBuilder?

//int numValidNodes = 0;
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Single node ways

[Brett Henderson]

Hi Toby,

On 20 November 2012 16:30, Toby Murray toby.mur...@gmail.com wrote:

 I have started playing with this in a branch:
 https://github.com/ToeBee/osmosis/tree/invalid_geometry

 So far I managed to add a new command line option keepInvalidWays to
 the write-pgsql(-dump) tasks. It defaults to false so that current
 functionality of dropping zero and single node ways is preserved.


I'd actually be happier if the default position is to include all data
regardless of whether it is accurate or not.  I feel that is the behaviour
of least surprise, even if it is different to today.  The pgsnapshot schema
is intended to be a complete and accurate representation of OSM data, so
any deviation from that should be explicitly selected by the user.

Would it make sense to separate the dropping of ways from the dropping of
linestrings?  Perhaps two options like keepInvalidWays and
keepInvalidLinestrings?  If keepInvalidWays=true, but
keepInvalidLinestrings=false then the linestring column could be set to
NULL if the way doesn't contain two or more nodes.

The reason I say this is because there may be cases where you want an
accurate copy of OSM data, and only include the linestring as a means of
performing geo-spatial queries against that data.

Thoughts?  Either way, the keepInvalidWays should solve most current
issues, so is a higher priority.


 I think it should be possible to add it to the diff processing as well
 to keep invalid ways out of a replicated database. I likely won't get
 a chance to work on it much more before next week because of
 Thanksgiving travel plans but I may try to reimport my database with
 this option enabled while I am out of town to see exactly what effects
 it has.


Sounds great.  I'm keen to ensure that initial import and replication are
consistent, even if they're not today.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Gradle Build Scripts

[Brett Henderson]

Hi All,

I've just pushed my gradle-based build changes to the main Git repository.
https://github.com/openstreetmap/osmosis

Please note that if you're using Eclipse you will need to run ./gradlew
eclipse before you can open the projects.  It may be a good idea to run
./gradlew cleanEclipse first to ensure there are no modified eclipse
files lying around.

If you have any problems please let me know.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] --truncate-apidb also truncates users

[Brett Henderson]

Hi Shaun,

On 16 November 2012 22:17, Shaun McDonald sh...@shaunmcdonald.me.uk wrote:

 Based on the docs at

 http://wiki.openstreetmap.org/wiki/Osmosis/Detailed_Usage_0.42#--truncate-apidb_.28--td.29
 for --truncate-apidb, I wasn't expecting that it would truncate the user
 table and all the other related tables, that are no API docs.

 However based on the code at

 https://github.com/openstreetmap/osmosis/blob/master/apidb/src/main/java/org/openstreetmap/osmosis/apidb/v0_6/ApidbTruncator.java#L23
 it does in fact truncate the user table too. Is this intentional
 behaviour? Could it be change such that only Osmosis generated users are
 removed or users are left behind?


The documentation could probably be more explicit, but it will truncate any
table that Osmosis populates during import.  The idea is that if you have a
populated database, you need to run truncate before you can perform a fresh
import.

The import process is very dumb and assumes an empty database.  Trying to
check for existing data would be more complicated and slower.

The current import process is just using insert statements, but longer term
I'd like to see it move over to using the PostgreSQL COPY command.  COPY is
*much* faster and is already used by the pgsnapshot schema.  If COPY is
used then the database tables must be empty before import anyway.  This is
another reason that I'm hesitant to add any existing user checks to the
current implementation.

Why is this causing you problems?  If you have special needs, performing a
manual truncate is fairly trivial.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Invalid actions during replication

[Brett Henderson]

Hi Paweł,

On 11 November 2012 21:24, Paweł Paprota ppa...@fastmail.fm wrote:

 Hi all,

 On Friday I have implemented a simple validation in the OWL plugin that
 checks whether given entity action (create/modify/delete) can be applied to
 the current database state. See implementation: [1].

 By invalid action I mean a situation when there is a delete action in
 the pipeline but there is no entity in the database to delete. Or modify
 action with version X when there is no version X - 1 in the database but
 some other version.

 I've been running replication for over a day now (in LOG mode, on a
 database with planet imported) and results are quite interesting. There is
 more invalid actions reported than I thought there would be. Some random
 examples:

 INFO: Invalid action: Node 1342627235 - Trying to modify entity that does
 not exist!
 INFO: Invalid action: Node 964649444 - Trying to modify entity that does
 not exist!
 INFO: Invalid action: Way 11133255 - Trying to modify entity that does not
 exist!
 INFO: Invalid action: Way 9105159 - Trying to modify entity that does not
 exist!

 I have not yet analyzed closely those entities that are reported but in
 general there should be no invalid actions at all on a full planet database
 with correct replication setup, am I correct in this assumption?


It's possible that there could be errors during initial replication, but I
would have thought they'd disappear eventually.  It seems a bit odd.

Unfortunately you will need to look into some examples one at a time to try
to trace the source of the issue.  It can be tedious to get to the bottom
of replication problems.  The pgsnapshot replication functionality has been
around for a long time now and has been fairly reliable, but perhaps
there's an issue in there somewhere.



 Also, if anyone's interested, this functionality would be trivial to port
 over to the pgsnapshot Osmosis plugin (--write-pgsql-change) since the OWL
 plugin is built from that plugin. I can do this if you think it could be
 useful.

 [1] https://github.com/ppawel/**osmosis/blob/owldb-plugin/**
 owldb/src/main/java/org/**openstreetmap/osmosis/owldb/**
 v0_6/impl/ChangeWriter.java#**L256https://github.com/ppawel/osmosis/blob/owldb-plugin/owldb/src/main/java/org/openstreetmap/osmosis/owldb/v0_6/impl/ChangeWriter.java#L256

 Implementation notes: basically there is a invalidActionsMode paramater
 for the --write-owldb-change task (which is derived from
 --write-pgsql-change but with OWL-specific additions) that can be IGNORE
 (does nothing), LOG (logs information about the action) or BREAK (stops
 processing when invalid action occurs).


I don't mind something like this being added, but I'd like to understand
the source of your current issues first.  If it turns out that it's normal
behaviour and can't be fixed then there's not much point raising warnings.
On the other hand, if there is a real issue here then it will be valuable.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

[osmosis-dev] Gradle Build Scripts

[Brett Henderson]

Hi All,

I'm keen to switch the Osmosis build over to using the Gradle build tool
instead of Ant.  I've just checked in some new Gradle scripts.  The current
Ant build works reasonably well, but I've lost many hours getting it to its
current point, and introducing new features is always a lot of effort.

The new scripts are on my github repo on the gradle branch.  There are some
basic instructions on its use in the project README.
https://github.com/brettch/osmosis/tree/gradle

Gradle for those who don't know it is a newer build tool written in the
Groovy language.  It provides its own DSL (Domain Specific Language) for
defining build scripts, but anything not directly supported is a short bit
of Groovy script away.  It provides a number of plugins for common tasks
such as java builds and eclipse IDE support.  As such it provides a lot of
the power of Maven, but is much less rigid when it doesn't quite meet your
needs.

Below are some pros and cons.

Advantages of Gradle:
* Simpler multi-project build support.
* Greatly reduced custom build scripting.
* Eclipse project files are auto-generated (and removed from source
control).
* Source is included on all dependencies in Eclipse (great debugging aid).
* Considerably faster build times (on my desktop a build excluding tests
takes 6 seconds versus 47 seconds in ant, full build just over 1 minute
versus just over 2.5 minutes for ant).
* No need to publish and resolve artefacts between projects using Ivy which
leads to faster builds and less consumed disk space.
* Very effective conditional compilation, much faster during development.
* Good set of pre-built plugins.
* No need to install gradle, the wrapper script bootstraps itself to the
correct version.

Disadvantages of Gradle:
* Much more magic involved, and less readable.
* Can't tweak build order to move tests as late as possible in build
process.
* Project names in Eclipse no longer have an Osmosis prefix.

There are probably other disadvantages that I haven't tripped over yet, but
so far it is working well.

If anybody has any thoughts either positive or negative please let me know.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Gradle Build Scripts

[Brett Henderson]

Hi Greg,

On 12 November 2012 03:24, Greg Troxel g...@ir.bbn.com wrote:


 Does gradle require anything else (other than a JDK/JRE) in terms of
 dependencies?


No.  You just need a JDK 1.6 or higher.  It has less dependencies than the
Ant build because it requires Ant to be installed along with a number of
ant modules.  Gradle on the other hand will bootstrap itself using a
wrapper script included in the source tree.  You can however use a locally
installed gradle (invoked via gradle instead of ./gradlew) if you wish.



 If someone doesn't use eclipse, does using gradle become more awkward?


Gradle has no dependency on Eclipse.  It just helps configure your Eclipse
environment.  In the existing Ant build, the Eclipse project files are all
manually maintained and have to be manually kept in sync with the Ivy
dependencies.  This change means that you have to run a gradle eclipse
command before you can import the projects into Eclipse.



 Is Groovy implemented in java?  If not, what barriers does this raise
 (beyond the existing java-isn't-really-portable issues)?


Yes, it's a JVM-based language.  I've been running it on Windows, Mac and
Linux for a while now with no issues.



 Does this add a requirement to have a network connection to do builds?


It requires a network connection, however this was always the case.  The
existing Ant/Ivy build also requires a network connection.  It should be
possible to copy a gradle cache directory between machines in a similar
fashion to the current ivy cache if you wish to operate in disconnected
mode.

Note that Gradle uses Ivy internally anyway, so under the covers the Gradle
and Ant build have many similarities in how they retrieve dependencies.



 Does this whole scheme have enough mindshare that it is really clear
 that it will be staying around, rather than being a flash in the pan?
 (E.g., arch was a really interesting VCS, but it's gone.  It pretty
 clearly inspired others, but AFAIK no one uses it any more.)


I wouldn't call Gradle a flash in the pan.  It's been around for several
years now and is continuing to become more popular.  A number of
well-recognised projects use it now including Spring.  I've used in on a
couple of commercial projects now and have found it to work quite well.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Incorrect string value: '\xF0\x90\x8D\x83\xF0\x90...' for column 'v' at row 6

[Brett Henderson]

On 8 November 2012 20:40, Simon Nuttall i...@cyclestreets.net wrote:

 Regarding my initial request on this:

 http://lists.openstreetmap.org/pipermail/osmosis-dev/2012-November/001409.html

 I have fixed the problem by updating our debian server, which has
 changed the MySQL version from 5.1.49-3-log  to 5.1.63-0+squeeze1-log.


That's a great result.  I wasn't sure where to begin with this error.  It's
been a while since I've had to tackle character set issues, I was hoping
they'd gone forever :-)

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Single node ways

[Brett Henderson]

Hi Toby,

On 5 November 2012 16:25, Toby Murray toby.mur...@gmail.com wrote:

 It seems osmosis intentionally drops ways with only one node in them
 when importing[1]. From another thread[2] on this mailing list, I
 understand this is because postgis does not handle such things very
 well. However, when consuming diffs, single node ways do make it into
 the database. In some situations this seems to cause errors when
 querying the data, but not every time.

 So first of all, there is an inconsistency between how the load
 process and the diff process handles these ways. But is there any
 other way to handle this besides dropping them completely? There are a
 couple of uses of the pgnsapshot schema that assume that it is a
 lossless way of storing OSM data and as far as I know, it is - except
 for this one exception. While single node ways are definitely a bug,
 they happen with unfortunate regularity, mostly due to bugs in editing
 software. Would it be acceptable to add in a phantom node 3cm away
 from the first oneor something, just while building the geometry? I
 think that happens in SQL while applying diffs which would make it
 more difficult since you don't actually want to insert nodes that
 don't exist... Thoughts?

 [1]
 https://github.com/ToeBee/osmosis/blob/master/pgsnapshot/src/main/java/org/openstreetmap/osmosis/pgsnapshot/v0_6/impl/CopyFilesetBuilder.java#L186
 [2]
 http://gis.19327.n5.nabble.com/osmosis-dev-one-node-ways-td5338301.html


Dropping the single node ways is messy.  A phantom node in the generated
linestring might be a better workaround.  If nobody has any better ideas
I'm open to it being changed.  Are you able to create a patch?

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Android?

[Brett Henderson]

Hi Adrià,

On 7 November 2012 03:41, Adrià Ribatallada Torelló
achtungw...@gmail.comwrote:

 Hi there!

 I was trying to use the osmosis tool in an android device, and
 after including all the required libraries to an android project and make
 it compile and install in the device I'v encountered some runtime
 obstacles that made me consider that the best approach might be to adapt
 the osmosis source code, adapting it to use the android libraries.

 (the most severe obstacle is a No validating SAXParser implementation
 available while trying to read the file plugin.xml. Is the same error
 explained here with more detail by this other guy:
 http://stackoverflow.com/questions/1030/trying-to-use-osmosis-in-android-environment-duplicate-file-error-with-jar-file/13251374#13251374
 ).


I'm not sure if this helps, but the plugin.xml loading code is used by JPF
(Java Plugin Framework) support.  JPF provides one way of packaging Osmosis
plugins, but is rarely used.  The majority of plugins use a simpler
mechanism involving a properties file called osmosis-plugin.conf bundled in
each plugin jar file.  If you don't need JPF you could remove the JPF code
from Osmosis completely and eliminate the error you're receiving.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Osmosis OSM wiki content

[Brett Henderson]

Hi Jeff,

On 22 October 2012 08:10, Jeff Meyer j...@gwhat.org wrote:

 Hi - n00b here.
 There's a lot of good content on the wiki about Osmosis, but as a new guy
 (fng), it seems like it could be organized a little more cleanly. I've put
 together some suggestions at:
 https://wiki.openstreetmap.org/wiki/Talk:Osmosis#Wiki_Osmosis_organization
 The intent isn't to change things too much, and certainly not to delete
 anything, but to consolidate similar information and reduce redundancy of
 page purpose a little.
 If there's support and/or suggestions of alternate approaches, I'd be glad
 to take care of it.
 If it's a bad idea, I'll sit tight.
 Also... if this isn't the right way to propose wiki mods, please let me
 know.


There's no right way to propose anything, this seems like as good a way as
any :-)

Regarding the wiki structure, about the only page I really care about is
the detailed usage page because it is the one page that I try to keep 100%
accurate and up to date.  I'm very hesitant to split it up, re-arrange it,
etc.  I take your point about the name detailed usage reading more like a
man page than a usage page, it's just a name that seemed like a good idea
at the time.  I like the content of the page, but I don't particularly care
what it's called.

As for the rest of the pages, I haven't been a great curator of the wiki so
I won't get in the way of somebody trying to improve it.  I'm also not the
best judge of what users need to know.  There is a bunch of stuff on there,
some better organised than others.  The various examples are based on
whatever people have found useful at the time so it's grown fairly
organically.  In particular some of the older stuff like API 0.5 support,
Java 1.5 support, Notes, etc are a bit distracting and could possibly move
elsewhere.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Osmosis error

[Brett Henderson]

Hi Laura,

On 19 October 2012 23:56, Laura Passigni laurapassi...@yahoo.it wrote:

 Hi Brett,
 thank you for your answer.
 Probably the problem is related to the huge size of the database.

 In fact the total database is about 850GB. Currently there is about 1
 billion of records in current_nodes and about the same size in
 current_way_nodes tables.


I've done some testing with the FileBasedSort class with large numbers of
sort items.  It turns out there are some issues with it that *might* have
caused your issue.  FileBasedSort should only use up to about 512 file
handles in a worst case, but in fact was using considerably more than that
on large sorts.  I would be surprised if it used anywhere near 32758
though.  On the other hand there might be several sorts being processed at
the same time which might make the problem worse.

Anyway, I've addressed the issue and committed a fix.  It is included in
the latest development build.  Try it out and let me know how it goes.
http://wiki.openstreetmap.org/wiki/Osmosis#Development_Build

If that fails we may need to monitor the files being opened by Osmosis.
You can list files held open by a process using the lsof -a -p
process_id command.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] java.lang.AbstractMethodError: org.openstreetmap.osm.data.osmbin.v1_0.OsmBinV10Writer.initialize

[Brett Henderson]

Hi Kim,

On 15 October 2012 19:18, KHOO KIM HWA khoofu...@yahoo.com wrote:

 Thanks Brett,

 I downgraded to osmosis 0.39 and the osm to osmbin file worked.

 But there are only attrnames, nodes.id2, nodes.idx, nodes.obm and
 osmbin.properties which are generated. The corresponding files of *.id2,
 *.idx and *.obm for ways and relations are not generated. I found
 osmbin.version=v1.1 in the osmbin.properties.

 Can someone give me some ideas on how to generate the corresponding files
 for ways and relations?


Sorry, I can't help.  The osmbin plugin is developed separately to Osmosis
and I haven't used it.  Perhaps others can comment.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] java.lang.AbstractMethodError: org.openstreetmap.osm.data.osmbin.v1_0.OsmBinV10Writer.initialize

[Brett Henderson]

Hi Kim,

On 13 October 2012 00:36, KHOO KIM HWA khoofu...@yahoo.com wrote:

 Hello,

 I got some errors when I try to convert .osm to osmbin file by executing:
 osmosis --read-xml file=idf_main_roads_new.osm --write-osmbin-0.6
 dir=./osmbin-full-map-dir

 Thread for task 1-read-xml failed
 java.lang.AbstractMethodError:
 org.openstreetmap.osm.data.osmbin.v1_0.OsmBinV10W
 riter.initialize(Ljava/util/Map;)V
 at
 org.openstreetmap.osmosis.xml.v0_6.XmlReader.run(XmlReader.java:95)
 at java.lang.Thread.run(Unknown Source)

 Can someone give me some ideas to fix the problems?


This error appears to be coming from an Osmosis plugin, not Osmosis
itself.  The plugin will need to be updated to support the latest version
of Osmosis.  If you can't get a newer version of the plugin you'll need to
downgrade to version 0.39 of Osmosis.  Recent versions of Osmosis require
all tasks to support a new initialize method.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Streaming Replication

[Brett Henderson]

Hi Jochen,

On 14 October 2012 06:17, Jochen Topf joc...@remote.org wrote:

 Very interesting work!

 How do you handle new incoming requests. They have to start from a known
 point
 so I guess you have to do an SQL query for each of them? Or do you just
 read
 the existing .osc files from disk and stream them out? This could take a
 long
 time...


It just reads existing .osc files.  The server-side is made up of two
processes.  The first process extracts data from the database and writes
.state.txt and .osc files in a similar way to existing replication but it
runs continuously using a single database connection.  The second process
serves the data to clients.  The two processes talk via an internal
HTTP-based channel so that the extracter can notify the data server when
new intervals have been processed.  The two processes can be run in a
single Osmosis process, but I usually run them separately.

It could take a long time to download a long time interval, but it is
perhaps faster than you'd expect.  I suspect that in most cases the
bottleneck will be client side trying to consume the data.  I've thought
about writing the data into a database instead, but it's more effort to
both develop and manage.  I'm planning to wait to see if it becomes an
issue.  Downloading bulk data should be relatively rare because most
connected clients should be up to date and just waiting for new data to
arrive.

One other thing to note is that it supports a tree of servers where one
master server feeds data to any number of slaves which in turn feed data to
end users.  Example commands are here:
http://wiki.openstreetmap.org/wiki/Osmosis/Replication#Streaming_Caching

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

[osmosis-dev] Streaming Replication

[Brett Henderson]

Hi All,

For those of you who currently use the minute diffs to keep a local
database up to date, you may be interested to know that a new form of
replication has hit the street.

The current replication system is based on a series of static replication
files that are placed on a web server for clients to download as described
here:
http://wiki.openstreetmap.org/wiki/Planet.osm/diffs#Using_the_replication_diffs

It is a very simple mechanism and works well for the existing daily, hourly
and minutely replication feeds.  Unfortunately it doesn't work well for
sub-minute replication because it becomes far too chatty.  On the server
side, the current feeds are generated from cron which also works well down
to one minute intervals, but the overhead of launching a new process and
connecting to the database for every replication interval also becomes too
inefficient for shorter intervals.

To solve this, a new streaming replication mechanism has been developed.
Under the covers the same database queries are utilised, but the process
performing the queries runs continously and polls the database for changes
at a shorter interval.  It is currently set to poll every 10 seconds, but
it can be reduced further if required.  The network transport is also
continuous and holds a single HTTP connection open for the lifetime of
communication between the server and client.  It is all implemented within
the latest version of Osmosis 0.41.  If you wish to experiment with the
server-side tasks however, several bugs have been fixed in the latest
development version.  Internally it uses the JBoss Netty framework which
means that it's all event-driven (ie. doesn't require a thread per client)
and should theoretically support a large number of concurrent clients.

To quickly see this in action, point your browser at this URL and you
should see new replication state data become available approximately
every 10 seconds.
http://planet.openstreetmap.org/replication/streaming/replicationState/current/tail

New Osmosis tasks have been developed to consume this data.  For some basic
instructions to help you get started, refer to this link:
http://wiki.openstreetmap.org/wiki/Osmosis/Replication#Client-side_Streaming

If you don't wish to use Osmosis, some basic documentation on the wire
protocol is available here:
http://wiki.openstreetmap.org/wiki/Osmosis/Replication#Streaming_Replication_Wire_Protocol

This is very much experimental and bugs will undoubtedly be encountered to
please be wary about trusting it to update your database if you've just
spent two weeks importing a planet file.  However, I'd love to see it get
some usage and would welcome any feedback.  This is not intended for use in
updating a local planet file as the existing daily files are better suited
to that.  For databases that can tolerate a minute delay, the existing
mechanism is very simple and has proven to be fairly reliable.  But if you
really need current access to data, and can cope with the additional
complexity, this should be useful.  The current 10 second delay is not a
lower limit, but is a good starting point for now.

Cheers,
Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Problems with the tag transform plugin

[Brett Henderson]

I've taken a look at tagtransform and added it to the main Osmosis codebase
here to get it working again:
https://github.com/brettch/osmosis/tree/tagtransform

Note that I have violated the original GPL licence by copying it into the
Public Domain Osmosis codebase so this may only stay here temporarily.
I've asked Dave separately to see if he'll allow it to be moved into Public
Domain.  If he's not comfortable with that it will need to remain separate.

I've made a large number of changes to the codebase, however the actual
compilation problem preventing compilation against 0.41 is fixed in this
commit.  This commit could probably be applied to the existing SVN-based
tagtransform codebase.
https://github.com/brettch/osmosis/commit/25711901df41e6950613dea901e3a15ad7ce93a1
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Error with read-pbf

[Brett Henderson]

On 20 September 2012 05:59, Stephan Knauss o...@stephans-server.de wrote:

 On 18.09.2012 13:07, Brett Henderson wrote:

 On 12 September 2012 08:20, Jean-Michel Hélary

 C:\maperitive\MesCartesC:\**osmosis-latest\osmosis-0.41\**
 bin\osmosis

 --read-pbf file=languedoc-roussillon.**osm.pbf
 --bounding-box top=43.70 left=3.7792 bottom=43.5292 right=3.98
 --write-xml file=new-montpellier.osm

 I don't have access to a Windows machine right now to validate your

 command line.  It seems to work fine under Linux.


 I verified the command with windows. Works fine.

 D:\Downloadsd:\osm\osmosis-0.**41\bin\osmosis --read-pbf
 file=planet-120912.osm.pbf --bounding-box top=43.70 left=3.7792
 bottom=43.5292 rig
 ht=3.98 --write-xml file=new-montpellier.osm
 Sep 19, 2012 9:07:15 PM org.openstreetmap.osmosis.**core.Osmosis run
 INFO: Osmosis Version 0.41
 Sep 19, 2012 9:07:16 PM org.openstreetmap.osmosis.**core.Osmosis run
 INFO: Preparing pipeline.
 Sep 19, 2012 9:07:16 PM org.openstreetmap.osmosis.**core.Osmosis run
 INFO: Launching pipeline execution.
 Sep 19, 2012 9:07:16 PM org.openstreetmap.osmosis.**core.Osmosis run
 INFO: Pipeline executing, waiting for completion.
 Sep 19, 2012 9:23:16 PM org.openstreetmap.osmosis.**core.Osmosis run
 INFO: Pipeline complete.
 Sep 19, 2012 9:23:16 PM org.openstreetmap.osmosis.**core.Osmosis run
 INFO: Total execution time: 960872 milliseconds.

 D:\Downloadsver

 Microsoft Windows [Version 6.0.6002]


 @Jean-Michel: What version of Windows are you using? Any special command
 interpreter?

 @Brett: Could the exception be extended to be a bit more verbosive of what
 is unrecognized? It could highlight the unrecognized option and print the
 whole command line it received.


Yes, the error message could definitely be more helpful.  I've just made a
simple change to include the name of the option that it doesn't recognise.
For example if you added the option -bogusoption to the command line, it
would now say that it didn't recognise bogusoption as an option.  This
could be improved on as you suggest by showing the entire command line and
highlighting the problem option, but that will take some more effort.
Patches welcome as always.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Problems with the tag transform plugin

[Brett Henderson]

On 28 September 2012 03:27, Shaun McDonald sh...@shaunmcdonald.me.ukwrote:

 Hi,

 There are two problems with the tag transform plugin for Osmosis. First
 the latest code in SVN doesn't build with the current version of osmosis
 (but does build with 0.39) due to the following error:

 ubuntu@ip-10-0-0-72:~/tagtransform$ ant
 Buildfile: /home/ubuntu/tagtransform/build.xml

 build:
 [javac] /home/ubuntu/tagtransform/build.xml:29: warning:
 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set
 to false for repeatable builds
 [javac] Compiling 24 source files to
 /home/ubuntu/tagtransform/build/classes
 [javac]
 /home/ubuntu/tagtransform/src/uk/co/randomjunk/osmosis/transform/v0_6/TransformChangeTask.java:12:
 uk.co.randomjunk.osmosis.transform.v0_6.TransformChangeTask is not abstract
 and does not override abstract method
 initialize(java.util.Mapjava.lang.String,java.lang.Object) in
 org.openstreetmap.osmosis.core.task.v0_6.Initializable
 [javac] public class TransformChangeTask extends
 TransformHelperChangeSink implements ChangeSinkChangeSource {
 [javac]^
 [javac]
 /home/ubuntu/tagtransform/src/uk/co/randomjunk/osmosis/transform/v0_6/TransformTask.java:10:
 uk.co.randomjunk.osmosis.transform.v0_6.TransformTask is not abstract and
 does not override abstract method
 initialize(java.util.Mapjava.lang.String,java.lang.Object) in
 org.openstreetmap.osmosis.core.task.v0_6.Initializable
 [javac] public class TransformTask extends TransformHelperSink
 implements SinkSource {
 [javac]^
 [javac] Note:
 /home/ubuntu/tagtransform/src/uk/co/randomjunk/osmosis/transform/impl/TransformHelper.java
 uses or overrides a deprecated API.
 [javac] Note: Recompile with -Xlint:deprecation for details.
 [javac] 2 errors

 BUILD FAILED
 /home/ubuntu/tagtransform/build.xml:29: Compile failed; see the compiler
 error output for details.

 Total time: 1 second


All tasks in Osmosis must now support the initialize method, so older
plugins such as tagtransform will require some minor fixes to be compatible
with this change.


 Also the latest built version linked on the wiki is built with too new a
 version of java, for the std version of java on Ubuntu and Mac OS X, thus
 I'm getting a version mismatch:

 osmosis -v
 27-Sep-2012 17:14:58 org.openstreetmap.osmosis.core.Osmosis run
 INFO: Osmosis Version 0.39
 27-Sep-2012 17:14:58 org.openstreetmap.osmosis.core.Osmosis main
 SEVERE: Execution aborted.
 java.lang.UnsupportedClassVersionError:
 uk/co/randomjunk/osmosis/transform/TransformPlugin : Unsupported
 major.minor version 51.0
 at java.lang.ClassLoader.defineClass1(Native Method)
 at java.lang.ClassLoader.defineClass(ClassLoader.java:634)
 at
 java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
 at java.net.URLClassLoader.defineClass(URLClassLoader.java:277)
 at java.net.URLClassLoader.access$000(URLClassLoader.java:73)
 at java.net.URLClassLoader$1.run(URLClassLoader.java:212)
 at java.security.AccessController.doPrivileged(Native Method)
 at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
 at
 org.codehaus.plexus.classworlds.realm.ClassRealm.loadClassFromSelf(ClassRealm.java:386)
 at
 org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy.loadClass(SelfFirstStrategy.java:42)
 at
 org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass(ClassRealm.java:244)
 at
 org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass(ClassRealm.java:230)
 at
 org.openstreetmap.osmosis.core.TaskRegistrar.loadPluginClass(TaskRegistrar.java:325)
 at
 org.openstreetmap.osmosis.core.TaskRegistrar.loadPlugin(TaskRegistrar.java:306)
 at
 org.openstreetmap.osmosis.core.TaskRegistrar.loadBuiltInPlugins(TaskRegistrar.java:123)
 at
 org.openstreetmap.osmosis.core.TaskRegistrar.initialize(TaskRegistrar.java:80)
 at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:81)
 at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:37)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
 at
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:616)
 at
 org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:329)
 at
 org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:239)
 at
 org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
 at
 org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
 at org.codehaus.classworlds.Launcher.main(Launcher.java:31)

 Reference wiki page:
 http://wiki.openstreetmap.org/wiki/Osmosis/TagTransform


Presumably it's been 

Re: [osmosis-dev] Feature request - a margin around a bbox

[Brett Henderson]

Hi Richard,

On 11 September 2012 22:41, Richard Mann
richard.mann.westoxf...@gmail.comwrote:

 Dunno if this is the right place for a feature request, but here goes...

 I'm extracting the data to render a particular z8 tile, and rather than
 using completeWays, I'm calculating the lonlats for a bbox that has a 2%
 margin around the z8 tile (because that's quicker to run).

 So the feature request is for an additional margin option to --bb that
 takes the specified leftright (or the ones inferred from x1 or x1x2), and
 adjusts them to add a percentage margin, and similarly with topbottom (or
 y1 or y1y2). If only one of left/right, or only one of top/bottom is
 specified, no adjustment would apply.

 This would allow the command to be shortened from:
 --bb top=51.63564009 left=-1.434375 bottom=50.7188239087754
 right=0.028125

 to:
 --bb zoom=8 x1=127 y1=86 margin=2

 and which has the distinct virtue of avoiding lots of external lon/lat
 calculations.

 Any chance of that being implemented?


It seems reasonable to me.  It'll get done a lot quicker if you can send a
patch though, I'm not likely to get it done any time soon.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Error with read-pbf

[Brett Henderson]

On 12 September 2012 08:20, Jean-Michel Hélary 
jean-michel.hel...@club-internet.fr wrote:

  Le 11/09/2012 23:30, Igor Podolskiy a écrit :

 Hi Jean-Michel,

 I'm new on this mailing list, but I've been using osmosis during one
 year now.

 Presently, I'm using osmosis 0.41 to extract a bounding box from a pbf
 file (downloaded from geofabrik). I'm using the following command :

 osmosis --read-pbf file=languedoc-roussillon.osm.pbf --bounding-box
 top=43.70 left=3.7792bottom=43.5292 right=3.98 --write-xml
 file=new-montpellier.osm

 but it fails (GRAVE :execution aborted) and throws the following
 exception : org.openstreetmap.osmosis.core.OsmosisRuntimeException:
 *Argument 1 specifies an unrecognized option*.

 could you please post the full error output (with stack trace) so we can
 see where the exception originates?

 Bye
 Igor

  Hi Igor,

 Here is the full output :


 C:\maperitive\MesCartesC:\osmosis-latest\osmosis-0.41\bin\osmosis
 --read-pbf file=languedoc-roussillon.osm.pbf
 --bounding-box top=43.70 left=3.7792 bottom=43.5292 right=3.98 --write-xml
 file=new-montpellier.osm
 12 sept. 2012 00:08:49 org.openstreetmap.osmosis.core.Osmosis main
 GRAVE: Execution aborted.

 org.openstreetmap.osmosis.core.OsmosisRuntimeException: Argument 1
 specifies an
 unrecognised option.
 at
 org.openstreetmap.osmosis.core.cli.CommandLineParser.parse(CommandLineParser.java:92)
 at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:74)
 at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:37)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
 at java.lang.reflect.Method.invoke(Unknown Source)
 at
 org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:329)
 at
 org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:239)
 at
 org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
 at
 org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
 at org.codehaus.classworlds.Launcher.main(Launcher.java:47)


That error is caused by passing a single - option to Osmosis.  Single
hyphen options specify global options such as -v or -q that aren't
related to specific tasks.

Is it possible that you have a osmosis.bat file somewhere under your home
directory that is passing additional options to Osmosis?

I don't have access to a Windows machine right now to validate your command
line.  It seems to work fine under Linux.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Error reading API DB with --rd

[Brett Henderson]

There was a change made to the apidb schema around November 2011.  The
Osmosis code change was made on November 2nd.  This was after 0.39 was
released (April 2011), but prior to 0.40 being released (December 2011).

You will need to either continue using Osmosis 0.39 with your old schema,
or upgrade your schema to the latest version.

Which wiki page(s) are you expecting to see the apidb schema changes
described on?

On 12 September 2012 20:50, Кирилл Бестужев bestou...@gmail.com wrote:

 No, should I?

 I see no changes to DB schema on the wiki!

 Kirill


 2012/9/12 Shaun McDonald sh...@shaunmcdonald.me.uk

 Hi,

 Have you updated the rails_port code to the latest version?

 Shaun


___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Running osmosis on a specific Postgresql schema

[Brett Henderson]

Hi Jocelyn,

On 13 September 2012 05:09, Jocelyn Jaubert jocelyn.jaub...@gmail.comwrote:

 Hi,

 I would like to use osmosis on a schema of a Postgresql database. I'm
 currently doing this by setting this:
 ALTER USER osmosis SET search_path TO osmosis,public;

 But it isn't really useful as I would like to have different osmosis
 schema in the same database - the idea is to have a different country
 or French region inside each. I could have a different user for each
 schema, bit I would prefer not to have to.


 So, it is possible to specify a schema when running
 osmosis --write-pgsql-change ?


It's not supported in the current implementation.

It doesn't look easy to implement.  Osmosis uses standard JDBC
functionality to access the database, and it looks like you have to issue
an explicit query such as set search path to 'schema' to change the
schema.  Given that Osmosis uses data sources and Spring Framework wrapper
classes, it may be difficult to find a way to issue that statement on
creation of all connections.

Is it possible to explicitly specify the schema name every time you specify
a table?  If so, that would be possible but would require some significant
code changes to propagate the schema name through to everywhere that builds
query strings.

I'm not keen to do this myself, but I'm more than happy to accept patches.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] applying diffs to the new planet

[Brett Henderson]

On 17 September 2012 02:57, Frederik Ramm frede...@remote.org wrote:

 Hi,


 On 16.09.2012 18:38, Christian H. Bruhn wrote:

 I downloaded the ODbL-Planet (PBF-format). How can apply ALL the
 (hourly-)diffs? How should the stat-file look like?


 Simply set the sequence number to 0 and it should work.


  So I tried without a state-file.


 I didn't know that it works without a state file at all!


If no state file is available, it will initialise itself to the latest
sequence.  In other words, on first run it will download the latest state
file and exit without making any changes.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Odd rate numbers with write-pgsql-dump

[Brett Henderson]

Hi Paul,

On 15 September 2012 18:37, Paul Norman penor...@mac.com wrote:

 I am using the --write-pgsql-dump in preparation for importing the planet
 into a pgsnapshot database and have noticed some oddities with ways.

 I am reading from the latest .pbf and using enableBboxBuilder=no
 enableLinestringBuilder=yes nodeLocationStoreType=TempFile. I tried
 InMemory
 but 16G of ram is not sufficient with the latest planet.

 My --log-progress task reported nodes being processed at about 250k
 nodes/second. It has now moved to the ways and the reports are interesting.

 It was reporting approximately 150-200 objects/second but when I manually
 take compute the ways/second from the way numbers I get figures in the
 realm
 of 5k+ objects/second. I know there may be some differences from deleted
 ways skewing the numbers, but this seems beyond what that would account
 for.

 It has taken approximately 20 minutes to reach way 4.5m, about 4k
 ways/second.

 iotop reports java reading at 5-10 M/s and writing at 150-400 K/s
 iostat reports the array with the .pbf and tempfile at 300 r/s 28 w/s 6000
 rkB/s 330 wkB/s, 90 %util.

 Although the progress seems okay (day or so to finish) I'm stumped by the
 objects/sec number.

 As an aside, about a year ago it took 5 hours to do it with InMemory but
 I'd
 have to go up to 24 GB RAM to do that again.


The numbers do seem a little odd, but I can't explain it.  The logic is the
same for all object types so if nodes are calculated correctly then ways
should also be calculated correctly.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] possible concurrency issue when using --apply-change

[Brett Henderson]

On 11 September 2012 22:11, Andrew Harvey andrew.harv...@gmail.com wrote:

 Hi,

 I'm using Osmosis to apply an .osc osm change file to an existing .osm
 file to produce a new .osm file with the changes from the .osc with:

 osmosis --read-xml-change file=change.osc.gz --sort-change
 --simplify-change --read-xml file=base.osm.gz --sort --apply-change
 --write-xml file=patched.osm.gz

 However I find that sometimes this works and sometimes it fails. When I
 say sometimes I mean if I run the same command line on the same source
 files multiple tiles maybe 20% of the time it will crash with messages
 like that attached. That really threw me off so my only thought is it is
 a concurrency issue at play.

 I will keep trying to find the problem but any hints or thoughts would
 help.

 Thank you.


This appears to have happened before:
http://lists.openstreetmap.org/pipermail/osmosis-dev/2011-January/000891.html

Unfortunately I have no idea what causes it.  I've never seen it myself.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

[osmosis-dev] Fast PBF Reader

[Brett Henderson]

Hi All,

I've been playing with the PBF reader implementation in Osmosis to see if I
can improve its performance.

The nice thing about the PBF format is that the data stream is broken into
coarse chunks that can be processed using multiple threads without thread
synchronisation being a major overhead.  I've just checked in a new
--read-pbf-fast implementation which does just that.  It is a complete
rewrite of the existing PBF implementation (it was easier to re-write than
to retrofit threading into the current implementation).  From an end user
perspective, it is similar to the existing --read-pbf task but has an
additional argument called workers which defines the number of worker
threads to use for processing.  It defaults to 1, but increasing it to
match your number of cores gives a significant performance boost.  On my
quad-core system (no hyper-threading) I get a 2-3 times performance
increase when just reading the file and discarding the contents.
Real-world usage with a longer pipeline will be less dramatic.

My command line in testing looks like this:
osmosis --read-pbf-fast myfile.pbf workers=4 --b bufferCapacity=1 --wn

A large buffer is very important.  The task implementation uses the master
thread to split the input stream *and* send results to the sink.  Using a
buffer is essential if you have any downstream tasks connected.

It's only available in Git for now, and is documented on the development
version of detailed documentation.
http://wiki.openstreetmap.org/wiki/Osmosis/Detailed_Usage_0.41#--read-pbf-fast_.28--rbf.29

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] GRAVE: Execution aborted - Task type repa already exists

[Brett Henderson]

On 3 May 2012 19:46, Cyprien 
c.y.p.r.i.e.n.p.a.r.i.s.+.n.o.s.p@gmail.com wrote:

 Hello,

 Every time I run osmosis with or without any parameters I get the
 following text error:

 28 mars 2012 08:48:17 org.openstreetmap.osmosis.core.Osmosis run
 INFO: Osmosis Version 0.39
 28 mars 2012 08:48:18 org.openstreetmap.osmosis.core.Osmosis main
 GRAVE: Execution aborted.
 org.openstreetmap.osmosis.core.OsmosisRuntimeException: Task type
 repa already exists.
at
 org.openstreetmap.osmosis.core.pipeline.common.TaskManagerFactoryRegister.register(TaskManagerFactoryRegister.java:44)
at
 org.openstreetmap.osmosis.core.TaskRegistrar.loadPluginClass(TaskRegistrar.java:350)
at
 org.openstreetmap.osmosis.core.TaskRegistrar.loadPlugin(TaskRegistrar.java:306)
at
 org.openstreetmap.osmosis.core.TaskRegistrar.loadBuiltInPlugins(TaskRegistrar.java:123)
at
 org.openstreetmap.osmosis.core.TaskRegistrar.initialize(TaskRegistrar.java:80)
at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:81)
at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:37)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at
 org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:329)
at
 org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:239)
at
 org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
at
 org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)


 I run it on Windows 7 64 bit up-to-date, with the latest Java version
 (64 bit version).  I also tried with the latest Java version in 32
 bits or on older Java versions, with no success.  But osmosis worked
 on the same computer 6 months ago, so this is very odd.

 So, what's the problem here?  I can't understand the error it generated.


Hi Cyprien,

Can you try running the latest version of Osmosis?  It may not fix the
problem, but it will make it easier to diagnose the problem.

Where did you get your copy of Osmosis and how did you install it?  The
error means that is has found two implementations of the --repa task on
your classpath which seems very odd.  That shouldn't happen if you're using
a pre-built copy of Osmosis.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Build failed in Jenkins: osmosis #32

[Brett Henderson]

Please ignore this failure.  I've just added the mailing list to broken
build notifications from the build server.
http://dev.openstreetmap.de:23457/hudson/job/osmosis/

The error below was a test to make sure it was working.  Build failures are
a very infrequent event so it shouldn't generate much list noise.

On 10 June 2012 22:23, hud...@dev.openstreetmap.de wrote:

 See http://dev.openstreetmap.de:23457/hudson/job/osmosis/32/

 --
 [...truncated 866 lines...]
 [ivy:resolve] .. (30kB)
 [ivy:resolve] .. (0kB)
 [ivy:resolve]   [SUCCESSFUL ]
 org.openstreetmap.osmosis#osmosis-set;0.40.1-120-g974b6b7!osmosis-set.jar
 (2ms)
 [ivy:resolve] :: resolution report :: resolve 557ms :: artifacts dl 18ms
 [ivy:resolve]   :: evicted modules:
 [ivy:resolve]   commons-logging#commons-logging;1.0.4 by
 [commons-logging#commons-logging;1.1.1] in [test]

  -
|  |modules||   artifacts
 |
|   conf   | number| search|dwnlded|evicted||
 number|dwnlded|

  -
|  default |   10  |   3   |   1   |   0   ||   10  |   1
 |
|  master  |   0   |   0   |   0   |   0   ||   0   |   0
 |
|  compile |   10  |   3   |   1   |   0   ||   10  |   1
 |
| provided |   0   |   0   |   0   |   0   ||   0   |   0
 |
|  runtime |   10  |   3   |   1   |   0   ||   10  |   1
 |
|   test   |   18  |   3   |   1   |   1   ||   17  |   1
 |
|  system  |   0   |   0   |   0   |   0   ||   0   |   0
 |
|  sources |   0   |   0   |   0   |   0   ||   0   |   0
 |
|  javadoc |   0   |   0   |   0   |   0   ||   0   |   0
 |
| optional |   0   |   0   |   0   |   0   ||   0   |   0
 |
|   distribution   |   0   |   0   |   0   |   0   ||   0   |   0
 |

  -
 [ivy:retrieve] :: retrieving ::
 org.openstreetmap.osmosis#osmosis-replication [sync]
 [ivy:retrieve]  confs: [default, master, compile, provided, runtime, test,
 system, sources, javadoc, optional, distribution]
 [ivy:retrieve]  47 artifacts copied, 0 already retrieved (13204kB/3051ms)
 [ivy:report] Processing
 /home/osmosisci/.ivy2/cache/org.openstreetmap.osmosis-osmosis-replication-default.xml
 to
 /osm/osmosis-continuous-integration/dot-hudson/jobs/osmosis/workspace/replication/report/ivy/org.openstreetmap.osmosis-osmosis-replication-default.html
 [ivy:report] Processing
 /home/osmosisci/.ivy2/cache/org.openstreetmap.osmosis-osmosis-replication-master.xml
 to
 /osm/osmosis-continuous-integration/dot-hudson/jobs/osmosis/workspace/replication/report/ivy/org.openstreetmap.osmosis-osmosis-replication-master.html
 [ivy:report] Processing
 /home/osmosisci/.ivy2/cache/org.openstreetmap.osmosis-osmosis-replication-compile.xml
 to
 /osm/osmosis-continuous-integration/dot-hudson/jobs/osmosis/workspace/replication/report/ivy/org.openstreetmap.osmosis-osmosis-replication-compile.html
 [ivy:report] Processing
 /home/osmosisci/.ivy2/cache/org.openstreetmap.osmosis-osmosis-replication-provided.xml
 to
 /osm/osmosis-continuous-integration/dot-hudson/jobs/osmosis/workspace/replication/report/ivy/org.openstreetmap.osmosis-osmosis-replication-provided.html
 [ivy:report] Processing
 /home/osmosisci/.ivy2/cache/org.openstreetmap.osmosis-osmosis-replication-runtime.xml
 to
 /osm/osmosis-continuous-integration/dot-hudson/jobs/osmosis/workspace/replication/report/ivy/org.openstreetmap.osmosis-osmosis-replication-runtime.html
 [ivy:report] Processing
 /home/osmosisci/.ivy2/cache/org.openstreetmap.osmosis-osmosis-replication-test.xml
 to
 /osm/osmosis-continuous-integration/dot-hudson/jobs/osmosis/workspace/replication/report/ivy/org.openstreetmap.osmosis-osmosis-replication-test.html
 [ivy:report] Processing
 /home/osmosisci/.ivy2/cache/org.openstreetmap.osmosis-osmosis-replication-system.xml
 to
 /osm/osmosis-continuous-integration/dot-hudson/jobs/osmosis/workspace/replication/report/ivy/org.openstreetmap.osmosis-osmosis-replication-system.html
 [ivy:report] Processing
 /home/osmosisci/.ivy2/cache/org.openstreetmap.osmosis-osmosis-replication-sources.xml
 to
 /osm/osmosis-continuous-integration/dot-hudson/jobs/osmosis/workspace/replication/report/ivy/org.openstreetmap.osmosis-osmosis-replication-sources.html
 [ivy:report] Processing
 /home/osmosisci/.ivy2/cache/org.openstreetmap.osmosis-osmosis-replication-javadoc.xml
 to
 /osm/osmosis-continuous-integration/dot-hudson/jobs/osmosis/workspace/replication/report/ivy/org.openstreetmap.osmosis-osmosis-replication-javadoc.html
 [ivy:report] Processing
 /home/osmosisci/.ivy2/cache/org.openstreetmap.osmosis-osmosis-replication-optional.xml
 to
 

Re: [osmosis-dev] [PATCH] Remove changing=true from fixed-revision Ivy dependencies

[Brett Henderson]

Hi Richard,

Thanks for the patch.  It's applied in Git.

Cheers,
Brett

On 11 March 2012 10:59, Richard Hansen rhan...@bbn.com wrote:

 In ivy.xml, the setting 'changing=true' tells Ivy that the
 dependency's jar might change despite its version not changing.  This
 causes Ivy to always query the repository, even if the jar is
 available in the local cache.  This setting is often used to express a
 dependency on the latest snapshot (e.g., nightly build) of a 3rd party
 project.

 Unfortunately, this setting will cause offline builds to fail if the
 dependency comes from a non-local repository, even if the dependency
 is available in the local cache.  Thus, it should be avoided for
 fixed-revision dependencies.

 The versions of the osmpbf and netty dependencies do not represent
 snapshots that could change over time, so this change removes
 'changing=true from the dependency declarations.
 ---
  pbf/ivy.xml  |2 +-
  replication-http/ivy.xml |2 +-
  2 files changed, 2 insertions(+), 2 deletions(-)

 diff --git a/pbf/ivy.xml b/pbf/ivy.xml
 index 6f6a0af..cebf4bf 100644
 --- a/pbf/ivy.xml
 +++ b/pbf/ivy.xml
 @@ -27,7 +27,7 @@
dependency org=org.openstreetmap.**osmosis name=osmosis-core
 rev=${project.version} conf=compile-default changing=true/

dependency org=com.google.protobuf name=protobuf-java
 rev=${dependency.version.**protobuf} conf=compile-default/
 -   dependency org=crosby name=osmpbf rev=${dependency.version.*
 *osmpbf} conf=compile-default changing=true/
 +   dependency org=crosby name=osmpbf rev=${dependency.version.*
 *osmpbf} conf=compile-default/

 dependency org=org.openstreetmap.**osmosis name=osmosis-xml
 rev=${project.version} conf=test-default changing=true/
 dependency org=org.openstreetmap.**osmosis
 name=osmosis-testutil rev=${project.version} conf=test-default
 changing=true/
 diff --git a/replication-http/ivy.xml b/replication-http/ivy.xml
 index 885275a..8844c6d 100644
 --- a/replication-http/ivy.xml
 +++ b/replication-http/ivy.xml
 @@ -28,7 +28,7 @@
dependency org=org.openstreetmap.**osmosis
 name=osmosis-replication rev=${project.version} conf=compile-default
 changing=true/
dependency org=org.openstreetmap.**osmosis name=osmosis-xml
 rev=${project.version} conf=compile-default changing=true/

 -   dependency org=org.jboss.netty name=netty
 rev=${dependency.version.**netty} conf=compile-default
 changing=true/
 +   dependency org=org.jboss.netty name=netty
 rev=${dependency.version.**netty} conf=compile-default/

dependency org=org.openstreetmap.**osmosis
 name=osmosis-testutil rev=${project.version} conf=test-default
 changing=true/
 dependency org=junit name=junit 
 rev=${dependency.version.**junit}
 conf=test-default/
 --
 1.7.9.2



 __**_
 osmosis-dev mailing list
 osmosis-dev@openstreetmap.org
 http://lists.openstreetmap.**org/listinfo/osmosis-devhttp://lists.openstreetmap.org/listinfo/osmosis-dev

___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] read-pbf output error after upgrade from 0.39 to 0.40.1

[Brett Henderson]

On 15 February 2012 19:45, Simon Nuttall simon.nutt...@gmail.com wrote:

  Do you have to use MySQL?  Osmosis still contains the MySQL but it is no
  longer tested.

 All of CycleStreets uses MySQL so abandoning that would require a lot
 of work. I've learned my way around MySQL now and know what to avoid
 and what to use. I'd have to go through the same process with Postgres
 - and we did start down that path a couple of years ago when it looked
 like you were abandoning MySQL.

   It may be necessary to update the MySQL schema to match the
  PostgreSQL schema.

 OK, I might look into that, but I think it is more likely that we'll
 just stick with 0.39 of Osmosis for the time being.


Okay, that's fair enough.

I think renaming your columns should be sufficient, but let me know if you
try it and it turns out to be more complicated.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] read buffer output error after moving from 0.39 to 0.40.1

[Brett Henderson]

On 15 February 2012 12:19, Simon Nuttall simon.nutt...@gmail.com wrote:

   Also, is it possible to reduce the size of your
  command line by breaking it into sub-commands.  If I'm interpreting your
  command line properly it should be possible to read your PBF files and
  perform filtering in separate Osmosis invocations writing to intermediate
  files, then merge into a single file, then write to the database.
 

 Yes, I'm very willing to change to whatever is the recommended way of
 doing it. I've just added extra small countries on to this process as
 interest in CycleStreets has grown.


Sorry, my comment wasn't clear.  I only suggested breaking it into separate
tasks to make it easier to isolate the problem you were experiencing.
Running it all as one command is fine when everything is working correctly.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] read buffer output error after moving from 0.39 to 0.40.1

[Brett Henderson]

Hi Simon,

On 8 February 2012 23:17, Simon Nuttall simon.nutt...@gmail.com wrote:

snip


 Produces:


 08-Feb-2012 12:03:27 org.openstreetmap.osmosis.core.Osmosis run
 INFO: Osmosis Version 0.40.1
 08-Feb-2012 12:03:27 org.openstreetmap.osmosis.core.TaskRegistrar
 loadBuiltInPlugins
 FINER: Loading plugin configuration file from url

 jar:file:/usr/local/osmosis/current/lib/default/osmosis-areafilter-0.40.1.jar!/osmosis-plugins.conf.
 .
 .
 .08-Feb-2012 12:03:37
 org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager
 waitForCompletion
 FINE: Waiting for task 1-read-pbf to complete.
 08-Feb-2012 12:03:46
 org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager
 waitForCompletion
 SEVERE: Thread for task 1-read-pbf failed
 org.openstreetmap.osmosis.core.OsmosisRuntimeException: An output
 error has occurred, aborting.
at
 org.openstreetmap.osmosis.core.store.DataPostbox.checkForOutputErrors(DataPostbox.java:78)
at
 org.openstreetmap.osmosis.core.store.DataPostbox.populateCentralQueue(DataPostbox.java:134)
at
 org.openstreetmap.osmosis.core.store.DataPostbox.put(DataPostbox.java:184)
at
 org.openstreetmap.osmosis.set.v0_6.impl.DataPostboxSink.process(DataPostboxSink.java:34)
at
 org.openstreetmap.osmosis.core.sort.v0_6.SortedEntityPipeValidator.process(SortedEntityPipeValidator.java:58)
at
 org.openstreetmap.osmosis.tagfilter.v0_6.TagFilter.process(TagFilter.java:133)
at
 org.openstreetmap.osmosis.tagfilter.v0_6.TagFilter.process(TagFilter.java:133)
at
 crosby.binary.osmosis.OsmosisBinaryParser.parseDense(OsmosisBinaryParser.java:139)
at crosby.binary.BinaryParser.parse(BinaryParser.java:124)
at crosby.binary.BinaryParser.handleBlock(BinaryParser.java:68)
at crosby.binary.file.FileBlock.process(FileBlock.java:135)
at
 crosby.binary.file.BlockInputStream.process(BlockInputStream.java:34)
at crosby.binary.osmosis.OsmosisReader.run(OsmosisReader.java:37)
at java.lang.Thread.run(Thread.java:636)
 08-Feb-2012 12:03:46
 org.openstreetmap.osmosis.core.pipeline.common.PassiveTaskManager
 waitForCompletion
 FINE: Task 2-tag-filter is passive, no completion wait required.


Do other exceptions occur earlier in the console output?  The above
exception is from a different thread to where the original exception
occurred.  One thread has encountered an exception and failed, then another
thread trying to write into a buffer that they share fails with the above
exception.

Can you provide a more complete console output, or at least look for the
cause exception message?  Also, is it possible to reduce the size of your
command line by breaking it into sub-commands.  If I'm interpreting your
command line properly it should be possible to read your PBF files and
perform filtering in separate Osmosis invocations writing to intermediate
files, then merge into a single file, then write to the database.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] read-pbf output error after upgrade from 0.39 to 0.40.1

[Brett Henderson]

Hi Simon,

The relevant error in your full log is below.  All other errors in the log
are caused by other threads failing when they can't write to their
destination (ie. the db writing thread is downstream of all other threads).

SEVERE: Thread for task 25-buffer failed
 org.openstreetmap.osmosis.
core.OsmosisRuntimeException: Unable to
 insert a new changeset for user with id -1.
at
org.openstreetmap.osmosis.apidb.v0_6.impl.ChangesetManager.addChangeset(ChangesetManager.java:160)
at
org.openstreetmap.osmosis.apidb.v0_6.impl.ChangesetManager.addChangesetIfRequired(ChangesetManager.java:175)
at
org.openstreetmap.osmosis.apidb.v0_6.ApidbWriter.process(ApidbWriter.java:1084)
at
org.openstreetmap.osmosis.core.buffer.v0_6.EntityBuffer.run(EntityBuffer.java:74)
at java.lang.Thread.run(Thread.java:679)
 Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException:
 Unknown column 'changeset_id' in 'field list'
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.Util.getInstance(Util.java:386)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1052)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3609)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3541)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2002)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2163)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2624)
at
com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2127)
at
com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2427)
at
com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2345)
at
com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2330)
at
org.openstreetmap.osmosis.apidb.v0_6.impl.ChangesetManager.addChangeset(ChangesetManager.java:148)
... 4 more

There were some changes made to the apidb schema recently where columns
named id were renamed to _id.  For example, the nodes table now
contains a node_id column instead of id column.  I'm guessing this is what
is tripping you up.

Do you have to use MySQL?  Osmosis still contains the MySQL but it is no
longer tested.  It may be necessary to update the MySQL schema to match the
PostgreSQL schema.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] calling DataPostbox.release() multiple times

[Brett Henderson]

On 19 January 2012 10:17, Richard Hansen rhan...@bbn.com wrote:


  release() and outputRelease() have been designed to support re-use.  I
 don't think I've tested it, but I believe it will work.  The two
 variables you mention are initialized near the start of those methods
 and only read after the Released flags have been set so their
 initial state isn't important.  However I can see that is confusing so
 I've modified their initial state to match their final state of true
 (I'll check it in shortly).


 I agree with that change.  The only other thing I would suggest is a
 comment indicating that the class is designed to support reuse.  (Or are
 all tasks supposed to be reusable?)


These days I try to support re-use for all tasks, but I wouldn't be
surprised if many of the older tasks don't work.

I've updated the comments to describe the re-use support.  Let me know if
it doesn't make sense.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] calling DataPostbox.release() multiple times

[Brett Henderson]

On 16 January 2012 07:22, Richard Hansen rhan...@bbn.com wrote:

 On 2012-01-15 07:34, Brett Henderson wrote:

 You're correct in saying that Releasable classes can be released
 multiple times, however that is not typically how they're used in the
 Osmosis pipeline.


 Does it make sense to change the Releasable API definition to say that
 release can only be called once?  Or might that break multiple-sink stages
 like merge?


I've updated the Releasable API definition to say that implementations
should support release being called multiple times but that it isn't
mandatory and cannot be relied on by clients.

Multiple sink stages like merge expose two separate sink instances to the
two upstream pipes (through MultiSink interface which allows multiple Sink
instances to be obtained) which feed into separate DataPostbox instances
internally.  Each sink instance only has its release method called once.
So a single release call limitation shouldn't break anything.



  Looking at the code a bit more I think it might only need minor changes.
  I had thought that release() and outputRelease() changed the state to
 equal that of a freshly constructed DataPostbox (to support reuse?), but
 that's not true:  inputExit and outputExit are false after construction but
 true after release().  So simply checking inputExit at the beginning of
 release() should take care of it.  See the attached (untested) patch.


release() and outputRelease() have been designed to support re-use.  I
don't think I've tested it, but I believe it will work.  The two variables
you mention are initialized near the start of those methods and only read
after the Released flags have been set so their initial state isn't
important.  However I can see that is confusing so I've modified their
initial state to match their final state of true (I'll check it in shortly).

Your patch would probably work but I'd prefer to continue to support re-use
rather than to use that variable for a second purpose.



 An alternative is to increase the burden on the users of DataPostbox. They
 could be required to either ensure that the output thread restarts or avoid
 calling DataPostbox.release() once the output thread exits. For example,
 EntityBuffer.run() could set an 'outputThreadIsRunning' boolean at the top
 and clear it at the end.  Then EntityBuffer.release() would be modified to
 only call buffer.release() if outputThreadIsRunning is true.


Race conditions between threads are tough to avoid, and I believe the above
change would introduce one.  If the output thread hasn't entered
EntityBuffer.run yet and the outputThreadIsRunning flag hasn't been set,
but the calling thread encounters an early exception and calls
EntityBuffer.release, the calling thread will think that the output thread
isn't running and there is no need to perform release functionality.  This
would then lead to the output thread blocking when it finally does start.

The way the class is constructed now, each thread will synchronise at the
initialize, complete and release methods.  If either thread skips straight
to release then the other thread detects that as an error condition and
aborts appropriately, but in all cases both threads will synchronise around
release before exiting.  The only reason to synchronise around release is
to support re-use which may never be used in practice but it should work
well.  We can't even rely on the initialize methods being called because it
is possible for a thread to fail before that is called.

I'd rather leave the class as it is designed now.  It's a tradeoff between
supporting re-use and incurring the limitation that every call to release
must be matched by a call to outputRelease in another thread.  I don't
think it's possible to have it both ways.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] [PATCH] --log-progress --buffer results in OsmosisRuntimeException: initialize has not been called

[Brett Henderson]

On 15 January 2012 08:11, Richard Hansen rhan...@bbn.com wrote:

 Hi all,

 With the latest Git revision (43089ed), I get an OsmosisRuntimeException
 when I use --log-progress followed by --buffer.  For example:

 osmosis -v \
--rb massachusetts.osm.pbf --sort \
--rb connecticut.osm.pbf --sort \
--merge --log-progress --buffer --wb _combined.osm.pbf
 ...
 org.openstreetmap.osmosis.**core.OsmosisRuntimeException: initialize has
 not been called
   at org.openstreetmap.osmosis.**core.store.DataPostbox.put(**
 DataPostbox.java:288)
   at org.openstreetmap.osmosis.**core.buffer.v0_6.EntityBuffer.**
 process(EntityBuffer.java:48)
   at org.openstreetmap.osmosis.**core.progress.v0_6.**
 EntityProgressLogger.process(**EntityProgressLogger.java:70)
   at org.openstreetmap.osmosis.**core.sort.v0_6.EntitySorter.**
 complete(EntitySorter.java:71)
   at crosby.binary.osmosis.**OsmosisBinaryParser.complete(**
 OsmosisBinaryParser.java:35)
   at crosby.binary.file.**BlockInputStream.process(**
 BlockInputStream.java:37)
   at crosby.binary.osmosis.**OsmosisReader.run(**
 OsmosisReader.java:45)
   at java.lang.Thread.run(Thread.**java:679)

 If I remove the --log-progress argument, it works as expected.

 I chased this down to a missing call to sink.initialize() in
 EntityProgressLogger.**initialize().  Attached is a patch that I believe
 fixes this bug.


Hi Richard,

I've applied the patch.  Thanks.

I've also updated the ChangeProgressLogger class which had the same issue.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] calling DataPostbox.release() multiple times

[Brett Henderson]

On 15 January 2012 14:09, Richard Hansen rhan...@bbn.com wrote:

 Hi all,

 I believe I have found a bug in DataPostbox.release().  (I'm using git
 master, currently 43089ed.)

 According to the javadoc comments, Releaseable.release() can be called
 multiple times.  However, DataPostbox.release() can only be called once.
  If it is called a second time, it blocks forever waiting for the output
 thread (which no longer exists) to release.


Good find, thanks for the report.

You're correct in saying that Releasable classes can be released multiple
times, however that is not typically how they're used in the Osmosis
pipeline.

DataPostbox is a bit tricky because it is the main synchronisation point
between multiple threads.  Two threads communicating via a DataPostbox
perform handshakes during the initialize, complete, and release methods
(the process methods run mostly independently).  It is the release method
where each threads tells the other thread that they are about to exit.
DataPostbox does technically support multiple calls to release, but it
would require each thread to restart after the release calls and this isn't
supported (or even desirable) by the Osmosis pipeline.



 I noticed this because the pbf reader can call release() twice, triggering
 the lockup.  To recreate, you can run the following:

osmosis --read-pbf /dev/null --buffer --write-null

 The two calls to DataPostbox.release() have the following stack traces:

 java.lang.Exception: Stack trace
at java.lang.Thread.dumpStack(**Thread.java:1266)
at org.openstreetmap.osmosis.**core.store.DataPostbox.**
 release(DataPostbox.java:335)
at org.openstreetmap.osmosis.**core.buffer.v0_6.EntityBuffer.**
 release(EntityBuffer.java:64)
at crosby.binary.osmosis.**OsmosisBinaryParser.complete(**
 OsmosisBinaryParser.java:36)
at crosby.binary.file.**BlockInputStream.process(**
 BlockInputStream.java:37)
at crosby.binary.osmosis.**OsmosisReader.run(**
 OsmosisReader.java:45)
at java.lang.Thread.run(Thread.**java:679)
 java.lang.Exception: Stack trace
at java.lang.Thread.dumpStack(**Thread.java:1266)
at org.openstreetmap.osmosis.**core.store.DataPostbox.**
 release(DataPostbox.java:335)
at org.openstreetmap.osmosis.**core.buffer.v0_6.EntityBuffer.**
 release(EntityBuffer.java:64)
at crosby.binary.osmosis.**OsmosisReader.run(**
 OsmosisReader.java:50)
at java.lang.Thread.run(Thread.**java:679)

 Deleting line 36 of 
 pbf/src/main/java/crosby/**binary/osmosis/**OsmosisBinaryParser.java
 will cause DataPostbox.release() to be called only once, but that just
 avoids the bug.


I've removed line 36 as you've suggested because it shouldn't be there.
Release should only be called in a finally block that gets called
regardless of success or failure, the complete method only gets called on
success.  I think it was me that added the other release in OsmosisReader
recently when I added initialize, but apparently I didn't make a
corresponding change in OsmosisBinaryParser.

Please let me know if you see any further issues.



 I believe a proper fix to DataPostbox.release() will require a minor
 redesign of the DataPostbox lifetime management code.  I'm not familiar
 enough with the code to feel comfortable making such a change.  I can take
 a crack at it, but I'm hoping an expert has the time to take a look.


What changes would you suggest?  I believe it's working as I originally
intended, but that's not to say there isn't a better way :-)

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] --simplify-change with a conflictResolutionMethod

[Brett Henderson]

Hi Andrew,

The SortedHistoryChangePipeValidator is also used by the --merge-change
task so can't be changed without breaking the assumptions that that task
makes.

Can I ask how you've ended up with two changes with the same version number
yet different timestamps and changeset numbers?  This data shouldn't be
possible to create using normal OSM tools and processes.  The various
validators in Osmosis act as protection against this type of unexpected
data and I'm very hesitant to relax those validations.

Brett

On 8 January 2012 12:15, Andrew Harvey andrew.harv...@gmail.com wrote:

 I tried editing

 osmosis-0.40.1/core/src/main/java/org/openstreetmap/osmosis/core/sort/v0_6/SortedHistoryChangePipeValidator.java
 at line 49 to be  rather than =, altering the validation hence
 removing the error message. This seems to remove one of the duplicated
 way items (which is what I want for --simplify-change), in my case it
 removed the first one.

 On Sun, Jan 8, 2012 at 9:38 AM, Andrew Harvey andrew.harv...@gmail.com
 wrote:
  I have the following .osc file. The important part is it has two ways
  with both the same id and version, except different dates and
  changesets.
 
  ?xml version='1.0' encoding='UTF-8'?
  osmChange version=0.6 generator=Osmosis 0.40.1
   modify
 way id=1 version=1 timestamp=2011-06-25T18:39:36Z uid=1
  user=user changeset=1
   nd ref=1/
   nd ref=1/
   tag k=foo v=bar/
 /way
   /modify
   modify
 way id=1 version=1 timestamp=2011-06-25T19:39:36Z uid=1
  user=user changeset=2
   nd ref=1/
   nd ref=1/
   tag k=foo v=bar_bar/
 /way
   /modify
  /osmChange
 
  Running it through osmosis with,
 
  osmosis --read-xml-change in.osc --sort-change --simplify-change
  --write-xml-change out.osc
 
  fails.
 
  Can it be expected that --simplify-change be able to handle this and
  offer a conflictResolutionMethod argument like --merge? Or is there
  some other way to handle this case?

 ___
 osmosis-dev mailing list
 osmosis-dev@openstreetmap.org
 http://lists.openstreetmap.org/listinfo/osmosis-dev

___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] nodes column in way table and way_nodes table

[Brett Henderson]

On 26 December 2011 01:43, Michel Seuthe m.seu...@googlemail.com wrote:

 Ya, I used a similar statement. It took a few hours to build the
 linestrings for germany.
 I tried to use osmosis to build the geometries, but it always threw
 SystemOutOfMemory Exception when the linestring column was added to ways
 table. The box osmosis was running on got 16GB RAM and I used InMemory
 option.


Did you specify the -Xmx option to java?  Were you using a 64-bit JVM?
Java by default will automatically set a maximum available memory size
which is fairly small compared to overall system RAM.
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] nodes column in way table and way_nodes table

[Brett Henderson]

Hi Michel,

Yes, there is redundant data stored in the pgsnapshot schema.  It's there
for performance reasons.  If you compare the pgsnapshot schema to the
earlier pgsimple schema you'll notice that the way.nodes column is new, but
that the way_nodes table has always existed.  Both way-node mechanisms are
used for different purposes.

The pgsnapshot schema relies heavily on the PostgreSQL CLUSTER feature
where table contents can be sorted to align with one of the indexes.  The
nodes and ways tables are both organised by geographical location which
drastically improves performance when performing bounding box style queries
because data in the same geographical area will be stored close together on
disk.  The way_nodes table can't make use of geographical clustering which
means that joining the ways table to the nodes table via way_nodes is very
slow for large amounts of data due to large amounts of disk seeks.
Therefore the way-node information is duplicated inside the ways table for
fast lookup of nodes within ways.  This schema includes tags directly
against the node, way and relation tables in hstore columns for exactly the
same reason.  On typical SATA disks, the pgsnapshot schema performs
approximately 10 times faster than the pgsimple schema for bounding box
queries.

I can't drop the way_node table because it is still needed for two reasons
(may be others I've forgotten).  1. It is used for replication when a node
changes and the related way geometries have to be updated. 2. It is used
for bounding box queries if the ways table doesn't have any geometric
columns configured.

I hope that makes it clearer.
Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] bound/bounds element in OSM XML

[Brett Henderson]

On 20 December 2011 20:37, Igor Podolskiy
igor.podols...@vwi-stuttgart.dewrote:

 Hi Brett,


  I just tried to build the branch and received a test failure. You
 relaxed the Bound constructor validation checks to allow a null origin,
 but this has caused the BoundTest.testConstructor10 test to fail. I'd
 just delete the test, but I'm not 100% sure if I've messed up somewhere
 because I *think* I built it successfully last night. I'm wondering if
 I'm going crazy ;-)

 No you aren't :) I indeed forgot to remove that test. And while I was at
 it, I now also found and fixed some code that assumed Bound.origin != null.
 Sorry I didn't get that right the first time.

 I updated the xml-bounds branch in my repo, it is now at commit 59b0e54.


It turned out that I hadn't built it last night, I'd only done a build
without running tests.

Cool, that's looking better.

But unfortunately there's another failure now in the apidb code.  I've
setup a job to build from your repo in Jenkins.
http://dev.openstreetmap.de:23457/hudson/job/osmosis-podolsir/

Building from your xml-bounds branch failed.
http://dev.openstreetmap.de:23457/hudson/job/osmosis-podolsir/1/

There's some sort of Locale problem.  I'll look into it when I get a chance
if you don't beat me to it.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] bound/bounds element in OSM XML

[Brett Henderson]

Hi Igor,

On 16 December 2011 04:15, Igor Podolskiy
igor.podols...@vwi-stuttgart.dewrote:

 Hi @osmosis-dev,


  That's why it took longer than I thought, but now I'm almost done and
 I'll upload it today or tomorrow - just writing now to avoid duplicate
 effort.


 the fix for the bound/bounds issue is now in my github repo in the
 xml-bounds branch:

 https://github.com/podolsir/**osmosis/tree/xml-boundshttps://github.com/podolsir/osmosis/tree/xml-bounds

 Feel free to comment and/or pull :)


I just tried to build the branch and received a test failure.  You relaxed
the Bound constructor validation checks to allow a null origin, but this
has caused the BoundTest.testConstructor10 test to fail.  I'd just delete
the test, but I'm not 100% sure if I've messed up somewhere because I
*think* I built it successfully last night.  I'm wondering if I'm going
crazy ;-)

Does the BoundTest.testConstructor10 method need to be removed?

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Osmosis replication fails

[Brett Henderson]

On 13 December 2011 06:31, Martijn van Exel m...@rtijn.org wrote:

 Hi all,

 I have a replication task set up, initially with a longer interval
 because my planet file is a few weeks old. I have had it running in a
 cron job with a two hour interval but I get a lot of errors similar to
 this one:

 SEVERE: Thread for task 1-rri failed
 org.openstreetmap.osmosis.core.OsmosisRuntimeException: Unable to
 parse xml file /tmp/change8301792328184763034.tmp.  publicId=(null),
 systemId=(null), lineNumber=7384, columnNumber=3.
 
 Caused by: org.xml.sax.SAXParseException; lineNumber: 746;
 columnNumber: 3; The element type osmChange must be terminated by
 the matching end-tag /osmChange.


It sounds like the change files are incomplete.  Perhaps some of the
downloads are failing.  Is your network connection usually reliable?



 Out of 50 executions, this error appeared 23 times.
 I have my replication interval set to one day, could that be the problem?
 Processing (when it succeeds) takes about 90 minutes. I have the cron
 job set to execute every two hours.


How is your replication configured?  Specifically which replication files
are you using (ie. minute, hour or day)?  It may be worth switching to
files with a longer interval if you are patching a file to reduce the
number of downloads required.  Minute replication files would typically be
more suitable to patching a database where small files can be applied
quickly.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] bound/bounds element in OSM XML

[Brett Henderson]

On 11 December 2011 15:00, Tobias Knerr o...@tobias-knerr.de wrote:

 Hello,

 I'm using Osmosis as a library for reading OSM files in OSM2World. It
 works well for this task and allows me to easily support various input
 formats. Recently, though, I added handling of files' bounds to
 OSM2World. This worked well with .osm.pbf files, but not with .osm
 files. Bounds in the latter were ignored entirely.

 Looking at the Osmosis source code on GitHub, I noticed several
 differences to the XML format as described on
 http://wiki.openstreetmap.org/wiki/OSM_XML

 The first difference is that OsmElementProcessor [1] looks for a
 bound, rather than bounds, element.

 More differences are visible in BoundElementProcessor [2], particularly
 that the origin attribute is considered required by Osmosis, and that
 Osmosis expects a box attribute with 4 comma-separated values, rather
 than the 4 attributes minlat, minlon, maxlat and maxlon used in
 the wiki.

 My question is: Why do these differences exist? Are they intentional?


I'm not sure of the exact history here, but I think bound existed before
bounds.  bounds is a newer concept with different attributes that has never
been implemented in Osmosis.  I'm not sure what would be involved in
updating Osmosis to support it, but perhaps it's just a matter of updating
the XML tasks to use the different element and attributes.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev