Re: [osmosis-dev] Osmosis and Osmium-enhanced PBF files with way node locations

[Locke, Jonathan]

Thanks for the feedback, Brett. I'm on vacation tomorrow, but I'll try to 
address your concerns on Tuesday.


   Jon


From: Jochen Topf 
Sent: Sunday, March 11, 2018 5:56:54 AM
To: Brett Henderson
Cc: Locke, Jonathan; Van Exel, Martijn; osmosis-dev@openstreetmap.org
Subject: Re: [osmosis-dev] Osmosis and Osmium-enhanced PBF files with way node 
locations

On Sun, Mar 11, 2018 at 10:19:21AM +, Brett Henderson wrote:
> *osmformat.proto*
> Are these changes mastered somewhere else?  In other words, are these new
> fields the same ones used by Osmium in its implementation?  I'm wondering
> if we need to re-sync from the main OSM-binary project.  The osmosis
> version is the same as that in https://github.com/scrosby/OSM-binary.  The
> repo https://github.com/brettch/OSM-binary is a fork of that, and the
> osmosis branch there *is* the same code as the osmosis-osm-binary directory
> in the osmosis repo, just with some re-arranging to fit inside the Osmosis
> structure and fit inside the Osmosis java package namespace.
>
> *Jochen* (if you're reading), where does Osmium source its osmformat.proto
> file from?

The PBF implementation in Osmium doesn't need the .proto file, so this
didn't come up there.

> Long story short, rather than make changes directly to the file in Osmosis
> and create a fork, should we apply them to upstream first and then re-sync
> Osmosis with that?

Yes, we should do this upstream in https://github.com/scrosby/OSM-binary.
If somebody creates a pull request, I'll merge it there...

Jochen
--
Jochen Topf  joc...@remote.org  
https://linkprotect.cudasvc.com/url?a=https://www.jochentopf.com/=E,1,HZFs9cweUVxpTLJQ4cO5l0bBsPZCxUql3lJmd3REbm5WoZilH4Cd7sTEj0hlrb-qeHzuiAi4u5gnj2ZpgPntBsyW7BF_R6_foHVx7qkNssf1vtzLlOQQ=1
  +49-351-31778688
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Osmosis and Osmium-enhanced PBF files with way node locations

[Jochen Topf]

On Sun, Mar 11, 2018 at 10:19:21AM +, Brett Henderson wrote:
> *osmformat.proto*
> Are these changes mastered somewhere else?  In other words, are these new
> fields the same ones used by Osmium in its implementation?  I'm wondering
> if we need to re-sync from the main OSM-binary project.  The osmosis
> version is the same as that in https://github.com/scrosby/OSM-binary.  The
> repo https://github.com/brettch/OSM-binary is a fork of that, and the
> osmosis branch there *is* the same code as the osmosis-osm-binary directory
> in the osmosis repo, just with some re-arranging to fit inside the Osmosis
> structure and fit inside the Osmosis java package namespace.
> 
> *Jochen* (if you're reading), where does Osmium source its osmformat.proto
> file from?

The PBF implementation in Osmium doesn't need the .proto file, so this
didn't come up there.

> Long story short, rather than make changes directly to the file in Osmosis
> and create a fork, should we apply them to upstream first and then re-sync
> Osmosis with that?

Yes, we should do this upstream in https://github.com/scrosby/OSM-binary.
If somebody creates a pull request, I'll merge it there...

Jochen
-- 
Jochen Topf  joc...@remote.org  https://www.jochentopf.com/  +49-351-31778688

___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Osmosis and Osmium-enhanced PBF files with way node locations

[Brett Henderson]

Hmm, my previous email was probably hard to read.  Let me know if it
doesn't make sense.

One other thing I should mention.  GIven that you are just adding read
support, have you looked at the osmosis-pbf2 sub-project?  It contains an
alternative PBF reading implementation that I wrote to support
multi-threaded reading.  The task is registered as *--read-pbf-fast*.  The
class *org.openstreetmap.osmosis.pbf2.v0_6.impl.PbfBlobDecoder* in method
*processWays* contains the WayNode parsing logic.


On Sun, 11 Mar 2018 at 21:19 Brett Henderson  wrote:

> Hi Jon,
>
> Thanks for sending through the patch.  I've just taken a look at it.  Some
> comments:
>
> *WayNode*
> The WayNode class now has the new optional latitude and longitude fields
> which makes sense.  Can you update the store method and (StoreReader,
> StoreClassRegister) constructor to persist and load those parameters
> again?  They're needed in case the pipeline does any functionality that
> requires creating temp files such as sorting.
>
> The class is now mutable which may cause problems in a multi-threaded
> pipeline if task implementations are tempted to modify state on the fly.
> Some of the Osmosis entity classes are mutable (an historical decision
> which I regrettably allowed through), but they're protected from concurrent
> modification through a somewhat elaborate read/write protection mechanism
> (see CommonEntityData for details ...).  In this case, can we keep the
> class immutable by adding those additional fields to an overloaded
> constructor?
>
> *osmformat.proto*
> Are these changes mastered somewhere else?  In other words, are these new
> fields the same ones used by Osmium in its implementation?  I'm wondering
> if we need to re-sync from the main OSM-binary project.  The osmosis
> version is the same as that in https://github.com/scrosby/OSM-binary.
> The repo https://github.com/brettch/OSM-binary is a fork of that, and the
> osmosis branch there *is* the same code as the osmosis-osm-binary directory
> in the osmosis repo, just with some re-arranging to fit inside the Osmosis
> structure and fit inside the Osmosis java package namespace.
>
> *Jochen* (if you're reading), where does Osmium source its
> osmformat.proto file from?
>
> Long story short, rather than make changes directly to the file in Osmosis
> and create a fork, should we apply them to upstream first and then re-sync
> Osmosis with that?
>
> Otherwise, the changes look relatively straightforward.  I don't have many
> strong opinions on how to test it.  Osmosis doesn't have an amazing test
> suite, it started out as a hacked together tool and grew into something
> bigger than I planned.  I mostly rely on some basic end to end testing for
> each task that creates files and reads then back again.  That's not
> possible if you only have read support for the new file format.  We may
> need to check in a small test file with a couple of ways created by Osmium.
>
> Cheers,
> Brett
>
>
> On Tue, 6 Mar 2018 at 04:18 Locke, Jonathan  wrote:
>
>> Hi Brett,
>>
>>
>> Attached is a patch that adds *WayNode* location (latitude/longitude)
>> support to *OsmosisReader*. It seems to work fine. You can check it out
>> by uncommenting the *@Test* annotation on the test I added and supplying
>> the path to your own PBF file (I would have added a full unit test there,
>> but I didn't know how you wanted to handle data for test cases in this
>> project, so I just left it like this for now). You'll want to create your
>> PBF file with a command similar to this:
>>
>>
>> *osmium add-locations-to-ways --keep-untagged-nodes -o
>> new-mexico-latest-with-way-nodes.osm.pbf new-mexico-latest.osm.pbf*
>>
>>
>> Only one technical question for you: is there any way to detect from the
>> header of the PBF file whether the file contains way node locations? It
>> would be nice not to have to read nodes for 45 minutes before discovering
>> that the PBF file doesn't have way node locations. Once there's an osmosis
>> release with the way node location feature in it (and ideally a data drop
>> with way node locations), we hope to switch to consuming only PBF files
>> with way node locations and we'd remove support from our apps for PBF files
>> without this information (thus the need to detect if the file has way
>> nodes).
>>
>>
>> Best,
>>
>>
>> Jon
>> --
>> *From:* Brett Henderson 
>> *Sent:* Sunday, March 4, 2018 3:40:02 PM
>> *To:* Locke, Jonathan
>> *Cc:* osmosis-dev@openstreetmap.org
>>
>> *Subject:* Re: [osmosis-dev] Osmosis and Osmium-enhanced PBF files with
>> way node locations
>> It's always nice to hear that your software is useful :-)  Thanks!  Yell
>> out if you run into any problems and I'll do my best to point you in the
>> right direction.
>>
>> Cheers,
>> Brett
>>
>> On Fri, 2 Mar 2018 at 11:32 Locke, Jonathan 
>> wrote:
>>
>> Hi Brett,
>>
>> From our perspective, it's 

Re: [osmosis-dev] Osmosis and Osmium-enhanced PBF files with way node locations

[Brett Henderson]

Hi Jon,

Thanks for sending through the patch.  I've just taken a look at it.  Some
comments:

*WayNode*
The WayNode class now has the new optional latitude and longitude fields
which makes sense.  Can you update the store method and (StoreReader,
StoreClassRegister) constructor to persist and load those parameters
again?  They're needed in case the pipeline does any functionality that
requires creating temp files such as sorting.

The class is now mutable which may cause problems in a multi-threaded
pipeline if task implementations are tempted to modify state on the fly.
Some of the Osmosis entity classes are mutable (an historical decision
which I regrettably allowed through), but they're protected from concurrent
modification through a somewhat elaborate read/write protection mechanism
(see CommonEntityData for details ...).  In this case, can we keep the
class immutable by adding those additional fields to an overloaded
constructor?

*osmformat.proto*
Are these changes mastered somewhere else?  In other words, are these new
fields the same ones used by Osmium in its implementation?  I'm wondering
if we need to re-sync from the main OSM-binary project.  The osmosis
version is the same as that in https://github.com/scrosby/OSM-binary.  The
repo https://github.com/brettch/OSM-binary is a fork of that, and the
osmosis branch there *is* the same code as the osmosis-osm-binary directory
in the osmosis repo, just with some re-arranging to fit inside the Osmosis
structure and fit inside the Osmosis java package namespace.

*Jochen* (if you're reading), where does Osmium source its osmformat.proto
file from?

Long story short, rather than make changes directly to the file in Osmosis
and create a fork, should we apply them to upstream first and then re-sync
Osmosis with that?

Otherwise, the changes look relatively straightforward.  I don't have many
strong opinions on how to test it.  Osmosis doesn't have an amazing test
suite, it started out as a hacked together tool and grew into something
bigger than I planned.  I mostly rely on some basic end to end testing for
each task that creates files and reads then back again.  That's not
possible if you only have read support for the new file format.  We may
need to check in a small test file with a couple of ways created by Osmium.

Cheers,
Brett


On Tue, 6 Mar 2018 at 04:18 Locke, Jonathan  wrote:

> Hi Brett,
>
>
> Attached is a patch that adds *WayNode* location (latitude/longitude)
> support to *OsmosisReader*. It seems to work fine. You can check it out
> by uncommenting the *@Test* annotation on the test I added and supplying
> the path to your own PBF file (I would have added a full unit test there,
> but I didn't know how you wanted to handle data for test cases in this
> project, so I just left it like this for now). You'll want to create your
> PBF file with a command similar to this:
>
>
> *osmium add-locations-to-ways --keep-untagged-nodes -o
> new-mexico-latest-with-way-nodes.osm.pbf new-mexico-latest.osm.pbf*
>
>
> Only one technical question for you: is there any way to detect from the
> header of the PBF file whether the file contains way node locations? It
> would be nice not to have to read nodes for 45 minutes before discovering
> that the PBF file doesn't have way node locations. Once there's an osmosis
> release with the way node location feature in it (and ideally a data drop
> with way node locations), we hope to switch to consuming only PBF files
> with way node locations and we'd remove support from our apps for PBF files
> without this information (thus the need to detect if the file has way
> nodes).
>
>
> Best,
>
>
> Jon
> --
> *From:* Brett Henderson 
> *Sent:* Sunday, March 4, 2018 3:40:02 PM
> *To:* Locke, Jonathan
> *Cc:* osmosis-dev@openstreetmap.org
>
> *Subject:* Re: [osmosis-dev] Osmosis and Osmium-enhanced PBF files with
> way node locations
> It's always nice to hear that your software is useful :-)  Thanks!  Yell
> out if you run into any problems and I'll do my best to point you in the
> right direction.
>
> Cheers,
> Brett
>
> On Fri, 2 Mar 2018 at 11:32 Locke, Jonathan  wrote:
>
> Hi Brett,
>
> From our perspective, it's definitely worth adding this feature because we 
> use OsmosisReader in a host of custom Java applications (dozens of them). I 
> think at this point, Osmosis code is running on our servers 24/7/365 doing 
> various kinds of back-end processing for different groups around the world.
>
> I totally understand the part about not having time. I am the author of 
> Apache Wicket and I've stepped away from that project for what are probably 
> similar reasons (OSS really does soak up time like mad!). So, I will spend 
> some time developing a patch for OsmosisReader that supports this new 
> location-enhanced format and I'll get in touch when my patch is ready for 
> your review. With luck, I shouldn't have too many 

Re: [OSM-dev] GSoC

[Simon Poole]

Hi Nafiz

Where you thinking of a specific app? There is no "Android version of
OSM" as such, just a number of applications for different purposes
(routing, map display, editing).

Simon


Am 10.03.2018 um 21:18 schrieb nafiz Chowdhury:
>
> Dear sir,
> I have a project suggestion for Google Summer of Code. This is more of
> an utility feature for phone users of OSM. Could a location reminder
> feature be added to the Android version of OSM where a user can set a
> location for a task so that whenever the user is in that region, they
> will be given a notification.
> Can this be considered as a valid project? If so, what libraries would
> I have to look into to implement something like this. 
> Thanks. 
>
>
>
>
> ___
> dev mailing list
> dev@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/dev



signature.asc
Description: OpenPGP digital signature
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev