Re: [OSM-talk] Downloading Version 3 of all bus stops in a country

[Safwat Halaby]

On Wed, 2017-09-27 at 09:18 +0200, Jo wrote:
> 2017-09-27 8:30 GMT+02:00 Safwat Halaby :
> 
> > On Tue, 2017-09-26 at 11:46 +0200, Jo wrote:
> > 
> > > Then load that in PostGIS and create scripts to read GTFS into
> > > PostGIS.
> > > 
> > > Then compare the data in the DB and produce output and ideally a
> > > UI.
> > > 
> > > I started doing something like that here:
> > > 
> > > https://github.com/osmbe/public_transport
> > > 
> > > Let me know if you see ways of working on that or another way to
> > > tackle the
> > > problem together.
> > > 
> > > Jo
> > 
> > I will check the project out. Thanks for the link. Would you mind
> > explaining what it is capable of? The readme is not so descriptive.
> > 
> 
> For the time being not so much yet. Before the summer I made some
> progress
> migrating scripts I had created for an import of Belgian bus stops
> and
> lines, but during the summer I got a bit 'distracted' by mentoring
> the
> PT_Assistant plugin.
> 
> The basic idea is to take operator data, either GTFS or a dump of
> their
> internal DB.
> 
> Then compare all the stops regarding tags and position. For the stops
> the
> other tables routes, trips and segments, can be used to calculate
> route_ref.
> 
> Then create OSM route relations based on their data and compare to
> what is
> present in OSM.
> 
> This is where it becomes trickier to keep track of their versions and
> ours,
> especially if the intent is to both give feedback to the operators
> and have
> a platform showing mappers which stops, routes and route_masters are
> in
> need of attention.
> 
> Polyglot

That's very interesting. My script is far less ambitious (no routes,
only plain stops).

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Downloading Version 3 of all bus stops in a country

[Safwat Halaby]

I think my last two replies never got through and were sent privately
instead. Here's a rephrasing. (which is possibly better anyways).
__

On Wed, 2017-09-27 at 10:53 +0200, Marc Gemis wrote:
> isn't it possible that the 2017 contains data from e.g. 2014, which
> has not been reviewed in the meantime ?
> Which would then mean that the 2015 edit is more recent.
> 
> Or is there a review date in the "database" ?

There is no internal review date, and what you describe could happen.
This is the only drawback I can think of. The first run would be
imperfect in that regard, but the government publishes new GTFS dumps
on a daily basis, so after the first run this wouldn't be an issue at
all, and the "review date" would be "yesterday" or at worst "during the
last few weeks".

This is still better than any practically possible manual work
considering our mapper density and the amount of stops. No one has been
able to manually review or edit all those stops and it's been 5 years
and the stops are horribly stale.

Note that:

1. extra tags (shelter, wheelchair) will be preserved despite the
drawback
2. the new database is considered pretty good quality, so issues will
likely be negligible or nonexistent.

__

On Wed, 2017-09-27 at 11:07 +0200, Jo wrote:
> If there is a conflict regarding position or tags, they should be
> resolved
> by a human mapper. If I were to apply the newer is better approach,
> we
> would constantly be reverting back to the positions the operators
> think
> their stops are at.
> 
> It's important to respect the mappers work, because without mappers
> OSM
> quickly dies off.
> 
> It might be complex to put such a solution in place, but it should be
> obvious that if data flows in 2 directions, too much simplification
> doesn't
> work.

The script will not "constantly revert".

1. Government publishes database with bad edit X: script adds it
2. user fixes it with fix Y
3. Goverment publishes database which still has bad edit X:

The script ignores 3, because "newer is better" and the user edit is
newer. No constant reverts are involved.

However, in the following scenario, the script would indeed introduce a
bad edit to OSM:

1. Government publishes database with bad edit X: script adds it
2. user fixes it with fix Y
3. Goverment publishes database with a NEW BAD EDIT Z.

So the script rests on the assumption that whenever a government
updates a bus stop, it's because the bus stop has actually changed or
because a newer better measurement was made, so Z is always expected to
be better than X (and usually better than Y because it's newer in time)
and this scenario shouldn't exist. Z is always assumed better.

If a provider is unreliable such that it makes random edits in its new
database that are LESS accurate than its older database, then this
script is not suitable for dealing with such a provider and the
complexity of your project is needed. This does not seem to be the case
in Israel. To quote anonymous_gushdan_mapper from the forums:

> Israel has something like 30,000 bus stops,
> and they change daily all across the country.
> There's no way human mappers could ever verify
> the accuracy of all of them, unless you have
> someone working full-time on
> this. However, the data is considered extremely
> accurate, and inaccuracies are quite rare.
> We do have a system that announces the name
> of the next stop, which uses this data.

> I think people from other countries don't realize that this
> is not a single, private operator data, nor it's a single
> city data - it's government generated data that controls
> the entire public transportation network in the country.
> If a bus stop is not in this dataset, it doesn't exist.
> There will never be anything more accurate for bus stops
> in Israel than this dataset.

> If accuracy is important to us, we *must* implement this
> importing script, otherwise the data on OSM will get
> stale quickly - just like the current data is stale
> and shows a lot of bus stops that have been
> since then moved or canceled.

source: 
https://forum.openstreetmap.org/viewtopic.php?pid=665570#p665570

So, I could probably naively always trust the GTFS and it'll still be
mostly fine. But I still want to give mappers the ability to fix
coordinates and such, without overriding their edits the next run
(unless they've been updated by the government to newer values during
that time). 

Lastly, I'd like to point out that no local mappers are complaining
about this and the feedback is so far positive.

Thanks.

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Downloading Version 3 of all bus stops in a country

[Jo]

If there is a conflict regarding position or tags, they should be resolved
by a human mapper. If I were to apply the newer is better approach, we
would constantly be reverting back to the positions the operators think
their stops are at.

It's important to respect the mappers work, because without mappers OSM
quickly dies off.

It might be complex to put such a solution in place, but it should be
obvious that if data flows in 2 directions, too much simplification doesn't
work.

Jo

2017-09-27 10:46 GMT+02:00 Safwat Halaby :

> On Wed, 2017-09-27 at 09:12 +0200, Jo wrote:
> > Deleting data on OpenStreetMap and replacing it by imported data is
> > obviously never the acceptable approach.
> >
> > What I don't understand is why you don't create something that
> > compares the
> > latest version of all the bus stops in OSM with the latest version of
> > the
> > GTFS data from upstream.
> >
> > Why compare with an inbetween version?
>
> I'm taking a "Newer is better" approach for conflict resolution.
>
> Let's say a user made a 2015 edit for a coordinate (We'll call this
> version X). And your 2017 database has a different coordinate (We'll
> call this version Y). How do you determine which coordinate to keep?
>
> If you had a 2012 database that would be easy.
>
> Case 1:
> 2012 database: Y
> 2015 user edit: X
> 2017 database: Y
>
> This means the Y has not been updated since 2012. Newer is better, so
> trust the user edit and set the coordinates to X.
>
> Case 2:
> 2012 database: T
> 2015 user edit: X
> 2017 database: Y
>
> This means Y is the newest value, and we should override the user edit.
>
> Without two database, we'd have to guess or resort to manual user
> intervention via fancy web services.
>
>
> > What I think is needed is a (web) service that stands between the
> > operator
> > data, be it GTFS or DB dumps and OpenStreetMap where comparison is
> > made and
> > which can be used by mappers to either improve OSM, or to send
> > feedback to
> > the operators that there are issues with the data they provide. Or
> > where
> > the operators can request flagged stops in bulk.
> >
>
> I am a huge advocate of simplicity. In my humble opinion, web services
> and SQL databases are overkill for what I'm trying to do. Your project
> spans dozens of files while mine is a single .js file for the JOSM
> scripting plugin, which reads two textual stops.txt files from the old
> and the newer GTFS databases.
>
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Downloading Version 3 of all bus stops in a country

[Safwat Halaby]

On Wed, 2017-09-27 at 09:12 +0200, Jo wrote:
> Deleting data on OpenStreetMap and replacing it by imported data is
> obviously never the acceptable approach.
> 
> What I don't understand is why you don't create something that
> compares the
> latest version of all the bus stops in OSM with the latest version of
> the
> GTFS data from upstream.
> 
> Why compare with an inbetween version?

I'm taking a "Newer is better" approach for conflict resolution.

Let's say a user made a 2015 edit for a coordinate (We'll call this
version X). And your 2017 database has a different coordinate (We'll
call this version Y). How do you determine which coordinate to keep?

If you had a 2012 database that would be easy.

Case 1:
2012 database: Y
2015 user edit: X
2017 database: Y

This means the Y has not been updated since 2012. Newer is better, so
trust the user edit and set the coordinates to X.

Case 2:
2012 database: T
2015 user edit: X
2017 database: Y

This means Y is the newest value, and we should override the user edit.

Without two database, we'd have to guess or resort to manual user
intervention via fancy web services.


> What I think is needed is a (web) service that stands between the
> operator
> data, be it GTFS or DB dumps and OpenStreetMap where comparison is
> made and
> which can be used by mappers to either improve OSM, or to send
> feedback to
> the operators that there are issues with the data they provide. Or
> where
> the operators can request flagged stops in bulk.
> 

I am a huge advocate of simplicity. In my humble opinion, web services
and SQL databases are overkill for what I'm trying to do. Your project
spans dozens of files while mine is a single .js file for the JOSM
scripting plugin, which reads two textual stops.txt files from the old
and the newer GTFS databases.

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Downloading Version 3 of all bus stops in a country

[Jo]

2017-09-27 8:30 GMT+02:00 Safwat Halaby :

> On Tue, 2017-09-26 at 11:46 +0200, Jo wrote:
>
> > Then load that in PostGIS and create scripts to read GTFS into
> > PostGIS.
> >
> > Then compare the data in the DB and produce output and ideally a UI.
> >
> > I started doing something like that here:
> >
> > https://github.com/osmbe/public_transport
> >
> > Let me know if you see ways of working on that or another way to
> > tackle the
> > problem together.
> >
> > Jo
>
> I will check the project out. Thanks for the link. Would you mind
> explaining what it is capable of? The readme is not so descriptive.
>

For the time being not so much yet. Before the summer I made some progress
migrating scripts I had created for an import of Belgian bus stops and
lines, but during the summer I got a bit 'distracted' by mentoring the
PT_Assistant plugin.

The basic idea is to take operator data, either GTFS or a dump of their
internal DB.

Then compare all the stops regarding tags and position. For the stops the
other tables routes, trips and segments, can be used to calculate route_ref.

Then create OSM route relations based on their data and compare to what is
present in OSM.

This is where it becomes trickier to keep track of their versions and ours,
especially if the intent is to both give feedback to the operators and have
a platform showing mappers which stops, routes and route_masters are in
need of attention.

Polyglot
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Downloading Version 3 of all bus stops in a country

[Jo]

Deleting data on OpenStreetMap and replacing it by imported data is
obviously never the acceptable approach.

What I don't understand is why you don't create something that compares the
latest version of all the bus stops in OSM with the latest version of the
GTFS data from upstream.

Why compare with an inbetween version?

What I think is needed is a (web) service that stands between the operator
data, be it GTFS or DB dumps and OpenStreetMap where comparison is made and
which can be used by mappers to either improve OSM, or to send feedback to
the operators that there are issues with the data they provide. Or where
the operators can request flagged stops in bulk.

The same goes for the lines (route_master) and the various itineraries
(route relations).

Polyglot

2017-09-27 7:42 GMT+02:00 Safwat Halaby :

> Hi John Whelan,
>
> As implied in the forum thread, not wanting to destroy user data is
> exactly why I'm building a relatively complex script. The naive
> approach is to destroy all-bus stops are re-import, everytime a GTFS
> update is released. But I don't want that.
>
> Instead of doing that, the script preserves all user-submitted data
> such as shelter, wheelchair, and GPS coordinate fixes and provides
> incremental updates rather than the destruction of all bus stops per
> import. The incremental updates do not destroy user changes. In order
> to achieve that, the older and the newer GTFS database need to be
> compared per update.
>
> Yesterday I didn't have the database which was used in the old 2012
> import, so I couldn't locally test-run my script. which is what lead to
> my original question in this thread. But now I managed to reverse-build
> the old GTFS database from version 3 of the bus stops in Israel by
> downloading the bus stops through the relevant changesets.
>
> Here's some of the discussion:
>
> >Here's a merging idea.
> >
> >Problem: Dealing with conflicts between mapper
> edits and gtfs data.
> >Solution: "The most recent version is the correct
> version" philosophy.
> >
> >- The first gtfs update would update everything.
> Conflicts are
> >  resolved by prioritizing the gtfs file's version. This
> is a
> >  "necessary evil" but is >only needed once.  (edit: I might be
> >
> able to mitigate this by tracing bus stop OSM history).
> >- Some time
> passes, and users update some of the bus stops.
> >- The ministry of
> transportation updates some bus stops in
> >  its database and publishes a
> new gtfs file.
> >- The next gtfs update would inspect the difference
> between
> >  the new gtfs file and the older gtfs file. Only bus stops
> >
> that have had their data (in >the gtfs file) changed since the
> >  last
> file are updated. So, conflicts are resolved by prioritizing
> >  the gtfs
> file version, but only for the bus stops that were changed
> >  by the
> ministry since the last update. The rest of the bus stops
> >  are left
> intact.
>
> (source: https://forum.openstreetmap.org/viewtopic.php?id=16738=3)
>
> Also, we know the data can be used in OSM.
>
> https://forum.openstreetmap.org/viewtopic.php?pid=244096#p244096
>
> ___
> talk mailing list
> talk@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk
>
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Downloading Version 3 of all bus stops in a country

[Safwat Halaby]

On Tue, 2017-09-26 at 11:46 +0200, Jo wrote:

> Then load that in PostGIS and create scripts to read GTFS into
> PostGIS.
> 
> Then compare the data in the DB and produce output and ideally a UI.
> 
> I started doing something like that here:
> 
> https://github.com/osmbe/public_transport
> 
> Let me know if you see ways of working on that or another way to
> tackle the
> problem together.
> 
> Jo

I will check the project out. Thanks for the link. Would you mind
explaining what it is capable of? The readme is not so descriptive.

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Downloading Version 3 of all bus stops in a country

[Safwat Halaby]

Thanks for the info, mmd!

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Downloading Version 3 of all bus stops in a country

[Safwat Halaby]

Hi John Whelan,

As implied in the forum thread, not wanting to destroy user data is
exactly why I'm building a relatively complex script. The naive
approach is to destroy all-bus stops are re-import, everytime a GTFS
update is released. But I don't want that.

Instead of doing that, the script preserves all user-submitted data
such as shelter, wheelchair, and GPS coordinate fixes and provides
incremental updates rather than the destruction of all bus stops per
import. The incremental updates do not destroy user changes. In order
to achieve that, the older and the newer GTFS database need to be
compared per update.

Yesterday I didn't have the database which was used in the old 2012
import, so I couldn't locally test-run my script. which is what lead to
my original question in this thread. But now I managed to reverse-build 
the old GTFS database from version 3 of the bus stops in Israel by
downloading the bus stops through the relevant changesets.

Here's some of the discussion:

>Here's a merging idea.
>
>Problem: Dealing with conflicts between mapper
edits and gtfs data.
>Solution: "The most recent version is the correct
version" philosophy.
>
>- The first gtfs update would update everything.
Conflicts are
>  resolved by prioritizing the gtfs file's version. This
is a
>  "necessary evil" but is >only needed once.  (edit: I might be
> 
able to mitigate this by tracing bus stop OSM history).
>- Some time
passes, and users update some of the bus stops.
>- The ministry of
transportation updates some bus stops in
>  its database and publishes a
new gtfs file.
>- The next gtfs update would inspect the difference
between
>  the new gtfs file and the older gtfs file. Only bus stops
> 
that have had their data (in >the gtfs file) changed since the
>  last
file are updated. So, conflicts are resolved by prioritizing
>  the gtfs
file version, but only for the bus stops that were changed
>  by the
ministry since the last update. The rest of the bus stops 
>  are left
intact.

(source: https://forum.openstreetmap.org/viewtopic.php?id=16738=3)

Also, we know the data can be used in OSM.

https://forum.openstreetmap.org/viewtopic.php?pid=244096#p244096

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Downloading Version 3 of all bus stops in a country

[john whelan]

One last point have you confirmed the Open Data license is acceptable to
OSM?  It took me about five years and a great deal of effort to ensure and
confirm the licenses were compatible to get my GTFS bus stops in.

The OSM legal working group eventually agreed they were but noted they
would want to confirm every other Canadian municipal license Open Data
licence before agreeing they were compatible.

The LWG are volunteers and I understand there is a small backlog.

Cheerio John

On 26 Sep 2017 5:18 pm, "john whelan"  wrote:

> Just a clarification, I'm not against imports and I have been involved in
> a number as I'm sure Frederik will recall but it's important to me that the
> data to be imported is of good quality, is correctly licensed and is merged
> with existing data.  In the case of bus stops that means including tags
> previously mapped such as shelter and wheelchair=yes and not just deleting
> the old and dropping in the new.
>
> Mappers who have spent time tagging deserve their efforts should be
> recorded.  Although I sometimes have my doubts about a mapper who has
> mapped once for half an hour and added a very inaccurate building tagged as
> an apartment block in a small African village but in general it should be
> followed.
>
> Cheerio John
>
> On 26 Sep 2017 5:08 pm, "john whelan"  wrote:
>
>> >We are well aware that GTFS accuracy is not perfect. (Though nobody Reported
>> a 100-meter deviation so far). A potential solution, which could also
>> work globally, is discussed below.
>>
>> Some University researchers who were looking at the GTFS files noted the
>> problem sometime ago in the US.  One provider was a couple of hundred
>> metres out on bus stop location.  Locally they were a little better but it
>> was only when the bus announcement system came in and they very very
>> carefully measured the location of each bus stop to comply with the
>> provincial instructions that the GTFS file became at all usable and even
>> then it took a dedicated team to do it and it wasn't a straight import.
>>
>> Personally unless you can confirm the accuracy of the bus stops my
>> feeling is no data is better than poor data that cannot be trusted.
>>
>> Please do not do this globally.
>>
>> Cheerio John
>>
>>
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Downloading Version 3 of all bus stops in a country

[john whelan]

Just a clarification, I'm not against imports and I have been involved in a
number as I'm sure Frederik will recall but it's important to me that the
data to be imported is of good quality, is correctly licensed and is merged
with existing data.  In the case of bus stops that means including tags
previously mapped such as shelter and wheelchair=yes and not just deleting
the old and dropping in the new.

Mappers who have spent time tagging deserve their efforts should be
recorded.  Although I sometimes have my doubts about a mapper who has
mapped once for half an hour and added a very inaccurate building tagged as
an apartment block in a small African village but in general it should be
followed.

Cheerio John

On 26 Sep 2017 5:08 pm, "john whelan"  wrote:

> >We are well aware that GTFS accuracy is not perfect. (Though nobody Reported
> a 100-meter deviation so far). A potential solution, which could also
> work globally, is discussed below.
>
> Some University researchers who were looking at the GTFS files noted the
> problem sometime ago in the US.  One provider was a couple of hundred
> metres out on bus stop location.  Locally they were a little better but it
> was only when the bus announcement system came in and they very very
> carefully measured the location of each bus stop to comply with the
> provincial instructions that the GTFS file became at all usable and even
> then it took a dedicated team to do it and it wasn't a straight import.
>
> Personally unless you can confirm the accuracy of the bus stops my feeling
> is no data is better than poor data that cannot be trusted.
>
> Please do not do this globally.
>
> Cheerio John
>
>
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Downloading Version 3 of all bus stops in a country

[Safwat Halaby]

On Tue, 2017-09-26 at 17:08 -0400, john whelan wrote:
> Please do not do this globally.
> 

Of course not. I wouldn't deploy anything globally without prior
discussion.

What I meant was the script is not Israel-specific, and anyone who
knows their provider has some reasonable accuracy could use it. It
allows incremental GTFS updates where mapper-submitted accuracy fixes
are not overridden by the next update.

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Downloading Version 3 of all bus stops in a country

[john whelan]

>We are well aware that GTFS accuracy is not perfect. (Though nobody Reported
a 100-meter deviation so far). A potential solution, which could also work
globally, is discussed below.

Some University researchers who were looking at the GTFS files noted the
problem sometime ago in the US.  One provider was a couple of hundred
metres out on bus stop location.  Locally they were a little better but it
was only when the bus announcement system came in and they very very
carefully measured the location of each bus stop to comply with the
provincial instructions that the GTFS file became at all usable and even
then it took a dedicated team to do it and it wasn't a straight import.

Personally unless you can confirm the accuracy of the bus stops my feeling
is no data is better than poor data that cannot be trusted.

Please do not do this globally.

Cheerio John
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Downloading Version 3 of all bus stops in a country

[Safwat Halaby]

On Tue, 2017-09-26 at 08:50 -0400, john whelan wrote:
> I suggest pulling in the country file and if need be chop it up to
> load
> into JOSM.  Load up the latest bus stops in a different layer then
> use the
> todo plugin and go through them one at a time cleaning them up that
> way.
> 
> Note that the GTFS file bus stop location may not be accurate, some
> bus
> stops are known to be 100 meters out, it depends on the provider.  If
> you
> have a system that announces bus stops for partially sighted people
> then
> that is an indication they should be fairly good but check with the
> provider.  Even when they are accurate they can still be out by a few
> meters, locally one was in the middle of a road junction so they do
> need
> verifying.
> 

We are well aware that GTFS accuracy is not perfect. (Though nobody
reported a 100-meter deviation so far). A potential solution, which
could also work globally, is discussed below. I already have a
(somewhat) working prototype. I'll look for other solutions too.

https://forum.openstreetmap.org/viewtopic.php?id=16738=3

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Downloading Version 3 of all bus stops in a country

[Safwat Halaby]

link fix:
 https://forum.openstreetmap.org/viewtopic.php?id=16738

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Downloading Version 3 of all bus stops in a country

[Safwat Halaby]

I'd like to point out (as I pointed out in the sub-threads), that I
solved this problem simply by fetching the changeset which introduced
v3.

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Downloading Version 3 of all bus stops in a country

[Safwat Halaby]

On Tue, 2017-09-26 at 14:12 +0200, Frederik Ramm wrote:
> 
> Are all these automatic edits at least discussed on the Israeli
> mailing
> list, or is this a free-for-all where anyone who knows how to write a
> script runs over all bus stops in Israel again and again?
> 
> Bye
> Frederik
> 

Why are you automatically assuming the worst? I have, and I am,
extensively discussing fixing the import mess. Even the previously
messy imports by the other users were discussed and were fairly
coordinated. See this: https://forum.openstreetmap.org/viewtopic.php?id
=16738



___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Downloading Version 3 of all bus stops in a country

[mmd]

Am 26.09.2017 um 12:39 schrieb Safwat Halaby:
> On Tue, 2017-09-26 at 11:46 +0200, Jo wrote:

>>
>> Overpass API is definitely your friend. Version 3 doesn't mean much
>> though.
>> Do you mean all bus stops with a public_transport tag?
> 
> Thanks for the reply.
> 
> By version 3, I mean a particular historic version of the OSM element. 
> 
> For instance, see "Version #3" of this bus stop:
> 
> https://www.openstreetmap.org/node/1802982884/history
> 
> I need all "version #3" of all bus stops that have a
> "source=israel_gtfs_v1" in Israel. As far as I know. Overpass can only
> fetch the latest version. Can overpass directly fetch version #3?
> 

That's not yet possible with release 0.7.54.

A future Overpass release will allow you to filter for a specific
version of an object, which exists at a given point in time. That's
either NOW, or the date you specify via [date: ...].

A query for all version 3 objects would only work, if you can find such
a point in time, where all of those objects are valid. If there isn't
such date, you need to try several queries with varying date.

Here's some to try out in the meantime: http://overpass-turbo.eu/s/rZc


-- 




___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Downloading Version 3 of all bus stops in a country

[john whelan]

I suggest pulling in the country file and if need be chop it up to load
into JOSM.  Load up the latest bus stops in a different layer then use the
todo plugin and go through them one at a time cleaning them up that way.

Note that the GTFS file bus stop location may not be accurate, some bus
stops are known to be 100 meters out, it depends on the provider.  If you
have a system that announces bus stops for partially sighted people then
that is an indication they should be fairly good but check with the
provider.  Even when they are accurate they can still be out by a few
meters, locally one was in the middle of a road junction so they do need
verifying.

Cheerio John

On 26 September 2017 at 08:12, Frederik Ramm  wrote:

> Hi,
>
> On 26.09.2017 13:31, Safwat Halaby wrote:
> > V3 was an automatic import for all nodes. it was made pretty quickly
> > after v1 and v2
>
> This whole import is a terrible mess:
>
> https://www.openstreetmap.org/node/1802982151/history
>
> It starts with the version 1 already being titled "attempt 3", then in
> version 2 a "fixme = check/adjust position and/or merge with existing
> stop if exists" is added which of course is another wording for "this
> import should not have happened in the first place"! Later all name tags
> were duplicated in name:he and the latest three versions are also some
> form of automated doctoring.
>
> A month ago you've removed the "fixme=..." from 30k nodes and replaced
> it with a "gtfs:reviewed=no", creating yet another version of the whole
> data set in our database. What makes you think that, after the "fixme"
> tags have *not* been acted on since the original import 5 years ago,
> changing this to "gtfs:reviewed=no" will suddenly have an effect?
>
> Are all these automatic edits at least discussed on the Israeli mailing
> list, or is this a free-for-all where anyone who knows how to write a
> script runs over all bus stops in Israel again and again?
>
> Bye
> Frederik
>
> --
> Frederik Ramm  ##  eMail frede...@remote.org  ##  N49°00'09" E008°23'33"
>
> ___
> talk mailing list
> talk@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk
>
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Downloading Version 3 of all bus stops in a country

[Frederik Ramm]

Hi,

On 26.09.2017 13:31, Safwat Halaby wrote:
> V3 was an automatic import for all nodes. it was made pretty quickly
> after v1 and v2

This whole import is a terrible mess:

https://www.openstreetmap.org/node/1802982151/history

It starts with the version 1 already being titled "attempt 3", then in
version 2 a "fixme = check/adjust position and/or merge with existing
stop if exists" is added which of course is another wording for "this
import should not have happened in the first place"! Later all name tags
were duplicated in name:he and the latest three versions are also some
form of automated doctoring.

A month ago you've removed the "fixme=..." from 30k nodes and replaced
it with a "gtfs:reviewed=no", creating yet another version of the whole
data set in our database. What makes you think that, after the "fixme"
tags have *not* been acted on since the original import 5 years ago,
changing this to "gtfs:reviewed=no" will suddenly have an effect?

Are all these automatic edits at least discussed on the Israeli mailing
list, or is this a free-for-all where anyone who knows how to write a
script runs over all bus stops in Israel again and again?

Bye
Frederik

-- 
Frederik Ramm  ##  eMail frede...@remote.org  ##  N49°00'09" E008°23'33"

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Downloading Version 3 of all bus stops in a country

[Jo]

It is a bit odd that you are not interested in the latest versions of these
objects, but OK. (As long as you don't plan to use them for an updated
import to OSM, of course)

Overpass API also makes it possible to fetch data for a given point in time.

Jo

2017-09-26 13:31 GMT+02:00 Safwat Halaby :

> On Tue, 2017-09-26 at 12:15 +0100, Tom Hughes wrote:
> > On 26/09/17 11:41, Safwat Halaby wrote:
> > > On Tue, 2017-09-26 at 11:31 +0200, Michael Reichert wrote:
> > > > Hi,
> > > >
> > > > Am 2017-09-26 um 09:19 schrieb SwiftFast:
> > > > > I need to download all version 3 bus stops in a country which
> > > > > has
> > > > > about
> > > > > 30,000 bus stops. Version 3 exists since a 2012 import. What's
> > > > > the
> > > > > recommended way to accomplish this? I have two ways in mind.
> > > >
> > > > Version 3 of what?
> > > >
> > > > Best regards
> > > >
> > >
> > > Version 3 of all Israeli bus stops that have
> > > "source=israel_gtfs_v1".
> > > See this: https://www.openstreetmap.org/node/1802982884/history and
> > > look at "version #3".
> >
> > But how do you know it's version 3 you want for every one?
> >
> > It sounds like you're making an assumption that all these objects
> > have
> > only been edited in a particular way by some automated process and
> > that
> > nobody has ever touched one by hand and hence added an extra version.
> >
> > Tom
> >
>
> V3 was an automatic import for all nodes. it was made pretty quickly
> after v1 and v2 and it's very unlikely someone edited during that time.
> But I solved my problems simply by fetching the responsible changeset.
> I was overcomplicating this problem. All I needed was:
>
> get https://api.openstreetmap.org/api/0.6/changeset/14265835/download
>
> Thanks for the help!
>
> ___
> talk mailing list
> talk@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk
>
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Downloading Version 3 of all bus stops in a country

[Safwat Halaby]

On Tue, 2017-09-26 at 12:15 +0100, Tom Hughes wrote:
> On 26/09/17 11:41, Safwat Halaby wrote:
> > On Tue, 2017-09-26 at 11:31 +0200, Michael Reichert wrote:
> > > Hi,
> > > 
> > > Am 2017-09-26 um 09:19 schrieb SwiftFast:
> > > > I need to download all version 3 bus stops in a country which
> > > > has
> > > > about
> > > > 30,000 bus stops. Version 3 exists since a 2012 import. What's
> > > > the
> > > > recommended way to accomplish this? I have two ways in mind.
> > > 
> > > Version 3 of what?
> > > 
> > > Best regards
> > > 
> > 
> > Version 3 of all Israeli bus stops that have
> > "source=israel_gtfs_v1".
> > See this: https://www.openstreetmap.org/node/1802982884/history and
> > look at "version #3".
> 
> But how do you know it's version 3 you want for every one?
> 
> It sounds like you're making an assumption that all these objects
> have 
> only been edited in a particular way by some automated process and
> that 
> nobody has ever touched one by hand and hence added an extra version.
> 
> Tom
> 

V3 was an automatic import for all nodes. it was made pretty quickly
after v1 and v2 and it's very unlikely someone edited during that time.
But I solved my problems simply by fetching the responsible changeset.
I was overcomplicating this problem. All I needed was:

get https://api.openstreetmap.org/api/0.6/changeset/14265835/download

Thanks for the help!

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Downloading Version 3 of all bus stops in a country

[Safwat Halaby]

I think I've needlessly overcomplicated this. All I need to do was to
do is fetch changeset #14265835, which introduced those historic nodes.

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Downloading Version 3 of all bus stops in a country

[Tom Hughes]

On 26/09/17 11:41, Safwat Halaby wrote:

On Tue, 2017-09-26 at 11:31 +0200, Michael Reichert wrote:

Hi,

Am 2017-09-26 um 09:19 schrieb SwiftFast:

I need to download all version 3 bus stops in a country which has
about
30,000 bus stops. Version 3 exists since a 2012 import. What's the
recommended way to accomplish this? I have two ways in mind.


Version 3 of what?

Best regards



Version 3 of all Israeli bus stops that have "source=israel_gtfs_v1".
See this: https://www.openstreetmap.org/node/1802982884/history and
look at "version #3".


But how do you know it's version 3 you want for every one?

It sounds like you're making an assumption that all these objects have 
only been edited in a particular way by some automated process and that 
nobody has ever touched one by hand and hence added an extra version.


Tom

--
Tom Hughes (t...@compton.nu)
http://compton.nu/

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Downloading Version 3 of all bus stops in a country

[Safwat Halaby]

On Tue, 2017-09-26 at 11:31 +0200, Michael Reichert wrote:
> Hi,
> 
> Am 2017-09-26 um 09:19 schrieb SwiftFast:
> > I need to download all version 3 bus stops in a country which has
> > about
> > 30,000 bus stops. Version 3 exists since a 2012 import. What's the
> > recommended way to accomplish this? I have two ways in mind.
> 
> Version 3 of what?
> 
> Best regards
> 

Version 3 of all Israeli bus stops that have "source=israel_gtfs_v1".
See this: https://www.openstreetmap.org/node/1802982884/history and
look at "version #3".

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Downloading Version 3 of all bus stops in a country

[Safwat Halaby]

On Tue, 2017-09-26 at 11:46 +0200, Jo wrote:
> Hi Safwat,
> 
> Overpass API is definitely your friend. Version 3 doesn't mean much
> though.
> Do you mean all bus stops with a public_transport tag?

Thanks for the reply.

By version 3, I mean a particular historic version of the OSM element. 

For instance, see "Version #3" of this bus stop:

https://www.openstreetmap.org/node/1802982884/history

I need all "version #3" of all bus stops that have a
"source=israel_gtfs_v1" in Israel. As far as I know. Overpass can only
fetch the latest version. Can overpass directly fetch version #3?

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Downloading Version 3 of all bus stops in a country

[Jo]

Hi Safwat,

Overpass API is definitely your friend. Version 3 doesn't mean much though.
Do you mean all bus stops with a public_transport tag?

I would go about it in a slightly different way and download everything
related to public transport, not just the stops using the query found on
this page:

https://wiki.openstreetmap.org/wiki/JOSM/Plugins/PT_Assistant/Mapping_Public_Transport_with_JOSM#Downloading_data

Then load that in PostGIS and create scripts to read GTFS into PostGIS.

Then compare the data in the DB and produce output and ideally a UI.

I started doing something like that here:

https://github.com/osmbe/public_transport

Let me know if you see ways of working on that or another way to tackle the
problem together.

Jo

2017-09-26 9:19 GMT+02:00 SwiftFast :

> I need to download all version 3 bus stops in a country which has about
> 30,000 bus stops. Version 3 exists since a 2012 import. What's the
> recommended way to accomplish this? I have two ways in mind.
>
> 1. Fetch all stops with Overpass API, then, for each
>stop id, fetch version 3 from the main API.
>- Pros: I already know how to do that
>- Cons: Potentially too much load on the OSM server.
>Will I be rate-limited?
>Requires some script writing which could take
>more time than the way below.
>
> 2. Download a history-supporting planet file,
>and extract the desired bus stops through Osmosis.
>- Pros: Less API load
>- Cons: I'm not sure what the proper Osmosis commands are,
>and if this  is a supported operation. Also, I need
>to download a big file (the oldest history supporting
>import is 40GB from 2013).
>
> 3. Other suggestions?
>
> Why I need this:
>
> I am creating a script[1] for incrementally updating GTFS imports.
> It'll initially be used in Israel but it can be adopted worldwide. We
> have an import from 2012 which was "dumped and forgotten" and is now
> very stale. I want to forever fix this.
>
> The ministry of transportation publishes new GTFS files daily. My
> script compares an older GTFS file with newer one, and only updates the
> bus stops that were changed/added/deleted. The idea is to feed it a
> newer GTFS database daily or weekly. However, I do not possess the file
> that was used in the 2012 import, so, in order to bootstrap the script,
> I need to recreate that file from the old OSM data.
>
> [1] https://forum.openstreetmap.org/viewtopic.php?id=16738=3
> (note: SwiftFast and SafwatHalaby are both me)
>
> Thanks in advance.
>
> ___
> talk mailing list
> talk@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk
>
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Downloading Version 3 of all bus stops in a country

[Michael Reichert]

Hi,

Am 2017-09-26 um 09:19 schrieb SwiftFast:
> I need to download all version 3 bus stops in a country which has about
> 30,000 bus stops. Version 3 exists since a 2012 import. What's the
> recommended way to accomplish this? I have two ways in mind.

Version 3 of what?

Best regards

Michael


-- 
Per E-Mail kommuniziere ich bevorzugt GPG-verschlüsselt. (Mailinglisten
ausgenommen)
I prefer GPG encryption of emails. (does not apply on mailing lists)



signature.asc
Description: OpenPGP digital signature
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk