Re: [OSM-dev] OSM->PostGIS Python tool?

2018-12-21 Thread Martijn van Exel
Jo — interesting, I wasn’t aware (or had forgotten..) that there was another 
Python Overpass interface. It seems we have
https://github.com/DinoTools/python-overpy 
 and the one I maintain  
https://github.com/mvexel/overpass-api-python-wrapper 
 .

It looks like they offer similar functionality. I would be curious to find out 
if you (or anyone here) has tried both and prefers one over the other, and why? 
Does anyone know the developer of overpy? I would prefer it if we could combine 
efforts rather than maintain two separate projects. The python package world is 
already messy and I hate contributing to that :/ 

Martijn

> On Dec 21, 2018, at 11:15 AM, Jo  wrote:
> 
> Hi Spencer,
> 
> I'm using Overpy to fetch data using Overpass API, then process it, then send 
> it to JOSM using its remote control feature:
> 
> https://github.com/osmbe/public_transport.git 
> 
> https://github.com/osmbe/public_transport/blob/master/scripts/OSM_data_model.py
>  
> 
> https://github.com/osmbe/public_transport/blob/master/scripts/Agency.py 
> 
> 
> You can experiment with it in this Notebook:
> 
> https://colab.research.google.com/drive/14ggsAKrs9gs03AVSNIGlWsLWJEq7EAaF#scrollTo=4GOsvegsGPxM
>  
> 
> 
> 
> Maybe it helps. In a previous iteration all this did involve loading the data 
> into a PostGIS database:
> 
> https://github.com/PolyglotOpenstreetmap/OSM_PublicTransportRoutes 
> 
> 
> Polyglot
> 
> On Fri, Dec 21, 2018 at 2:16 PM Spencer Gardner  > wrote:
> I'm researching options for a Python-based tool that uses OSM data. From what 
> I can gather there's no native Python library for OSM imports to a PostGIS 
> database. (Yes, imposm is developed in Python but there's no documentation I 
> can find on how to use it as a Python library--it appears to be intended as a 
> command line tool). This seems odd to me since there's such a large community 
> of OSM users. I feel like I'm missing something. Are there other viable 
> Python libraries I'm not aware of? (Viable = large-ish user base and history 
> of bugfixing.) Is there documentation for using imposm within Python that 
> I've overlooked?
> 
> Bonus points for:
> - Windows compatibility
> - No external (i.e. non-Python) libraries needed
> 
> Thanks for any help!
> ___
> dev mailing list
> dev@openstreetmap.org 
> https://lists.openstreetmap.org/listinfo/dev 
> 
> ___
> dev mailing list
> dev@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/dev

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] OSM->PostGIS Python tool?

2018-12-21 Thread Sarah Hoffmann
Hi,

On Thu, Dec 20, 2018 at 08:54:34PM -0800, Spencer Gardner wrote:
> I'm researching options for a Python-based tool that uses OSM data. From
> what I can gather there's no native Python library for OSM imports to a
> PostGIS database. (Yes, imposm is developed in Python but there's no
> documentation I can find on how to use it as a Python library--it appears
> to be intended as a command line tool). This seems odd to me since there's
> such a large community of OSM users. I feel like I'm missing something. Are
> there other viable Python libraries I'm not aware of? (Viable = large-ish
> user base and history of bugfixing.) Is there documentation for using
> imposm within Python that I've overlooked?
> 
> Bonus points for:
> - Windows compatibility
> - No external (i.e. non-Python) libraries needed

If you are looking for something purely Python, then you can combine
pyosmium for reading OSM data with SQLAlchemy for writing it into
the database. This gives you maximum flexibility for the database
schema.

For an example on how to do it:
https://github.com/waymarkedtrails/osgende/blob/master/tools/osgende-import

This tool only imports the raw OSM data. If you want preprocessed
geometries then you would need to have a look at pyosmiums shapely
example. 

This is quite a bit slower than the Java and C++ importers mentioned in this
thread but that would only be relevant if you do regular full planet
imports.

Regarding Windows compatibility: pyosmium does have a native library part.
We do create pre-compiled versions for installation via pip but only for
a handful of python versions and so far I had neither positive nor negative
feedback if that actually works as intended.

Kind regards

Sarah

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] OSM->PostGIS Python tool?

2018-12-21 Thread Jo
Hi Spencer,

I'm using Overpy to fetch data using Overpass API, then process it, then
send it to JOSM using its remote control feature:

https://github.com/osmbe/public_transport.git
https://github.com/osmbe/public_transport/blob/master/scripts/OSM_data_model.py
https://github.com/osmbe/public_transport/blob/master/scripts/Agency.py

You can experiment with it in this Notebook:

https://colab.research.google.com/drive/14ggsAKrs9gs03AVSNIGlWsLWJEq7EAaF#scrollTo=4GOsvegsGPxM


Maybe it helps. In a previous iteration all this did involve loading the
data into a PostGIS database:

https://github.com/PolyglotOpenstreetmap/OSM_PublicTransportRoutes

Polyglot

On Fri, Dec 21, 2018 at 2:16 PM Spencer Gardner 
wrote:

> I'm researching options for a Python-based tool that uses OSM data. From
> what I can gather there's no native Python library for OSM imports to a
> PostGIS database. (Yes, imposm is developed in Python but there's no
> documentation I can find on how to use it as a Python library--it appears
> to be intended as a command line tool). This seems odd to me since there's
> such a large community of OSM users. I feel like I'm missing something. Are
> there other viable Python libraries I'm not aware of? (Viable = large-ish
> user base and history of bugfixing.) Is there documentation for using
> imposm within Python that I've overlooked?
>
> Bonus points for:
> - Windows compatibility
> - No external (i.e. non-Python) libraries needed
>
> Thanks for any help!
> ___
> dev mailing list
> dev@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/dev
>
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] OSM->PostGIS Python tool?

2018-12-21 Thread Martijn van Exel
Spencer, 

I've done numerous Python based projects around OSM data, but I usually treat 
the data loading as a separate process outside of my applications.

For bulk loading OSM data into PostGIS I would recommend osmosis if you want a 
lossless representation with optional direct access to geometry. osmosis can 
also be used to keep your image of OSM up-to-date. Note that initial loading of 
the entire OSM planet takes quite some time (more than a day last time I did 
it).

If you just require access to small chunks of data at any given time (city 
level or smaller) you can ingest the data on-demand using the Overpass server, 
for which I maintain a python interface. Overpass also allows you to target 
very specific subsets of OSM data. Depending on your use case that may be a 
preferred option.
--
 Martijn van Exel
 m...@rtijn.org


On Fri, Dec 21, 2018, at 06:16, Spencer Gardner wrote:
> I'm researching options for a Python-based tool that uses OSM data. From what 
> I can gather there's no native Python library for OSM imports to a PostGIS 
> database. (Yes, imposm is developed in Python but there's no documentation I 
> can find on how to use it as a Python library--it appears to be intended as a 
> command line tool). This seems odd to me since there's such a large community 
> of OSM users. I feel like I'm missing something. Are there other viable 
> Python libraries I'm not aware of? (Viable = large-ish user base and history 
> of bugfixing.) Is there documentation for using imposm within Python that 
> I've overlooked?
> 
> Bonus points for:
> - Windows compatibility
> - No external (i.e. non-Python) libraries needed
> 
> Thanks for any help!
> ___
> dev mailing list
> dev@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/dev
> 
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] OpenStreetMap Carto release v4.18.0

2018-12-21 Thread Daniel Koć
W dniu 21.12.2018 o 16:49, Dave F pisze:
> What does 'later' mean:
>
> On 21/12/2018 15:36, Daniel Koć wrote:
>> Moving railway=tram_stop and station=subway later


We usually refer this way to higher zoom levels. In this case it was
moving from z12+ and z13+ respectively to z14+ (and "+" means "starting
from this level to all the higher ones").


-- 
"I see dead people" [Sixth Sense]



___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] OpenStreetMap Carto release v4.18.0

2018-12-21 Thread Dave F

Hi

What does 'later' mean:

On 21/12/2018 15:36, Daniel Koć wrote:

Moving railway=tram_stop and station=subway later


DaveF

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


[OSM-dev] OpenStreetMap Carto release v4.18.0

2018-12-21 Thread Daniel Koć
Dear all,

Today, v4.18.0 of the OpenStreetMap Carto stylesheet (the default
stylesheet on the OSM website) has been released. Once changes are
deployed on the openstreetmap.org it will take couple of days before
all tiles show the new rendering.

Changes include


- Adding rendering for healthcare tag scheme
  - Changing hospital icon
  - Changing healthcare color to red
  - Changing natural=scrub color
  - Changing landuse=allotments color and pattern
- Adding rendering for natural=cape
- Rendering leisure=ice_rink
- Adding rendering for man_made=crane
- Adding icons for shop=fabric and shop=carpet
- Updating icons for amenity=arts_centre, leisure=slipway,
amenity=restaurant/amenity=food_court and
man_made=storage_tank/man_made=silo
- Using dedicated icon for artwork_type=bust
- Rendering railway pattern on z12
- Showing labels of big states (like Alaska)
- Moving railway=tram_stop and station=subway later
- Adding rendering for more private POIs
- Removing smoothing in leisure=track and attraction=water_slide
- Using subway bridge style for subway construction bridges
- Rendering wind turbines names and other tweaks
- Changing man_made gray and text color, making text-dy uniform
- Small documentation and code fixes

Thanks to all the contributors for this release including tpetillon, a
new contributor.

For a full list of commits, see
https://github.com/gravitystorm/openstreetmap-carto/compare/v4.17.0...v4.18.0

As always, we welcome any bug reports at
https://github.com/gravitystorm/openstreetmap-carto/issues

-- 
"Excuse me, I have some growing up to do" [P. Gabriel]



___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] OSM->PostGIS Python tool?

2018-12-21 Thread Pierre Béland
For Building quality analysis, I extract data from Overpass-API and I import 
into PostgreSQL/PostGIS schemas,  I use a solution based on osmosis 
pgsnapshot_schema_0.6. This solution similar to OSM2Pgsql is well documented 
and facilitates a lot importing OSM database extracs into PostGIS.

Telenav uses a similar approach that they documented on Github
See https://github.com/TelenavMapping/verifying_our_edits
The Osmosis instruction to import in  PostgreSQL, do not let specify a 
particular database schema. Like in the main.sh script, the solution is to 
first load data in the public schema. Once loaded, it is moved it to a 
particular schema. 

 
Pierre 
 

Le vendredi 21 décembre 2018 09 h 08 min 41 s HNE, Michael Reichert 
 a écrit :  
 
 Hi Spencer,

Am 21/12/2018 um 05.54 schrieb Spencer Gardner:
> I'm researching options for a Python-based tool that uses OSM data. From
> what I can gather there's no native Python library for OSM imports to a
> PostGIS database. (Yes, imposm is developed in Python but there's no
> documentation I can find on how to use it as a Python library--it appears
> to be intended as a command line tool). This seems odd to me since there's
> such a large community of OSM users. I feel like I'm missing something. Are
> there other viable Python libraries I'm not aware of? (Viable = large-ish
> user base and history of bugfixing.) Is there documentation for using
> imposm within Python that I've overlooked?

There are currently two serious import programmes for OSM data into
PostGIS databases – Osm2pgsql and Imposm 3. The first one is written in
C++ (formerly C), the second one in Go.

I am not aware of any well-known Python library for an import. Given the
typical use cases and the speed of existing solutions, it is unlikely
that there is any. There are Python bindings of the C++ library
Libosmium. The bindings are called Pyosmium. Libosmium is the fastest
library to deal with OSM data.

Why do you need a library for the import? Which use cases are not
covered by Osm2pgsql or Imposm 3?

Please keep in mind that OSM data is large (our planet is about 40 GB in
.osm.pbf format). Using a interpreted language is a waste of time and
energy for most purposes.

> Bonus points for:
> - Windows compatibility
> - No external (i.e. non-Python) libraries needed

Osm2pgsql runs under Windows.

Best regards

Michael

-- 
Per E-Mail kommuniziere ich bevorzugt GPG-verschlüsselt. (Mailinglisten
ausgenommen)
I prefer GPG encryption of emails. (does not apply on mailing lists)
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev
  ___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] OSM->PostGIS Python tool?

2018-12-21 Thread Michael Reichert
Hi Spencer,

Am 21/12/2018 um 05.54 schrieb Spencer Gardner:
> I'm researching options for a Python-based tool that uses OSM data. From
> what I can gather there's no native Python library for OSM imports to a
> PostGIS database. (Yes, imposm is developed in Python but there's no
> documentation I can find on how to use it as a Python library--it appears
> to be intended as a command line tool). This seems odd to me since there's
> such a large community of OSM users. I feel like I'm missing something. Are
> there other viable Python libraries I'm not aware of? (Viable = large-ish
> user base and history of bugfixing.) Is there documentation for using
> imposm within Python that I've overlooked?

There are currently two serious import programmes for OSM data into
PostGIS databases – Osm2pgsql and Imposm 3. The first one is written in
C++ (formerly C), the second one in Go.

I am not aware of any well-known Python library for an import. Given the
typical use cases and the speed of existing solutions, it is unlikely
that there is any. There are Python bindings of the C++ library
Libosmium. The bindings are called Pyosmium. Libosmium is the fastest
library to deal with OSM data.

Why do you need a library for the import? Which use cases are not
covered by Osm2pgsql or Imposm 3?

Please keep in mind that OSM data is large (our planet is about 40 GB in
.osm.pbf format). Using a interpreted language is a waste of time and
energy for most purposes.

> Bonus points for:
> - Windows compatibility
> - No external (i.e. non-Python) libraries needed

Osm2pgsql runs under Windows.

Best regards

Michael

-- 
Per E-Mail kommuniziere ich bevorzugt GPG-verschlüsselt. (Mailinglisten
ausgenommen)
I prefer GPG encryption of emails. (does not apply on mailing lists)



signature.asc
Description: OpenPGP digital signature
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


[OSM-dev] OSM->PostGIS Python tool?

2018-12-21 Thread Spencer Gardner
I'm researching options for a Python-based tool that uses OSM data. From
what I can gather there's no native Python library for OSM imports to a
PostGIS database. (Yes, imposm is developed in Python but there's no
documentation I can find on how to use it as a Python library--it appears
to be intended as a command line tool). This seems odd to me since there's
such a large community of OSM users. I feel like I'm missing something. Are
there other viable Python libraries I'm not aware of? (Viable = large-ish
user base and history of bugfixing.) Is there documentation for using
imposm within Python that I've overlooked?

Bonus points for:
- Windows compatibility
- No external (i.e. non-Python) libraries needed

Thanks for any help!
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] renderd queue

2018-12-21 Thread Martin Koppenhoefer
Am Do., 20. Dez. 2018 um 19:20 Uhr schrieb Kojo :

> Hi all,
>
> How do I check the renderd queue?
>
> I am having a huge database load, even if there are no incoming requests
> for not rendered tiles.
> I presume that there is a queue in process using my database (Postgres]
>  resources. Is it correct?
>


maybe autovacuum? It is enabled by default.

Cheers,
Martin
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev