Re: [OSM-dev] OSM->PostGIS Python tool?

2018-12-21 Thread Martijn van Exel
Jo — interesting, I wasn’t aware (or had forgotten..) that there was another 
Python Overpass interface. It seems we have
https://github.com/DinoTools/python-overpy 
 and the one I maintain  
https://github.com/mvexel/overpass-api-python-wrapper 
 .

It looks like they offer similar functionality. I would be curious to find out 
if you (or anyone here) has tried both and prefers one over the other, and why? 
Does anyone know the developer of overpy? I would prefer it if we could combine 
efforts rather than maintain two separate projects. The python package world is 
already messy and I hate contributing to that :/ 

Martijn

> On Dec 21, 2018, at 11:15 AM, Jo  wrote:
> 
> Hi Spencer,
> 
> I'm using Overpy to fetch data using Overpass API, then process it, then send 
> it to JOSM using its remote control feature:
> 
> https://github.com/osmbe/public_transport.git 
> 
> https://github.com/osmbe/public_transport/blob/master/scripts/OSM_data_model.py
>  
> 
> https://github.com/osmbe/public_transport/blob/master/scripts/Agency.py 
> 
> 
> You can experiment with it in this Notebook:
> 
> https://colab.research.google.com/drive/14ggsAKrs9gs03AVSNIGlWsLWJEq7EAaF#scrollTo=4GOsvegsGPxM
>  
> 
> 
> 
> Maybe it helps. In a previous iteration all this did involve loading the data 
> into a PostGIS database:
> 
> https://github.com/PolyglotOpenstreetmap/OSM_PublicTransportRoutes 
> 
> 
> Polyglot
> 
> On Fri, Dec 21, 2018 at 2:16 PM Spencer Gardner  > wrote:
> I'm researching options for a Python-based tool that uses OSM data. From what 
> I can gather there's no native Python library for OSM imports to a PostGIS 
> database. (Yes, imposm is developed in Python but there's no documentation I 
> can find on how to use it as a Python library--it appears to be intended as a 
> command line tool). This seems odd to me since there's such a large community 
> of OSM users. I feel like I'm missing something. Are there other viable 
> Python libraries I'm not aware of? (Viable = large-ish user base and history 
> of bugfixing.) Is there documentation for using imposm within Python that 
> I've overlooked?
> 
> Bonus points for:
> - Windows compatibility
> - No external (i.e. non-Python) libraries needed
> 
> Thanks for any help!
> ___
> dev mailing list
> dev@openstreetmap.org 
> https://lists.openstreetmap.org/listinfo/dev 
> 
> ___
> dev mailing list
> dev@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/dev

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] OSM->PostGIS Python tool?

2018-12-21 Thread Sarah Hoffmann
Hi,

On Thu, Dec 20, 2018 at 08:54:34PM -0800, Spencer Gardner wrote:
> I'm researching options for a Python-based tool that uses OSM data. From
> what I can gather there's no native Python library for OSM imports to a
> PostGIS database. (Yes, imposm is developed in Python but there's no
> documentation I can find on how to use it as a Python library--it appears
> to be intended as a command line tool). This seems odd to me since there's
> such a large community of OSM users. I feel like I'm missing something. Are
> there other viable Python libraries I'm not aware of? (Viable = large-ish
> user base and history of bugfixing.) Is there documentation for using
> imposm within Python that I've overlooked?
> 
> Bonus points for:
> - Windows compatibility
> - No external (i.e. non-Python) libraries needed

If you are looking for something purely Python, then you can combine
pyosmium for reading OSM data with SQLAlchemy for writing it into
the database. This gives you maximum flexibility for the database
schema.

For an example on how to do it:
https://github.com/waymarkedtrails/osgende/blob/master/tools/osgende-import

This tool only imports the raw OSM data. If you want preprocessed
geometries then you would need to have a look at pyosmiums shapely
example. 

This is quite a bit slower than the Java and C++ importers mentioned in this
thread but that would only be relevant if you do regular full planet
imports.

Regarding Windows compatibility: pyosmium does have a native library part.
We do create pre-compiled versions for installation via pip but only for
a handful of python versions and so far I had neither positive nor negative
feedback if that actually works as intended.

Kind regards

Sarah

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] OSM->PostGIS Python tool?

2018-12-21 Thread Jo
Hi Spencer,

I'm using Overpy to fetch data using Overpass API, then process it, then
send it to JOSM using its remote control feature:

https://github.com/osmbe/public_transport.git
https://github.com/osmbe/public_transport/blob/master/scripts/OSM_data_model.py
https://github.com/osmbe/public_transport/blob/master/scripts/Agency.py

You can experiment with it in this Notebook:

https://colab.research.google.com/drive/14ggsAKrs9gs03AVSNIGlWsLWJEq7EAaF#scrollTo=4GOsvegsGPxM


Maybe it helps. In a previous iteration all this did involve loading the
data into a PostGIS database:

https://github.com/PolyglotOpenstreetmap/OSM_PublicTransportRoutes

Polyglot

On Fri, Dec 21, 2018 at 2:16 PM Spencer Gardner 
wrote:

> I'm researching options for a Python-based tool that uses OSM data. From
> what I can gather there's no native Python library for OSM imports to a
> PostGIS database. (Yes, imposm is developed in Python but there's no
> documentation I can find on how to use it as a Python library--it appears
> to be intended as a command line tool). This seems odd to me since there's
> such a large community of OSM users. I feel like I'm missing something. Are
> there other viable Python libraries I'm not aware of? (Viable = large-ish
> user base and history of bugfixing.) Is there documentation for using
> imposm within Python that I've overlooked?
>
> Bonus points for:
> - Windows compatibility
> - No external (i.e. non-Python) libraries needed
>
> Thanks for any help!
> ___
> dev mailing list
> dev@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/dev
>
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] OSM->PostGIS Python tool?

2018-12-21 Thread Martijn van Exel
Spencer, 

I've done numerous Python based projects around OSM data, but I usually treat 
the data loading as a separate process outside of my applications.

For bulk loading OSM data into PostGIS I would recommend osmosis if you want a 
lossless representation with optional direct access to geometry. osmosis can 
also be used to keep your image of OSM up-to-date. Note that initial loading of 
the entire OSM planet takes quite some time (more than a day last time I did 
it).

If you just require access to small chunks of data at any given time (city 
level or smaller) you can ingest the data on-demand using the Overpass server, 
for which I maintain a python interface. Overpass also allows you to target 
very specific subsets of OSM data. Depending on your use case that may be a 
preferred option.
--
 Martijn van Exel
 m...@rtijn.org


On Fri, Dec 21, 2018, at 06:16, Spencer Gardner wrote:
> I'm researching options for a Python-based tool that uses OSM data. From what 
> I can gather there's no native Python library for OSM imports to a PostGIS 
> database. (Yes, imposm is developed in Python but there's no documentation I 
> can find on how to use it as a Python library--it appears to be intended as a 
> command line tool). This seems odd to me since there's such a large community 
> of OSM users. I feel like I'm missing something. Are there other viable 
> Python libraries I'm not aware of? (Viable = large-ish user base and history 
> of bugfixing.) Is there documentation for using imposm within Python that 
> I've overlooked?
> 
> Bonus points for:
> - Windows compatibility
> - No external (i.e. non-Python) libraries needed
> 
> Thanks for any help!
> ___
> dev mailing list
> dev@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/dev
> 
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] OSM->PostGIS Python tool?

2018-12-21 Thread Pierre Béland
For Building quality analysis, I extract data from Overpass-API and I import 
into PostgreSQL/PostGIS schemas,  I use a solution based on osmosis 
pgsnapshot_schema_0.6. This solution similar to OSM2Pgsql is well documented 
and facilitates a lot importing OSM database extracs into PostGIS.

Telenav uses a similar approach that they documented on Github
See https://github.com/TelenavMapping/verifying_our_edits
The Osmosis instruction to import in  PostgreSQL, do not let specify a 
particular database schema. Like in the main.sh script, the solution is to 
first load data in the public schema. Once loaded, it is moved it to a 
particular schema. 

 
Pierre 
 

Le vendredi 21 décembre 2018 09 h 08 min 41 s HNE, Michael Reichert 
 a écrit :  
 
 Hi Spencer,

Am 21/12/2018 um 05.54 schrieb Spencer Gardner:
> I'm researching options for a Python-based tool that uses OSM data. From
> what I can gather there's no native Python library for OSM imports to a
> PostGIS database. (Yes, imposm is developed in Python but there's no
> documentation I can find on how to use it as a Python library--it appears
> to be intended as a command line tool). This seems odd to me since there's
> such a large community of OSM users. I feel like I'm missing something. Are
> there other viable Python libraries I'm not aware of? (Viable = large-ish
> user base and history of bugfixing.) Is there documentation for using
> imposm within Python that I've overlooked?

There are currently two serious import programmes for OSM data into
PostGIS databases – Osm2pgsql and Imposm 3. The first one is written in
C++ (formerly C), the second one in Go.

I am not aware of any well-known Python library for an import. Given the
typical use cases and the speed of existing solutions, it is unlikely
that there is any. There are Python bindings of the C++ library
Libosmium. The bindings are called Pyosmium. Libosmium is the fastest
library to deal with OSM data.

Why do you need a library for the import? Which use cases are not
covered by Osm2pgsql or Imposm 3?

Please keep in mind that OSM data is large (our planet is about 40 GB in
.osm.pbf format). Using a interpreted language is a waste of time and
energy for most purposes.

> Bonus points for:
> - Windows compatibility
> - No external (i.e. non-Python) libraries needed

Osm2pgsql runs under Windows.

Best regards

Michael

-- 
Per E-Mail kommuniziere ich bevorzugt GPG-verschlüsselt. (Mailinglisten
ausgenommen)
I prefer GPG encryption of emails. (does not apply on mailing lists)
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev
  ___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] OSM->PostGIS Python tool?

2018-12-21 Thread Michael Reichert
Hi Spencer,

Am 21/12/2018 um 05.54 schrieb Spencer Gardner:
> I'm researching options for a Python-based tool that uses OSM data. From
> what I can gather there's no native Python library for OSM imports to a
> PostGIS database. (Yes, imposm is developed in Python but there's no
> documentation I can find on how to use it as a Python library--it appears
> to be intended as a command line tool). This seems odd to me since there's
> such a large community of OSM users. I feel like I'm missing something. Are
> there other viable Python libraries I'm not aware of? (Viable = large-ish
> user base and history of bugfixing.) Is there documentation for using
> imposm within Python that I've overlooked?

There are currently two serious import programmes for OSM data into
PostGIS databases – Osm2pgsql and Imposm 3. The first one is written in
C++ (formerly C), the second one in Go.

I am not aware of any well-known Python library for an import. Given the
typical use cases and the speed of existing solutions, it is unlikely
that there is any. There are Python bindings of the C++ library
Libosmium. The bindings are called Pyosmium. Libosmium is the fastest
library to deal with OSM data.

Why do you need a library for the import? Which use cases are not
covered by Osm2pgsql or Imposm 3?

Please keep in mind that OSM data is large (our planet is about 40 GB in
.osm.pbf format). Using a interpreted language is a waste of time and
energy for most purposes.

> Bonus points for:
> - Windows compatibility
> - No external (i.e. non-Python) libraries needed

Osm2pgsql runs under Windows.

Best regards

Michael

-- 
Per E-Mail kommuniziere ich bevorzugt GPG-verschlüsselt. (Mailinglisten
ausgenommen)
I prefer GPG encryption of emails. (does not apply on mailing lists)



signature.asc
Description: OpenPGP digital signature
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


[OSM-dev] OSM->PostGIS Python tool?

2018-12-21 Thread Spencer Gardner
I'm researching options for a Python-based tool that uses OSM data. From
what I can gather there's no native Python library for OSM imports to a
PostGIS database. (Yes, imposm is developed in Python but there's no
documentation I can find on how to use it as a Python library--it appears
to be intended as a command line tool). This seems odd to me since there's
such a large community of OSM users. I feel like I'm missing something. Are
there other viable Python libraries I'm not aware of? (Viable = large-ish
user base and history of bugfixing.) Is there documentation for using
imposm within Python that I've overlooked?

Bonus points for:
- Windows compatibility
- No external (i.e. non-Python) libraries needed

Thanks for any help!
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev