Re: [OSM-dev] Overpass API - Memory Error on 16GB RAM

2018-10-23 Thread Michael Reichert
Hi Leon,

Am 23.10.18 um 19:33 schrieb mmd:
> Am 23.10.18 um 09:17 schrieb Leon Boehmer:
>> We are trying a query which returns all highways in Europe on our
>> on-premise Overpass.
> 
> I really have no idea what your ultimate goal is, but in any case I
> would strongly recommend to download a Europe extract (e.g. from
> Geofabrik [1]), and use some filtering tool like osmium-tool [2] to
> extract highways. Store the result in whatever DB you like, could be
> even Overpass API.

osmium tags-filter -o highways-in-europe.osm.pbf europe-latest.osm.pbf
w/highway

would be the command you need.

Best regards

Michael

-- 
Per E-Mail kommuniziere ich bevorzugt GPG-verschl├╝sselt. (Mailinglisten
ausgenommen)
I prefer GPG encryption of emails. (does not apply on mailing lists)



signature.asc
Description: OpenPGP digital signature
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] Overpass API - Memory Error on 16GB RAM

2018-10-23 Thread mmd
Am 23.10.18 um 09:17 schrieb Leon Boehmer:
> We are trying a query which returns all highways in Europe on our
> on-premise Overpass.

I really have no idea what your ultimate goal is, but in any case I
would strongly recommend to download a Europe extract (e.g. from
Geofabrik [1]), and use some filtering tool like osmium-tool [2] to
extract highways. Store the result in whatever DB you like, could be
even Overpass API.

If you insist on using Overpass API for the task at hand, start with
much smaller bounding boxes.

Also, please consider adding the term "Overpass API" to your email
subject. This list covers so many different tools, i.e. "Memory Error on
16GB RAM" isn't exactly helpful.


[1] http://download.geofabrik.de/europe.html
[2] https://osmcode.org/osmium-tool/


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] Memory Error on 16GB RAM

2018-10-23 Thread Marco Boeringa

Hi Leon,


I am actually wondering why you use an on-premise Overpass instance to 
return "all highways in Europe"...? I am certainly not an expert here, 
but I don't think Overpass was ever designed to handle that kind and 
size of request, and maybe even more so the potential client app you are 
using to visualize the data. I am thus also wondering what client app 
you are using to access the data?



E.g. considering you are talking about a local instance of the database, 
wouldn't it be a more logical fit, and direct route, to access the data 
in QGIS via an ODBC connection and write your queries in standard 
(PostGIS) SQL, instead of through Overpass?



Marco


Op 23-10-2018 om 09:17 schreef Leon Boehmer:

Hi All,

We are trying a query which returns all highways in Europe on our 
on-premise Overpass. Unfortunately this query currently fails with 
"runtime error: open64: 0 No error information /osm3s_v0.7.55_osm_base 
Dispatcher_Client::request_read_and_idx::timeout. The server is 
probably too busy to handle your request.". By setting maxsize to 5GB 
and timeout to 900 we have managed to avoid other errors, but we see 
this error far sooner than the 15 minutes we have given the query. We 
have managed to trace this message to dispatcher_client.cc 
line 224 and see some sort of hardcoded timeout of 
100 (counter) x 300ms. Unfortunately my understanding of C++ doesn't 
suffice to actually determine what the code is precisely waiting for 
and how we can mitigate this particular condition.
The VM Overpass is running on is equipped with 16GB of RAM as well as 
SSDs and serves no other query at the same time, so we are pretty 
confident it should be able to execute it in a timely fashion.


Kind Regards,

Leon Boehmer.


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


[osmosis-dev] Osmosis 0.47 Release

2018-10-23 Thread Brett Henderson
Hi All,

I've just released Osmosis 0.47.
http://bretth.dev.openstreetmap.org/osmosis-build/osmosis-0.47.tgz
http://bretth.dev.openstreetmap.org/osmosis-build/osmosis-0.47.zip

>From changes.txt:


   - Build using Java 11 now that Java has moved to a 6 month release cycle
   with no (free) long term support.
   - Use https for all replication URLs.
   - Use single-threaded PBF reading by default.
   - Add support for PBF way location attributes.
   - Remove customdb dataset implementation.  It performed poorly and was
   superseded long ago by pgsimple/pgsnapshot.
   - Fix munin replication lag shell script.
   - Modify --write-apidb to commit every 1 to allow large databases to
   be populated.
   - Fix --write-apidb-change to correctly write way history.


Let me know if you see any issues.

It's a good time to mention that this will probably be the last Osmosis
release I make.  I first starting hacking on Osmosis in April 2007, and
after 11 years it feels like time to finally let it go.  I haven't spent
much time in the OpenStreetMap world for a number of years, but Osmosis has
been largely superseded by better/faster tools such as Osmium.  Osmosis
itself is really showing its age.  In its day it helped move OpenStreetMap
forward, in particular making it possible to replicate data around in a
timely and efficient manner, but that's a well solved problem now.  The
version of Osmosis building the daily/hourly/minute changesets is _many_
years old and yet remains compatible with the latest version, not much has
changed.

I've kept the lights on for a number of years by accepting PRs, doing basic
maintenance such as library and java updates, and making it easier to
build/test through changes such as Dockerising the build.  But even that
can be time consuming, and I struggle to find the time to do it properly.

The code is all in the public domain, so if anybody wishes to continue to
hack away on it they are very welcome to do so.  If anybody wants to take
on release management duties, let me know, and I'll help them get access to
some of the key infrastructure such as the dev server for hosting
distribution binaries, Maven Central artefact upload  process, and GitHub
git repository.

Cheers,
Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev


JOSM and lack of memory

2018-10-23 Thread Bob Hawkins
Vincent
I have had my 64-bit PC since 2013, about which I keep a log of all events.  
Several notes mention installing 32-bit Java.  I believe I misunderstood the 
instructions that recommended it should be installed on 64-bit machines under 
certain circumstances.  JOSM operated well, nevertheless, until a year ago, 
when I encountered the lack of memory issue first.
I uninstalled Java 8u191, downloaded its 64-bit equivalent and installed it.  I 
am delighted to report everything has returned to normal.
I began using JOSM in 2008, following the purchase of my first GPSr.  I sing 
its praises at every opportunity and continue to obtain great pleasure from 
using it.
I thank you for determining the solution to this setback.  Hurrah! for mailing 
lists and forums.
With regards
Bob Hawkins


---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus


[OSM-dev] Memory Error on 16GB RAM

2018-10-23 Thread Leon Boehmer
Hi All,

We are trying a query which returns all highways in Europe on our on-premise 
Overpass. Unfortunately this query currently fails with "runtime error: open64: 
0 No error information /osm3s_v0.7.55_osm_base 
Dispatcher_Client::request_read_and_idx::timeout. The server is probably too 
busy to handle your request.". By setting maxsize to 5GB and timeout to 900 we 
have managed to avoid other errors, but we see this error far sooner than the 
15 minutes we have given the query. We have managed to trace this message to 
dispatcher_client.cc line 224 and see some sort of hardcoded 
timeout of 100 (counter) x 300ms. Unfortunately my understanding of C++ doesn't 
suffice to actually determine what the code is precisely waiting for and how we 
can mitigate this particular condition.
The VM Overpass is running on is equipped with 16GB of RAM as well as SSDs and 
serves no other query at the same time, so we are pretty confident it should be 
able to execute it in a timely fashion.

Kind Regards,

Leon Boehmer.
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev