Re: [OSM-dev] [meta] Problem with reply to messages from mailing list on Gmail
On 11/12/2014 17:47, Tom Hughes wrote: The tagging list is probably configured to set Reply-To and this one isn't (it's a per list option). Well, I learnt something new today. Thanks. I always assumed it was settings bug in Thunderbird. Shame all clients don't have the Reply: Sender/All/List option. Dave F. --- This email has been checked for viruses by Avast antivirus software. http://www.avast.com ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OSM with Hadoop
Stephen, previous discussions of combining NoSQL *or* massively parallel storage with OSM were often less driven by the approach "let's investigate solid future storage models for OSM" but rather by "hey there's a cool new technology I'd like to play with and I'm sure it can somehow work with OSM". The results were often, if there were any at all, along the lines of "hey this particular very specific use case is now 2% faster than before", but looking closer you'd see that the same speedup could have been had with an old-fashioned un-sexy "create index" statement if the author had known anything about PostgreSQL/PostGIS (*), or maybe that the data import took five weeks unless you had massive hardware, or somesuch. I was therefore a bit skeptical reading your message, but relieved when I found that you're keeping an open mind about the results and plan to thoroughly analyse whether using a massively parallel storage will indeed perform better than plain old PostgreSQL/PostGIS for what are OSM's everyday use cases. (I'd like to see the word "cost-effective" thrown in somewhere - and for data reading we have a sufficiently well scaling data replication in place already. As far as the central database is concerned, OSM is very interested in making it easy for everyone to run their own local copy.) Bye Frederik (*) It is an often overlooked fact that the amount of actual geo information in the central database is small - just the node coordinates - everything else is plain old relational stuff. Therefore the OSM database doesn't even need or use the PostGIS spatial extensions - but they are often used for analysing OSM data after importing them in a separate database. -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OSM with Hadoop
A very, very interesting and necessary project! Hadoop seems to be the mature choice. I have done some experiments based on this CouchDB scripts: http://wiki.openstreetmap.org/wiki/OSMCouch (there exists also an older modified imposm which can export directly to CouchDB) While this kind of storage is not made for edits, a student at the Technical University of Vienna is currently looking into representing the OSM data model in a better (more flexible) way with CouchDB. Am 2015-01-09 um 16:27 schrieb Paweł Paprota: > Hi Stephen, > > I read your document - it sounds like an interesting project! Even more > so that, as you can see, not much has been done in this area so far. > Pretty much every approach to OSM data storage/handling I've seen is > Postgres + some variation of a database schema optimized towards a > specific data usage (like routing, rendering, some specific types of > read queries). > > I'd be very interested to see completely different approaches to this > problem as the database size grows, it becomes extremely hard to build a > service which works with all OSM data, meaning a service with global, > not just local, coverage. Not to mention a service which works with all > *historic* OSM data - that's basically a suicide mission at this point > because (1) you probably won't fit all the data in your local dev > environment and hence (2) if you only have a small part of the data > locally then you can't easily extrapolate from that to how (or rather - > IF) your service will work if you build an infrastructure to support > full dataset. > > That is quite a challenge since it really puts heavy brakes on > innovation in OSM and new services being offered. > > Paweł > > On Thu, Jan 8, 2015, at 16:33, Pieren wrote: >> On Fri, Jan 2, 2015 at 7:38 PM, Stephen Knox >> wrote: >> >>> So firstly I am wondering if I am missing any previous posts / research on >>> the topic? >> Here are my findings about hadoop quoted in MLs: >> https://lists.openstreetmap.org/pipermail/dev/2009-August/016554.html >> https://lists.openstreetmap.org/pipermail/dev/2010-March/018648.html >> https://lists.openstreetmap.org/pipermail/dev/2010-July/019964.html >> https://lists.openstreetmap.org/pipermail/osmosis-dev/2010-August/000682.html >> https://lists.openstreetmap.org/pipermail/dev/2012-March/024468.html >> https://lists.openstreetmap.org/pipermail/talk/2012-March/062444.html >> https://lists.openstreetmap.org/pipermail/talk/2012-November/065171.html >> >> Pieren >> >> ___ >> dev mailing list >> dev@openstreetmap.org >> https://lists.openstreetmap.org/listinfo/dev > ___ > dev mailing list > dev@openstreetmap.org > https://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
[OSM-dev] osm2pgsql: Unable to import planet
Hello,
unfortunately I'm unable to import current planet files using osm2pgsql.
This might be a psql issue because the strange error message I get seems to
come from postgress rather than osm2pgsql:
osm2pgsql -U postgres -s --number-processes=8 -C4000 -S hstore-match-only.style
--flat-nodes /tmp/flatnodefile --hstore --hstore-match-only -d digmt
planet-150105.osm.pbf
osm2pgsql SVN version 0.87.1 (64bit id space)
Using built-in tag processing pipeline
Using projection SRS 900913 (Spherical Mercator)
Setting up table: planet_osm_point
Setting up table: planet_osm_line
Setting up table: planet_osm_polygon
Setting up table: planet_osm_roads
Allocating memory for dense node cache
Allocating dense node cache in one big chunk
Allocating memory for sparse node cache
Sharing dense sparse
Node-cache: cache=4000MB, maxblocks=512000*8192, allocation method=11
Mid: loading persistent node cache from /tmp/flatnodefile
Allocated space for persistent node cache file
Maximum node in persistent node cache: 0
Mid: pgsql, scale=100 cache=4000
Setting up table: planet_osm_nodes
Setting up table: planet_osm_ways
Setting up table: planet_osm_rels
Reading in file: planet-150105.osm.pbf
Processing: Node(2672693k 1631.7k/s) Way(267143k 7.32k/s) Relation(2297370
57.27/s)Maximum node in persistent node cache: 3270508543
node cache: stored: 499159094(18.68%), storage efficiency: 95.21% (dense
blocks: 463882, sparse nodes: 24636458), hit rate: 13.42%
Osm2pgsql failed due to ERROR: get_way_list failed: FEHLER: invalid memory
alloc request size 18446744073709551613
(7)
Arguments were:
{158316322,37669753,156819792,156819720,245694537,156819742,158316324,126760837,95242622,95242612,31404881,158316333,29171707,26142674,37626540,29178874,26202582,29020170,57208434,57208429,26142794,29200328,29200230,293588151,29200312,29070293,24277202,34616458,24277204,61594153,158316940,158316941,158316934,35878683,35147653,57737110,158316942,26963683,57737104,95351513,95351552,95351505,95351547,95351519,95351544,35998299,158316920,158316943,158316935,127949408,35999803,127949407,314868523,124719357,36824502,36887582,158319685,158319677,36826753,119960436,119960555,119960556,158319690,158319678,158319703,39432345,39432346,231742109,42071595,42071598},
Any Idea?
Sven
--
"linux is evolution, not intelligent design"
(Linus Torvalds)
/me is giggls@ircnet, http://sven.gegg.us/ on the Web
- End forwarded message -
--
"linux is evolution, not intelligent design"
(Linus Torvalds)
/me is giggls@ircnet, http://sven.gegg.us/ on the Web
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev
