Re: [OSM-dev] Munitely updated regional osm2pgsql database
On 02/07/2014 15:14, Ilya Zverev wrote: So, https://github.com/zverik/regional ... I hope it will be useful to somebody Thanks! Just installed that this morning, and it makes a significant difference, both to speed of diff import and to database growth. On Ubuntu 14.04 it also needed: sudo apt-get install python-psycopg2 sudo apt-get install python-shapely But once those were in place it "just worked". Cheers, Andy ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Munitely updated regional osm2pgsql database
The next best thing that could happen... is to have osm2pgsql (and maybe
similar other tools like imposm) to take advantage of augmented diffs
produced by overpass-API.
These augmented diffs have 2 benefits:
- they are created thru your own query (so you can get them for any area,
like the bbox of your choice)
- they are greatly self-contained*, making most of the awful slim tables**
not required anymore
We simply need someone to implement their support... I expected to have a
student doing that this summer, but it will not be the case.
This is on top of my "OSM dev wish list":
https://cquest.hackpad.com/OSM-dev-wish-list-3OhxQgDuiEK
* they contain ways related to modified nodes, as well as nodes related to
ways... making the planet_osm_ways table useless for updated
** as a reminder: osm2pgsql "slim" tables... they take 250GB of space on a
planet database, where you have around 150-200GB of useful
point/line/roads/polygon data.
2014-07-03 17:11 GMT+02:00 Sachin Dole :
> thanks sly, that was good info for me. and thanks Ilya for asking the
> question.
>
>
> On Thu, Jul 3, 2014 at 10:01 AM, sly (sylvain letuffe) > wrote:
>
>> On jeudi 3 juillet 2014, Ilya Zverev wrote:
>>
>> > If, with comments above, this
>>
>> > script is still not the best way to have regional minutely updates,
>>
>> > please point me to an alternative.
>>
>>
>>
>> I wasn't saying that what you did is or isn't the best way ("best" beeing
>> relative to a need) to have regional diffs, I was just pointing out, for
>> you, or for anyone with a need of "subregional diffs" who might not know,
>> that there exists those 2 alternatives I pointed out.
>>
>>
>>
>> --
>>
>> sly, direct contact : sylv...@letuffe.org
>>
>> http://wiki.openstreetmap.org/wiki/User:Sletuffe
>>
>> ___
>> dev mailing list
>> dev@openstreetmap.org
>> https://lists.openstreetmap.org/listinfo/dev
>>
>>
>
>
> --
> President | Genvega Inc. | www.genvega.com | tel: 630.290.2561
>
> ___
> dev mailing list
> dev@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/dev
>
>
--
Christian Quest - OpenStreetMap France
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Munitely updated regional osm2pgsql database
thanks sly, that was good info for me. and thanks Ilya for asking the
question.
On Thu, Jul 3, 2014 at 10:01 AM, sly (sylvain letuffe)
wrote:
> On jeudi 3 juillet 2014, Ilya Zverev wrote:
>
> > If, with comments above, this
>
> > script is still not the best way to have regional minutely updates,
>
> > please point me to an alternative.
>
>
>
> I wasn't saying that what you did is or isn't the best way ("best" beeing
> relative to a need) to have regional diffs, I was just pointing out, for
> you, or for anyone with a need of "subregional diffs" who might not know,
> that there exists those 2 alternatives I pointed out.
>
>
>
> --
>
> sly, direct contact : sylv...@letuffe.org
>
> http://wiki.openstreetmap.org/wiki/User:Sletuffe
>
> ___
> dev mailing list
> dev@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/dev
>
>
--
President | Genvega Inc. | www.genvega.com | tel: 630.290.2561
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Munitely updated regional osm2pgsql database
On jeudi 3 juillet 2014, Ilya Zverev wrote:
> If, with comments above, this
> script is still not the best way to have regional minutely updates,
> please point me to an alternative.
I wasn't saying that what you did is or isn't the best way ("best" beeing
relative to a need) to have regional diffs, I was just pointing out, for you,
or for anyone with a need of "subregional diffs" who might not know, that
there exists those 2 alternatives I pointed out.
--
sly, direct contact : sylv...@letuffe.org
http://wiki.openstreetmap.org/wiki/User:Sletuffe
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Munitely updated regional osm2pgsql database
sly (sylvain letuffe) wrote: >> and since diffs are only available for the whole planet > > FYI, that isn't true anymore : > http://wiki.openstreetmap.org/wiki/Planet.osm/diffs#Regionally_limited_diffs > Geofabrik is providing daily diffs for "any subregion you can imagine" ...daily diffs. And, e.g. just for a part of Russia, not the whole. > and french association provides minute diffs for several countries, and the > list can be increased on case by cases along with a real need. Which need is real and which is imaginary? What do I do if I have a polygon, then increase it as space allows, then change it once more when I have more money to spare — every time I would ask french association (which doesn't have any reason to help me) to alter a cutting polygon? I prefer to rely only on official services when possible, thus no augmented diff processing, for example. If, with comments above, this script is still not the best way to have regional minutely updates, please point me to an alternative. IZ ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Munitely updated regional osm2pgsql database
On mercredi 2 juillet 2014, Ilya Zverev wrote: > and since diffs are only available for the whole planet FYI, that isn't true anymore : http://wiki.openstreetmap.org/wiki/Planet.osm/diffs#Regionally_limited_diffs Geofabrik is providing daily diffs for "any subregion you can imagine" and french association provides minute diffs for several countries, and the list can be increased on case by cases along with a real need. -- sly, direct contact : sylv...@letuffe.org http://wiki.openstreetmap.org/wiki/User:Sletuffe ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Munitely updated regional osm2pgsql database
Hi! I think it's unfair that the entry level to producing OSM tiles is prohibitively high: you cannot just install postgis/mapnik/mod_tile on a $10 vps and have minutely updated country tiles. There certainly won't be enough memory to run osm2pgsql, and since diffs are only available for the whole planet, your database would grow 600 MB a day. I ended up reloading it from a local machine every 10 days, and quickly grew tired of this. So, https://github.com/zverik/regional — some scripts that I wrote for my server, including trim_osc.py, which processes osmChange files, removing nodes and ways outside a given polygon. It obviously isn't very effective (it cannot touch ways with no node information, i.e. those which have only their tags modified), but still, after I installed it, daily database size increase dropped from 600 to 100 megabytes (which is around 10 times more than the actual extract growth). I hope it will be useful to somebody, who's rendering tiles for fun and not wanting to spare more than 5-10 dollars a month on it. IZ ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
