Re: [OSM-dev] How to support name:* in MapOSMatic?

[Peter Körner]

On the toolserver we imported with this style [1] and so have the name, 
  name:de, name:fr, etc. columns. Then we used the script [2] to created 
one view for each language and for each table:

SELECT planet_osm_point.osm_id, planet_osm_point.access, 
planet_osm_point."addr:flats", planet_osm_point."addr:housenumber", 
planet_osm_point."addr:interpol
ation", planet_osm_point.admin_level, planet_osm_point.aerialway, 
planet_osm_point.aeroway, planet_osm_point.amenity, 
planet_osm_point.area, planet_osm_point
.barrier, planet_osm_point.bicycle, planet_osm_point.bridge, 
planet_osm_point.boundary, planet_osm_point.building, 
planet_osm_point.capital, planet_osm_point
.construction, planet_osm_point.cutting, planet_osm_point.disused, 
planet_osm_point.ele,
planet_osm_point.embankment, planet_osm_point.foot, 
planet_osm_point.highway, planet_osm_point.historic, 
planet_osm_point.horse, planet_osm_point.junction, 
planet_osm_point.landuse, planet_osm_point.layer, 
planet_osm_point.learning, planet_osm_point.leisure, 
planet_osm_point.lock, planet_osm_point.man_made, 
planet_osm_point.military, planet_osm_point.motorcar, 
planet_osm_point.cycleway, planet_osm_point.operator, 
planet_osm_point.surveillance, planet_osm_point.website, 
planet_osm_point.wikipedia, planet_osm_point.opening_hours, 
planet_osm_point.phone, planet_osm_point.fax, planet_osm_point.url, 
planet_osm_point.importance, planet_osm_point.fee, 
planet_osm_point.information, planet_osm_point.shop, 
planet_osm_point.path, planet_osm_point.tower, planet_osm_point.lit,
 CASE
 WHEN planet_osm_point."name:de" IS NOT NULL THEN
 planet_osm_point."name:de"
 ELSE
 planet_osm_point.name
 END AS name,
planet_osm_point."natural", planet_osm_point.oneway, 
planet_osm_point.poi, planet_osm_point.power, 
planet_osm_point.power_source, planet_osm_point.place, 
planet_osm_point.railway, planet_osm_point.ref, 
planet_osm_point.religion, planet_osm_point.residence, 
planet_osm_point.route, planet_osm_point.service, 
planet_osm_point.sport, planet_osm_point.tourism, 
planet_osm_point.tunnel, planet_osm_point.waterway, 
planet_osm_point.width, planet_osm_point.wood, planet_osm_point.z_order, 
planet_osm_point.way FROM planet_osm_point;


Maybe you can access them more easily.

Peter


[1]


___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] amazon AMIs for a full blown openstreetmap.org-like server

[jamesmikedup...@googlemail.com]

Ok, this looks great.
So, now, that would be my suggestion to use on this amazon webservice.
A lot of small osm files, the size of the smallest tile.

Now, on thing did occur to me while walking the dog, we could render
multiple zoom levels at once.
Every time you render the smallest unit (tile) you would render the
sections of all the zoom levels and that should prevent you from
touching the data at the same time.

mike


On Fri, Jan 8, 2010 at 6:08 PM, Dane Springmeyer  wrote:
>
> On Jan 8, 2010, at 7:19 AM, jamesmikedup...@googlemail.com wrote:
>
>> http://wiki.openstreetmap.org/wiki/Mapnik#Data_Sources
>> This webpage only talkes about posgis, is there any documentation that
>> covers the osm direct rendering?
>
> http://trac.mapnik.org/wiki/OsmPlugin
>
>

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] amazon AMIs for a full blown openstreetmap.org-like server

[Dane Springmeyer]

On Jan 8, 2010, at 7:19 AM, jamesmikedup...@googlemail.com wrote:

> http://wiki.openstreetmap.org/wiki/Mapnik#Data_Sources
> This webpage only talkes about posgis, is there any documentation that
> covers the osm direct rendering?

http://trac.mapnik.org/wiki/OsmPlugin


___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] When are ways/relations in a changeset?

[Frederik Ramm]

Hi,

> When I change the position of a node, will the way be in the changeset, too?

No.

> When I change members of relations (without adding / removing members), 
> will the relation be in the changeset, too?

No.

Bye
Frederik

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] amazon AMIs for a full blown openstreetmap.org-like server

[jamesmikedup...@googlemail.com]

http://wiki.openstreetmap.org/wiki/Mapnik#Data_Sources
This webpage only talkes about posgis, is there any documentation that
covers the osm direct rendering?
mike

On Fri, Jan 8, 2010 at 4:08 PM, jamesmikedup...@googlemail.com
 wrote:
> So this renderes directly from an osm file?
>
> On Fri, Jan 8, 2010 at 4:02 PM, Lennard  wrote:
>> jamesmikedup...@googlemail.com wrote:
>>> Ok, well I am going to first work on making the mapnik be able to
>>> render an osm file standalone without postgis,
>>> will report back when that works.
>>
>> *cough* http://trac.mapnik.org/browser/trunk/plugins/input/osm
>>
>> Will not work with the postgis-based stylesheet.
>
> I have not looked into all of this, as I said. From what I know about
> mapnick, it renders from the database.
>
> Compiling now.
> mike
>

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] amazon AMIs for a full blown openstreetmap.org-like server

[jamesmikedup...@googlemail.com]

So this renderes directly from an osm file?

On Fri, Jan 8, 2010 at 4:02 PM, Lennard  wrote:
> jamesmikedup...@googlemail.com wrote:
>> Ok, well I am going to first work on making the mapnik be able to
>> render an osm file standalone without postgis,
>> will report back when that works.
>
> *cough* http://trac.mapnik.org/browser/trunk/plugins/input/osm
>
> Will not work with the postgis-based stylesheet.

I have not looked into all of this, as I said. From what I know about
mapnick, it renders from the database.

Compiling now.
mike

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] When are ways/relations in a changeset?

[Matthias Julius]

Andreas Kalsch  writes:

> When I change the position of a node, will the way be in the changeset, too?
> When I change members of relations (without adding / removing members), 
> will the relation be in the changeset, too?

When you change a node only the node is uploaded to the API and
therefore only the node is part of a changeset regardless whether it
is used by a way or is a member of a relation.  The same is true when
you add an existing node to a way for example.  Then, only the way
itself is changed and not its nodes or relations it is a member of.

Matthias

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] amazon AMIs for a full blown openstreetmap.org-like server

[Lennard]

jamesmikedup...@googlemail.com wrote:
> Ok, well I am going to first work on making the mapnik be able to
> render an osm file standalone without postgis,
> will report back when that works.

*cough* http://trac.mapnik.org/browser/trunk/plugins/input/osm

Will not work with the postgis-based stylesheet.

-- 
Lennard

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] amazon AMIs for a full blown openstreetmap.org-like server

[jamesmikedup...@googlemail.com]

Ok, well I am going to first work on making the mapnik be able to
render an osm file standalone without postgis,
will report back when that works.
mike

On Fri, Jan 8, 2010 at 3:50 PM, Stefan de Konink  wrote:
> Op 08-01-10 15:39, jamesmikedup...@googlemail.com schreef:
>>>
>>> Tiny would have to be an osmtile in binary format, agreed?
>>
>> Yes, well if you sort the data properly then mapnik would just render
>> the data in a sax callback,  right?
>
> Jup, but you will never get better results of that storing in parts than
> having it stored in one file and having an index to it. So in that respect
> you have to mmap the file, for example a post/plane parser, extract the
> childs, send the childs to Mapnik.
>
> I have tested this using MonetDB/XML, that wasn't really a succes ;) Either
> way you need something that stores that creates an index by a bounding box.
> You can use rtree's for that sure. But we already have shown that sideway
> cracking gives far better results. [google: sideway cracking databases]
>
>>> You don't want to store it in XML, it will get huge... and requires
>>> again the parsing overhead. And as I pointed out before, we prefer to
>>> render 64 tiles at a time.
>>
>> So, you would have 64 tiles of data in one page How big is that?
>> the size of a city.
>
> All depending on the zoom level :)
>
>> Well I have gotten the parsing down to very small using sax2, it is
>> just a bit slower than reading the file.
>> If you sort the xml data topologically and pack it all in, it could be
>> very easy to process.
>
> But how are you going to store this, for example if we take a *perfect*
> filesystem. Will you store everything by nodeid, wayid, relationid? OR will
> you store everything that is in a tile, in a file marking the tile, having
> the same structure as the tile server uses?
>
>
>> I will have to work through the entire pipeline of processing to see
>> if it fits my model, I risk having to put my foot in my mouth here
>
> Sure :) Just start with some smaller country and try to prevent bottlenecks.
> The worse thing that can happen is TomH saying you are a 'Bonker' (whatever
> that may mean).
>
>
> Stefan
>

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] amazon AMIs for a full blown openstreetmap.org-like server

[Stefan de Konink]

Op 08-01-10 15:39, jamesmikedup...@googlemail.com schreef:
>> Tiny would have to be an osmtile in binary format, agreed?
>
> Yes, well if you sort the data properly then mapnik would just render
> the data in a sax callback,  right?

Jup, but you will never get better results of that storing in parts than 
having it stored in one file and having an index to it. So in that 
respect you have to mmap the file, for example a post/plane parser, 
extract the childs, send the childs to Mapnik.

I have tested this using MonetDB/XML, that wasn't really a succes ;) 
Either way you need something that stores that creates an index by a 
bounding box. You can use rtree's for that sure. But we already have 
shown that sideway cracking gives far better results. [google: sideway 
cracking databases]

>> You don't want to store it in XML, it will get huge... and requires
>> again the parsing overhead. And as I pointed out before, we prefer to
>> render 64 tiles at a time.
>
> So, you would have 64 tiles of data in one page How big is that?
> the size of a city.

All depending on the zoom level :)

> Well I have gotten the parsing down to very small using sax2, it is
> just a bit slower than reading the file.
> If you sort the xml data topologically and pack it all in, it could be
> very easy to process.

But how are you going to store this, for example if we take a *perfect* 
filesystem. Will you store everything by nodeid, wayid, relationid? OR 
will you store everything that is in a tile, in a file marking the tile, 
having the same structure as the tile server uses?


> I will have to work through the entire pipeline of processing to see
> if it fits my model, I risk having to put my foot in my mouth here

Sure :) Just start with some smaller country and try to prevent 
bottlenecks. The worse thing that can happen is TomH saying you are a 
'Bonker' (whatever that may mean).


Stefan

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] amazon AMIs for a full blown openstreetmap.org-like server

[jamesmikedup...@googlemail.com]

On Fri, Jan 8, 2010 at 3:23 PM, Stefan de Konink  wrote:
> Op 08-01-10 15:09, jamesmikedup...@googlemail.com schreef:
>> I dont know all the details of mapnik, but from what I have seen,
>> using the postgres database is not needed in all cases.
>> I am thinking about
>
> It is not required to use the database, but the overhead is in the
> rendering. I do agree that PostgreSQL will not always give you the
> fastest results ;)
>
>> Well you mean a 4k page in memory. But you dont need to have all that
>> data memory mapped.
>> It could be just 4k of data on disk in an array. It could also just be
>> a tiny osm file that is parsed when needed.
>
> Tiny would have to be an osmtile in binary format, agreed?

Yes, well if you sort the data properly then mapnik would just render
the data in a sax callback,
right?
It would be a single pass over the data to render it, maybe with a
second pass to do some touch ups.
But given a street you could just draw the street. It has a vector of nodes.

imagine the following sequence of events in a sax xml callback :

begin tile  :
process nodes for next street (marking intersections)
Process all the intersecting streets (there we need to preprocess the data)
process street (process intersections as they go)
next street.

So you need to look for street crossings, and need to mark the points
where they cross as such. I can imagine that might need some work.
I am just working this out right now, but I can imagine that if you
sorted the streets and knew all the intersections and marked each
intersection, storing a ref to it like in  and also had those
segments available, then you could just draw the street.



>
>> Well again, I have not really gotten into mapnik. But I can.
>
> Please do so :)

Ok, well I will have to.!

>
>> But lets try and define the problem as rendering an osm file to a
>> tile. That osm file is updated , and rerendered.
>> All the data needed to render is just stored in osmxml in a nice
>> sorted way. You never need all the data at once because you only
>> render a tile at a time.
>
> You don't want to store it in XML, it will get huge... and requires
> again the parsing overhead. And as I pointed out before, we prefer to
> render 64 tiles at a time.

So, you would have 64 tiles of data in one page How big is that?
the size of a city.

Well I have gotten the parsing down to very small using sax2, it is
just a bit slower than reading the file.
If you sort the xml data topologically and pack it all in, it could be
very easy to process.

>
>
>> but you dont need a full postgres database functionality, you just
>> have very basic update of pages of data.
>
> In your idea, how would you *update* the storage?

My vision of a distributed GIT repository would have the user update
the storage when they save it...
Now if you have a web user that commits the data or uploads it, then
the page of data would be updated by that process
the rendering would be triggered and the whole thing checked in.
Each tile would be checked in and older versions discarded as needed.

Now, if you just have a nice xml file that is compact  or an ASN.1
file for higher speed binary or something else, you need to update
that. but how often are we talking about update here? Most people are
using daily diff files from the main database. those would be split up
into the tiles, and the xml would be replaced by a new page, sorting
would have to take place. I need to work this all out, but it is
possible.

I will have to work through the entire pipeline of processing to see
if it fits my model, I risk having to put my foot in my mouth here

mike

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

[OSM-dev] When are ways/relations in a changeset?

[Andreas Kalsch]

When I change the position of a node, will the way be in the changeset, too?
When I change members of relations (without adding / removing members), 
will the relation be in the changeset, too?

I have currently no data to test these cases, so does anybody know how 
the results of these operations is?

Andi

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] amazon AMIs for a full blown openstreetmap.org-like server

[Stefan de Konink]

Op 08-01-10 15:09, jamesmikedup...@googlemail.com schreef:
> I dont know all the details of mapnik, but from what I have seen,
> using the postgres database is not needed in all cases.
> I am thinking about

It is not required to use the database, but the overhead is in the 
rendering. I do agree that PostgreSQL will not always give you the 
fastest results ;)

> Well you mean a 4k page in memory. But you dont need to have all that
> data memory mapped.
> It could be just 4k of data on disk in an array. It could also just be
> a tiny osm file that is parsed when needed.

Tiny would have to be an osmtile in binary format, agreed?

> Well again, I have not really gotten into mapnik. But I can.

Please do so :)

> But lets try and define the problem as rendering an osm file to a
> tile. That osm file is updated , and rerendered.
> All the data needed to render is just stored in osmxml in a nice
> sorted way. You never need all the data at once because you only
> render a tile at a time.

You don't want to store it in XML, it will get huge... and requires 
again the parsing overhead. And as I pointed out before, we prefer to 
render 64 tiles at a time.


> but you dont need a full postgres database functionality, you just
> have very basic update of pages of data.

In your idea, how would you *update* the storage?



Stefan

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] amazon AMIs for a full blown openstreetmap.org-like server

[jamesmikedup...@googlemail.com]

I dont know all the details of mapnik, but from what I have seen,
using the postgres database is not needed in all cases.
I am thinking about


> I would find it difficult to believe that with for example a memorymapped
> file, that allocates for example 4k for a node would outperform a database
> method. I would see a better future in automatic analysing a stylesheet,
> determing enums, and using databases.

Well you mean a 4k page in memory. But you dont need to have all that
data memory mapped.
It could be just 4k of data on disk in an array. It could also just be
a tiny osm file that is parsed when needed.


>> It will depend on your architecture, but I think on a distributed
>> system where you have many processors,
>> that lots of files , if connected to the processors will work well.
>> Ideally you would have a set of processors to deal with a set of data
>> and any requests would come to them and the whole thing would be in
>> cache all the time...
>
> That would assume a big disk cache. For NL, it would be theoretically
> possible to use multiple renderd instances out of the box. But that would
> require to run renderd on an TCP/IP socket, instead of a Unixsocket.
>
> In my perspective the query time can be reduced anyway significantly if we
> want it. Mapnik is much harder to optimise, unless you want to go GPU
> accelerated rendering.

Well again, I have not really gotten into mapnik. But I can.

But lets try and define the problem as rendering an osm file to a
tile. That osm file is updated , and rerendered.
All the data needed to render is just stored in osmxml in a nice
sorted way. You never need all the data at once because you only
render a tile at a time.

I think that you could optimize some... there are ways to store that
data so it is just copied into memory and does not need parsing.. you
could use shared memory as discussed before to store the tiles

but you dont need a full postgres database functionality, you just
have very basic update of pages of data.

I think that a multile user postgis is a bit overkill for a simple
rendering process...

or am I missing something, cause I have not looked at the code in detail.

mike

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] How to support name:* in MapOSMatic?

[David MENTRE]

Hello Brian,

2010/1/8 Brian Quinion :
> That is how the gazetteer output from osm2pgsql handles it already.

Thanks a lot!

Regards,
d.

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] amazon AMIs for a full blown openstreetmap.org-like server

[Stefan de Konink]

Now both to list.

Op 08-01-10 14:53, jamesmikedup...@googlemail.com schreef:
> On Fri, Jan 8, 2010 at 2:50 PM, Stefan de Konink  wrote:
>> Op 08-01-10 14:46, jamesmikedup...@googlemail.com schreef:
>>>
>>> I have been working on c++ code with rtrees and other nice things.
>>> depending on what you want to do with it, I think it is fair to make
>>> updates very slow if the rendering is very fast...
>>
>> Do you mean small files as render tiles?
>
> something like that. I wonder if we were to store all the osm data for
> one tile and its index on a file system and just keep them all in
> synch. Anyone who edits the data would update all of them as needed.

You would have to take in to account what data is required for 
rendering. This is basically what the column wise PostgreSQL Schema 
tries to achieve.

I would find it difficult to believe that with for example a 
memorymapped file, that allocates for example 4k for a node would 
outperform a database method. I would see a better future in automatic 
analysing a stylesheet, determing enums, and using databases.


>> This is already what we do on tile.openstreetmap.nl; I have modified renderd
>> to render 'metatiles', but not to save them as metatiles. So on disk I have
>> all seperate tiles. My webserver, based if the file exists, serves the file,
>> or otherwise shoots in a render request like mod_tile would do.
>
> define metatile?

Metatile is basically a tile that spawns a (much) greater space than the 
invidivual tile, for example 8x8. This reduces mapnik-overhead, since 
you always have to render a bit bigger than the tile anyway.
An additional assumption has been made that it would be faster to store 
and server metatiles by extracting data, than invidivual tiles. 
Currently there is no benchmark that can confirm or deny that.


>> So the main difference is that I never extract tiles, and never compress
>> tiles. We still should plan some sort of benchmark to see if this method is
>> better or worse then meta tiles. I think its better because the webserver is
>> on less load, and the disk speed or filesystem should be tuned anyway.
>
> It will depend on your architecture, but I think on a distributed
> system where you have many processors,
> that lots of files , if connected to the processors will work well.
> Ideally you would have a set of processors to deal with a set of data
> and any requests would come to them and the whole thing would be in
> cache all the time...

That would assume a big disk cache. For NL, it would be theoretically 
possible to use multiple renderd instances out of the box. But that 
would require to run renderd on an TCP/IP socket, instead of a Unixsocket.

In my perspective the query time can be reduced anyway significantly if 
we want it. Mapnik is much harder to optimise, unless you want to go GPU 
accelerated rendering.


Stefan

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] How to support name:* in MapOSMatic?

[Brian Quinion]

On Fri, Jan 8, 2010 at 12:57 PM, David MENTRE  wrote:
> The ideal configuration would be to have all names (name:fr, name:ar,
> ...) into a single "name" column and use it. Or have a kind of couple
> (name, country_code) that would store all different names for a
> street. But I don't know how to do this.

That is how the gazetteer output from osm2pgsql handles it already.

It creates a type of keyvalue as:

CREATE TYPE keyvalue AS (
  key TEXT,
  value TEXT
);

and then has a column of type keyvalue[] (keyvalue array)

All name values are then merged into this field during the osm2pgsql
import.  It ends up with a field value like:

{"(name:en,Peschanka)","(name,Песчанка)","(alt_name,Pesnanka)"}

Then a function is used to retrieve the name in the required value by
specifying a preference order:

CREATE OR REPLACE FUNCTION get_name_by_language(name keyvalue[],
languagepref TEXT[]) RETURNS TEXT
  AS $$
DECLARE
  search TEXT[];
  found BOOLEAN;
BEGIN

  IF (array_upper(name, 1) is null) THEN
return null;
  END IF;

  search := languagepref;

  FOR j IN 1..array_upper(search, 1) LOOP
FOR k IN 1..array_upper(name, 1) LOOP
  IF (name[k].key = search[j] AND trim(name[k].value) != '') THEN
return trim(name[k].value);
  END IF;
END LOOP;
  END LOOP;

  RETURN null;
END;
$$
LANGUAGE plpgsql IMMUTABLE;

select get_name_by_language(namearray,
ARRAY['name:en','name:de','name']) from table;

--
 Brian

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] amazon AMIs for a full blown openstreetmap.org-like server

[Stefan de Konink]

Op 08-01-10 14:46, jamesmikedup...@googlemail.com schreef:
> I have been working on c++ code with rtrees and other nice things.
> depending on what you want to do with it, I think it is fair to make
> updates very slow if the rendering is very fast...

Do you mean small files as render tiles?

This is already what we do on tile.openstreetmap.nl; I have modified 
renderd to render 'metatiles', but not to save them as metatiles. So on 
disk I have all seperate tiles. My webserver, based if the file exists, 
serves the file, or otherwise shoots in a render request like mod_tile 
would do.

So the main difference is that I never extract tiles, and never compress 
tiles. We still should plan some sort of benchmark to see if this method 
is better or worse then meta tiles. I think its better because the 
webserver is on less load, and the disk speed or filesystem should be 
tuned anyway.


Stefan


___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] amazon AMIs for a full blown openstreetmap.org-like server

[jamesmikedup...@googlemail.com]

I have been working on c++ code with rtrees and other nice things.
depending on what you want to do with it, I think it is fair to make
updates very slow if the rendering is very fast...
mike

On Fri, Jan 8, 2010 at 2:39 PM, Stefan de Konink  wrote:
> Op 08-01-10 13:49, jamesmikedup...@googlemail.com schreef:
>> I would like to have an account.
>>
>> We could try out some ideas for replacing the database with a lot of
>> small files.
>
> We could just switch to MonetDB get deduplication, and don't require a
> lot of space ;)
>
> ...in other news did anyone try SphinxSearch on OSM data? Since I
> dislike PostgreSQL's geo-non-indiced functionality. Hence: an index on
> IsValid(...) is not materialised, I wonder if anyone succesfully applied
> stuff like:
>
> select planet_osm_point.osm_id, planet_osm_polygon.name,
> planet_osm_point.name from planet_osm_point, planet_osm_polygon where
> ST_Intersects(planet_osm_point.way, planet_osm_polygon.way) and
> planet_osm_polygon.name <> '' and planet_osm_point.name <> ''  and
> planet_osm_polygon.name <> planet_osm_point.name;
>
> sql_query = select planet_osm_roads.osm_id, planet_osm_polygon.name,
> planet_osm_roads.name from planet_osm_roads, planet_osm_polygon where
> st_isvalid(planet_osm_roads.way) AND st_isvalid(planet_osm_polygon.way)
> AND ST_Intersects(planet_osm_roads.way, planet_osm_polygon.way) and
> planet_osm_polygon.name <> '' and planet_osm_roads.name <> ''  and
> planet_osm_polygon.name <> planet_osm_roads.name;
>
> sql_query = select p2.osm_id, p1.name, p2.name from planet_osm_polygon
> as p2, planet_osm_polygon as p1 where ST_Intersects(p1.way, p2.way) and
> p1.name <> '' and p2.name <> ''  and p1.name <> p2.name;
>
>
> Stefan
>
> ps. TomH thanks for the comment on IRC :D
>
>
> ___
> dev mailing list
> dev@openstreetmap.org
> http://lists.openstreetmap.org/listinfo/dev
>

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] amazon AMIs for a full blown openstreetmap.org-like server

[Stefan de Konink]

Op 08-01-10 13:49, jamesmikedup...@googlemail.com schreef:
> I would like to have an account.
>
> We could try out some ideas for replacing the database with a lot of
> small files.

We could just switch to MonetDB get deduplication, and don't require a 
lot of space ;)

...in other news did anyone try SphinxSearch on OSM data? Since I 
dislike PostgreSQL's geo-non-indiced functionality. Hence: an index on 
IsValid(...) is not materialised, I wonder if anyone succesfully applied 
stuff like:

select planet_osm_point.osm_id, planet_osm_polygon.name, 
planet_osm_point.name from planet_osm_point, planet_osm_polygon where 
ST_Intersects(planet_osm_point.way, planet_osm_polygon.way) and 
planet_osm_polygon.name <> '' and planet_osm_point.name <> ''  and 
planet_osm_polygon.name <> planet_osm_point.name;

sql_query = select planet_osm_roads.osm_id, planet_osm_polygon.name, 
planet_osm_roads.name from planet_osm_roads, planet_osm_polygon where 
st_isvalid(planet_osm_roads.way) AND st_isvalid(planet_osm_polygon.way) 
AND ST_Intersects(planet_osm_roads.way, planet_osm_polygon.way) and 
planet_osm_polygon.name <> '' and planet_osm_roads.name <> ''  and 
planet_osm_polygon.name <> planet_osm_roads.name;

sql_query = select p2.osm_id, p1.name, p2.name from planet_osm_polygon 
as p2, planet_osm_polygon as p1 where ST_Intersects(p1.way, p2.way) and 
p1.name <> '' and p2.name <> ''  and p1.name <> p2.name;


Stefan

ps. TomH thanks for the comment on IRC :D


___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] How to support name:* in MapOSMatic?

[Greg Troxel]

  We would like to support multiple names for a given street: name:fr,
  name:ar, etc. As far as I have understood, we could configure
  osm2pgsql to create columns "name:fr", "name:ar", etc. and change our
  SQL requests to use those columns. That does not seems very scalable.

  The ideal configuration would be to have all names (name:fr, name:ar,
  ...) into a single "name" column and use it. Or have a kind of couple
  (name, country_code) that would store all different names for a
  street. But I don't know how to do this.

Not well thought out, but:

  add a name table

id(SERIAL), name(lang), name(text)

  for each such named object, get a new SERIAL in the name table, and
  add in all of the language/value pairs

  put the id in the main database



pgpDWLmvFNWAv.pgp
Description: PGP signature
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

[OSM-dev] How to support name:* in MapOSMatic?

[David MENTRE]

Hello,

I'm part of MapOSMatic developers. Currently, we are using osm2pgsql
to translate OSM data into Posgresql, as usual. We are using the
default configuration for osm2pgsql. In particular, the tag "name" is
translated into a pgsql column "name".

We would like to support multiple names for a given street: name:fr,
name:ar, etc. As far as I have understood, we could configure
osm2pgsql to create columns "name:fr", "name:ar", etc. and change our
SQL requests to use those columns. That does not seems very scalable.

The ideal configuration would be to have all names (name:fr, name:ar,
...) into a single "name" column and use it. Or have a kind of couple
(name, country_code) that would store all different names for a
street. But I don't know how to do this.

Would any reader of this list have a suggestion on the best way to
handle this issue?

Best regards,
david

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] amazon AMIs for a full blown openstreetmap.org-like server

[jamesmikedup...@googlemail.com]

I would like to have an account.

We could try out some ideas for replacing the database with a lot of
small files.

mike


On Fri, Jan 8, 2010 at 1:42 PM, Emilie Laffray  wrote:
>
>
> 2010/1/8 Jonathan-David SCHRODER 
>>
>> Hello,
>> Please tell me if I'm replicating any initiative or doing something the
>> wrong or a not-so-good way.
>>
>> I've started this week from an EBS backed (or "boot")  Ubuntu 8.04 image
>> ami-f6ad409f.
>> I agree to give the .pem file for ssh connection to an instance to anyone
>> willing to collaborate.
>>
>> My idea is to provide an EC2 EBS image with rails port(done) + tile
>> rendering setup(started) + rails port->tile rendering sync scripts(todo) +
>> easy config scripts (why not triggerable on startup with JSON config ala
>> tilecache.org) (todo, not sure I'll have time to it).
>> The list of commands to install is very long. I note down all the commands
>> I type so as to be able to one or more bash scripts to be able to reinstall
>> a similar solution on an Ubuntu N.NN later (by assuming that
>> commands/package names for 8.04 are similar to those for 9.04 etc..). Maybe
>> I'll have 3 bash scripts such as : install_railsport_u804.sh,
>> install_mapnik_u804.sh, install_railsport_mapnik_sync.sh. And I'd need other
>> scripts to go along : setup_rails_port_u804.sh, setup_mapnik_u804.sh and
>> setup_rails_mapnik_sync_u804.sh...
>> What do you think of this ?
>>
>> A recall on the requirements (reply to change them if you'd like to
>> collaborate or request something):
>> - postgres+osmosis+rails port (with 3 dbs : development, testing and
>> production) : filled with nothing for now. setup_rails_port_u804.sh would
>> have a simple calling syntax like :
>>    sudo setup_rails_port_u804.sh
>> [--bz2url=http://domain/someRegion.osm.bz2|--bz2file=/path/to/file.osm.bz2]
>> [--forceResetDB (false if omitted)]
>> [--dbName=development(default),other,dbs,to,make,identical,too]
>> [--guessDBLoginPassFromRailsConfig (default is true)]
>> - mapnik tile rendering setup. I want to have a simple script to insert a
>> bz2 + tell to use some style sheets + to tell to render at z=N (max zoom)
>> - cron'able synced file making the Mapnik setup be filled with the Rails
>> ports's data. I plan to get inspiration from the myanmar scripts for this.
>>
>> I'm sorry to repeat this, though, please reply if you know of some linux
>> image (on EC2 or not) hanging somewhere which almost does what I'm trying to
>> achieve and that image is easy to redeploy (ex : .iso or vmware/virtual bos
>> EC2 ami).
>
> Well, one quick point: database performance are going to be bad. Someone is
> currently trying to do something similar in one of the mailing list. Cloud
> vendor are very good for serving content like a web page. They are extremely
> poor for database performance. The IO is usually catastrophic.
> I wouldn't expect any real good performance out of such a system especially
> for rendering. I think that using cloud technology is for the moment at
> least a bit pointless.
>
> Emilie Laffray
>
>
>
> ___
> dev mailing list
> dev@openstreetmap.org
> http://lists.openstreetmap.org/listinfo/dev
>
>

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] amazon AMIs for a full blown openstreetmap.org-like server

[Emilie Laffray]

2010/1/8 Jonathan-David SCHRODER 

> Hello,
> Please tell me if I'm replicating any initiative or doing something the
> wrong or a not-so-good way.
>
> I've started this week from an EBS backed (or "boot")  Ubuntu 8.04 image
> ami-f6ad409f.
> I agree to give the .pem file for ssh connection to an instance to anyone
> willing to collaborate.
>
> My idea is to provide an EC2 EBS image with rails port(done) + tile
> rendering setup(started) + rails port->tile rendering sync scripts(todo) +
> easy config scripts (why not triggerable on startup with JSON config ala
> tilecache.org) (todo, not sure I'll have time to it).
> The list of commands to install is very long. I note down all the commands
> I type so as to be able to one or more bash scripts to be able to reinstall
> a similar solution on an Ubuntu N.NN later (by assuming that
> commands/package names for 8.04 are similar to those for 9.04 etc..). Maybe
> I'll have 3 bash scripts such as : install_railsport_u804.sh,
> install_mapnik_u804.sh, install_railsport_mapnik_sync.sh. And I'd need other
> scripts to go along : setup_rails_port_u804.sh, setup_mapnik_u804.sh and
> setup_rails_mapnik_sync_u804.sh...
> What do you think of this ?
>
> A recall on the requirements (reply to change them if you'd like to
> collaborate or request something):
> - postgres+osmosis+rails port (with 3 dbs : development, testing and
> production) : filled with nothing for now. setup_rails_port_u804.sh would
> have a simple calling syntax like :
>sudo setup_rails_port_u804.sh [--bz2url=
> http://domain/someRegion.osm.bz2|--bz2file=/path/to/file.osm.bz2]
> [--forceResetDB (false if omitted)]
> [--dbName=development(default),other,dbs,to,make,identical,too]
> [--guessDBLoginPassFromRailsConfig (default is true)]
> - mapnik tile rendering setup. I want to have a simple script to insert a
> bz2 + tell to use some style sheets + to tell to render at z=N (max zoom)
> - cron'able synced file making the Mapnik setup be filled with the Rails
> ports's data. I plan to get inspiration from the myanmar scripts for this.
>
> I'm sorry to repeat this, though, please reply if you know of some linux
> image (on EC2 or not) hanging somewhere which almost does what I'm trying to
> achieve and that image is easy to redeploy (ex : .iso or vmware/virtual bos
> EC2 ami).
>

Well, one quick point: database performance are going to be bad. Someone is
currently trying to do something similar in one of the mailing list. Cloud
vendor are very good for serving content like a web page. They are extremely
poor for database performance. The IO is usually catastrophic.
I wouldn't expect any real good performance out of such a system especially
for rendering. I think that using cloud technology is for the moment at
least a bit pointless.

Emilie Laffray
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] amazon AMIs for a full blown openstreetmap.org-like server

[Jonathan-David SCHRODER]

Hello,
Please tell me if I'm replicating any initiative or doing something the
wrong or a not-so-good way.

I've started this week from an EBS backed (or "boot")  Ubuntu 8.04 image
ami-f6ad409f.
I agree to give the .pem file for ssh connection to an instance to anyone
willing to collaborate.

My idea is to provide an EC2 EBS image with rails port(done) + tile
rendering setup(started) + rails port->tile rendering sync scripts(todo) +
easy config scripts (why not triggerable on startup with JSON config ala
tilecache.org) (todo, not sure I'll have time to it).
The list of commands to install is very long. I note down all the commands I
type so as to be able to one or more bash scripts to be able to reinstall a
similar solution on an Ubuntu N.NN later (by assuming that commands/package
names for 8.04 are similar to those for 9.04 etc..). Maybe I'll have 3 bash
scripts such as : install_railsport_u804.sh, install_mapnik_u804.sh,
install_railsport_mapnik_sync.sh. And I'd need other scripts to go along :
setup_rails_port_u804.sh, setup_mapnik_u804.sh and
setup_rails_mapnik_sync_u804.sh...
What do you think of this ?

A recall on the requirements (reply to change them if you'd like to
collaborate or request something):
- postgres+osmosis+rails port (with 3 dbs : development, testing and
production) : filled with nothing for now. setup_rails_port_u804.sh would
have a simple calling syntax like :
   sudo setup_rails_port_u804.sh [--bz2url=
http://domain/someRegion.osm.bz2|--bz2file=/path/to/file.osm.bz2]
[--forceResetDB (false if omitted)]
[--dbName=development(default),other,dbs,to,make,identical,too]
[--guessDBLoginPassFromRailsConfig (default is true)]
- mapnik tile rendering setup. I want to have a simple script to insert a
bz2 + tell to use some style sheets + to tell to render at z=N (max zoom)
- cron'able synced file making the Mapnik setup be filled with the Rails
ports's data. I plan to get inspiration from the myanmar scripts for this.

I'm sorry to repeat this, though, please reply if you know of some linux
image (on EC2 or not) hanging somewhere which almost does what I'm trying to
achieve and that image is easy to redeploy (ex : .iso or vmware/virtual bos
EC2 ami).

Jonathan

On Sun, Dec 20, 2009 at 6:20 PM, Jonathan-David SCHRODER <
jonathan.schro...@gmail.com> wrote:

> Hello,
>
> I'd like to create a platform like opencyclemap.org for indoor stuff
> easily, and more on more people on the Internet
> seem to want to setup their own openstreetmap server.
> Some tutorials exist describing the steps to obtain an OSM DB with data in
> it + a first Mapnik rendering :
> http://weait.com/content/build-your-own-openstreetmap-server
> Now what I would sincerely like is a quick and easy solution providing me
> an openstreetmap server distribution/image that :
> - ships on liveCD / CD / DVD or not but preferably for me as an AMI (Amazon
> Machine Image)
> - has :
> - an OSM psgql db setup with [SomeGeoData]
> - a mapnik or other renderer
> - an HTTP Rails port so as to have something like osm.org's
> slippymap+potlatch+HTTP API (at least TMS tiling API for tiles streaming and
> v0.6 features filtering API for POIs search and display) for mobiles
> applications&OSM editors (JOSM, Merkaartor etc..)
> ...where [SomeGeoData] is a flexible set (imported/synchronized frequently
> from other sources, or stored on machine and editable by remote over HTTP
> with a JOSM-like editor) of :
> -  layers (containing features that will be drawn but "not-clickable"),
> - POIs (clickable features),
> - interchangeable GeoStyleSheets or "how map should be drawn"-rules
>
> ... and that [SomeGeoData]'s flexible set of layers, POIs, stylesheets is
> remotely editable through a web interface, an HTTP interface or some shell
> scripts to run on the host.
>
> As far as LiveCDs / distributions are concerned, though I've seen only a
> few (one is Fedora based :
> https://fedoraproject.org/wiki/User:Ynemoy/Fedora_Geo_Spin ).
>
> I've experimented with tiledrawer.com for nearly a week on Amazon EC2.
> That server contains :
> - an openstreetmap "database" (postgresql + postGIS + the "primary
> interfacing" tools osm2pgsql / osmosis)
> - the TileCache python server which serves png tiles over http the TMS way*
> * this is openstreetmap.org's way of providing tiles for the Slippy Map
> one just needs to download each tile http://publicDNS/tilecache/x/y/z.png; 
> see also
> http://en.wikipedia.org/wiki/Tile_Map_Service
> I need the Rails port on top of that.
>
> I can give free access to an amazon EC2 instance based on an Alestic Ubuntu
> EC2 image... based on TileDrawer.com's image or not.
>
> Who can comment on my need/request for help/proposal for collaboration ?
>
> Thanks in advance !
>
> Take care !
>
> Jonathan-David SCHRODER
>
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org