Re: [OSM-dev] what server next?
Grant Slater wrote: Politely, hell no. Compare total hardware usage and cost of operating and coordinating ti...@home* layer versus mapnik layer. Mapnik layer still operates from 1 server! hey, so does t...@h. It operates from 1 server. Just uses a few more clients :). * t...@h got us here, but mapnik can carry up further. It would be excellent to see the performance APIs which grew from t...@h growing more useful, maybe towards a feature set like xapi? I would love to see more ROMAs and TRAPIs put in place (fast read only mirror), also XAPIs that allow convenient filtering might be nice) What about the search server. Is that hardware sufficient? Indexing and searching is quite crucial and I seem to recall that the index was only updated every few months due to perf reasons. spaetz signature.asc Description: OpenPGP digital signature ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] what server next?
On Wed, Sep 9, 2009 at 7:26 PM, Graham Jones grahamjones...@googlemail.com wrote: I might give this a bit more thought if I get chance - does anyone have a feel for the load on the XAPI servers? - queries per unit time and data rate? There is the munin graphs, but they only give an overview of how much the OSM servers serve. See: http://wiki.openstreetmap.org/wiki/Servers In ancient times there was an anonimized log available that contained lots of data but it has been removed. It would be great to have an updated version of this. http://wiki.openstreetmap.org/wiki/Database#Access_logs And yes having a much faster /api/0.6/map request would be wonderful (I.E. XAPI or ROMA). -- /emj ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] what server next?
In article 4aa91558.1040...@sspaeth.de sebast...@sspaeth.de writes: I would love to see more ROMAs and TRAPIs put in place (fast read only mirror), also XAPIs that allow convenient filtering might be nice) As far as I can tell, it looks like a single Trapi server could handle all t...@h requests, if it could keep up to date reliably. Unfortunatly, it appears that a single consumer-grade SATA disk is no longer fast enough to keep up with the number of updates beeing done. Combined with the problems generating change files (minute diffs are sometimes delayed, especially when planet file is being generated -- hopefully new dev will solve this) and the change files missing stuff, Trapi hasn't been all that reliable lately. I'm also running into some snags on the new version, that I hoped to have ready months ago. One appears to be a perl bug, and the other is in the fastcgi that only occurs on one of the two systems I've tested on. For a new Trapi server, I think the following is minimal: fast net connection that can handle large volume fast disk: raid0 of 3 or more drives, or enterprise SSD. 100GB is enough. (additional 100GB or more of slow disk is also useful for planet files and logs) 2GB memory. More is better. 2Ghz cpu. Faster/additional cpu has some, but not much, impact. (The first requrement is dropped if you are only running Trapi for a local render farm. I found Trapi used less bandwidth to keep updated than a couple of t...@h renderers.) Note that there isn't any particular reason to colocate with other OSM servers, other than Mat's load-ballancer if that isn't replaced. -- Blars Blarson blar...@scd.debian.net With Microsoft, failure is not an option. It is a standard feature. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] what server next?
Tom, I am surprised that you think reliability is the issue - I thought we could make it quite reliable by having multiple redundant servers. I expected the criticism to be complexity - to make it work you need something along the lines of the following: 1. A 'master' which maintains a database of the available servers, whether they are alive or not, and how fast they are. 2. Some 'deputies' which synchronise themselves with the master and take over its duty if the master dies (the 'take over' bit is the bit I don't know how to do - will have to do something with DNS I suppose...). 3. When a request comes in the current master decides which sever to send the request to and passes it on. Therefore I think the main issue is that this is rather complicated and may be difficult to maintain, rather than a reliability issue as such? There is of course the additional processing step of choosing a server, which could slow things down, but I suppose it is just a scan through the bounding boxes of the various servers to see which ones can meet this request, so it might not add much onto the overall request. Graham. On Sep 8, 2009 8:46pm, Tom Hughes t...@compton.nu wrote: On 08/09/09 19:41, Graham Jones wrote: This would work for a single 'main' server, but I like the idea of it being distributed with lots of little ones (for example the computer in my attic could serve Northern England, someone else could do Belgium etc.). I don't know how to deal with re-directing the requests without a central main server though...any ideas? That kind of distribution is, in my opinion, a terrible idea. Such a system will just never be reliable. Tom -- Tom Hughes (t...@compton.nu) http://www.compton.nu/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] what server next?
Graham, If one of the slaves goes down in the time until the master finds out the slave will keep getting requests even so it isn't available. It is also difficult to reliably keep lots of machines in sync so that they all return the same data, especially when there are lots of different admins for the system, each admining their own slave inconsistently with the other slave. If it was an easy task all 3 xapi servers would have an high uptime. Shaun On 9 Sep 2009, at 08:08, grahamjones...@googlemail.com wrote: Tom, I am surprised that you think reliability is the issue - I thought we could make it quite reliable by having multiple redundant servers. I expected the criticism to be complexity - to make it work you need something along the lines of the following: 1. A 'master' which maintains a database of the available servers, whether they are alive or not, and how fast they are. 2. Some 'deputies' which synchronise themselves with the master and take over its duty if the master dies (the 'take over' bit is the bit I don't know how to do - will have to do something with DNS I suppose...). 3. When a request comes in the current master decides which sever to send the request to and passes it on. Therefore I think the main issue is that this is rather complicated and may be difficult to maintain, rather than a reliability issue as such? There is of course the additional processing step of choosing a server, which could slow things down, but I suppose it is just a scan through the bounding boxes of the various servers to see which ones can meet this request, so it might not add much onto the overall request. Graham. On Sep 8, 2009 8:46pm, Tom Hughes t...@compton.nu wrote: On 08/09/09 19:41, Graham Jones wrote: This would work for a single 'main' server, but I like the idea of it being distributed with lots of little ones (for example the computer in my attic could serve Northern England, someone else could do Belgium etc.). I don't know how to deal with re-directing the requests without a central main server though...any ideas? That kind of distribution is, in my opinion, a terrible idea. Such a system will just never be reliable. Tom -- Tom Hughes (t...@compton.nu) http://www.compton.nu/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev smime.p7s Description: S/MIME cryptographic signature ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] what server next?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Shaun McDonald schreef: If it was an easy task all 3 xapi servers would have an high uptime. I guess you got an open wound there. Because those servers are feed by what the main server is producing. If we look in the past two months and the amount of outages because of bad encoded changesets the actual problem is not the management but the information to it that is unvalidated. Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEAREKAAYFAkqncB8ACgkQYH1+F2Rqwn0pAQCdFeFDNZEQMrizj1FuxZCFplAT 9WMAn0Jp0VyLYDad5Hhl2hn0tC1NQkK6 =LBOQ -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] what server next?
On Tue, Sep 8, 2009 at 10:09 PM, Micha Ruhgnub...@gmail.com wrote: Hi 2009/9/8 Frederik Ramm frede...@remote.org Your idea suffers from the fundamental problem that it only counts those having last touched an object. No, it doesn't. I thought of a mechanism to avoid that: Users adding ways/pois get a score of 3 per way/poi added, users adding more tags to existing way/pois get a score-point of 1 for each edit. Import sources get a 0.1 score-point per way/poi. One could also think of more sophisticated factors to calculate score-points. - initial creation: 3 score-point factor - adding name: 2 - adding other tag: 1 - import sources: 0.1 - mass changes: 0.1 / change - bots: 0 What about a limit for score-point gaining per hour/day for a user? * Of course, the score-point factors are subject to fine-tuning! They must be adjusted, when one creates such a solution. And welcome to the real world -- if you put up a metric for people to achieve they'll happily manipulate it to show up on top. On your original you only made it mean I have to make 3 times as many useless edits to own each tile. Adding name, 2 points... excellent, I guess I should first delete the name, then readd it. 1 point for a tag... thank you utf8 character set! 3 points for initial creation... well, why waste time putting stuff in the right place, someone else can move it for 0 points at some future time. How do we find bots? What's the definition of a mass change that's somehow different to a prolific mapper? Does it seem worth the effort? Dave ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] what server next?
On Tue, Sep 08, 2009 at 08:35:51AM +0100, Tom Hughes wrote: Subject: Re: [OSM-dev] what server next? On 08/09/09 08:21, Florian Lohoff wrote: Sometimes i'd like to actively contact mappers in that distant region and mostly i dont care what they are called - simply - send email to top most recently active mappers in that region. This has been discussed a number of times - it's not something that needs any new hardware though. The problems with it are all about software and about the risk of it being abused. The possibility is already there - i wrote my own scripts around it - selecting users from my database mirror and then using some perl magic to post the messages to the OSM message system. Also I'd like to be able to answer to messages via mail not the web frontend. I dont like to touch my mouse for writing emails ... I am already thinking of doing some procmail/perl stuff to rewrite from adress on OSM messages and replying to a virtual gateway of mine which will repost the message to the OSM message system. If OSM usernames would not allow spaces and be case sensitive it would be possible to make the OSM usernames an email adress. Flo -- Florian Lohoff f...@rfc822.org Es ist ein grobes Missverständnis und eine Fehlwahrnehmung, dem Staat im Internet Zensur- und Überwachungsabsichten zu unterstellen. - - Bundesminister Dr. Wolfgang Schäuble -- 10. Juli in Berlin signature.asc Description: Digital signature ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] what server next?
On 09/09/09 15:41, Florian Lohoff wrote: Also I'd like to be able to answer to messages via mail not the web frontend. I dont like to touch my mouse for writing emails ... I am already thinking of doing some procmail/perl stuff to rewrite from adress on OSM messages and replying to a virtual gateway of mine which will repost the message to the OSM message system. Yes, and that's somewhere on my to-do list (as the person that gets all the replies from the people that can't read) but it's not trivial as our email lives in an entirely separate location to the database. If OSM usernames would not allow spaces and be case sensitive it would be possible to make the OSM usernames an email adress. Well if we suppressed a zillion other characters as well. Actually case sensitivity is not an issue - everything to the left of the @ is not (by standard) case insensitive. Many servers treat it as such but the standards do not require that so any mail client should not rely on it being case insensitive when replying. Tom -- Tom Hughes (t...@compton.nu) http://www.compton.nu/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] what server next?
On Wed, Sep 9, 2009 at 2:41 PM, Florian Lohofff...@rfc822.org wrote: If OSM usernames would not allow spaces and be case sensitive it would be possible to make the OSM usernames an email adress. RFC 822 is a lot more flexible about E-Mail addresses than most people think. My username which contains spaces is e.g. a valid RFC 822 E-Mail address if you put it in quotations: Ævar Arnfjörð Bjarmason@users.openstreetmap.org Given the hostname part of your E-Mail you probably know some of this:) But can you provide an example of a username that isn't a valid in a RFC 822 E-Mail address when double-quoted in this manner? That's leaving aside that you could of course just give people nicknames / send to the OSM user ID if you wanted to implement this. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] what server next?
On Wed, Sep 09, 2009 at 06:07:35PM +, Ævar Arnfjörð Bjarmason wrote: On Wed, Sep 9, 2009 at 2:41 PM, Florian Lohofff...@rfc822.org wrote: If OSM usernames would not allow spaces and be case sensitive it would be possible to make the OSM usernames an email adress. RFC 822 is a lot more flexible about E-Mail addresses than most people think. My username which contains spaces is e.g. a valid RFC 822 E-Mail address if you put it in quotations: Ævar Arnfjörð Bjarmason@users.openstreetmap.org Given the hostname part of your E-Mail you probably know some of this:) But can you provide an example of a username that isn't a valid in a RFC 822 E-Mail address when double-quoted in this manner? That's leaving aside that you could of course just give people nicknames / send to the OSM user ID if you wanted to implement this. Currently OSM allows most characters: 2 usernames with in their name 35 users with ' in their name 49 users with @ in their name 4527 users with space in their name And there are tons of software out there which does not support anything than AlphaNumeric + handful of specials like -_+-. f...@de Would also be a valid email address although its of no use as most entry forms will only accept a domain part with at least a single dot ... Also the case sensitivity worrys me a bit - there is software out there which will fix the case for you on sending - IIRC Lotus Notes has got this (bad) habit ... Flo -- Florian Lohoff f...@rfc822.org Es ist ein grobes Missverständnis und eine Fehlwahrnehmung, dem Staat im Internet Zensur- und Überwachungsabsichten zu unterstellen. - - Bundesminister Dr. Wolfgang Schäuble -- 10. Juli in Berlin signature.asc Description: Digital signature ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] what server next?
Hi all, We have recently had: db (smaug new) (1), rails (sarel, norbert draco. extra cpu) (2), tile (yevaud new), dev (errol new), xapi (fafnir. 2x300GB 10kRPM WD Raptor) (2), pg_namefinder (katie. 2x300GB 10kRPM WD Raptor) (2, 5), OSMF (Ridley. 2x750GB WD) (2) and soon wiki (?. HPDL360G4P,2xDualCPU2.8Ghz4MB,12GB,2x73GB 10kRPM, OOB iLO). (3) After all of that First of all: Thanks you (and all the other) for the work you put into it to make the current setup work. What is next? What are we going to need next? Faster Web Frontend? Better Load-Balancing (with fuchur) (4), Faster api/map call?, something complete different something new? Two ideas come to my mind: (ok, actually three) The first one would be a (decently fast) OSMXAPI server. Since the 0.6 API switch there seems to be a shortage of XAPI servers. If there was a stable, fast and up to date XAPI server it would help lots of people and it might reduce load on the main API. The second one would be another tile server (say b.tile.. and d.tile...) located network topologically far away. That should provide faster loading time for many users and would be the first step towards a redundant setup. Biggest problem is probably going to be hosting. The third one would be a faster namefinder / geocoder. But that is most likely not a server but a software issue. Perhaps one of the developers could speak up how far there efforts have gotten and what kind of server they would need. Patrick Petschge Kilian ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] what server next?
On Tue, Sep 08, 2009 at 05:47:04AM +0200, Micha Ruh wrote: Score-points get calculated per supertile, top 4 contributors for each supertile including their score-points get stored in a database, the attribution-db. Keys and indexes on zoom level and new supertile index (maybe central tile x/y). OpenLayer gets extended in a way that while requesting tiles an additional request to the attribution-db is issued, contributors for the corresponding supertile get loaded and nicely presented in the lower right corner of the view. OSM runs the attribution-db and spreads the extended OpenLayer lib. Sometimes i'd like to actively contact mappers in that distant region and mostly i dont care what they are called - simply - send email to top most recently active mappers in that region. Flo -- Florian Lohoff f...@rfc822.org Es ist ein grobes Missverständnis und eine Fehlwahrnehmung, dem Staat im Internet Zensur- und Überwachungsabsichten zu unterstellen. - - Bundesminister Dr. Wolfgang Schäuble -- 10. Juli in Berlin signature.asc Description: Digital signature ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] what server next?
On 08/09/09 08:21, Florian Lohoff wrote: Sometimes i'd like to actively contact mappers in that distant region and mostly i dont care what they are called - simply - send email to top most recently active mappers in that region. This has been discussed a number of times - it's not something that needs any new hardware though. The problems with it are all about software and about the risk of it being abused. Tom -- Tom Hughes (t...@compton.nu) http://www.compton.nu/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] what server next?
On 08/09/09 08:02, Patrick Petschge Kilian wrote: The first one would be a (decently fast) OSMXAPI server. Since the 0.6 API switch there seems to be a shortage of XAPI servers. If there was a stable, fast and up to date XAPI server it would help lots of people and it might reduce load on the main API. We've already given XAPI a server, and before we give it any more hardware I would need some serious convincing that XAPI as it currently exists is actually workable and scalable in some sensible way because as far as I can tell at the moment it isn't. As I understand it the database takes so long to load that if there are any working servers they are way out of date, and even when a server is working it can only serve a couple of users at a time which, when a XAPI query can often take minutes or hours to run is clearly not practical for serving a large community. There are also issues with the run time that the code uses - it's horrible ancient and crufty and requires various kernel security features to be turned off. Tom -- Tom Hughes (t...@compton.nu) http://www.compton.nu/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] what server next?
El Martes, 8 de Septiembre de 2009, Grant Slater escribió: We also get slightly more OSM operated hardware outside UCL soon. Any friendly large hosts out there that can offer us some well bandwidth fed hardware/rack space? Maybe a university in North America? The spanish local chapter *may* be able to get some rack space and connection to a backbone, either free or dirty cheap. Supposing that we get the rack space, what kind of machine and services would you like us to host here? Just a mapnik renderer to balance load and improve tile serving uptime? -- -- Iván Sánchez Ortega i...@sanchezortega.es http://ivan.sanchezortega.es Proudly running Debian Linux with 2.6.30-1-amd64 kernel, KDE 3.5.10, and PHP 5.2.10-2.2 generating this signature. Uptime: 12:32:40 up 2 days, 19:42, 4 users, load average: 1.22, 0.92, 0.90 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] what server next?
2009/9/8 Iván Sánchez Ortega i...@sanchezortega.es: Supposing that we get the rack space, what kind of machine and services would you like us to host here? Just a mapnik renderer to balance load and improve tile serving uptime? Hosting OpenLayers.js and some kind of web page would be useful as well so people with simple web pages could embed a map in an iframe. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] what server next?
El Martes, 8 de Septiembre de 2009, Micha Ruh escribió: [...] A server calculates user contribution for sets of 3x3 tiles (supertiles) for each zoom level 12-18. Meh. Precalculating all that would be too time-consuming. I'd rather go 8x8 mod_tile-like meta-tiles and a real-time API for z12, with a cache on top. Heck, you could even do just z12 tiles and get away with that. The calculation is done once a week (maybe done by t...@h clients). Meh. Mark the cache as dirty every week. And if mod_tile+mapnik can render stuff in near-real-time, there is no reason you could not just count stuff in near-real-time. Heck, mod_tile has to load data for 8x8 meta-tiles every time a meta-tile is marked dirty. Use that opportunity to count the attribution data there: just add a couple of fields to osm2pgsql and make mod_tile count the attribution data. Users adding ways/pois get a score of 3 per way/poi added, users adding more tags to existing way/pois get a score-point of 1 for each edit. Import sources get a 0.1 score-point per way/poi. Good luck telling those apart, given a planet dump. Why not just 1 node/way/relation = 1 point?? OpenLayer gets extended in a way that while requesting tiles an additional request to the attribution-db is issued, contributors for the corresponding supertile get loaded and nicely presented in the lower right corner of the view. No need to. A bit of javascript magic can do that. Cheers, -- -- Iván Sánchez Ortega i...@sanchezortega.es http://ivan.sanchezortega.es Proudly running Debian Linux with 2.6.30-1-amd64 kernel, KDE 3.5.10, and PHP 5.2.10-2.2 generating this signature. Uptime: 12:39:41 up 2 days, 19:49, 4 users, load average: 0.95, 0.96, 0.92 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] what server next?
2009/9/8 John Smith deltafoxtrot...@gmail.com: Hosting OpenLayers.js and some kind of web page would be useful as well so people with simple web pages could embed a map in an iframe. * osm.org * Export tab * Select Embeddable Html -- Regards, Thomas Wood (Edgemaster) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] what server next?
Am 08.09.2009 um 12:38 schrieb Iván Sánchez Ortega i...@sanchezortega. es: El Martes, 8 de Septiembre de 2009, Grant Slater escribió: We also get slightly more OSM operated hardware outside UCL soon. Any friendly large hosts out there that can offer us some well bandwidth fed hardware/rack space? Maybe a university in North America? The spanish local chapter *may* be able to get some rack space and connection to a backbone, either free or dirty cheap. Supposing that we get the rack space, what kind of machine and services would you like us to host here? Just a mapnik renderer to balance load and improve tile serving uptime? We also have an offer for a server including traffic and hardware from a german ISP for a few month now. Grant also knows already about this for a while. Jonas -- -- Iván Sánchez Ortega i...@sanchezortega.es http://ivan.sanchezortega.es Proudly running Debian Linux with 2.6.30-1-amd64 kernel, KDE 3.5.10, and PHP 5.2.10-2.2 generating this signature. Uptime: 12:32:40 up 2 days, 19:42, 4 users, load average: 1.22, 0.92, 0.90 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] what server next?
On 22/07/28164 20:59, Iván Sánchez Ortega wrote: El Martes, 8 de Septiembre de 2009, Grant Slater escribió: We also get slightly more OSM operated hardware outside UCL soon. Any friendly large hosts out there that can offer us some well bandwidth fed hardware/rack space? Maybe a university in North America? The spanish local chapter *may* be able to get some rack space and connection to a backbone, either free or dirty cheap. Supposing that we get the rack space, what kind of machine and services would you like us to host here? Just a mapnik renderer to balance load and improve tile serving uptime? My guess would be that providing a caching proxy for mapnik tiles and potentially the wiki would be the easiest for some external hosting. It would help quite a bit with bandwidth (and hopefully also with uptime) as the mediawiki proxy test showed and should be comparatively moderate on server hardware and admin time. Setting up a global mapnik render server would need alot more server power and might not be worth it unless you want to offer localised or alternative rendering styles. Kai ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] what server next?
2009/9/8 Thomas Wood grand.edgemas...@gmail.com: 2009/9/8 John Smith deltafoxtrot...@gmail.com: Hosting OpenLayers.js and some kind of web page would be useful as well so people with simple web pages could embed a map in an iframe. * osm.org * Export tab * Select Embeddable Html I'm not suggesting a limited subset of OSM as a static image. It would be nice if there was a full dynamic embedable iframe option. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] what server next?
2009/9/8 John Smith deltafoxtrot...@gmail.com: 2009/9/8 Thomas Wood grand.edgemas...@gmail.com: 2009/9/8 John Smith deltafoxtrot...@gmail.com: Hosting OpenLayers.js and some kind of web page would be useful as well so people with simple web pages could embed a map in an iframe. * osm.org * Export tab * Select Embeddable Html I'm not suggesting a limited subset of OSM as a static image. It would be nice if there was a full dynamic embedable iframe option. Sorry, I meant the whole point of having distributed servers is to allow for greater uptime of this, using a static image is the only way to get that uptime at present. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] what server next?
El Martes, 8 de Septiembre de 2009, John Smith escribió: I'm not suggesting a limited subset of OSM as a static image. It would be nice if there was a full dynamic embedable iframe option. Have you actually had a look at the export tab? -- -- Iván Sánchez Ortega i...@sanchezortega.es Aviso: Este e-mail es confidencial y no debería ser usado por nadie que no sea el destinatario original. No se permite la reproducción mediante fotocopia, walkie-talkie, emisora de radioaficionado, satélite, televisión por cable, proyector, señales de humo, código morse, braille, lenguaje de signos, taquigrafía o cualquier otro medio. Bajo ningún concepto debe traducirse al francés este e-mail. Este e-mail no puede ser ridiculizado, parodiado, juzgado en una competición, o leído en voz alta con un acento gracioso llevando un bigote falso y/o cualquier tipo de sombrero, incluyendo pero no limitándose a pañuelos. No inciten ni provoquen a este e-mail. Si está medicándose, puede experimentar nauseas, desorientación, histeria, vómitos, pérdida temporal de la memoria a corto plazo y malestar general al leer este e-mail. Consulte a su médico o farmacéutico antes de leer este e-mail. Todas las modelos descritas en este e-mail son mayores de 18 años. Si ha recibido este e-mail por error es probablemente porque estaba borracho cuando escribí la dirección del destinatario. attachment: export_tab.png___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] what server next?
2009/9/8 Iván Sánchez Ortega i...@sanchezortega.es: Have you actually had a look at the export tab? This whole topic is about servers, and if the map server goes down that embedded option is of no use. I should have been clearer in my previous email. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] what server next?
2009/9/8 Thomas Wood grand.edgemas...@gmail.com: 2009/9/8 John Smith deltafoxtrot...@gmail.com: Hosting OpenLayers.js and some kind of web page would be useful as well so people with simple web pages could embed a map in an iframe. * osm.org * Export tab * Select Embeddable Html Maybe the embedded stuff needs to be on it's own hostname instead of www so that the basic html and js can be duplicated across multiple sites that also host tile servers etc. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] what server next?
El Martes, 8 de Septiembre de 2009, John Smith escribió: Maybe the embedded stuff needs to be on it's own hostname instead of www so that the basic html and js can be duplicated across multiple sites that also host tile servers etc. DNS load balancing should be able to take care of that. -- -- Iván Sánchez Ortega i...@sanchezortega.es Aviso: Este e-mail es confidencial y no debería ser usado por nadie que no sea el destinatario original. No se permite la reproducción mediante fotocopia, walkie-talkie, emisora de radioaficionado, satélite, televisión por cable, proyector, señales de humo, código morse, braille, lenguaje de signos, taquigrafía o cualquier otro medio. Bajo ningún concepto debe traducirse al francés este e-mail. Este e-mail no puede ser ridiculizado, parodiado, juzgado en una competición, o leído en voz alta con un acento gracioso llevando un bigote falso y/o cualquier tipo de sombrero, incluyendo pero no limitándose a pañuelos. No inciten ni provoquen a este e-mail. Si está medicándose, puede experimentar nauseas, desorientación, histeria, vómitos, pérdida temporal de la memoria a corto plazo y malestar general al leer este e-mail. Consulte a su médico o farmacéutico antes de leer este e-mail. Todas las modelos descritas en este e-mail son mayores de 18 años. Si ha recibido este e-mail por error es probablemente porque estaba borracho cuando escribí la dirección del destinatario. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] what server next?
2009/9/8 Iván Sánchez Ortega i...@sanchezortega.es: El Martes, 8 de Septiembre de 2009, John Smith escribió: Maybe the embedded stuff needs to be on it's own hostname instead of www so that the basic html and js can be duplicated across multiple sites that also host tile servers etc. DNS load balancing should be able to take care of that. Exactly, but the embedded stuff is only a couple of files so the entire website wouldn't need to be duplicated. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] what server next?
Iván Sánchez Ortega wrote: Supposing that we get the rack space, what kind of machine and services would you like us to host here? Just a mapnik renderer to balance load and improve tile serving uptime? I'd suggest: * A localised front page * Cacheing proxy of the main Mapnik render * A localised render -- Jonathan (Jonobennett) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] what server next?
I would like to look into a distributed version of XAPI - I agree with Tom that the current XAPI server is not ideal (but this may be because I can't work out how it works). I envisage an XAPI server utilising a 'standard' PostgreSQL database produced by osmosis, kept up to date by the daily/hourly diffs etc, along with a simple web server front-end to do the XAPI to SQL translation. I made a start coding a simply python based server to do this, but haven't got very far (made the mistake of deciding not to write my own parser but use a parser generator - terrible mistake!). Maybe I should give up and write it in C - I'm sure the production version will be in C for speed anyway This would work for a single 'main' server, but I like the idea of it being distributed with lots of little ones (for example the computer in my attic could serve Northern England, someone else could do Belgium etc.). I don't know how to deal with re-directing the requests without a central main server though...any ideas? Having a nice fast XAPI system would make more dynamic maps (e.g. a vector layer of POIs obtained from XAPI over a static mapnik background) a bit more usable - all of the examples I have seen so far are a bit sluggish. Graham 2009/9/8 Tom Hughes t...@compton.nu On 08/09/09 08:02, Patrick Petschge Kilian wrote: The first one would be a (decently fast) OSMXAPI server. Since the 0.6 API switch there seems to be a shortage of XAPI servers. If there was a stable, fast and up to date XAPI server it would help lots of people and it might reduce load on the main API. We've already given XAPI a server, and before we give it any more hardware I would need some serious convincing that XAPI as it currently exists is actually workable and scalable in some sensible way because as far as I can tell at the moment it isn't. As I understand it the database takes so long to load that if there are any working servers they are way out of date, and even when a server is working it can only serve a couple of users at a time which, when a XAPI query can often take minutes or hours to run is clearly not practical for serving a large community. There are also issues with the run time that the code uses - it's horrible ancient and crufty and requires various kernel security features to be turned off. Tom -- Tom Hughes (t...@compton.nu) http://www.compton.nu/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev -- Dr. Graham Jones Hartlepool, UK email: grahamjones...@gmail.com ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] what server next?
On 08/09/09 19:41, Graham Jones wrote: This would work for a single 'main' server, but I like the idea of it being distributed with lots of little ones (for example the computer in my attic could serve Northern England, someone else could do Belgium etc.). I don't know how to deal with re-directing the requests without a central main server though...any ideas? That kind of distribution is, in my opinion, a terrible idea. Such a system will just never be reliable. Tom -- Tom Hughes (t...@compton.nu) http://www.compton.nu/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] what server next?
Hi, Micha Ruh wrote: Last night I dreamed about a solution to the cc-by-sa attribution problem. It would be soo nice to have appropriate attribution displayed in OpenLayer while browsing the map. Your idea suffers from the fundamental problem that it only counts those having last touched an object. The guy who went there on his bike and painstakingly recorded the geometry - zero points. The guy who later added an is_in tag - 100 points. We already have users who upload tens of thousands of objects completely unchanged just so that their name comes out on top (see http://www.openstreetmap.org/user/pfoten-weg/edits) - I don't want to encourage that by giving the last-editing user an even more prominent place. Bye Frederik ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] what server next?
Hi 2009/9/8 Frederik Ramm frede...@remote.org Your idea suffers from the fundamental problem that it only counts those having last touched an object. No, it doesn't. I thought of a mechanism to avoid that: Users adding ways/pois get a score of 3 per way/poi added, users adding more tags to existing way/pois get a score-point of 1 for each edit. Import sources get a 0.1 score-point per way/poi. One could also think of more sophisticated factors to calculate score-points. - initial creation: 3 score-point factor - adding name: 2 - adding other tag: 1 - import sources: 0.1 - mass changes: 0.1 / change - bots: 0 What about a limit for score-point gaining per hour/day for a user? * Of course, the score-point factors are subject to fine-tuning! They must be adjusted, when one creates such a solution. Regards Micha ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] what server next?
Dev, We have recently had: db (smaug new) (1), rails (sarel, norbert draco. extra cpu) (2), tile (yevaud new), dev (errol new), xapi (fafnir. 2x300GB 10kRPM WD Raptor) (2), pg_namefinder (katie. 2x300GB 10kRPM WD Raptor) (2, 5), OSMF (Ridley. 2x750GB WD) (2) and soon wiki (?. HPDL360G4P,2xDualCPU2.8Ghz4MB,12GB,2x73GB 10kRPM, OOB iLO). (3) After all of that What is next? What are we going to need next? Faster Web Frontend? Better Load-Balancing (with fuchur) (4), Faster api/map call?, something complete different something new? / Grant 1: http://wiki.openstreetmap.org/wiki/Servers/smaug 2: http://wiki.openstreetmap.org/wiki/Servers/Upgrades/082009 3: http://wiki.openstreetmap.org/wiki/Servers/Upgrades/092009 4: http://wiki.openstreetmap.org/wiki/Servers/fuchur 5: http://trac.openstreetmap.org/browser/applications/utils/export/osm2pgsql/gazetteer ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] what server next?
2009/9/8 Grant Slater openstreet...@firefishy.com: What is next? What are we going to need next? Faster Web Frontend? Better Load-Balancing (with fuchur) (4), Faster api/map call?, something complete different something new? Distributed web/tile server? :) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] what server next?
2009/9/8 Grant Slater openstreet...@firefishy.com: Politely, hell no. Compare total hardware usage and cost of operating and coordinating ti...@home* layer versus mapnik layer. Mapnik layer still operates from 1 server! * t...@h got us here, but mapnik can carry up further. It would be excellent to see the performance APIs which grew from t...@h growing more useful, maybe towards a feature set like xapi? I think you misunderstood me, I'm not talking about t...@h, I'm talking about the ability to have multiple tile servers/web front ends in multiple data centres so people can actually use OSM maps to embed them in websites etc and not worry about 2 day outages. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] what server next?
On Mon, Sep 7, 2009 at 9:02 PM, Grant Slater openstreet...@firefishy.comwrote: Politely, hell no. Compare total hardware usage and cost of operating and coordinating ti...@home* layer versus mapnik layer. Mapnik layer still operates from 1 server! * t...@h got us here, but mapnik can carry up further. It would be excellent to see the performance APIs which grew from t...@h growing more useful, maybe towards a feature set like xapi? Was going to suggest exactly that: a server to run data queries like that would be great. Something like what cloudmade does: generate weekly extracts of important regions or features. Important being discussed on the list and/or voted on by the community. Perhaps someone wants a shapefile with all the motorways in Western Europe or a MySQL dump of all amenity=* POI for the state of Alaska. Make it easier to get access (in OSM and more popular formats) to our data. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] what server next?
2009/9/8 John Smith deltafoxtrot...@gmail.com: I think you misunderstood me, I'm not talking about t...@h, I'm talking about the ability to have multiple tile servers/web front ends in multiple data centres so people can actually use OSM maps to embed them in websites etc and not worry about 2 day outages. True sorry. Tile going down was unfortunate, but there was a plan to keep serving but it did not work out. I had too many balls and I dropped one, the rest of the weekend went well. We also get slightly more OSM operated hardware outside UCL soon. Any friendly large hosts out there that can offer us some well bandwidth fed hardware/rack space? Maybe a university in North America? / Grant ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] what server next?
2009/9/8 Grant Slater openstreet...@firefishy.com What is next? What are we going to need next? Last night I dreamed about a solution to the cc-by-sa attribution problem. It would be soo nice to have appropriate attribution displayed in OpenLayer while browsing the map. There it goes.. A server calculates user contribution for sets of 3x3 tiles (supertiles) for each zoom level 12-18. The calculation is done once a week (maybe done by t...@h clients). Users adding ways/pois get a score of 3 per way/poi added, users adding more tags to existing way/pois get a score-point of 1 for each edit. Import sources get a 0.1 score-point per way/poi. Score-points get calculated per supertile, top 4 contributors for each supertile including their score-points get stored in a database, the attribution-db. Keys and indexes on zoom level and new supertile index (maybe central tile x/y). OpenLayer gets extended in a way that while requesting tiles an additional request to the attribution-db is issued, contributors for the corresponding supertile get loaded and nicely presented in the lower right corner of the view. OSM runs the attribution-db and spreads the extended OpenLayer lib. * Information which user created a way is only available in history, so initial calculation is heavy-load, but must only be done once. * attribution information for mutliple supertiles could be aggregated into attribution-pages which are stored in the attribution-db. this attribution-pages are sent to the client, which sorts out the appropriate attribution with some .js magic for the current displayed view. This could save the attribution-db from load. * attribution score-point changes could be calculated on-the-fly, once an initial load of the attribution-db is done. :D Need more sleep, now. Regards Micha ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] what server next?
2009/9/8 Micha Ruh gnub...@gmail.com: OpenLayer gets extended in a way that while requesting tiles an additional request to the attribution-db is issued, contributors for the corresponding supertile get loaded and nicely presented in the lower right corner of the view. Wouldn't this only serve to clutter things up and detract away from the important information? ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] what server next?
2009/9/8 John Smith deltafoxtrot...@gmail.com Wouldn't this only serve to clutter things up and detract away from the important information? Nasa World Wind et. al. does a fine job in displaying copyright information without distract the experience. if they'd donate some decent kit in return for their logo being displayed etc. Wouldn't this only serve to clutter things up and detract away from the important information? :D Regards Micha ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] what server next?
2009/9/8 Micha Ruh gnub...@gmail.com: Wouldn't this only serve to clutter things up and detract away from the important information? No more than the existing logos on the OSM website ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
