Re: [OSM-dev] what server next?
Tom, I am surprised that you think reliability is the issue - I thought we could make it quite reliable by having multiple redundant servers. I expected the criticism to be complexity - to make it work you need something along the lines of the following: 1. A 'master' which maintains a database of the available servers, whether they are alive or not, and how fast they are. 2. Some 'deputies' which synchronise themselves with the master and take over its duty if the master dies (the 'take over' bit is the bit I don't know how to do - will have to do something with DNS I suppose...). 3. When a request comes in the current master decides which sever to send the request to and passes it on. Therefore I think the main issue is that this is rather complicated and may be difficult to maintain, rather than a reliability issue as such? There is of course the additional processing step of choosing a server, which could slow things down, but I suppose it is just a scan through the bounding boxes of the various servers to see which ones can meet this request, so it might not add much onto the overall request. Graham. On Sep 8, 2009 8:46pm, Tom Hughes t...@compton.nu wrote: On 08/09/09 19:41, Graham Jones wrote: This would work for a single 'main' server, but I like the idea of it being distributed with lots of little ones (for example the computer in my attic could serve Northern England, someone else could do Belgium etc.). I don't know how to deal with re-directing the requests without a central main server though...any ideas? That kind of distribution is, in my opinion, a terrible idea. Such a system will just never be reliable. Tom -- Tom Hughes (t...@compton.nu) http://www.compton.nu/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] what server next?
Graham, If one of the slaves goes down in the time until the master finds out the slave will keep getting requests even so it isn't available. It is also difficult to reliably keep lots of machines in sync so that they all return the same data, especially when there are lots of different admins for the system, each admining their own slave inconsistently with the other slave. If it was an easy task all 3 xapi servers would have an high uptime. Shaun On 9 Sep 2009, at 08:08, grahamjones...@googlemail.com wrote: Tom, I am surprised that you think reliability is the issue - I thought we could make it quite reliable by having multiple redundant servers. I expected the criticism to be complexity - to make it work you need something along the lines of the following: 1. A 'master' which maintains a database of the available servers, whether they are alive or not, and how fast they are. 2. Some 'deputies' which synchronise themselves with the master and take over its duty if the master dies (the 'take over' bit is the bit I don't know how to do - will have to do something with DNS I suppose...). 3. When a request comes in the current master decides which sever to send the request to and passes it on. Therefore I think the main issue is that this is rather complicated and may be difficult to maintain, rather than a reliability issue as such? There is of course the additional processing step of choosing a server, which could slow things down, but I suppose it is just a scan through the bounding boxes of the various servers to see which ones can meet this request, so it might not add much onto the overall request. Graham. On Sep 8, 2009 8:46pm, Tom Hughes t...@compton.nu wrote: On 08/09/09 19:41, Graham Jones wrote: This would work for a single 'main' server, but I like the idea of it being distributed with lots of little ones (for example the computer in my attic could serve Northern England, someone else could do Belgium etc.). I don't know how to deal with re-directing the requests without a central main server though...any ideas? That kind of distribution is, in my opinion, a terrible idea. Such a system will just never be reliable. Tom -- Tom Hughes (t...@compton.nu) http://www.compton.nu/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev smime.p7s Description: S/MIME cryptographic signature ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Keeping .osm database up-to-date
When you say read the diffs. Do you mean download the weekly planet from osm.org, and then run it though osmosis. Would you be happy sending me an example of your set-up? Richard. 2009/9/8 Stephan Knauss o...@stephans-server.de Richard Ive wrote: Is it easy to run a cron that downloads the daily diff which you can then run an: ./osm2pgsql -a -m -d gis ../planet-date.osm.gz on? (I'd assume you add the -a to add the data, rather than clear the db) my setup uses osmosis to read the diffs and pipe the results to osm2pgsql. Mind to use the slim mode on initial import to support updates. Stephan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev -- Regards, Richard Ive. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] what server next?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Shaun McDonald schreef: If it was an easy task all 3 xapi servers would have an high uptime. I guess you got an open wound there. Because those servers are feed by what the main server is producing. If we look in the past two months and the amount of outages because of bad encoded changesets the actual problem is not the management but the information to it that is unvalidated. Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEAREKAAYFAkqncB8ACgkQYH1+F2Rqwn0pAQCdFeFDNZEQMrizj1FuxZCFplAT 9WMAn0Jp0VyLYDad5Hhl2hn0tC1NQkK6 =LBOQ -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] what server next?
On Tue, Sep 8, 2009 at 10:09 PM, Micha Ruhgnub...@gmail.com wrote: Hi 2009/9/8 Frederik Ramm frede...@remote.org Your idea suffers from the fundamental problem that it only counts those having last touched an object. No, it doesn't. I thought of a mechanism to avoid that: Users adding ways/pois get a score of 3 per way/poi added, users adding more tags to existing way/pois get a score-point of 1 for each edit. Import sources get a 0.1 score-point per way/poi. One could also think of more sophisticated factors to calculate score-points. - initial creation: 3 score-point factor - adding name: 2 - adding other tag: 1 - import sources: 0.1 - mass changes: 0.1 / change - bots: 0 What about a limit for score-point gaining per hour/day for a user? * Of course, the score-point factors are subject to fine-tuning! They must be adjusted, when one creates such a solution. And welcome to the real world -- if you put up a metric for people to achieve they'll happily manipulate it to show up on top. On your original you only made it mean I have to make 3 times as many useless edits to own each tile. Adding name, 2 points... excellent, I guess I should first delete the name, then readd it. 1 point for a tag... thank you utf8 character set! 3 points for initial creation... well, why waste time putting stuff in the right place, someone else can move it for 0 points at some future time. How do we find bots? What's the definition of a mass change that's somehow different to a prolific mapper? Does it seem worth the effort? Dave ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Feed reader stupidities with our Atom feeds
On Tue, 8 Sep 2009 22:45:53 +1000 Hugh Barnes list@hughbris.com wrote: , but I'll check on Feedreader for Windows tomorrow @ work. It did the right thing. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] what server next?
On Tue, Sep 08, 2009 at 08:35:51AM +0100, Tom Hughes wrote: Subject: Re: [OSM-dev] what server next? On 08/09/09 08:21, Florian Lohoff wrote: Sometimes i'd like to actively contact mappers in that distant region and mostly i dont care what they are called - simply - send email to top most recently active mappers in that region. This has been discussed a number of times - it's not something that needs any new hardware though. The problems with it are all about software and about the risk of it being abused. The possibility is already there - i wrote my own scripts around it - selecting users from my database mirror and then using some perl magic to post the messages to the OSM message system. Also I'd like to be able to answer to messages via mail not the web frontend. I dont like to touch my mouse for writing emails ... I am already thinking of doing some procmail/perl stuff to rewrite from adress on OSM messages and replying to a virtual gateway of mine which will repost the message to the OSM message system. If OSM usernames would not allow spaces and be case sensitive it would be possible to make the OSM usernames an email adress. Flo -- Florian Lohoff f...@rfc822.org Es ist ein grobes Missverständnis und eine Fehlwahrnehmung, dem Staat im Internet Zensur- und Überwachungsabsichten zu unterstellen. - - Bundesminister Dr. Wolfgang Schäuble -- 10. Juli in Berlin signature.asc Description: Digital signature ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] what server next?
On 09/09/09 15:41, Florian Lohoff wrote: Also I'd like to be able to answer to messages via mail not the web frontend. I dont like to touch my mouse for writing emails ... I am already thinking of doing some procmail/perl stuff to rewrite from adress on OSM messages and replying to a virtual gateway of mine which will repost the message to the OSM message system. Yes, and that's somewhere on my to-do list (as the person that gets all the replies from the people that can't read) but it's not trivial as our email lives in an entirely separate location to the database. If OSM usernames would not allow spaces and be case sensitive it would be possible to make the OSM usernames an email adress. Well if we suppressed a zillion other characters as well. Actually case sensitivity is not an issue - everything to the left of the @ is not (by standard) case insensitive. Many servers treat it as such but the standards do not require that so any mail client should not rely on it being case insensitive when replying. Tom -- Tom Hughes (t...@compton.nu) http://www.compton.nu/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Best way to select from a region
This one is 224495 - nine-five at the end instead of five-nine - though. 224459 has both a name and admin_level, it's returned by: [...] Thank you for pointing me to that. In fact it missing relation revealed a larger bug in the system: no relation beyond 187000 is considered as a potential area. I'm at the moment trying to fix it. Ok, the problem should now be fixed. Please try it again. Cheers, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql slim mode, postgis, and hard disk spindles
On Tue, Sep 8, 2009 at 4:42 PM, Stephan Knausso...@stephans-server.de wrote: Richard Weait wrote: Jaunty, osm2pgsql from svn, p4, 3.4GHz single 150GB hard drive 50 hours. Wow, 50 hours to apply the changes from a day. Oops. Not the daily diff. 50 hours for the full weekly planet. Still slow, but ... ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql slim mode, postgis, and hard disk spindles
I've created another thread for this, however it seems topical to ask here. Do you automate uploading diffs? If so would you be able to provide me with an example of how do you, ie scripts and/or crons? 2009/9/9 Richard Weait rich...@weait.com On Tue, Sep 8, 2009 at 4:42 PM, Stephan Knausso...@stephans-server.de wrote: Richard Weait wrote: Jaunty, osm2pgsql from svn, p4, 3.4GHz single 150GB hard drive 50 hours. Wow, 50 hours to apply the changes from a day. Oops. Not the daily diff. 50 hours for the full weekly planet. Still slow, but ... ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev -- Regards, Richard Ive. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] broken utf8 in minute changeset 200907140650
2009/7/14 Richard Fairhurst rich...@systemed.net: Hello, I still see some issues with the UTF-8 workarounds made in Potlatch. If I want to correct a letter that was previously typed, after writing the new letter the cursor jumps always at least(?) over the next character. For instance, if I want to correct Soseaua to become Șoseaua (first letter is capital letter S with comma below), after removing S and typing Ș, the cursor ends up after Șos. If I want to change the first a in Calaretilor into ă (lowercase a with breve), the cursor ands up after Căla, so again, two farther than it should be. If I want to change the character ' into '„' in 'Abcdefgh', the cursor ends up after '„Abc'. It looks like the cursor moves with the same ammount of positions as bytes encoding the UTF-8 character. Further to the long-running Linux Flash Player brokenness, it would be very helpful for any Linux FP users to do this: 1. Go to http://www.systemeD.net/stuff/keycode.html 2. Type some non-ASCII characters into the top box (letters with accents, the sort of thing you might want to enter as an OSM tag value) 3. For each one, tell me what it returns in the next two boxes For completeness, I am adding the letters and characters used in Romanian: For ă â ș ț î Ă Â Ș Ț Î „ ” « », respectively, original how Flash stores it and how the server gets it: ă 3 c4 192 03 C3 84 C6 92 â e2 C3 A2 ș 3 c8 2122 03 C3 88 E2 84 A2 ț 3 c8 203a 03 C3 88 E2 80 BA î ee C3 AE Ă 3 c4 201a  3 c3 201a 03 C3 83 E2 80 9A Ș 3 c8 2dc 03 C3 88 CB 9C Ț 3 c8 161 03 C3 88 C5 A1 Î 3 c3 17d 03 C3 83 C5 BD „ 3 e2 20ac 17e 03 C3 A2 E2 82 AC C5 BE ” 3 e2 20ac 9d 03 C3 A2 E2 82 AC C2 9D « ab C2 AB » bb C2 BB Also, please note that pressing AltGr generates the extra initial noise 3 in the flash store line, but since there is another standard variant of the Romanian layout which doesn't involve pressing AltGr to get to the special characters, the initial 3 is not present. Thanks! cheers Richard -- View this message in context: http://www.nabble.com/broken-utf8-in-minute-changeset-200907140650-tp24475713p24487675.html Sent from the OpenStreetMap - Dev mailing list archive at Nabble.com. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev -- Regards, EddyP = Imagination is more important than knowledge A.Einstein ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] what server next?
On Wed, Sep 9, 2009 at 2:41 PM, Florian Lohofff...@rfc822.org wrote: If OSM usernames would not allow spaces and be case sensitive it would be possible to make the OSM usernames an email adress. RFC 822 is a lot more flexible about E-Mail addresses than most people think. My username which contains spaces is e.g. a valid RFC 822 E-Mail address if you put it in quotations: Ævar Arnfjörð Bjarmason@users.openstreetmap.org Given the hostname part of your E-Mail you probably know some of this:) But can you provide an example of a username that isn't a valid in a RFC 822 E-Mail address when double-quoted in this manner? That's leaving aside that you could of course just give people nicknames / send to the OSM user ID if you wanted to implement this. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] what server next?
On Wed, Sep 09, 2009 at 06:07:35PM +, Ævar Arnfjörð Bjarmason wrote: On Wed, Sep 9, 2009 at 2:41 PM, Florian Lohofff...@rfc822.org wrote: If OSM usernames would not allow spaces and be case sensitive it would be possible to make the OSM usernames an email adress. RFC 822 is a lot more flexible about E-Mail addresses than most people think. My username which contains spaces is e.g. a valid RFC 822 E-Mail address if you put it in quotations: Ævar Arnfjörð Bjarmason@users.openstreetmap.org Given the hostname part of your E-Mail you probably know some of this:) But can you provide an example of a username that isn't a valid in a RFC 822 E-Mail address when double-quoted in this manner? That's leaving aside that you could of course just give people nicknames / send to the OSM user ID if you wanted to implement this. Currently OSM allows most characters: 2 usernames with in their name 35 users with ' in their name 49 users with @ in their name 4527 users with space in their name And there are tons of software out there which does not support anything than AlphaNumeric + handful of specials like -_+-. f...@de Would also be a valid email address although its of no use as most entry forms will only accept a domain part with at least a single dot ... Also the case sensitivity worrys me a bit - there is software out there which will fix the case for you on sending - IIRC Lotus Notes has got this (bad) habit ... Flo -- Florian Lohoff f...@rfc822.org Es ist ein grobes Missverständnis und eine Fehlwahrnehmung, dem Staat im Internet Zensur- und Überwachungsabsichten zu unterstellen. - - Bundesminister Dr. Wolfgang Schäuble -- 10. Juli in Berlin signature.asc Description: Digital signature ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Keeping .osm database up-to-date
Richard Ive wrote: When you say read the diffs. Do you mean download the weekly planet from osm.org http://osm.org, and then run it though osmosis. Would you be happy sending me an example of your set-up? The initial import was in slim mode. Updates are done whis way, osmosis being configured to hourly diffs: osmosis --rci --buffer-change bufferCapacity=1000 --wxc - | D:\osm\PostgreSQL\osm2pgsql-test\osm2pgsql.exe -a -s -C 3800 -d osm -H localhost -U postgres -S D:\osm\PostgreSQL\osm2pgsql-test\default.style - Stephan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Keeping .osm database up-to-date
Hi, Richard Ive wrote: When you say read the diffs. Do you mean download the weekly planet from osm.org http://osm.org, and then run it though osmosis. Would you be happy sending me an example of your set-up? There is a detailed example at http://wiki.openstreetmap.org/wiki/Minutely_Mapnik This uses minuetly diffs but you can do the same with hourly. Bye Frederik ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
