Re: [OSM-dev] API/XAPI caching proxy server

On Thu, Dec 16, 2010 at 3:17 PM, Ian Dees wrote: > On Thu, Dec 16, 2010 at 3:10 PM, Peter Körner wrote: > >> >> >> Am 16.12.2010 22:05, schrieb Stefan Keller: >> >>> Rewriting xapi on the other hand is an excellent idea. >>> >>> Yes, good idea. >>> >>> I would probably start with osm2pgsql a

Re: [OSM-dev] API/XAPI caching proxy server

Hi Stefan, On Fri, Dec 17, 2010 at 11:45 AM, Stefan Keller wrote: > Hi Brett > > Thanks very much for your detailed instructions. > > > In my experience the biggest limitation in performance is disk seeking, > > rather than the amount of data returned. > > If that's the bottleneck (or the amount

Re: [OSM-dev] API/XAPI caching proxy server

On 17 December 2010 00:45, Stefan Keller wrote: > Hi Brett > > Thanks very much for your detailed instructions. > > > In my experience the biggest limitation in performance is disk seeking, > > rather than the amount of data returned. > > If that's the bottleneck (or the amount of data returned b

Re: [OSM-dev] API/XAPI caching proxy server

Hi Brett Thanks very much for your detailed instructions. > In my experience the biggest limitation in performance is disk seeking, > rather than the amount of data returned. If that's the bottleneck (or the amount of data returned before processing), then pl/pgsql or pl/python could help, since

Re: [OSM-dev] API/XAPI caching proxy server

On Thu, Dec 16, 2010 at 5:39 PM, Brett Henderson wrote: > > In my experience the biggest limitation in performance is disk seeking, > rather than the amount of data returned. The earlier pgsimple schema used > to be well indexed, but due to the order in which data is created in OSM > (ie. light

Re: [OSM-dev] API/XAPI caching proxy server

I'll add some background info to the discussion which may be useful. If you do use a relational database, the latest pgsnapshot schema (ie. pgsimple with hstore tags) for Osmosis is at least worth a look. Note that I still called it the simple schema in Osmosis 0.38, but I am now differentiating

Re: [OSM-dev] API/XAPI caching proxy server

On 16/12/10 22:01, Wyo wrote: Tom Hughes wrote: It's not the number of requests which matters, it's the amount of data that is returned. To trigger the limit you have to fetch several hundred megabytes of data in a relatively short space of time. I'm abashed ;-? It can't be that I've downloade

Re: [OSM-dev] API/XAPI caching proxy server

Tom Hughes wrote: It's not the number of requests which matters, it's the amount of data that is returned. To trigger the limit you have to fetch several hundred megabytes of data in a relatively short space of time. I'm abashed ;-? It can't be that I've downloaded megabytes of data, there must

Re: [OSM-dev] API/XAPI caching proxy server

On 16/12/10 20:24, Erik Johansson wrote: That's not that amusing when the large requests happen to include a high density area. Is it really that bad to make 2 requests/second in a rural area? It's not the number of requests which matters, it's the amount of data that is returned. To trigger

Re: [OSM-dev] API/XAPI caching proxy server

Hi, Peter Körner wrote: The osmosis simple schema would be a better start. It also supports hstore and an geometry column but stores full meta information. I don't want to interfere with anybody's plans but I would recommend to not automatically assume that a relational database (or a relatio

Re: [OSM-dev] API/XAPI caching proxy server

Am 16.12.2010 22:17, schrieb Ian Dees: The osmosis simple schema would be a better start. It also supports hstore and an geometry column but stores full meta information. > I'm working on this exact setup as we speak. I wrote an XAPI querystring parser in ANTLR (https://github.com/iande

Re: [OSM-dev] API/XAPI caching proxy server

On Thu, 2010-12-16 at 22:10 +0100, Peter Körner wrote: > > Am 16.12.2010 22:05, schrieb Stefan Keller: > >> Rewriting xapi on the other hand is an excellent idea. > > > > Yes, good idea. > > > > I would probably start with osm2pgsql and setup a mirror (incuding > > hstore). Since osm2pgsql uses th

Re: [OSM-dev] API/XAPI caching proxy server

Am 16.12.2010 22:19, schrieb Stefan de Konink: Op 16-12-10 22:10, Peter Körner schreef: The osm2pgsql scheme does not store meta information (version, creator, timestamp), which are required to create valid .osm files. Valid sounds like someone updated a XSD file already?

Re: [OSM-dev] API/XAPI caching proxy server

-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Op 16-12-10 22:10, Peter Körner schreef: > The osm2pgsql scheme does not store meta information (version, creator, > timestamp), which are required to create valid .osm files. Valid sounds like someone updated a XSD file already?

Re: [OSM-dev] API/XAPI caching proxy server

On Thu, Dec 16, 2010 at 3:10 PM, Peter Körner wrote: > > > Am 16.12.2010 22:05, schrieb Stefan Keller: > >> Rewriting xapi on the other hand is an excellent idea. >>> >> >> Yes, good idea. >> >> I would probably start with osm2pgsql and setup a mirror (incuding >> hstore). Since osm2pgsql uses the

Re: [OSM-dev] API/XAPI caching proxy server

Am 16.12.2010 22:05, schrieb Stefan Keller: Rewriting xapi on the other hand is an excellent idea. Yes, good idea. I would probably start with osm2pgsql and setup a mirror (incuding hstore). Since osm2pgsql uses the PostGIS database I think it should'nt be too hard to turn predicates to WHER

Re: [OSM-dev] API/XAPI caching proxy server

> Rewriting xapi on the other hand is an excellent idea. Yes, good idea. I would probably start with osm2pgsql and setup a mirror (incuding hstore). Since osm2pgsql uses the PostGIS database I think it should'nt be too hard to turn predicates to WHERE clauses and bbox to a ST_Within query in Post

Re: [OSM-dev] API/XAPI caching proxy server

Hi, Erik Johansson wrote: It seems the excess detection process doesn't coop well with my mapping. So what do you suggest? Move slower or stop mapping? Load the data on a smaller zoom level so you're making one bigger request instead of tons of small requests? That's not that amusing when

Re: [OSM-dev] API/XAPI caching proxy server

On Thu, Dec 16, 2010 at 8:51 PM, Frederik Ramm wrote: > Hi, > > Wyo wrote: >> >> It seems the excess detection process doesn't coop well with my mapping. >> So what do you suggest? Move slower or stop mapping? > > Load the data on a smaller zoom level so you're making one bigger request > instead

Re: [OSM-dev] API/XAPI caching proxy server

Hi, Wyo wrote: It seems the excess detection process doesn't coop well with my mapping. So what do you suggest? Move slower or stop mapping? Load the data on a smaller zoom level so you're making one bigger request instead of tons of small requests? Bye Frederik -- Frederik Ramm ## eMai

Re: [OSM-dev] API/XAPI caching proxy server

Frederik Ramm wrote: On 12/15/10 19:54, Wyo wrote: Just tell me, how many people, who don't get paid for the work, have set up their own XAPI server? A handful or a dozens, wordwide? Sorry forgive me my harsh words, yet the current procedure to set up an XAPI server is useless. For most interest

Re: [OSM-dev] API/XAPI caching proxy server

Tom Hughes wrote: at http://www.openstreetmap.org/edit?lat=49.55544&lon=5.09523&zoom=17, one shift of the map and I hit the "Couldn't load map". It looks (based on the IP your mail was sent from) like you have been rate limited. How long does this persist? I'm not sure what sort of mapping

Re: [OSM-dev] API/XAPI caching proxy server

Peter, Our problem is money which osm could provide. Above you wrote you had 'Manpower as well as money'. What do you need even more money for? He used "we=osm" in one part of his message and "we=student group" in the other part. The student group has manpower and no money, and OSM, he b

Re: [OSM-dev] API/XAPI caching proxy server

Am 16.12.2010 17:18, schrieb Philipp Borgers: Sorry I don't agree that xapi is a non-essential service. It's one of the easiest ways to get osm data and we should provide easy access to our data in as many ways as possible. I general it I think osm should provide stronger infrastructure around al

Re: [OSM-dev] API/XAPI caching proxy server

Hi, Philipp Borgers wrote: Sorry I don't agree that xapi is a non-essential service. Let me share my defintion: "essential" is anything that absolutely *must* be done on the OSM servers, as part of the OSM infrastructure. Something that cannot possibly be done by a group of students on thei

Re: [OSM-dev] API/XAPI caching proxy server

On 16/12/10 16:18, Philipp Borgers wrote: > We are going to setup one tile cache and a database mirror. We would > like to "rewrite" search functionality and perhaps xapi ;) At the moment > we are writing some javascript/ajax/frontend-foo and analyse the logs of > osm tile servers and generating h

Re: [OSM-dev] API/XAPI caching proxy server

Am Mittwoch, den 15.12.2010, 20:10 +0100 schrieb Frederik Ramm: > Hi, > > On 12/15/10 19:54, Wyo wrote: > > Just tell me, how many people, who don't get paid for the work, have set > > up their own XAPI server? A handful or a dozens, wordwide? Sorry forgive > > me my harsh words, yet the current p

Re: [OSM-dev] API/XAPI caching proxy server

On 15/12/10 21:35, Wyo wrote: Tom Hughes wrote: I tried it when I first got your email and I've just tried again and on both occasions data was loading just fine - typical time to download a chunk of data is 10-30s at worst. Why then do I get it all the time? Since I've read your answer about

Re: [OSM-dev] API/XAPI caching proxy server

On 15/12/10 21:58, Ian Dees wrote: Well the link you sent wasn't for P2 editing... It is if you have P2 set as your default editor ;-) Tom -- Tom Hughes (t...@compton.nu) http://compton.nu/ ___ dev mailing list dev@openstreetmap.org http://lists.o

Re: [OSM-dev] API/XAPI caching proxy server

On Wed, Dec 15, 2010 at 3:56 PM, Otto Wyss wrote: > I just tried that link now (~10 minutes after you) and it took roughly >> 15 seconds to get map data back. There are no Yahoo imagery tiles, so >> the screen is filled with "We're sorry, the data you have requested is >> unavailable.", but the O

Re: [OSM-dev] API/XAPI caching proxy server

On Wed, Dec 15, 2010 at 3:35 PM, Wyo wrote: > Tom Hughes wrote: > >> I tried it when I first got your email and I've just tried again and on >> both occasions data was loading just fine - typical time to download a >> chunk of data is 10-30s at worst. >> >> Why then do I get it all the time? Sin

Re: [OSM-dev] API/XAPI caching proxy server

Tom Hughes wrote: I tried it when I first got your email and I've just tried again and on both occasions data was loading just fine - typical time to download a chunk of data is 10-30s at worst. Why then do I get it all the time? Since I've read your answer about 10 min ago, I tried it without

Re: [OSM-dev] API/XAPI caching proxy server

On 15/12/10 18:41, Wyo wrote: Well just about now it's impossible to work, in Potlatch2 I always get "Could'n load map". I've save my changes (that worked), left to viewing and again entered editing. The second shift of the map again brought the "Could'n load map". From now on till about 22:00h

Re: [OSM-dev] API/XAPI caching proxy server

Hi, Although I ranted before about the high barriers to entry in using the OSM data, I agree with Frederik on this one. With limited resources providing a good quality data query service isn't really a priority. I've implemented support for XAPI in both Kosmos and Maperitive and it's a flaky serv

Re: [OSM-dev] API/XAPI caching proxy server

Hi, On 12/15/10 19:54, Wyo wrote: Just tell me, how many people, who don't get paid for the work, have set up their own XAPI server? A handful or a dozens, wordwide? Sorry forgive me my harsh words, yet the current procedure to set up an XAPI server is useless. For most interested people this is

Re: [OSM-dev] API/XAPI caching proxy server

Frederik Ramm wrote: You're right that XAPI is in this thread's subject, but Wyo explicitly talked about mapping, for which XAPI is not required. I've talked about API/XAPI as you can see in the subject. And at least during high volume hours the API server is already on the brink, e.g. just now

Re: [OSM-dev] API/XAPI caching proxy server

Tom Hughes wrote: Oh please, do stop talking bollocks. I've actually been doing quite a bit of mapping the last few weeks at all sorts of different times and it's been absolutely fine. Well just about now it's impossible to work, in Potlatch2 I always get "Could'n load map". I've save my change

Re: [OSM-dev] API/XAPI caching proxy server

On Wed, 15 Dec 2010, Frederik Ramm wrote: On 12/15/10 08:53, Stefan de Konink wrote: It has been done in C already :) By different people, so what are we waiting for :) If there's a working instance that supports the XAPI protocol it should be added to http://wiki.openstreetmap.org/wiki/XA

Re: [OSM-dev] API/XAPI caching proxy server

Hi, On 12/15/10 08:53, Stefan de Konink wrote: It has been done in C already :) By different people, so what are we waiting for :) If there's a working instance that supports the XAPI protocol it should be added to http://wiki.openstreetmap.org/wiki/XAPI#Servers and also added to the automat

Re: [OSM-dev] API/XAPI caching proxy server

On Wed, 15 Dec 2010, Frederik Ramm wrote: But then again: XAPI has been written by a single person, in (what we consider) an esoteric programming language, in their spare time. So if someone really wanted to do it better, it should be a breeze to implement the same, or even an improved, interf

Re: [OSM-dev] API/XAPI caching proxy server

Hi, On 12/15/10 01:57, Stefan Keller wrote: > As stated by Wyo (and me) above: The technology and the way XAPI is > currently realized is out of discussion (MUMPS/GT.M). But then again: XAPI has been written by a single person, in (what we consider) an esoteric programming language, in their sp

Re: [OSM-dev] API/XAPI caching proxy server

Hi Frederik, You're right. I see that the priorities are elsewhere. There are still following questions which remain to me regarding XAPI: > Anyone who is unhappy with the performance of XAPI *can* set up their own > mirror server by using the minutely diffs we publish As stated by Wyo (and me)

Re: [OSM-dev] API/XAPI caching proxy server

Hi, Stefan Keller wrote: But at least regarding XAPI (as indicated in the thread title) it's true: Although http://wiki.openstreetmap.org/wiki/Platform_Status indicates green/OK, my tests and the remarks at http://wiki.openstreetmap.org/wiki/Xapi#Servers tell me that there's currently no XAPI se

Re: [OSM-dev] API/XAPI caching proxy server

But at least regarding XAPI (as indicated in the thread title) it's true: Although http://wiki.openstreetmap.org/wiki/Platform_Status indicates green/OK, my tests and the remarks at http://wiki.openstreetmap.org/wiki/Xapi#Servers tell me that there's currently no XAPI server answers in less than 60

Re: [OSM-dev] API/XAPI caching proxy server

On 14/12/10 20:51, Wyo wrote: I guess OSM has to use any of these technologies anywhen in the future. Even now it's almost impossible to map anything into OSM in the main traffic hours. The question is more where and how to start. Oh please, do stop talking bollocks. I've actually been doing q

Re: [OSM-dev] API/XAPI caching proxy server

Philipp Borgers wrote: Which wiki page exactly? I do if I find the page ;) I'm highly interested in DB-Clusters. We will play with it in the future. Suggestions and additional informations are welcomed. I think osm should use more of these distributed/cluster technologies... I guess OSM has to

Re: [OSM-dev] API/XAPI caching proxy server

Dear Wyo, Graham Jones wrote: > Last time there was serious trouble with XAPI servers I started to think about > setting up my own server (at least for a limited area), but got stuck with the > technology that XAPI uses. We also looked at XAPI technology and I can confirm that it's a zoo of techn

Re: [OSM-dev] API/XAPI caching proxy server

Which wiki page exactly? I do if I find the page ;) I'm highly interested in DB-Clusters. We will play with it in the future. Suggestions and additional informations are welcomed. I think osm should use more of these distributed/cluster technologies... best regards philipp Am Montag, den 13.12.

Re: [OSM-dev] API/XAPI caching proxy server

Philipp Borgers wrote: we use api, xapi, nominatim with a little proxy script written in php. We use memcache for caching. It's far from perfect... ;) Nice. For anybody providing their own map, this would be a good proxy script instead of the current suggested CGI version. Could you update the

Re: [OSM-dev] API/XAPI caching proxy server

Stefan de Konink wrote: DBSlayer is actually what you want here. Discussed before on this mailinglist, sadly at that time nobody wanted to do distributed databasing. I'm not sure if DBSlayer is usable solution for an XAPI proxy. I envision a solution where a proxy could act as a normal XAPI ser

Re: [OSM-dev] API/XAPI caching proxy server

Hi, we use api, xapi, nominatim with a little proxy script written in php. We use memcache for caching. It's far from perfect... ;) best regards philipp Am Samstag, den 11.12.2010, 11:41 +0100 schrieb Wyo: > After being told to create my own API/XAPI instance > (http://gis.638310.n2.nabble.co

Re: [OSM-dev] API/XAPI caching proxy server

On Sat, 11 Dec 2010, Wyo wrote: To have as many as possible proxies running in the internet means to stick with technologies easily available. That limits the choice almost entirely to a PHP/MySQL solution, at least for the beginning. DBSlayer is actually what you want here. Discussed before

Re: [OSM-dev] API/XAPI caching proxy server

Wyo, Last time there was serious trouble with XAPI servers I started to think about setting up my own server (at least for a limited area), but got stuck with the technology that XAPI uses. I did start to write my own version of it but didn't get far (I made the mistake of trying to use an off the