[OSM-talk] Question about Osmosis, Geofabrik and HTTP 504 Error
Dear list, I am getting the HTTP 504 error message *sometimes* when trying to get a state.txt file from Geofabrik with osmosis, but it happens sometimes and with some files. For example, the state.txt file for the Berlin updates fails to download and the state.txt file for the Bretagne updates works ok, and after a few minutes, the opposite situation happens. Could this be because the server is denying files by using some kind of file-oriented policy? Regards, Juan___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Question about Osmosis, Geofabrik and HTTP 504 Error
Hi, On 05/21/2014 04:36 PM, Juan Lucas Domínguez Rubio wrote: I am getting the HTTP 504 error message *sometimes* when trying to get a state.txt file from Geofabrik with osmosis, but it happens sometimes and with some files. For example, the state.txt file for the Berlin updates fails to download and the state.txt file for the Bretagne updates works ok, and after a few minutes, the opposite situation happens. Could this be because the server is denying files by using some kind of file-oriented policy? The Geofabrik download server applies rate limiting. If you try to download too many files in a short time - for example, if you were to run Osmosis in a loop where you'd check for a new state.txt every minute even though they only are renewed once a day - then the server will block access for your IP and issue 503 Service Unavailable errors (but not 504). Because of the throttling nature of this block, you might land in a situation where you can request one file every five minutes successfully, but then you get 503 again. Don't request so many files, and the problem will go away. If you check for 10 state.txt files every hour you won't have a problem but if you do that every five minutes then you might. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Question about Osmosis, Geofabrik and HTTP 504 Error
Hello, Thanks for the explanation. I have set up the cron so that it gets the *.osc.gz files from Geofabrik every night. I thought it would be a good idea to run osmosis three times, so in case the first does not work, the second perhaps will, and if one of them works, then the rest will do nothing because I am checking the lag every time. After your reply, I think I will try it only once every night, and if one night it does not work, it's not so important because the following night it will probably work. Any comments? Regards, Juan On Wednesday, May 21, 2014 5:46 PM, Frederik Ramm frede...@remote.org wrote: Hi, On 05/21/2014 04:36 PM, Juan Lucas Domínguez Rubio wrote: I am getting the HTTP 504 error message *sometimes* when trying to get a state.txt file from Geofabrik with osmosis, but it happens sometimes and with some files. For example, the state.txt file for the Berlin updates fails to download and the state.txt file for the Bretagne updates works ok, and after a few minutes, the opposite situation happens. Could this be because the server is denying files by using some kind of file-oriented policy? The Geofabrik download server applies rate limiting. If you try to download too many files in a short time - for example, if you were to run Osmosis in a loop where you'd check for a new state.txt every minute even though they only are renewed once a day - then the server will block access for your IP and issue 503 Service Unavailable errors (but not 504). Because of the throttling nature of this block, you might land in a situation where you can request one file every five minutes successfully, but then you get 503 again. Don't request so many files, and the problem will go away. If you check for 10 state.txt files every hour you won't have a problem but if you do that every five minutes then you might. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
