[OSM-talk] Bulk Uploading GPX files on the OSM website
Hi, A few of my issues with uploading gpx Files to OSM which I'd like to discuss, and put forward some suggestions. This may have a specific place to be discussed? I'm really not on top of mailing lists. Each few months I have about 100meg of gpx files to upload which is a very slow process on a slow internet connection. One option is to bundle them and upload, but they appear as one trace as stated here: http://wiki.openstreetmap.org/wiki/Batch_Upload Likewise if I upload as a batch I then have to find all the files and change all the tags for purpose of organisation. So I individually and painstakingly upload them and tag them up, with each file taking about 3 mins+, and I currently have a backlog of about 500 files. So the concept of sitting for 24+ hours of solid uploading/labelling files is unappealing! So it is always a case of taking a huge amount of time, the current options are just to either take that time up with a different process or compromise the organisation with a more basic process to save time. So in short, 'save time by not doing it fully', or to put it bluntly, 'cut corners'. Part of the issue is also having to move between pages to edit tags (if batched uploaded), or just to upload a new file if manually uploading files individually. Only a few seconds loading each page but it really adds up fast. This can be partly overcome by having multiple tabs open and setting up the next upload while the other one is underway, however the 3+mins per upload was considering this. Then moving files around and finding files is not possible. Finding files has been addressed and there is a request here: https://trac.openstreetmap.org/ticket/1999 so I'll just +1 that. For moving files, currently once a file is uploaded it's stuck in that order. Again depending on your level of organisation you may be able to relate or not, but sometimes it's infuriating to have to delete files you've painfully uploaded to upload another one in the correct place, so that the files are in date order (which is how I have them organised). It would be a huge time saver to be able to manually move the files up/down the list. Also the bulk uploader, and the wiki page is far beyond the ability of the average computer user, of which I am one. Within this project I will be one of the less computer(code) able people, however within the wider world, I am very able with computers. So from that perspective it should be common for me to read much I don't understand in this project, however I shouldn't have issues with using something like a bulk uploader, or a commonly needed tool/site etc. So that is my description of some bottlenecks. My idea, for simplicity I've stuck into an image form, which I'll give a rough description of, but hopefully the concept is clear. http://i57.photobucket.com/albums/g226/ben_robbins_/OSM-DetailsPage.png http://i57.photobucket.com/albums/g226/ben_robbins_/OSM-EditingPage.png There would be a bulk upload option in on the upload file page, and you then select a .rar/.zip file etc. It uploads untagged, or can inherit the file name for description, or put your own in for all of them being the same. Once all loaded a page opens (editing page) which lists all the files and allows you to add tags in mass. As for instance the tag 'UK' would likely appear on 100's of files in a row if you live in the UK which may have been uploaded. Likewise in smaller clumps with country/region/local region/town name etc. So if you could apply tags to ticked files, or manually type names in on the same page that would be ideal. Then for the file name a 'replace' function as in Excel would be ideal, as the track name can often be part of the file name. So if you inherit the file name, then replace, you can extract the necessary section easily as in example. The replace option would need to be able to look for something such as File - or **/**/2012 - for example. Then next to each file with it's tick box would be a small up and down arrow, for shifting the file around in the order, and likewise a delete button for quick removal rather than having to shift to a sub page and taking time. Finally within the current file listings page, a tick box method of getting multiple files into a multiple file editing table as just discribed would be needed. An advancement on this would be to be able to add to a list, by leaving the page and coming back to it with more selections. Also being able to remove things from the selections page, rather than just un-selecting. (Actually in rereading that, it really sounds similar, but a less graphical version, to the facebook bulk photo uploading interface.) I have no idea about ease of enabling such a feature/features or the demand that would have on the overall system, but I hope this highlights a few seemingly small inefficiencies which are huge in mass file uploading. Any one of the
Re: [OSM-talk] Bulk Uploading GPX files on the OSM website
On Tue, Aug 21, 2012 at 6:22 PM, Ben Robbins ben_robbi...@hotmail.com wrote: A few of my issues with uploading gpx Files to OSM which I'd like to discuss, and put forward some suggestions. This may have a specific place to be discussed? I'm really not on top of mailing lists. What I usually do is uploading them using a command line script found here: http://wiki.openstreetmap.org/wiki/Batch_Upload#CURL curl -u user:password -H Expect: -F file=@file.gpx -F description=description \ -F tags=tags -F visibility=public http://www.openstreetmap.org/api/0.6/gpx/create this can easily be automated, and you can set different tags for each one of the gpx. So far I've uploaded more than 1000 GPX: http://www.openstreetmap.org/user/simone/traces -- -S ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Bulk Uploading GPX files on the OSM website
(Copying to hot + dev because there are certainly many similar cases as I describe in hot's regions that would benefit more-than-average-places of more traces dev as that's where the brains/skills to fix this problem are. Replies to dev only?) I'll just +1 this post, its remarks and suggestions. As a similarly positioned tech savvy but not a coding-capable person I can only hope that someone with dev capabilities would pick up on the task. Another data point from behind bad connections of Haiti: I've been here now nearly two years and have caught myself uploading GPS traces only twice (en masse, a few other uploads of single tracks other than that). This has been due to what Ben describes and looking at the bigger picture--a recent quick analysis I did of the GPS traces downloaded for all the territory of Haiti--I am tempted to guess that this difficulty of uploading GPS traces leaves many traces unuploaded, which is a clear loss to the project. Why do I think so: a simple comparison of my Haiti GPS trace folder size with the size of all GPS traces for Haiti shows that I've uploaded almost half of the tad over 400mb of traces. Additionally the number of traces uploaded by someone else than me vs total (tagged with Haiti or haiti) is/was perhaps 40-50 (for the total history), iirc. .. And there have been hundreds of OSM contributors going around with GPSs here. Now, it's most certain that the technical inconvenience of uploading is only a part of the problem. .. But it _is_ part of the problem... I know that it's not a secret that the GPS trace functions of OSM both up- downloading are not as developed as data editing, map renderings available, etc and since can't really fix this problem I can only hope that someone else will (one day). Cheers from Haiti, -Jaakko Sent from my BlackBerry® device from Digicel -- Mobile: +509-37-26 91 54, Skype/GoogleTalk: jhelleranta -Original Message- From: Ben Robbins ben_robbi...@hotmail.com Date: Tue, 21 Aug 2012 17:22:31 To: talk@openstreetmap.org Subject: [OSM-talk] Bulk Uploading GPX files on the OSM website ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Bulk Uploading
On Sun, Jul 19, 2009 at 06:58, Andrew Ayrea...@britishideas.com wrote: I am converting public domain data into OSM files for bulk upload. This is data that currently is not in OSM. JOSM is a nightmare for uploading large amounts of data. I have a OSM file with 30k nodes (5Mb) and JOSM tells me it will take 15 hours... I tried bulk_upload.py and it fails with error 404 - no further information. I tried bulk_upload.php and it also failed with an error - can't remember the details now. Recently I've used the following in order to upload some converted data to osm... Data were generated from State's SHP using: http://svn.openstreetmap.org/applications/utils/import/shp2osm/polyshp2osm.py then: http://svn.openstreetmap.org/applications/utils/import/bulk_upload_06 Used a wrap-around to do directory traverse. -S ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Bulk Uploading
Hey, (just noticed your mail on the list) 2009/7/19 Andrew Ayre a...@britishideas.com: JOSM is a nightmare for uploading large amounts of data. I have a OSM file with 30k nodes (5Mb) and JOSM tells me it will take 15 hours... I tried bulk_upload.py and it fails with error 404 - no further information. I tried bulk_upload.php and it also failed with an error - can't remember the details now. I'm now trying the scripts at http://www.openstreetmap.pl/balrog/bulkupload/. However I don't want to upload incorrect or partial data to the public server. So I tried uploading to api06.dev.openstreetmap.org and mysql.dev.openstreetmap.org. Both servers returned the following error: 500: Internal Server Error (h2Application error/h2Rails application failed to start properly) All I want is a (moderately) fast and reliable way of uploading data. I have plenty of it ready to go. Can anyone help? I'm afraid the scripts won't be any faster than JOSM, all the scripts as well as JOSM use the same API and there's likely not going to be much difference in speed or in reliability. The 15h was probably a little pessimistic, though (depending on your connectivity, but it definitely should be less than that for 30k nodes). And reliability is assured by the API too. Cheers ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
[OSM-talk] Bulk Uploading
I am converting public domain data into OSM files for bulk upload. This is data that currently is not in OSM. JOSM is a nightmare for uploading large amounts of data. I have a OSM file with 30k nodes (5Mb) and JOSM tells me it will take 15 hours... I tried bulk_upload.py and it fails with error 404 - no further information. I tried bulk_upload.php and it also failed with an error - can't remember the details now. I'm now trying the scripts at http://www.openstreetmap.pl/balrog/bulkupload/. However I don't want to upload incorrect or partial data to the public server. So I tried uploading to api06.dev.openstreetmap.org and mysql.dev.openstreetmap.org. Both servers returned the following error: 500: Internal Server Error (h2Application error/h2Rails application failed to start properly) All I want is a (moderately) fast and reliable way of uploading data. I have plenty of it ready to go. Can anyone help? Do the Balrogg scripts really work? thanks, Andy -- Andy PGP Key ID: 0xDC1B5864 ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
