Re: [OSM-dev] Export of GPX data ("planet.gpx")
A quick status update. I've done some more work and pushed a first incomplete version to my bitbucket repository[1]. I've never tested this so there are bound to be bugs. Tom pointed out a mistake I made with the private traces. So there'll be one public.gpx with all the trackpoints from all private traces combined. Today I've implemented parts of the custom GPX writing code and I plan to finish the rest in the next few days and then test it. I've also been contacted about this tool and someone is interested in finishing or doing his own version of it. I've given him the code and permission I've written so far to do whatever he wants with it. He needs the planet.gpx data more desperately than me so he might chose to finish the tool if I take too long :) Either way I hope we'll have a solution sometime next week ready to be tested. Cheers, Lars [1] http://bitbucket.org/lfrancke/planetgpxdump/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Export of GPX data ("planet.gpx")
Okay I'm partly done (will upload it to bitbucket in the next few days). I just copied and pasted a lot of stuff from the program that dumps the history data for OSM. The format for the metadata XML is currently something like this: Description barfoofobar * I don't really care if I use attributes[1] for the tags or write it in the element body as above so if anyone has an opinion about this let me know and I'll gladly change it. * For "Public" traces I would omit the user and uid attributes * I only include inserted and visible files. I assume that "inserted" means that the file has been processed and the individual points have been inserted? * As per short discussion on IRC I write a second, plain-text, file that just includes one fileNames one each line so it can easily be used with tar --from-file (or something else) to compile the final package * I ignore the data_public flag for users - that was before my time so I don't really know if this flag was supposed to cover GPX files too but I assumed not because of the possibility to set visibility on a per GPX basis * Start latitude and longitude are saved as doubles for GPX files (opposed to ints for nodes) but I treat them the same with seven digit precision. Is that okay? I think I will do some work to include "Trackable" and "Private" traces too (anonymized of course). That means that I'd have to write custom GPX files. * I currently use the id as the file name (just as for all the other traces). If the id shouldn't be revealed I can generate some other value but I figured that'd be security by obscurity and deemed it unnecessary * How would you prefer the output/how are GPX files saved on the server? Should I compress them and if yes which compression algorithm. Or should I write a second plain-text files with all generated GPX files so you can process them yourself? * Which metadata to write? id(?), points and visibility can safely be shared. startLatitude, startLongitude and timestamp only for "Trackable"? This would mean the following for the XML elements: Identifiable: Public: Trackable: & no Tags or Description Private: & no Tags or Description Cheers, Lars [1] | ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Export of GPX data ("planet.gpx")
On Wed, Dec 9, 2009 at 00:04, Lars Francke wrote: >> I'd take a look at it if no one else is working on it but if someone >> else is interested I'd gladly step back :) > > As no one has stepped forward with any ideas or offers to implement > this I'll take a shot at it. This also means that I get to define the > format of the metadata. Once I've worked that out I'll post again. > > A quick question to the admins: Do I have anything to worry about > regarding the server infrastructure? Thanks Lars, this is longtime needed data. We have been asked several time for a GPX dump. thank you! -S ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Export of GPX data ("planet.gpx")
> I'd take a look at it if no one else is working on it but if someone > else is interested I'd gladly step back :) As no one has stepped forward with any ideas or offers to implement this I'll take a shot at it. This also means that I get to define the format of the metadata. Once I've worked that out I'll post again. A quick question to the admins: Do I have anything to worry about regarding the server infrastructure? Cheers, Lars ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Export of GPX data ("planet.gpx")
On Sat, Dec 5, 2009 at 10:55, Tom Hughes wrote: > On 05/12/09 01:27, Lars Francke wrote: > >> The easiest for now would be to just throw all the GPX files that are >> marked as Identifiable or Public in a compressed file and add a file >> with (XML?) metadata about those traces. > > Actually I said tar for a reason, to avoid trying to compress the archive > which will just waste CPU time without achieving much as the traces > themselves are already compressed. Thanks for the note! Didn't know they were already compressed. >> Tom said that we'd lose 25% of all GPX files this way (they are either >> Private or Trackable) but it would be the easiest option for now. In a >> second step we could use the GPS-point data from the database to write >> custom GPX-files that comply with the visibility settings. > > The specific counts currently are: > > private - 116458 > trackable - 5410 > public - 324558 > identifiable - 13855 Thanks again for those numbers. Looking at the visibility settings again, I've got a question about the Trackable and Private settings. What about the tags for those? I guess those will be private too as someone could use personal stuff there. Lars ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Export of GPX data ("planet.gpx")
On 05/12/09 01:27, Lars Francke wrote: > The easiest for now would be to just throw all the GPX files that are > marked as Identifiable or Public in a compressed file and add a file > with (XML?) metadata about those traces. Actually I said tar for a reason, to avoid trying to compress the archive which will just waste CPU time without achieving much as the traces themselves are already compressed. > Tom said that we'd lose 25% of all GPX files this way (they are either > Private or Trackable) but it would be the easiest option for now. In a > second step we could use the GPS-point data from the database to write > custom GPX-files that comply with the visibility settings. The specific counts currently are: private - 116458 trackable - 5410 public - 324558 identifiable - 13855 Tom -- Tom Hughes (t...@compton.nu) http://www.compton.nu/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
