Re: [OSM-dev] Inserting OSM data
Hi all, >> The ideal solution for use would be to send a WAY segment in OSM >> format to some service and receive a response with the IDs for the >> created way and the created nodes! But I understand this is not >> supported due to the obvious merging problem (avoid duplicates)!? > > With the API, you can upload new nodes/ways/relations, fetch the > existing ones, and update the existing ones (uploading a changeset). > > The problem is that you might upload a new way that overlaps an > existing OSM way. Either you upload it and merge with OSM tools, or > you download the OSM data in the neighborhood, check if it overlaps, > and update your data with the already-existing OSM data, etc etc. We are now about to implement the --> public OSM server filter and as a first measure would like to simply upload GPS traces generated by our employees. We have the trackpoints in our database. We can easily generate a gpx file from that http://www.topografix.com/GPX/1/0/";> ... and now are looking for a way to programmatically upload that to the OSM server for the community to review the data and conversion into actual ways and nodes. We plan to use a system() call from our application and execute the following command curl -v -v -d @tracks.gpx -H "X_HTTP_METHOD_OVERRIDE: POST" "http://api.openstreetmap.org/api/0.6/gpx/create"; but I assume that the above command is not entirely correct!? I especially not understand the "It expects the following POST parameters in a multipart/form-data HTTP message:" part on http://wiki.openstreetmap.org/wiki/API_v0.6. How has the above command line to be modified to work? In a later step we want to add functionality to our system that allows our users to directly create nodes and ways on the public OSM server but this requires a bunch of problems to be solved first. FOr now we would like to stick to uploading traces. How can this best be done from a unix application? Thanks a lot, Andreas ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OSM Data Update
Hi all, > I just had an issue with our map server. The maps between Lübeck and > Hamburg (Germany) were fine for a couple of weeks. I ran a script over > night that pulled some updates from the OSM server, deleted superfluous > ways, created new ways in our private database based on the downloaded > OSM and regenerated the tiles for this area. The result was that 20km > of A1 between Lübeck and Hamburg vanished. > > I am wondering whether this is due to some kind of bug in our software > or due to someone replacing all these ways with new ones!? Anyone > worked on this during the last 24h? > > I just would like to find the bug, if if there is one in our software. > But if this is to be expectd due to heavy OSM data changes ... Just following up on my own posting. Just just updated anotehr are that lacked some ways from the OSM server • internal database import • tile generation This time the end of the A25 in Geesthacht. The motorway ends too early on the field and has no conenction to the B5. When I check out this area on www.openstreetmap.org the (View pane) it looks just fine!? Does the maps diplayed on www.openstreetmap.org represent the current database contents or some older state (tiles generated every now and then)?? I am just trying to learn how save it is to update from the OSM server continusly. If ways vanish without warning and without immediate proper replacement that would be a great problem for us. Is thi sto be expected or should I further look for an error in our software (OSM scanner, import)? Thanks a lot, Andreas ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] OSM Data Update
Hi all, I just had an issue with our map server. The maps between Lübeck and Hamburg (Germany) were fine for a couple of weeks. I ran a script over night that pulled some updates from the OSM server, deleted superfluous ways, created new ways in our private database based on the downloaded OSM and regenerated the tiles for this area. The result was that 20km of A1 between Lübeck and Hamburg vanished. I am wondering whether this is due to some kind of bug in our software or due to someone replacing all these ways with new ones!? Anyone worked on this during the last 24h? I just would like to find the bug, if if there is one in our software. But if this is to be expectd due to heavy OSM data changes ... Thanks a lot, Andreas ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Hourly/Daily Updates
Hi all, >> I'll leave the daily changesets running for now because they are >> running with a much longer delay and shouldn't miss data. I need to incorporate changes into our map server and am currentlöy building on http://planet.openstreetmap.org/daily and plan down make a wget on wget http://planet.openstreetmap.org/daily/20100325-20100326.osc.gz once a day and then import that changes into our database. Is this save? Will http://planet.openstreetmap.org/daily be maintained in the future. Hourly files would be even better for us. Is this maintained, if so where. It seems to be disabled right now!? Thanks a lot, Andreas ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Removing Minutely and Hourly Changesets
Hi Peter, >>> The heavies task is 1. Maybe we could set up a database for this >>> task.. >>> Peter >> >> I rather liked Brett's earlier suggestion to have this solved on the >> diff >> server itself. >> >> So you could "wget http://diffserver/state/minute/20100301_00";, >> and >> the web service would find the right state file and return it to you. >> > I'm currently setting up such a service, may be already working > tonight. I just wanted to give this a try. Is this already working? What do I use for diffserver? Thanks, Andreas ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Removing Minutely and Hourly Changesets
Hi all,
> I just noticed, that the minute diffs are not here anymore.
> My tool http://datenkueche.com/osmlive did use this files.
>
> Is there a possibility to find a file if I have only the time?
>
> For the old system I used:
>
> $delta=(int) $_GET['delta'];
> $time1=time()-7200-360 -$delta;
> $time2=time()-7200-300 -$delta;
> $time3=time()-7200-420 -$delta;
> $timestring1=date("YmdHi",$time1);
> $timestring2=date("YmdHi",$time2);
> $timestring3=date("YmdHi",$time3);
>
>
> $filename= $timestring1."-".$timestring2.".osc.gz";
> --
>
> Any idea how to do that with the new system?
I would be interested inthat as well, ideally a file that is generated
every hour with a fixed named scheme we can easily track for. Can this
be reactivated? Why has it been disabled after all?
Thanks,
Andreas
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Inserting OSM data
Hi Peter, >> Thus the data we are producing is already of high >> value (could directly be inserted into the public OSM database). > Could you please provide an extract of this data so we can take a look > at the quality ourself? I can and will in a couple of days. The code that is generating the OSM ways in our database and the OSM export filter is just being implemented! Regards, Andreas ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Inserting OSM data
Hi Nic, > One solution is to add your own tag to the OSM files you generate e.g. > smartsoft_id=nnn. And publish the files for review somewhere. Then > scan the minutely updates at planet.openstreetmap.de for your id > numbers and when they appear you can delete them from your DB. Thanks, that sounds like a plan. Downloading and unzipping the files automatically (cronjob, expect script) will require some tinkering, but this is certainly doable with medium efforts. The OSM are supposed to be put on our webserver!? How do I announce a new OSM file to whom? > You can also upload data directly to OSM and get the IDs. But if you > don't allow us to at least see before hand what you are up to, you run > the risk of making us angry. Hmghh! I will check out both approaches! Thanks, Andreas ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Inserting OSM data
Hi John, > I'm going to go back to some of the primative concepts of OSM here, > just to make sure that everyone is on the same page... > > OSM has 2 completly destict concepts, tracks and ways: (I am > simplifying here, but you get the idea) > > Tracks are a collection of gps points, as they are recorded, generally > 1 per second, but there is a huge deal of variety here. it's slightly > more complicated than that, some people filter them, some people split > diferent journeys or date periods into separate tracks, but it's > basicaly that simple. > > Ways are (in this context) a road -- with just enough points along it > to mark the curvature, lots of metadata, and it's always just one > road, it will never span a bunch of roads describing a route. Totally aware of thi sdistinction. We are actually generating ways from our tracks before trying to send them to the public OSM server > The trouble here is that there is no link between tracks and ways -- > asside from the fact that they are close to each other. This is for > very important technical reasons. in general once a road has been > traversed several times, and it is clear that the GPS data is good, > somone will trace over the (say) 200 gps points that make up the road, > by adding a way containing (say) 4 nodes. From that point on, there is > no connection. > > It is very destructive simply convert traces directly into ways, > because it takes up huge amounts of valuable space in the database, > and leads to a lack of user filtering of bad data. Simply driving past > a large metalic building can thorugh GPS off by a significant distance > etc. 99.9% of the tracks/ways we are producing are in or close to forrests. We have already written code that reduces tracks with 80 records down to a track of only 20 records. And I as a database engineer are totally aware of the storage issue. It really looks that we won't even use the reduced track for creating a way but ask our users to do the click, click, click,... ting to manually enter a way with as few nodes as possible with the track being shown inthe background on the map he is clicking on. So we can assume high quality ways mit as few nodes as possible on our site. It's just that I need to keep our users within our ERP system. I cannot ask tem to log into OSM and use JOSM to edit tracks. That won't fly! :-( > I have personally looked into ways of doing exactly what (I think) you > are trying to do, and found that computers just arn't clever enough to > match the data up, and produce good ways from traces. > > That's not to say that we either don't want, or can't use your data. > what I would sugest is the following: > > You upload data you have as traces, I believe this can be done fairly > simply using the API (i've done it, using CURL, and got it to work) > this returns, I blieve an id that should be useful to you. > > You could add the metadata that you have about road staes and single > nodes to the main map data, also farily simple thoguht the API. > Still checking out the API described on http://wiki.openstreetmap.org/wiki/API_v0.6 Thanks, Andreas ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Inserting OSM data
Hi, >> we are collecting huge amounts of GPS-tracks with a fleet of vehicles >> with on-board computers. [...] We have to maintain our own >> database of tracks, add properties like a so a called pickupCondition, >> which determines how good a track is (e.g. paved, muddy, never use it >> when it's raining,...). > > Do you store all the tracks, or do you conflate them into road axis? > > I mean: if a truck has gone through a road twelve times, do you store > twelve lines or just one? Is there a way to get just one line per road? Yes! The user importing the track data in our system is responsible for selecting only those records that make up a way. If this is not convenient he can leave the track in the background and not at all use the data directly but create the way manually in a click, click, click,... manner. We can also provide functonlity in out app to first specifiy an existing (start) node and then do the click, click, click ... thing so that the newly created way has a defined binding to the existing street data. Thus the data we are producing is already of high value (could directly be inserted into the public OSM database). >> • our users cannot use the official OSM editor for several reasons (to >> complicated, no direct access to our tracks,...) > > Depending on your Java expertise, you could program some JOSM plugin. > Just an idea, though. None Java expertise. Unfortunately! We are doing everything in Objective-C! :-) > >> For that we need some kind of TCP-based XML interface so that we could >> send a track in some XML format to a TCP socket and get back the IDs >> assigned to objects by the public OSM database server. Only that would >> allow us the assign the returned IDs to our private database objects >> and thus avoid data duplication. Is there such an API (TCP-socket >> based)? > > Looky here: http://wiki.openstreetmap.org/wiki/API_v0.6 > > Whenever you (successfully) upload a new way, you get the way ID in > return. No problem there. Wow, I believe that's whatI was looking for!! Scanning this page ... Thanks a lot so far, Andreas ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Inserting OSM data
Hi Tom, >> However, we want to pass our track data to the OSM project and are >> looking for a neat way to do that. We can convert our data to any >> format (some kind of CSV or XML preferred). We just need a TCP >> interface to post them to or a destination email address where to send >> the files to. Simply uploading the tracks to some server is no good >> solution. When a user creates a track in our database it gets a >> primary >> key but of course can't be assigned an official OSM ID yet. So the >> upload process would need to be as follows: > > Can you clarify whether you're talking about uploading this as GPS > trace data (what we would normally recommend for raw GPS data) or as > actual OSM map data? OSM!! The data is already rich enough to contain street names, way kind and some other attributes for forrest tracks (e.g. never use when it's raining). >> Iterate through al newly created tracks >>- send a track with nodes to the OSM server and receive IDs for the >> way and the nodes >>- assign these IDs to our private database objects and thus marking >> them sent and no longer new >> >> For that we need some kind of TCP-based XML interface so that we could >> send a track in some XML format to a TCP socket and get back the IDs >> assigned to objects by the public OSM database server. Only that would >> allow us the assign the returned IDs to our private database objects >> and thus avoid data duplication. Is there such an API (TCP-socket >> based)? > > There are APIs for both, but not necessarily exactly what you want (in > particular for GPS trace data). The ideal solution for use would be to send a WAY segment in OSM format to some service and receive a response with the IDs for the created way and the created nodes! But I understand this is not supported due to the obvious merging problem (avoid duplicates)!? Still what is the best option for us? Regards, Andreas ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Inserting OSM data
Hi John, >> For that we need some kind of TCP-based XML interface so that we could >> send a track in some XML format to a TCP socket and get back the IDs >> assigned to objects by the public OSM database server. Only that would >> allow us the assign the returned IDs to our private database objects >> and thus avoid data duplication. Is there such an API (TCP-socket >> based)? > > It's not that simple, because you may need to deal with merging, > alternatively you could produce .osm files similar to what JOSM > produces and then people can load those files into JOSM and merge or > add data to OSM. Creating OSM files would be easy for us. But where would we sent them? Simply put them on our webserver and make an announcement on the osm list "hey, we have some osm data to be merged/imported! Have fun!"!? The other problem is how do we get to know that one of our ways got imported into the public OSM database and thus assigned an ID so that we can get rid of that way in our private database? I have just logged into www.openstreetmap.org and found the GPS traces pane and the page on which one can upload gps traces. However, where can I upload OSM files to with ways having street names, way kind, ...? Thanks, Andreas ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Inserting OSM data
Hi all, we are collecting huge amounts of GPS-tracks with a fleet of vehicles with on-board computers. These tracks are used internally for a bunch of purposes (vehicle monitoring, etc ). We have to maintain our own database of tracks, add properties like a so a called pickupCondition, which determines how good a track is (e.g. paved, muddy, never use it when it's raining,...). We would like to insert this collected data into the OSM database and am looking for the best way to do that. We have the data in our ERP system. Users are supposed to review the collected tracks, add properties (pickupCondition). What we plan to do is to use the public OSM data, add the self-collected tracks and use the resulting dataset to generate map, calculate routes, etc. We need to do it that way, because • our users cannot use the official OSM editor for several reasons (to complicated, no direct access to our tracks,...) • the "grab OSM data from public server and compile the data for our mapserver, routeserver,..."-cycle is too long. It would take 1-2 weeks until entered data would show up in our ERP. However, we want to pass our track data to the OSM project and are looking for a neat way to do that. We can convert our data to any format (some kind of CSV or XML preferred). We just need a TCP interface to post them to or a destination email address where to send the files to. Simply uploading the tracks to some server is no good solution. When a user creates a track in our database it gets a primary key but of course can't be assigned an official OSM ID yet. So the upload process would need to be as follows: Iterate through al newly created tracks - send a track with nodes to the OSM server and receive IDs for the way and the nodes - assign these IDs to our private database objects and thus marking them sent and no longer new For that we need some kind of TCP-based XML interface so that we could send a track in some XML format to a TCP socket and get back the IDs assigned to objects by the public OSM database server. Only that would allow us the assign the returned IDs to our private database objects and thus avoid data duplication. Is there such an API (TCP-socket based)? What's our best option? Thanks a lot, Andreas ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
