Re: [Talk-us] We have less than a week left to remap!
The 27th? Damn. I thought it was going to go read-only on the 1st... At least that's what I've been telling people who I've been contacting to see if they will accept the new CT (which is still being somewhat successful geting at least 2-3 people a day to accept). Ugh. - James Date: Wed, 21 Mar 2012 21:59:07 -0500 From: toby.mur...@gmail.com To: talk-us@openstreetmap.org Subject: [Talk-us] We have less than a week left to remap! I don't think this has been reposted to talk-us yet. According to the latest license change rebuild plan, the database will enter read-only mode on March 27th for the license rebuild: http://www.osmfoundation.org/wiki/License/Rebuild_Plan So that gives us a matter of days to do any more remapping. It seems like the US hasn't seen a concerted effort to remap dirty data like some other countries have. So here is my attempt to get that started. I'm not sure if some people still don't know about the license change of if people think someone else will take care of the important things... But *everything* is on the chopping block! Some highlights: 1) State/county borders. I've actually cleaned up a good chunk of these and hope to finish in the next few days. I may have to reimport some of the remaining ways. Anyone have thoughts on the best source for this information? 2) The interstate system. I've taken care of about 4 or 5 states here in the midwest but there is a large swath from Illinois down to South Carolina that will become unroutable on April 1st. This will set OSM back immensely in the eyes of anyone actually trying to use our data. 3) Specific cities: Los Angeles, Ausin, Seattle. Most major cities have some unclean blobs in them but these are particularly nasty. User blars has not responded to repeated requests for contact and I have reason to believe he is intentionally ignoring anything OSM related. His contributions are beyond huge in the LA area and really along a lot of the west coast. Seattle is mostly affected by user Sunny who has proven to be unreachable despite my best cyberstalking attempts. Austin, TX is also largely affected by one prolific user but it has actually gotten some attention so it isn't entirely doomed. 4) Shorelines. These were imported from a public domain source by an anonymous user. I'm tempted to largely mark this with odbl=clean and call it a day. Thoughts? In light of this, I would strongly encourage everyone reading this list to suspend your current mapping efforts and concentrate entirely on license remapping for the next few days. It's not the most fun work but it needs to get done. Tools to use: JOSM license change plugin. Select everything in an area and hit the License Check button. Start deleting red things and replacing them from clean sources: http://wiki.openstreetmap.org/wiki/JOSM/Plugins/LicenseChange Potlatch2 I think now has the license change layer enabled by default. If not, go into the options and make sure Show license status is checked. Again... remove anything with a red halo and replace. Badmap. Dirty interstates are clearly visible: http://cleanmap.poole.ch/?zoom=5lat=41.12823lon=-99.48465layers=00B0 OSM Inspector. Good for drilling down and finding individual objects to remap: http://tools.geofabrik.de/osmi/?view=wtfelon=-100.24219lat=40.17384zoom=4 TIGER 2011 road name tiles. Used as a background imagery layer while editing. Good to pull street names while remapping residential areas. Just be sure to expand the abbreviations while you're at it. http://wiki.openstreetmap.org/wiki/TIGER_2011 I have also written two blog posts on my own remapping efforts. The first focuses on interstates where there is existing clean data to work with. The biggest trick here is making sure you don't break the route relations. But honestly I wouldn't worry about it too much. As long as the ways are intact, routing will continue to work. We can come back later and clean up the route relations after the license change. http://ksmapper.blogspot.com/2012/01/license-change-mapping.html The second post is quite a bit more technical and I don't expect many others to follow it really. It was useful in that area of LA but it involves some traditional GIS tools, data analysis and a little python coding. For this type of work, most people would be better off using the TIGER 2011 road name tiles listed above. http://ksmapper.blogspot.com/2012/03/remapping-using-tiger-2011.html Honestly, it is kind of late to be sending out this email... I probably should have done it a couple of weeks ago but hopefully we can still make a meaningful dent in the dirty data. At least this whole process is almost over... Toby ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] We have less than a week left to remap!
On Thu, Mar 22, 2012 at 1:00 AM, James Mast rickmastfa...@hotmail.com wrote: The 27th? Damn. I thought it was going to go read-only on the 1st... At least that's what I've been telling people who I've been contacting to see if they will accept the new CT (which is still being somewhat successful geting at least 2-3 people a day to accept). Ugh. Yeah, that was my first reaction too. And several others have expressed concerns about the tight schedule for a couple of different reasons. So there is still a chance that it will be pushed back a little. But either way... not much time left. Toby ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] We have less than a week left to remap!
I don't think this has been reposted to talk-us yet. According to the latest license change rebuild plan, the database will enter read-only mode on March 27th for the license rebuild: http://www.osmfoundation.org/wiki/License/Rebuild_Plan It has been stated that it is a GOAL to pull the plug on April 1st. The Sword of Damocles wielded by the Powers That Be in OSM are under no hard-and-fast obligation to do so, they are just trying to reach a stated goal. If (as it is looking more and more to be true) that significant areas will disappear from the map, making routing and other important uses of these data seriously impaired, it is incumbent upon the Powers That Be to declare failure at reaching the stated goals. Yes, this is disappointing, but it is true. The only meaningful conclusion that can be reached is to give the OSM community more time and push back the April 1st (or March 27th) deadline. (One month? Three months?) Is this an eleventh hour request to do so? Well, yes! But every word of the above is true. The OSM community simply must have more time to complete these tasks. It is as simple as that. Don't cripple an extremely useful and becoming-more-mainstream and rather popular tool with arbitrary and capricious dates on a calendar. Plan and assess, yes, that has already happened, but now is the time to RE-assess, and reach the solemn conclusion that flipping the switch on significant data in barely a week is ill-advised at best, and will be a crippling body blow to the project at worst. Perhaps one from which the project will not fully recover. Nobody asked me to say it, but I'm asking for three more months. We can complete needed tasks by then. SteveA California ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] [Imports] Uploads to City of Salisbury, MD
Importing data correctly is hard, especially the first time. I'm glad you've come to the list. I'd be curious to know if you didn't notice the warnings on the wiki to contact the community first; I've gone ahead and added the warning banner to the ogr2osm, shp2osm, shp-to-osm, and shape2osm pages in case people find those before the other import pages. On Wed, Mar 21, 2012 at 10:50 PM, Paul Norman penor...@mac.com wrote: If there are duplicated ways and nodes, perhaps reverting is the best option? Unfortunately I'd agree that reverting these changesets will be the easiest and best course of action. Trust me that you'll spend a long time cleaning up dupe nodes and ways. Some of the buildings have up to four dupe nodes and 3 dupe ways, all from different changesets. I've done some reversion before with the JOSM plugin, so if you'd like I can help do this. I'd like to help you do another successful import, so please consider sharing the original GIS files and your ogr2osm translation files. I very much appreciate your interest in OSM and your effort, so please do not let this dissuade you from continuing! -Josh ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] [Imports] Uploads to City of Salisbury, MD
On Thu, Mar 22, 2012 at 6:40 AM, Josh Doe j...@joshdoe.com wrote: On Wed, Mar 21, 2012 at 10:50 PM, Paul Norman penor...@mac.com wrote: If there are duplicated ways and nodes, perhaps reverting is the best option? Unfortunately I'd agree that reverting these changesets will be the easiest and best course of action. Trust me that you'll spend a long time cleaning up dupe nodes and ways. Some of the buildings have up to four dupe nodes and 3 dupe ways, all from different changesets. This is what I can tell from the first few building changesets (not an extensive investigation): 10882159: 493 dangling nodes (REVERT) 10884039: 3517 dangling nodes (REVERT) 10885267: 1 building node (OK) 10891857: 5 dangling nodes (REVERT) 10891870: 8000 dangling nodes (REVERT) 10893364: building nodes and ways (OK?) -Josh ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] [Imports] Uploads to City of Salisbury, MD
I briefly downloaded all sby:bldgtype-tagged ways and relation of Maryland through the overpass-api. Then removed the ones having only a sby:bldgtype tag, run the validator and deleted the duplicated nodes and ways. This would result in a changeset to remove the roughly 71'000 duplicates nodes and ways. If the area was edited since the import and reverting gets tricky, this might be the option to go, at least the result looks ok at the first glance. Please also note that the conversion step seems to add a building=yes tag on on inner ring of building polygons () which is certainly bad tagging, despite the correct rendering (52 occurrences, so could be fixed manually). M On 22.03.2012, at 12:02, Josh Doe wrote: On Thu, Mar 22, 2012 at 6:40 AM, Josh Doe j...@joshdoe.com wrote: On Wed, Mar 21, 2012 at 10:50 PM, Paul Norman penor...@mac.com wrote: If there are duplicated ways and nodes, perhaps reverting is the best option? Unfortunately I'd agree that reverting these changesets will be the easiest and best course of action. Trust me that you'll spend a long time cleaning up dupe nodes and ways. Some of the buildings have up to four dupe nodes and 3 dupe ways, all from different changesets. This is what I can tell from the first few building changesets (not an extensive investigation): 10882159: 493 dangling nodes (REVERT) 10884039: 3517 dangling nodes (REVERT) 10885267: 1 building node (OK) 10891857: 5 dangling nodes (REVERT) 10891870: 8000 dangling nodes (REVERT) 10893364: building nodes and ways (OK?) -Josh ___ Imports mailing list impo...@openstreetmap.org http://lists.openstreetmap.org/listinfo/imports ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] [Imports] Uploads to City of Salisbury, MD
On Thu, Mar 22, 2012 at 8:04 AM, Marc Zoss marcz...@gmail.com wrote: I briefly downloaded all sby:bldgtype-tagged ways and relation of Maryland through the overpass-api. Then removed the ones having only a sby:bldgtype tag, run the validator and deleted the duplicated nodes and ways. This would result in a changeset to remove the roughly 71'000 duplicates nodes and ways. If the area was edited since the import and reverting gets tricky, this might be the option to go, at least the result looks ok at the first glance. Please also note that the conversion step seems to add a building=yes tag on on inner ring of building polygons () which is certainly bad tagging, despite the correct rendering (52 occurrences, so could be fixed manually). Thanks for doing that, as that was the next step I was going to try. I posted some regarding the changesets here: http://wiki.openstreetmap.org/wiki/User_talk:Nick_SPW#Salisbury.2C_Maryland_import I think perhaps we should revert a subset of the changesets, such as the dangling nodes, and then use your method to handle the rest. -Josh ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] [Imports] Uploads to City of Salisbury, MD
Josh and Marc, Thank you! I apologize that I'm unable to speak the OSM language as well as everyone, I'm working on it :) I posted on the Salisbury, Maryland Import page that Josh created to give more detail about my uploads. I didn't really think that I created so many duplicates, because I did a lot of things in JOSM before I actually chose to upload. One thing I know for sure is that I didn't I upload until I was actually able to - I was getting a proxy error and the uploads were timing out when I attempted to upload the entire batch. I assumed that these attempts were unsuccessful, which I might be wrong about and might have resulted in duplication. I assumed that my successful attempts started, maybe @ 10901673, when I realized I needed to break the original shapefile up tabularly into percentiles and upload 10 segments of the building footprint dataset, one after the other. These were all definitely successful, and were only done once per percentile. Josh, where are you finding the list of changesets in the format you posted? I can only figure out how to list them in my editor profile with my comments. If you believe that the method you mention that removes the 71,000 nodes is the best approach, please feel free to do so. I will also gladly manually fix the inner ring tagging issue as the data gets fixed. Please let me know what I can do to help. I am also willing to share the .osm files and/or shapefiles if that will help. Thanks. - Nick -Original Message- From: joshthephysic...@gmail.com [mailto:joshthephysic...@gmail.com] On Behalf Of Josh Doe Sent: Thursday, March 22, 2012 8:51 AM To: Marc Zoss Cc: impo...@openstreetmap.org; talk-us@openstreetmap.org; Nick Chamberlain Subject: Re: [Imports] [Talk-us] Uploads to City of Salisbury, MD On Thu, Mar 22, 2012 at 8:04 AM, Marc Zoss marcz...@gmail.com wrote: I briefly downloaded all sby:bldgtype-tagged ways and relation of Maryland through the overpass-api. Then removed the ones having only a sby:bldgtype tag, run the validator and deleted the duplicated nodes and ways. This would result in a changeset to remove the roughly 71'000 duplicates nodes and ways. If the area was edited since the import and reverting gets tricky, this might be the option to go, at least the result looks ok at the first glance. Please also note that the conversion step seems to add a building=yes tag on on inner ring of building polygons () which is certainly bad tagging, despite the correct rendering (52 occurrences, so could be fixed manually). Thanks for doing that, as that was the next step I was going to try. I posted some regarding the changesets here: http://wiki.openstreetmap.org/wiki/User_talk:Nick_SPW#Salisbury.2C_Maryl and_import I think perhaps we should revert a subset of the changesets, such as the dangling nodes, and then use your method to handle the rest. -Josh ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] [Imports] Uploads to City of Salisbury, MD
Nick and Josh thanks for the clarification on your upload strategy. With previous large uploads I have experience the same behaviour resulting in massive dupes. So I guess it is not a conversion issue. If you want me to commit the remove duplicates changeset, I can do so. But you will have to go through the data subsequently and check if the issues are resolved and no new ones emerged. M On 22.03.2012, at 14:12, Nick Chamberlain wrote: Josh and Marc, Thank you! I apologize that I'm unable to speak the OSM language as well as everyone, I'm working on it :) I posted on the Salisbury, Maryland Import page that Josh created to give more detail about my uploads. I didn't really think that I created so many duplicates, because I did a lot of things in JOSM before I actually chose to upload. One thing I know for sure is that I didn't I upload until I was actually able to - I was getting a proxy error and the uploads were timing out when I attempted to upload the entire batch. I assumed that these attempts were unsuccessful, which I might be wrong about and might have resulted in duplication. I assumed that my successful attempts started, maybe @ 10901673, when I realized I needed to break the original shapefile up tabularly into percentiles and upload 10 segments of the building footprint dataset, one after the other. These were all definitely successful, and were only done once per percentile. Josh, where are you finding the list of changesets in the format you posted? I can only figure out how to list them in my editor profile with my comments. If you believe that the method you mention that removes the 71,000 nodes is the best approach, please feel free to do so. I will also gladly manually fix the inner ring tagging issue as the data gets fixed. Please let me know what I can do to help. I am also willing to share the .osm files and/or shapefiles if that will help. Thanks. - Nick -Original Message- From: joshthephysic...@gmail.com [mailto:joshthephysic...@gmail.com] On Behalf Of Josh Doe Sent: Thursday, March 22, 2012 8:51 AM To: Marc Zoss Cc: impo...@openstreetmap.org; talk-us@openstreetmap.org; Nick Chamberlain Subject: Re: [Imports] [Talk-us] Uploads to City of Salisbury, MD On Thu, Mar 22, 2012 at 8:04 AM, Marc Zoss marcz...@gmail.com wrote: I briefly downloaded all sby:bldgtype-tagged ways and relation of Maryland through the overpass-api. Then removed the ones having only a sby:bldgtype tag, run the validator and deleted the duplicated nodes and ways. This would result in a changeset to remove the roughly 71'000 duplicates nodes and ways. If the area was edited since the import and reverting gets tricky, this might be the option to go, at least the result looks ok at the first glance. Please also note that the conversion step seems to add a building=yes tag on on inner ring of building polygons () which is certainly bad tagging, despite the correct rendering (52 occurrences, so could be fixed manually). Thanks for doing that, as that was the next step I was going to try. I posted some regarding the changesets here: http://wiki.openstreetmap.org/wiki/User_talk:Nick_SPW#Salisbury.2C_Maryl and_import I think perhaps we should revert a subset of the changesets, such as the dangling nodes, and then use your method to handle the rest. -Josh ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] [Imports] Uploads to City of Salisbury, MD
On 3/22/2012 9:12 AM, Nick Chamberlain wrote: I was getting a proxy error and the uploads were timing out when I attempted to upload the entire batch. This is a common problem with uploading large changesets with JOSM, where upload failures result in a partial upload. We're hesitant to put automatic failed changeset upload recovery and resolution into JOSM for fear that someone will create a mass conversion of shapefiles and upload it all without even trying to tune the data to OSM standards.g ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
[Talk-us] Duplicates in data uploads (using JOSM) -- was: Re: [Imports] Uploads to City of Salisbury, MD
With previous large uploads I have experience the same behaviour resulting in massive dupes. So I guess it is not a conversion issue. I don't have experience with conversions nor (mass) imports -- but I _have_ had massive dupes problems a number of times when uploading larger amounts of data with JOSM over a bad connection. The problem has always been related to the combination of large uploads and bad connections where (if I understand right) the JOSM data upload connection gets a hick-up at some point and isn't able to finish the job -- and doesn't leave a note for itself where it was left of. Then, because of reasons I don't _exactly_ understand there's duplication of data on the next upload(s (attempts)). My vague understanding is that this is due to at least the fact that JOSM uploads nodes first and only after that the information about ways (i.e. which nodes belong to which ways). And then when it hasn't gotten or confirmation for succesful uploads (or it hasn't recorded that to it's data file(?)) it considers the uploaded nodes to still be new at next upload(s (attempts)). I feel that duplication sometimes happens also to partial uploads where the ways have uploaded, too, resulting in duplicate uploaded ways but I haven't documented this well enough to say this solidly. If you have a bad connection / feel that this may be your problem it is a good idea to tweak the JOSM Advanced upload settings (Upload Advanced tab: Upload data in chunks of objects. Chunk size: , where is your number of objects per chunk. I use 200 in with my Haitian connection. Cheers, -Jaakko http://osm.org/user/jaakkoh -- jaa...@helleranta.com * Skype: jhelleranta * Mobile: +509-37-269154 * http://go.hel.cc/MyProfile On Thu, Mar 22, 2012 at 8:28 AM, Marc Zoss marcz...@gmail.com wrote: Nick and Josh thanks for the clarification on your upload strategy. With previous large uploads I have experience the same behaviour resulting in massive dupes. So I guess it is not a conversion issue. If you want me to commit the remove duplicates changeset, I can do so. But you will have to go through the data subsequently and check if the issues are resolved and no new ones emerged. M On 22.03.2012, at 14:12, Nick Chamberlain wrote: Josh and Marc, Thank you! I apologize that I'm unable to speak the OSM language as well as everyone, I'm working on it :) I posted on the Salisbury, Maryland Import page that Josh created to give more detail about my uploads. I didn't really think that I created so many duplicates, because I did a lot of things in JOSM before I actually chose to upload. One thing I know for sure is that I didn't I upload until I was actually able to - I was getting a proxy error and the uploads were timing out when I attempted to upload the entire batch. I assumed that these attempts were unsuccessful, which I might be wrong about and might have resulted in duplication. I assumed that my successful attempts started, maybe @ 10901673, when I realized I needed to break the original shapefile up tabularly into percentiles and upload 10 segments of the building footprint dataset, one after the other. These were all definitely successful, and were only done once per percentile. Josh, where are you finding the list of changesets in the format you posted? I can only figure out how to list them in my editor profile with my comments. If you believe that the method you mention that removes the 71,000 nodes is the best approach, please feel free to do so. I will also gladly manually fix the inner ring tagging issue as the data gets fixed. Please let me know what I can do to help. I am also willing to share the .osm files and/or shapefiles if that will help. Thanks. - Nick -Original Message- From: joshthephysic...@gmail.com [mailto:joshthephysic...@gmail.com] On Behalf Of Josh Doe Sent: Thursday, March 22, 2012 8:51 AM To: Marc Zoss Cc: impo...@openstreetmap.org; talk-us@openstreetmap.org; Nick Chamberlain Subject: Re: [Imports] [Talk-us] Uploads to City of Salisbury, MD On Thu, Mar 22, 2012 at 8:04 AM, Marc Zoss marcz...@gmail.com wrote: I briefly downloaded all sby:bldgtype-tagged ways and relation of Maryland through the overpass-api. Then removed the ones having only a sby:bldgtype tag, run the validator and deleted the duplicated nodes and ways. This would result in a changeset to remove the roughly 71'000 duplicates nodes and ways. If the area was edited since the import and reverting gets tricky, this might be the option to go, at least the result looks ok at the first glance. Please also note that the conversion step seems to add a building=yes tag on on inner ring of building polygons () which is certainly bad tagging, despite the correct rendering (52 occurrences, so could be fixed manually). Thanks for doing that, as that was the next step I was going to try. I
Re: [Talk-us] [Imports] Uploads to City of Salisbury, MD
Mark, if you could commit the remove duplicates changeset, that'd be great. I will do my best to check if the issues are resolved, and will gladly accept any guidance on the best ways to do so. Thanks. - Nick -Original Message- From: Marc Zoss [mailto:marcz...@gmail.com] Sent: Thursday, March 22, 2012 9:28 AM To: Nick Chamberlain; Josh Doe Cc: impo...@openstreetmap.org; talk-us@openstreetmap.org Subject: Re: [Imports] [Talk-us] Uploads to City of Salisbury, MD Nick and Josh thanks for the clarification on your upload strategy. With previous large uploads I have experience the same behaviour resulting in massive dupes. So I guess it is not a conversion issue. If you want me to commit the remove duplicates changeset, I can do so. But you will have to go through the data subsequently and check if the issues are resolved and no new ones emerged. M On 22.03.2012, at 14:12, Nick Chamberlain wrote: Josh and Marc, Thank you! I apologize that I'm unable to speak the OSM language as well as everyone, I'm working on it :) I posted on the Salisbury, Maryland Import page that Josh created to give more detail about my uploads. I didn't really think that I created so many duplicates, because I did a lot of things in JOSM before I actually chose to upload. One thing I know for sure is that I didn't I upload until I was actually able to - I was getting a proxy error and the uploads were timing out when I attempted to upload the entire batch. I assumed that these attempts were unsuccessful, which I might be wrong about and might have resulted in duplication. I assumed that my successful attempts started, maybe @ 10901673, when I realized I needed to break the original shapefile up tabularly into percentiles and upload 10 segments of the building footprint dataset, one after the other. These were all definitely successful, and were only done once per percentile. Josh, where are you finding the list of changesets in the format you posted? I can only figure out how to list them in my editor profile with my comments. If you believe that the method you mention that removes the 71,000 nodes is the best approach, please feel free to do so. I will also gladly manually fix the inner ring tagging issue as the data gets fixed. Please let me know what I can do to help. I am also willing to share the .osm files and/or shapefiles if that will help. Thanks. - Nick -Original Message- From: joshthephysic...@gmail.com [mailto:joshthephysic...@gmail.com] On Behalf Of Josh Doe Sent: Thursday, March 22, 2012 8:51 AM To: Marc Zoss Cc: impo...@openstreetmap.org; talk-us@openstreetmap.org; Nick Chamberlain Subject: Re: [Imports] [Talk-us] Uploads to City of Salisbury, MD On Thu, Mar 22, 2012 at 8:04 AM, Marc Zoss marcz...@gmail.com wrote: I briefly downloaded all sby:bldgtype-tagged ways and relation of Maryland through the overpass-api. Then removed the ones having only a sby:bldgtype tag, run the validator and deleted the duplicated nodes and ways. This would result in a changeset to remove the roughly 71'000 duplicates nodes and ways. If the area was edited since the import and reverting gets tricky, this might be the option to go, at least the result looks ok at the first glance. Please also note that the conversion step seems to add a building=yes tag on on inner ring of building polygons () which is certainly bad tagging, despite the correct rendering (52 occurrences, so could be fixed manually). Thanks for doing that, as that was the next step I was going to try. I posted some regarding the changesets here: http://wiki.openstreetmap.org/wiki/User_talk:Nick_SPW#Salisbury.2C_Mar yl and_import I think perhaps we should revert a subset of the changesets, such as the dangling nodes, and then use your method to handle the rest. -Josh ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] Duplicates in data uploads (using JOSM) -- was: Re: [Imports] Uploads to City of Salisbury, MD
Jaakko, Thank you for the explanation. I will tweak my chunk sizes further next time. I did so before, but they were still fairly large and took a few hours per upload. Reducing them might take longer, but if that fixes duplication I will do that. Thanks. - Nick From: Jaakko Helleranta.com [mailto:jaa...@helleranta.com] Sent: Thursday, March 22, 2012 10:47 AM To: Marc Zoss Cc: Nick Chamberlain; Josh Doe; impo...@openstreetmap.org; talk-us@openstreetmap.org Subject: Duplicates in data uploads (using JOSM) -- was: Re: [Imports] [Talk-us] Uploads to City of Salisbury, MD With previous large uploads I have experience the same behaviour resulting in massive dupes. So I guess it is not a conversion issue. I don't have experience with conversions nor (mass) imports -- but I _have_ had massive dupes problems a number of times when uploading larger amounts of data with JOSM over a bad connection. The problem has always been related to the combination of large uploads and bad connections where (if I understand right) the JOSM data upload connection gets a hick-up at some point and isn't able to finish the job -- and doesn't leave a note for itself where it was left of. Then, because of reasons I don't _exactly_ understand there's duplication of data on the next upload(s (attempts)). My vague understanding is that this is due to at least the fact that JOSM uploads nodes first and only after that the information about ways (i.e. which nodes belong to which ways). And then when it hasn't gotten or confirmation for succesful uploads (or it hasn't recorded that to it's data file(?)) it considers the uploaded nodes to still be new at next upload(s (attempts)). I feel that duplication sometimes happens also to partial uploads where the ways have uploaded, too, resulting in duplicate uploaded ways but I haven't documented this well enough to say this solidly. If you have a bad connection / feel that this may be your problem it is a good idea to tweak the JOSM Advanced upload settings (Upload Advanced tab: Upload data in chunks of objects. Chunk size: , where is your number of objects per chunk. I use 200 in with my Haitian connection. Cheers, -Jaakko http://osm.org/user/jaakkoh -- jaa...@helleranta.com * Skype: jhelleranta * Mobile: +509-37-269154 * http://go.hel.cc/MyProfile On Thu, Mar 22, 2012 at 8:28 AM, Marc Zoss marcz...@gmail.com wrote: Nick and Josh thanks for the clarification on your upload strategy. With previous large uploads I have experience the same behaviour resulting in massive dupes. So I guess it is not a conversion issue. If you want me to commit the remove duplicates changeset, I can do so. But you will have to go through the data subsequently and check if the issues are resolved and no new ones emerged. M On 22.03.2012, at 14:12, Nick Chamberlain wrote: Josh and Marc, Thank you! I apologize that I'm unable to speak the OSM language as well as everyone, I'm working on it :) I posted on the Salisbury, Maryland Import page that Josh created to give more detail about my uploads. I didn't really think that I created so many duplicates, because I did a lot of things in JOSM before I actually chose to upload. One thing I know for sure is that I didn't I upload until I was actually able to - I was getting a proxy error and the uploads were timing out when I attempted to upload the entire batch. I assumed that these attempts were unsuccessful, which I might be wrong about and might have resulted in duplication. I assumed that my successful attempts started, maybe @ 10901673, when I realized I needed to break the original shapefile up tabularly into percentiles and upload 10 segments of the building footprint dataset, one after the other. These were all definitely successful, and were only done once per percentile. Josh, where are you finding the list of changesets in the format you posted? I can only figure out how to list them in my editor profile with my comments. If you believe that the method you mention that removes the 71,000 nodes is the best approach, please feel free to do so. I will also gladly manually fix the inner ring tagging issue as the data gets fixed. Please let me know what I can do to help. I am also willing to share the .osm files and/or shapefiles if that will help. Thanks. - Nick -Original Message- From: joshthephysic...@gmail.com [mailto:joshthephysic...@gmail.com] On Behalf Of Josh Doe Sent: Thursday, March 22, 2012 8:51 AM To: Marc Zoss Cc: impo...@openstreetmap.org; talk-us@openstreetmap.org; Nick Chamberlain Subject: Re: [Imports] [Talk-us] Uploads to City of Salisbury, MD On Thu, Mar 22, 2012 at 8:04 AM, Marc Zoss marcz...@gmail.com wrote: I briefly downloaded all sby:bldgtype-tagged ways and relation of Maryland through the overpass-api. Then removed the ones having only a sby:bldgtype tag, run the validator and deleted the duplicated nodes and ways.
Re: [Talk-us] [Imports] Uploads to City of Salisbury, MD
On Thu, Mar 22, 2012 at 10:52 AM, Nick Chamberlain nchamberl...@ci.salisbury.md.us wrote: Mark, if you could commit the remove duplicates changeset, that'd be great. I will do my best to check if the issues are resolved, and will gladly accept any guidance on the best ways to do so. Thanks. I'm reverting a few of the changesets as we speak, so if Mark could hold off a few minutes, I'll update you all as I go. Since this only concerns the US, future messages will only be sent on the talk-us@ list. -Josh ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] [Imports] Uploads to City of Salisbury, MD
On Thu, Mar 22, 2012 at 8:49 PM, Josh Doe j...@joshdoe.com wrote: On Thu, Mar 22, 2012 at 10:52 AM, Nick Chamberlain nchamberl...@ci.salisbury.md.us wrote: Mark, if you could commit the remove duplicates changeset, that'd be great. I will do my best to check if the issues are resolved, and will gladly accept any guidance on the best ways to do so. Thanks. I'm reverting a few of the changesets as we speak, so if Mark could hold off a few minutes, I'll update you all as I go. Since this only concerns the US, future messages will only be sent on the talk-us@ list. I've reverted a few of them (see [0]), but when I checked #10901301 (2 nodes), it couldn't revert cleanly, since some nodes are danglers, while others are used for ways in subsequent uploads. I'll stop for now, but maybe Mark can take a look at things now? -Josh [0]: http://wiki.openstreetmap.org/wiki/User_talk:Nick_SPW#Salisbury.2C_Maryland_import ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
