Re: [Talk-GB] Automated Code-Point Open postcode editing (simple cases only)
On 19 July 2019 17:58:58 CEST, Devonshire wrote: > >I know that Robert is sincere in his views but the classic "don't add >data to OSM because it will spoil someone else's enjoyment" always >makes me chuckle. In most parts of the country the idea that the >current cohort of mappers can add accurate address data by hand is pie >in the sky. > >There are certainly issues with adding these postcodes to buildings in >dense town centres but in those areas you can often find postcodes by >other means anyway. I think adding postcodes to residential or rural >areas from these datasets is fine but I personally wouldn't add them >unless I had some on-the-ground knowledge of the area. I've been thinking about it during my summer holidays (much needed step back). I strongly encourage others to add as many postcodes as they can, both manually and from Code-point open/ONS in low density areas. The latter are the only realistic and legal way of bringing OSM to a state where users of e.g. Maps.me or OsmAnd can expect a search for a postcode to usually work rather than usually fail. This is in my opinion the single most important issue individual British OSM users (who can't or don't want to purchase a Code-point license) are facing now. Having all houses tagged with postcodes, house numbers and street names is of course a noble goal but we are talking about 10M addresses that have yet to be surveyed. In contrast, a single searchable postcode is all most users need for navigation and this information is readily available now. I will keep updating my osm files on Github and support other mappers but I will not do more bulk postcode mapping myself. Having added well over 50k unique postcodes I am happy with the state of Cambridgeshire and surrounding areas. But doing that for the rest of the country would take a lot of effort and frankly speaking I need to readjust my priorities. Best regards, ndrw6 ___ Talk-GB mailing list Talk-GB@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk-gb
Re: [Talk-GB] Automated Code-Point Open postcode editing (simple cases only)
On 20/07/2019 10:36, Tony Shield wrote: Hi Starting to get a little confused here - my brother lives in Northern Ireland and has a BT postcode which he has given me. Am I allowed to put into OSM? You can add the postcode to any address in OSM, if you know the full address from personal knowledge. That includes BT postcodes. What you can't do is bulk-import BT postcode centroids from a dataset. Mark ___ Talk-GB mailing list Talk-GB@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk-gb
Re: [Talk-GB] Automated Code-Point Open postcode editing (simple cases only)
Hi Starting to get a little confused here - my brother lives in Northern Ireland and has a BT postcode which he has given me. Am I allowed to put into OSM? TonyS999 On 20/07/2019 08:02, Mark Goodge wrote: On 19/07/2019 22:36, nd...@redhazel.co.uk wrote: On 19/07/2019 20:29, Mark Goodge wrote: ONS postcode products are also OGL, so can be reused in OSM and similar. They're also more useful than Code-Point Open in that they also include lookups to a number of other government codes (such as local authority GSS codes). It also differentiates between "large user" and normal postcodes, and includes an introduction date and, where applicable, a termination date for every valid postcode. That would be very useful indeed but what is the license of these extra features? OGL alone doesn't mean anything if they qualify it with "data may contain third party IP" or similar. It's OGL apart from BT postcodes, which are mentioned separately in the licence. Their website says: "Our postcode products (derived from Code-Point® Open) are subject to the Open Government Licence." and then: "If you also use the Northern Ireland data (postcodes starting with “BT”), you need a separate licence for commercial use direct from Land and Property Services." I understand it as only the part that already exists in Code-Point Open is free, extra information may or may not be free, depending if it comes from ONS own data or other sources. The other information is OGL as that's derived from other published data that's already OGL. For example, GSS codes and their associated names. The licence page is quite clear about that: postcode lookup data is OGL apart from BT postcodes. And those are easy enough to filter out if necessary. Mark ___ Talk-GB mailing list Talk-GB@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk-gb ___ Talk-GB mailing list Talk-GB@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk-gb
Re: [Talk-GB] Automated Code-Point Open postcode editing (simple cases only)
On 19/07/2019 22:36, nd...@redhazel.co.uk wrote: On 19/07/2019 20:29, Mark Goodge wrote: ONS postcode products are also OGL, so can be reused in OSM and similar. They're also more useful than Code-Point Open in that they also include lookups to a number of other government codes (such as local authority GSS codes). It also differentiates between "large user" and normal postcodes, and includes an introduction date and, where applicable, a termination date for every valid postcode. That would be very useful indeed but what is the license of these extra features? OGL alone doesn't mean anything if they qualify it with "data may contain third party IP" or similar. It's OGL apart from BT postcodes, which are mentioned separately in the licence. Their website says: "Our postcode products (derived from Code-Point® Open) are subject to the Open Government Licence." and then: "If you also use the Northern Ireland data (postcodes starting with “BT”), you need a separate licence for commercial use direct from Land and Property Services." I understand it as only the part that already exists in Code-Point Open is free, extra information may or may not be free, depending if it comes from ONS own data or other sources. The other information is OGL as that's derived from other published data that's already OGL. For example, GSS codes and their associated names. The licence page is quite clear about that: postcode lookup data is OGL apart from BT postcodes. And those are easy enough to filter out if necessary. Mark ___ Talk-GB mailing list Talk-GB@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk-gb
Re: [Talk-GB] Automated Code-Point Open postcode editing (simple cases only)
On 19/07/2019 20:29, Mark Goodge wrote: ONS postcode products are also OGL, so can be reused in OSM and similar. They're also more useful than Code-Point Open in that they also include lookups to a number of other government codes (such as local authority GSS codes). It also differentiates between "large user" and normal postcodes, and includes an introduction date and, where applicable, a termination date for every valid postcode. That would be very useful indeed but what is the license of these extra features? OGL alone doesn't mean anything if they qualify it with "data may contain third party IP" or similar. Their website says: "Our postcode products (derived from Code-Point® Open) are subject to the Open Government Licence." and then: "If you also use the Northern Ireland data (postcodes starting with “BT”), you need a separate licence for commercial use direct from Land and Property Services." I understand it as only the part that already exists in Code-Point Open is free, extra information may or may not be free, depending if it comes from ONS own data or other sources. Best regards, ndrw6 ___ Talk-GB mailing list Talk-GB@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk-gb
Re: [Talk-GB] Automated Code-Point Open postcode editing (simple cases only)
On 19/07/2019 15:15, Andrzej wrote: Indeed, Code-Point Open is less than ideal, the issues are almost always caused by lack of differentiation between residential and "large user" postcodes. On the other hand, it is the only legal source of postcodes we have, other than local knowledge, but the latter is realistically limited to a dozen or so postcodes per mapper. Businesses website could also be OK but they are usually copyrighted. Derived databases, like FHRS, are generally not OK, a unless also permitted by Royal Mail. ONS postcode products are also OGL, so can be reused in OSM and similar. They're also more useful than Code-Point Open in that they also include lookups to a number of other government codes (such as local authority GSS codes). It also differentiates between "large user" and normal postcodes, and includes an introduction date and, where applicable, a termination date for every valid postcode. And, unlike Code-Point Open, the ONSPD has a persistent URL that can be used to automate updates. https://hub.arcgis.com/datasets/ons::ons-postcode-directory-latest-centroids More generally, the ONS open data geography datasets are a goldmine. Another useful one is the Index of Place Names; that has obvious utility for OSM as a means of verifying the official spelling of names entered by mappers. Mark ___ Talk-GB mailing list Talk-GB@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk-gb
Re: [Talk-GB] Automated Code-Point Open postcode editing (simple cases only)
On Fri, Jul 19, 2019, at 3:21 PM, Andrzej wrote: > > Thank you for your opinion, Robert. I will suspend adding postcodes from > Code-point Open. > > Do others agree with it or would you rather have more postcodes in database > first and work on accuracy and completeness afterwards? > > Indeed, Code-Point Open is less than ideal, the issues are almost always > caused by lack of differentiation between residential and "large user" > postcodes. On the other hand, it is the only legal source of postcodes we > have, other than local knowledge, but the latter is realistically limited to > a dozen or so postcodes per mapper. Businesses website could also be OK but > they are usually copyrighted. Derived databases, like FHRS, are generally not > OK, a unless also permitted by Royal Mail. > > It's not that I don't care about complete addresses either. But my spare time > is limited, and I feel I can contribute more by adding missing postcodes in a > town vs adding complete addresses in a few streets. Others may have different > priorities. > > I disagree that having data from Code-Point Open outside OSM is sufficient. > Excluding surveyed information, everything in OSM is already publicly > available (or should be). Yet, we all keep using and working on OSM. Besides, > how to extend or combine information without adding it first? > Hi, I would love to see a comparison done between the accuracy of manually added postcodes vs. those added from the OS or ONS datasets. Someone manually added a bunch of postcodes near me and I am pretty certain quite a few of them are wrong but without going around knocking on people's doors they are probably going to stay wrong forever. I usually use the ONS Postcode dataset (https://geoportal.statistics.gov.uk/datasets/ons-postcode-directory-may-2019) rather than CodePoint Open. The datasets do have differences and which is more accurate I have no idea. The ONS centroids are mostly "snapped" to the nearest building within the postcode so are pretty easy to match up. I know that Robert is sincere in his views but the classic "don't add data to OSM because it will spoil someone else's enjoyment" always makes me chuckle. In most parts of the country the idea that the current cohort of mappers can add accurate address data by hand is pie in the sky. There are certainly issues with adding these postcodes to buildings in dense town centres but in those areas you can often find postcodes by other means anyway. I think adding postcodes to residential or rural areas from these datasets is fine but I personally wouldn't add them unless I had some on-the-ground knowledge of the area. Kevin ___ Talk-GB mailing list Talk-GB@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk-gb
Re: [Talk-GB] Automated Code-Point Open postcode editing (simple cases only)
On 19/07/2019 15:15, Andrzej wrote: Do others agree with it or would you rather have more postcodes in database first and work on accuracy and completeness afterwards? Andrez ... while the code-point table does provide a list against which missing post codes can be identified, the key piece of information that is needed is to add a road name to the post code, and that is not something that is easy to establish currently. If we all simply add address data to places we visit the gaps would fill up quite quickly but I'm guilty of not doing that. I've a list of postcode I have been looking up on OASAnd and not finding which I need to actually put in! -- Lester Caine - G8HFL - Contact - https://lsces.uk/wiki/Contact L.S.Caine Electronic Services - https://lsces.uk Model Engineers Digital Workshop - https://medw.uk Rainbow Digital Media - https://rainbowdigitalmedia.uk ___ Talk-GB mailing list Talk-GB@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk-gb
Re: [Talk-GB] Automated Code-Point Open postcode editing (simple cases only)
On 19 July 2019 09:58:52 BST, "Robert Whittaker (OSM lists)" wrote: > >I thus have to object not just to the new proposal but also any >continuation of the previous work to add single postcodes to buildings >under the centroid. Thank you for your opinion, Robert. I will suspend adding postcodes from Code-point Open. Do others agree with it or would you rather have more postcodes in database first and work on accuracy and completeness afterwards? Indeed, Code-Point Open is less than ideal, the issues are almost always caused by lack of differentiation between residential and "large user" postcodes. On the other hand, it is the only legal source of postcodes we have, other than local knowledge, but the latter is realistically limited to a dozen or so postcodes per mapper. Businesses website could also be OK but they are usually copyrighted. Derived databases, like FHRS, are generally not OK, a unless also permitted by Royal Mail. It's not that I don't care about complete addresses either. But my spare time is limited, and I feel I can contribute more by adding missing postcodes in a town vs adding complete addresses in a few streets. Others may have different priorities. I disagree that having data from Code-Point Open outside OSM is sufficient. Excluding surveyed information, everything in OSM is already publicly available (or should be). Yet, we all keep using and working on OSM. Besides, how to extend or combine information without adding it first? Best regards, ndrw6 ___ Talk-GB mailing list Talk-GB@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk-gb
Re: [Talk-GB] Automated Code-Point Open postcode editing (simple cases only)
On Tue, 16 Jul 2019 at 22:20, ndrw6 wrote: > Over past several months I've been adding postcodes from Code-Point > Open. I've streamlined the procedure a bit, so I can now add the tags > without spelling out every single one of them, but it is still a manual > and labour intensive process: > > https://github.com/ndrw6/import_postcodes/ > > While working on that, I've noticed there are a lot of simple cases > where automatic collation would have produced very similar results. For > example, in case of existing OSM buildings without an addr:postcode tag > located at or very near to a Code-Point Open centroid. > > Therefore I'm requesting permission to use the following automated edit > procedure: I'm afraid I think this new suggestion will be too prone to errors to outweigh the benefits, so I have to oppose it, unless there's significant manual curation. In which case though, you might as well not bother with the 15m limit and instead look to find the true extents of the postcode unit. I'm also not convinced that the previous work on adding postcodes to single buildings in isolation is that beneficial. In particular, the new proposal has the following issues: * Not all buildings are addressable properties, and so some (e.g. garages, outbuildings) shouldn't have a postcode. How would you avoid accidentally adding postcodes to these? * Large-user postcodes only belong to a single building, and shouldn't be added to any nearby buildings, however close. It's not clear how you would avoid these with the proposed method. (For example it's common for a bank to have its own postcode, but the buildings either side along the high street will all share a different postcode.) * Often the two sides of a street will have different postcodes. So in the case of terraced houses, you may have an opposite property within 15m belonging to a different postcode unit. There are also some problems with the existing method: * Sometimes OSM building polygons encompass several joined properties (e.g. a terrace or a row of shops). Adding a single postcode to such a building polygon might be incorrect, as the properties may not all share the same postcode. * Adding postcodes in this way removes a convenient way for other mappers to work on adding more detail, by looking for postcodes that are not yet in OSM. I would argue that the benefits of adding just a postcode to a building corresponding to the centroid are pretty small -- any users can already use code-point open to obtain this information. The real value in adding postcodes to OSM is when they combined with other address information (e.g. street names and house numbers to give a full address) and when human extrapolation is used to infer the full set of properties that have the same postcode. I thus have to object not just to the new proposal but also any continuation of the previous work to add single postcodes to buildings under the centroid. Instead I would suggest it would be better to proceed more slowly and take the time to add addr:street (and other addr tags) tags when adding postcodes, and also try to add the postcode to the full set of properties for each unit where this can be deduced. See the discussion at http://sk53-osm.blogspot.com/2013/12/british-postcodes-on-openstreetmap.html . For instance at http://overpass-turbo.eu/s/KRf only the highlighted buildings have postcodes and none have streets. It should be possible to infer the postcodes of most of the houses from the centroid locations, and also add the addr:street tags in that area. A tool that finds postcode units whose centroid lies on top of an existing OSM building would still be a good way of finding instances to apply this procedure to though. Better still if it could prioritise areas with a high density of small buildings (i.e. a systematic import/ tracing of buildings) that also lack postcodes. Best wishes, Robert. > 1. Open an osm file containing missing postcodes (from the above > website) in jOSM > > 1a. Select all points from the above dataset > > 2. Download OSM data in the area of interest > > 2a. Select all ways with a "building" tag of typical residential house > size and without an "addr:postcode" tag (search phrase: 'building > -"addr:postcode" type:way areasize:50-1000') > > 3. Use a collation plugin to collate both datasets with "centroid > distance" set to "< 15m". The condition is there to apply postcodes only > to small buildings in direct vicinity of the codepoint centroid. > > There are some caveats I've noticed, often not different from manual > editing: > > a) Some buildings have addresses added as separate points rather than > tags (automated edit will add addr:postcode tags directly to the > building, this is what I chose to do manually as well) > > b) Collation plugin doesn't support relations (these postcodes will get > ignored and can be added later manually) > > c) Often OSM buildings contain multiple addresses or postcodes and > should be split into several
Re: [Talk-GB] Automated Code-Point Open postcode editing (simple cases only)
On Wed, Jul 17, 2019, at 12:56 PM, Mateusz Konieczny wrote: > In that case maybe it would be a good idea to merge existing address-only > nodes > with building outlines as the first step? Some buildings (usually post office locations but also some others such as council offices) can have more than one postcode so I usually put those additional postcodes on separate nodes. Kevin___ Talk-GB mailing list Talk-GB@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk-gb
Re: [Talk-GB] Automated Code-Point Open postcode editing (simple cases only)
These should reasonably easy to remove from the dataset, easier than checking for existing addresses. Thanks for the suggestion. Best regards, ndrw6 On 17 July 2019 18:09:39 BST, David Woolley wrote: >On 16/07/2019 22:19, ndrw6 wrote: >> 3. Use a collation plugin to collate both datasets with "centroid >> distance" set to "< 15m". The condition is there to apply postcodes >only >> to small buildings in direct vicinity of the codepoint centroid. > >This algorithm will apply PO Box number postcodes to some buildings >adjacent to the post office. Similarly for other high use post codes, >which are close to residential areas. > >___ >Talk-GB mailing list >Talk-GB@openstreetmap.org >https://lists.openstreetmap.org/listinfo/talk-gb ___ Talk-GB mailing list Talk-GB@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk-gb
Re: [Talk-GB] Automated Code-Point Open postcode editing (simple cases only)
On 16/07/2019 22:19, ndrw6 wrote: 3. Use a collation plugin to collate both datasets with "centroid distance" set to "< 15m". The condition is there to apply postcodes only to small buildings in direct vicinity of the codepoint centroid. This algorithm will apply PO Box number postcodes to some buildings adjacent to the post office. Similarly for other high use post codes, which are close to residential areas. ___ Talk-GB mailing list Talk-GB@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk-gb
Re: [Talk-GB] Automated Code-Point Open postcode editing (simple cases only)
I will try to avoid adding postcodes in these cases. Basically, narrow down the list of postcodes to add to these that don't have any neighbouring objects with addr: tags. Best regards, ndrw6 On 17 July 2019 12:10:32 BST, Mateusz Konieczny wrote: >17 Jul 2019, 12:58 by nd...@redhazel.co.uk: > >> I haven't seen such guidelines for the UK myself. In general people >prefer having address tags on buildings. Separate address points are >more of a stop gap solution until building shapes are sufficiently >accurate. >> >In that case maybe it would be a good idea to merge existing >address-only nodes >with building outlines as the first step? ___ Talk-GB mailing list Talk-GB@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk-gb
Re: [Talk-GB] Automated Code-Point Open postcode editing (simple cases only)
17 Jul 2019, 12:58 by nd...@redhazel.co.uk: > I haven't seen such guidelines for the UK myself. In general people prefer > having address tags on buildings. Separate address points are more of a stop > gap solution until building shapes are sufficiently accurate. > In that case maybe it would be a good idea to merge existing address-only nodes with building outlines as the first step? ___ Talk-GB mailing list Talk-GB@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk-gb
Re: [Talk-GB] Automated Code-Point Open postcode editing (simple cases only)
I haven't seen such guidelines for the UK myself. In general people prefer having address tags on buildings. Separate address points are more of a stop gap solution until building shapes are sufficiently accurate. An exception is addresses on entrances, which some mappers seem to prefer. I can simply avoid such areas. Some clarifications: - Code-Point Open contains only one data entry (point) per postcode. It is located near the centroid and snapped to a nearest building (technically delivery point). So the scope is fairly small. - This is automatic editing, not an import. I can skip areas that use different addressing conventions. There is still over a hundred thousands of simple cases that can save a lot of manual work. - I ignore postcodes that are already in OSM near that location. So it is an additive process (no changes or deletions) - Manual methods are not perfect either, due to Code-Point Open limitations. But it is still the only legal source of postcodes in bulk we have (licensed by the owner). Best regards, ndrw6 On 17 July 2019 10:07:52 BST, Mateusz Konieczny wrote: > > > >16 lip 2019, 23:19 od nd...@redhazel.co.uk: > >> added as separate points rather than tags (automated edit will add >addr:postcode tags directly to the building, this is what I chose to do >manually as well) >> >Duplicating address data or adding >part to a separate node and part to >building outline seems incorrect to be. > >At least in Poland address imports >are obligated to handle this situation. ___ Talk-GB mailing list Talk-GB@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk-gb
Re: [Talk-GB] Automated Code-Point Open postcode editing (simple cases only)
16 lip 2019, 23:19 od nd...@redhazel.co.uk: > added as separate points rather than tags (automated edit will add > addr:postcode tags directly to the building, this is what I chose to do > manually as well) > Duplicating address data or adding part to a separate node and part to building outline seems incorrect to be. At least in Poland address imports are obligated to handle this situation.___ Talk-GB mailing list Talk-GB@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk-gb
[Talk-GB] Automated Code-Point Open postcode editing (simple cases only)
Hi, Over past several months I've been adding postcodes from Code-Point Open. I've streamlined the procedure a bit, so I can now add the tags without spelling out every single one of them, but it is still a manual and labour intensive process: https://github.com/ndrw6/import_postcodes/ While working on that, I've noticed there are a lot of simple cases where automatic collation would have produced very similar results. For example, in case of existing OSM buildings without an addr:postcode tag located at or very near to a Code-Point Open centroid. Therefore I'm requesting permission to use the following automated edit procedure: 1. Open an osm file containing missing postcodes (from the above website) in jOSM 1a. Select all points from the above dataset 2. Download OSM data in the area of interest 2a. Select all ways with a "building" tag of typical residential house size and without an "addr:postcode" tag (search phrase: 'building -"addr:postcode" type:way areasize:50-1000') 3. Use a collation plugin to collate both datasets with "centroid distance" set to "< 15m". The condition is there to apply postcodes only to small buildings in direct vicinity of the codepoint centroid. There are some caveats I've noticed, often not different from manual editing: a) Some buildings have addresses added as separate points rather than tags (automated edit will add addr:postcode tags directly to the building, this is what I chose to do manually as well) b) Collation plugin doesn't support relations (these postcodes will get ignored and can be added later manually) c) Often OSM buildings contain multiple addresses or postcodes and should be split into several buildings or building parts. This affects both manual and automated procedure, to minimize the impact I am setting relatively small "centroid distance" and building area limits. Best regards, ndrw6 ___ Talk-GB mailing list Talk-GB@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk-gb
