Re: [Talk-us] [Imports] Import WestCOG building footprints in south-west Connecticut

[Martin Machyna]

So to update this thread, I have integrated addresses from CT Open Data
dataset and also updated the wiki page (
https://wiki.openstreetmap.org/wiki/Connecticut/Western_COG_Building_Import)

The whole dataset can be looked at:
https://drive.google.com/file/d/10hUl09WSmK-I8h1hkMlELIGwDYsEziSe/view?usp=sharing
For quick loading I also made a subset for one town (Westport):
https://drive.google.com/file/d/1oMSbHXpPY5eLSHhGlikGy2uudyq3VyoV/view?usp=sharing

If there is any issue, please let us know.

On the side note of the CT Open Data buildings suitability for import I
found a way how to simplify all buildings in an automated way in python.
Here is a quick comparison of before/after
https://files.slack.com/files-pri/T029HV94T-F01AD0FNFD3/simplified.png (you
need to have Slack account)
It doesn't look so bad and we could consider it for a next round of import.

Just for a future reference in case someone would need to do the same, the
python code is:

import geopandas as gpd
import pandas as pd
from shapely import speedups
speedups.enable()

address =
gpd.read_file("Connecticut_Buildings_with_Addresses_experimental.shp")
simple = address.simplify(0.05, preserve_topology=True)
simple.to_file('Buildings-simplified.geojson', driver='GeoJSON')



On Sat, Aug 29, 2020 at 6:50 PM  wrote:

> I was going to look at the buildings too.  I’ve used a tool in ArcGIS to
> correct some pretty awful buildings, but I couldn’t download them either.
>   If there is no hurry, I’d check in again with the contact on Monday.  It
> would be nice to have the buildings with addresses on them.
>
>
>
> Joe
>
>
>
> *From:* Yury Yatsynovich 
> *Sent:* Saturday, August 29, 2020 5:05 PM
> *To:* Julien Lepiller 
> *Cc:* impo...@openstreetmap.org; talk-us@openstreetmap.org
> *Subject:* Re: [Talk-us] [Imports] Import WestCOG building footprints in
> south-west Connecticut
>
>
>
> Hi Julien,
>
> Unfortunately, I have limited knowledge on the data quality as I wasn't
> able to download it (the server returns error). I let the CT point of
> contact (Scott) know about the problem -- he mentioned in our communication
> that he forwarded the issue to the tech support team, but I haven't heard
> from them since then and I'm still unable to download it.
>
>
>
> On Sat, Aug 29, 2020, 4:57 PM Julien Lepiller  wrote:
>
> So, it's been a week since that last message. Do you think we should
> import addresses and buildings at the same time? Should we import the
> buildings first and care about addresses later?
>
> Yury, what are your thoughts about the data source quality? Do you
> think it's a good idea to import from WestCOG and maybe rely on CT data
> for the rest of CT? I tried playing with the data and I didn't see any
> difference between drawing the buildings from scratch and having to
> simplify and correct CT's data.
>
> Thanks!
>
> Le Sat, 22 Aug 2020 19:36:23 -0400,
> Martin Machyna  a écrit :
>
> > Thank Julien for pushing this forward!
> >
> > yeah, I tried to get addresses from here:
> >
> http://geodata-ctmaps.opendata.arcgis.com/datasets/bfa7da83da384c2aa809882179369dc4_0/features/305004
> > and add them on top of the westCOG buildings.
> >
> > The data is a big mess because it's a join_table of like 30 different
> > address databases. I lost a bit of motivation there, but I could have
> > a look at it again.
> >
> > Martin
> >
> > On Sat, Aug 22, 2020 at 2:19 PM Julien Lepiller 
> > wrote:
> >
> > > Le Sat, 22 Aug 2020 13:30:02 -0400,
> > > Yury Yatsynovich  a écrit :
> > >
> > > > Hi Julien,
> > > > The following communication that I've had recently with a CT
> > > > official might be of interest to you:
> > > >
> > > >
> > >
> > > Oh, great! I think we already saw this data (I tried to contact them
> > > too, but never got a reply :/). From what we saw (I think it was in
> > > February?) the footprints have simplification issues (see
> > > https://files.slack.com/files-pri/T029HV94T-FTDGDHXTM/image.png for
> > > instance) where they are too detailed, not square enough, etc. Some
> > > buildings also have holes in them, when there's none in the imagery.
> > >
> > > So I think it's too bad to be used directly, without a lot of manual
> > > effort to simplify, square and redraw the shapes. However, the
> > > address data is very interesting, so maybe we could extract from
> > > it? Or we could use a separate dataset if they have addresses
> > > separately.
> > >
> > > ___
> > > Imports mailing list
> > > impo...@o

Re: [Talk-us] [Imports] Import WestCOG building footprints in south-west Connecticut

[joe.sapletal]

I was going to look at the buildings too.  I’ve used a tool in ArcGIS to 
correct some pretty awful buildings, but I couldn’t download them either.If 
there is no hurry, I’d check in again with the contact on Monday.  It would be 
nice to have the buildings with addresses on them.

 

Joe

 

From: Yury Yatsynovich  
Sent: Saturday, August 29, 2020 5:05 PM
To: Julien Lepiller 
Cc: impo...@openstreetmap.org; talk-us@openstreetmap.org
Subject: Re: [Talk-us] [Imports] Import WestCOG building footprints in 
south-west Connecticut

 

Hi Julien,

Unfortunately, I have limited knowledge on the data quality as I wasn't able to 
download it (the server returns error). I let the CT point of contact (Scott) 
know about the problem -- he mentioned in our communication that he forwarded 
the issue to the tech support team, but I haven't heard from them since then 
and I'm still unable to download it. 

 

On Sat, Aug 29, 2020, 4:57 PM Julien Lepiller mailto:o...@lepiller.eu> > wrote:

So, it's been a week since that last message. Do you think we should
import addresses and buildings at the same time? Should we import the
buildings first and care about addresses later?

Yury, what are your thoughts about the data source quality? Do you
think it's a good idea to import from WestCOG and maybe rely on CT data
for the rest of CT? I tried playing with the data and I didn't see any
difference between drawing the buildings from scratch and having to
simplify and correct CT's data.

Thanks!

Le Sat, 22 Aug 2020 19:36:23 -0400,
Martin Machyna mailto:mach...@gmail.com> > a écrit :

> Thank Julien for pushing this forward!
> 
> yeah, I tried to get addresses from here:
> http://geodata-ctmaps.opendata.arcgis.com/datasets/bfa7da83da384c2aa809882179369dc4_0/features/305004
> and add them on top of the westCOG buildings.
> 
> The data is a big mess because it's a join_table of like 30 different
> address databases. I lost a bit of motivation there, but I could have
> a look at it again.
> 
> Martin
> 
> On Sat, Aug 22, 2020 at 2:19 PM Julien Lepiller  <mailto:o...@lepiller.eu> >
> wrote:
> 
> > Le Sat, 22 Aug 2020 13:30:02 -0400,
> > Yury Yatsynovich  > <mailto:yury.yatsynov...@gmail.com> > a écrit :
> >  
> > > Hi Julien,
> > > The following communication that I've had recently with a CT
> > > official might be of interest to you:
> > >
> > >  
> >
> > Oh, great! I think we already saw this data (I tried to contact them
> > too, but never got a reply :/). From what we saw (I think it was in
> > February?) the footprints have simplification issues (see
> > https://files.slack.com/files-pri/T029HV94T-FTDGDHXTM/image.png for
> > instance) where they are too detailed, not square enough, etc. Some
> > buildings also have holes in them, when there's none in the imagery.
> >
> > So I think it's too bad to be used directly, without a lot of manual
> > effort to simplify, square and redraw the shapes. However, the
> > address data is very interesting, so maybe we could extract from
> > it? Or we could use a separate dataset if they have addresses
> > separately.
> >
> > ___
> > Imports mailing list
> > impo...@openstreetmap.org <mailto:impo...@openstreetmap.org> 
> > https://lists.openstreetmap.org/listinfo/imports
> >  


___
Talk-us mailing list
Talk-us@openstreetmap.org <mailto:Talk-us@openstreetmap.org> 
https://lists.openstreetmap.org/listinfo/talk-us

___
Talk-us mailing list
Talk-us@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-us

Re: [Talk-us] [Imports] Import WestCOG building footprints in south-west Connecticut

[Yury Yatsynovich]

Hi Julien,
Unfortunately, I have limited knowledge on the data quality as I wasn't
able to download it (the server returns error). I let the CT point of
contact (Scott) know about the problem -- he mentioned in our communication
that he forwarded the issue to the tech support team, but I haven't heard
from them since then and I'm still unable to download it.

On Sat, Aug 29, 2020, 4:57 PM Julien Lepiller  wrote:

> So, it's been a week since that last message. Do you think we should
> import addresses and buildings at the same time? Should we import the
> buildings first and care about addresses later?
>
> Yury, what are your thoughts about the data source quality? Do you
> think it's a good idea to import from WestCOG and maybe rely on CT data
> for the rest of CT? I tried playing with the data and I didn't see any
> difference between drawing the buildings from scratch and having to
> simplify and correct CT's data.
>
> Thanks!
>
> Le Sat, 22 Aug 2020 19:36:23 -0400,
> Martin Machyna  a écrit :
>
> > Thank Julien for pushing this forward!
> >
> > yeah, I tried to get addresses from here:
> >
> http://geodata-ctmaps.opendata.arcgis.com/datasets/bfa7da83da384c2aa809882179369dc4_0/features/305004
> > and add them on top of the westCOG buildings.
> >
> > The data is a big mess because it's a join_table of like 30 different
> > address databases. I lost a bit of motivation there, but I could have
> > a look at it again.
> >
> > Martin
> >
> > On Sat, Aug 22, 2020 at 2:19 PM Julien Lepiller 
> > wrote:
> >
> > > Le Sat, 22 Aug 2020 13:30:02 -0400,
> > > Yury Yatsynovich  a écrit :
> > >
> > > > Hi Julien,
> > > > The following communication that I've had recently with a CT
> > > > official might be of interest to you:
> > > >
> > > >
> > >
> > > Oh, great! I think we already saw this data (I tried to contact them
> > > too, but never got a reply :/). From what we saw (I think it was in
> > > February?) the footprints have simplification issues (see
> > > https://files.slack.com/files-pri/T029HV94T-FTDGDHXTM/image.png for
> > > instance) where they are too detailed, not square enough, etc. Some
> > > buildings also have holes in them, when there's none in the imagery.
> > >
> > > So I think it's too bad to be used directly, without a lot of manual
> > > effort to simplify, square and redraw the shapes. However, the
> > > address data is very interesting, so maybe we could extract from
> > > it? Or we could use a separate dataset if they have addresses
> > > separately.
> > >
> > > ___
> > > Imports mailing list
> > > impo...@openstreetmap.org
> > > https://lists.openstreetmap.org/listinfo/imports
> > >
>
>
> ___
> Talk-us mailing list
> Talk-us@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk-us
>
___
Talk-us mailing list
Talk-us@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-us

Re: [Talk-us] [Imports] Import WestCOG building footprints in south-west Connecticut

[Julien Lepiller]

So, it's been a week since that last message. Do you think we should
import addresses and buildings at the same time? Should we import the
buildings first and care about addresses later?

Yury, what are your thoughts about the data source quality? Do you
think it's a good idea to import from WestCOG and maybe rely on CT data
for the rest of CT? I tried playing with the data and I didn't see any
difference between drawing the buildings from scratch and having to
simplify and correct CT's data.

Thanks!

Le Sat, 22 Aug 2020 19:36:23 -0400,
Martin Machyna  a écrit :

> Thank Julien for pushing this forward!
> 
> yeah, I tried to get addresses from here:
> http://geodata-ctmaps.opendata.arcgis.com/datasets/bfa7da83da384c2aa809882179369dc4_0/features/305004
> and add them on top of the westCOG buildings.
> 
> The data is a big mess because it's a join_table of like 30 different
> address databases. I lost a bit of motivation there, but I could have
> a look at it again.
> 
> Martin
> 
> On Sat, Aug 22, 2020 at 2:19 PM Julien Lepiller 
> wrote:
> 
> > Le Sat, 22 Aug 2020 13:30:02 -0400,
> > Yury Yatsynovich  a écrit :
> >  
> > > Hi Julien,
> > > The following communication that I've had recently with a CT
> > > official might be of interest to you:
> > >
> > >  
> >
> > Oh, great! I think we already saw this data (I tried to contact them
> > too, but never got a reply :/). From what we saw (I think it was in
> > February?) the footprints have simplification issues (see
> > https://files.slack.com/files-pri/T029HV94T-FTDGDHXTM/image.png for
> > instance) where they are too detailed, not square enough, etc. Some
> > buildings also have holes in them, when there's none in the imagery.
> >
> > So I think it's too bad to be used directly, without a lot of manual
> > effort to simplify, square and redraw the shapes. However, the
> > address data is very interesting, so maybe we could extract from
> > it? Or we could use a separate dataset if they have addresses
> > separately.
> >
> > ___
> > Imports mailing list
> > impo...@openstreetmap.org
> > https://lists.openstreetmap.org/listinfo/imports
> >  


___
Talk-us mailing list
Talk-us@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-us

Re: [Talk-us] [Imports] Import WestCOG building footprints in south-west Connecticut

[Martin Machyna]

Thank Julien for pushing this forward!

yeah, I tried to get addresses from here:
http://geodata-ctmaps.opendata.arcgis.com/datasets/bfa7da83da384c2aa809882179369dc4_0/features/305004
and add them on top of the westCOG buildings.

The data is a big mess because it's a join_table of like 30 different
address databases. I lost a bit of motivation there, but I could have a
look at it again.

Martin

On Sat, Aug 22, 2020 at 2:19 PM Julien Lepiller  wrote:

> Le Sat, 22 Aug 2020 13:30:02 -0400,
> Yury Yatsynovich  a écrit :
>
> > Hi Julien,
> > The following communication that I've had recently with a CT official
> > might be of interest to you:
> >
> >
>
> Oh, great! I think we already saw this data (I tried to contact them
> too, but never got a reply :/). From what we saw (I think it was in
> February?) the footprints have simplification issues (see
> https://files.slack.com/files-pri/T029HV94T-FTDGDHXTM/image.png for
> instance) where they are too detailed, not square enough, etc. Some
> buildings also have holes in them, when there's none in the imagery.
>
> So I think it's too bad to be used directly, without a lot of manual
> effort to simplify, square and redraw the shapes. However, the address
> data is very interesting, so maybe we could extract from it? Or we
> could use a separate dataset if they have addresses separately.
>
> ___
> Imports mailing list
> impo...@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/imports
>
___
Talk-us mailing list
Talk-us@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-us

Re: [Talk-us] [Imports] Import WestCOG building footprints in south-west Connecticut

[Julien Lepiller]

Le Sat, 22 Aug 2020 13:30:02 -0400,
Yury Yatsynovich  a écrit :

> Hi Julien,
> The following communication that I've had recently with a CT official
> might be of interest to you:
> 
> 

Oh, great! I think we already saw this data (I tried to contact them
too, but never got a reply :/). From what we saw (I think it was in
February?) the footprints have simplification issues (see
https://files.slack.com/files-pri/T029HV94T-FTDGDHXTM/image.png for
instance) where they are too detailed, not square enough, etc. Some
buildings also have holes in them, when there's none in the imagery.

So I think it's too bad to be used directly, without a lot of manual
effort to simplify, square and redraw the shapes. However, the address
data is very interesting, so maybe we could extract from it? Or we
could use a separate dataset if they have addresses separately.

___
Talk-us mailing list
Talk-us@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-us

Re: [Talk-us] [Imports] Import WestCOG building footprints in south-west Connecticut

[Yury Yatsynovich]

Hi Julien,
The following communication that I've had recently with a CT official might
be of interest to you:


*

Hi Yury,



At this point, yes, I think that’s correct. Obviously, we’d like if you
cite the state, but we don’t have set guidelines for that. I forwarded the
other issue with downloading data and will let you know what I hear.
Hide quoted text



Thanks,



Scott



*From:* Yury Yatsynovich 
*Sent:* Monday, August 3, 2020 8:53 AM
*To:* Gaul, Scott 
*Subject:* Re: Request for permission to import data into OpenStreetMap



Thank you,  Scott!

The above mentioned terms describe mostly (lack of) liabilities of the data
provider, but don't mention any constraints on who and how can use the
data. Does it mean that there are no such constraints? There is also no
attribution requirement -- does it mean that I'm not required to cite the
source of the data?

With best regards,



On Mon, Aug 3, 2020, 7:42 AM Gaul, Scott  wrote:

Hi Yury,

Thanks for asking – you can use the terms of use for the CT open data
portal, here: https://data.ct.gov/terms. That should also cover the GIS
portal. Let us know any questions.



Thanks,



Scott



*From:* Yury Yatsynovich 
*Sent:* Friday, July 31, 2020 3:57 PM
*To:* Gaul, Scott 
*Subject:* Request for permission to import data into OpenStreetMap



Hi Scott!

Are there any constraints on who and how is allowed to use the data posted
on http://geodata-ctmaps.opendata.arcgis.com

?

I'm asking this because I'm interested in importing the data on CT
buildings with addresses into OpenStreetMap (OSM) -- to do so I need to
make sure that the imported data's license is compatible with OSM.

With kind regards,

OSM contributor,

--

Yury Yatsynovich


On Sat, Aug 22, 2020, 1:17 PM Julien Lepiller  wrote:

> Hi!
>
> With other contributors in Connecticut, we would like to import
> building footprints. We have evaluated different data sources, and
> concluded that no Connecticut-wide sources were usable without a lot of
> manual work to fix building geometry.
>
> We found that the WestCOG has very accurate building footprint data on
> its territory (south west Connecticut), available online and with a
> compatible license (CC0):
> http://data.westcog.org:8080/GIS_data/Buildings.gdb.zip
> That's 513,141 buildings, and 296,423 building parts in some of them,
> that cover this territory:
>
> https://westcog.org/wp-content/uploads/2015/08/WestCOG_Locus_Map-e1439920850177-790x1024.jpg
>
> Other COGs unfortunately don't share this data online. We hope that a
> successful import could be a convincing argument for other COGs to open
> their data.
>
> We have documented our current plan at
> https://wiki.openstreetmap.org/wiki/Connecticut/Western_COG_Building_Import
> on the wiki.
>
> We have contacted local mappers of this part of
> Connecticut (we are based in and around New Haven, not is south-west
> Connecticut), with no negative feedback and 2-3 positive responses. We
> have created a process to convert the data from WestCOG to OSM tags and
> files that can easily be loaded in JOSM.
>
> We have never done an import before, so we'd appreciate any advice on
> how to properly do the import. From our reading the wiki, we should use
> a separate user to import the data. Is that one shared user for the
> import, or one user for each person importing data?
>
> We have identified potential issues with this import: roads and
> waterways come from an old import and might very well cross the
> buildings we'd like to import. For now, we have a task on the osmus
> task manager to try and correct road geometry (currently finished at
> 50%) that covers Fairfield County (WestCOG is a part of Fairfield
> county): https://tasks.openstreetmap.us/project/193. This is not going
> very fast, so we'd like to start the import even though we haven't
> finished the tasks. We plan to fix road and water issues as we
> encounter them instead, while encouraging people to go and fix them
> independently from our import.
>
> WestCOG currently has almost no building mapped, but obviously we plan
> to keep existing buildings and only import buildings that are not yet
> mapped.
>
> What is a good way to split this import? Is there a good size per
> changeset that you could recommend?
>
> Thanks for your help!
>
> Julien
>
> ___
> Imports mailing list
> impo...@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/imports
>
___
Talk-us mailing list
Talk-us@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-us