Thanks for catching the 'null' values. In geopandas data.frame each row/object must have assigned values for all fields. I was just assuming null values don't get exported. I guess this can be fixed with something like: sed 's/, "[^"]*": null//g'
How did you keep track of your import progress? Did you use taskscheduler? We are looking for something that, but without public access. On Fri, Sep 11, 2020 at 8:04 AM TC Haddad <[email protected]> wrote: > > Hi Martin, I have limited experience with imports, but I helped in > Portland, OR. I looked at your Westport file Two things I would pass on: > > - if the field is null, I don't think the tag should be in the import. > Null fields make the files unnecessarily large and provide no value > - we removed the buildings that conflicted with existing buildings in > advance > - we had a separate process to handle buildings with multiple addresses > - we tiled the geography into squares of no more than 500 buildings each. > This makes it possible for an import volunteer to have a workflow where > they import a square and then review every building for the conflicts you > mentioned (roads crossing buildings, etc). More buildings imported at one > time makes the review very difficult, especially if there are lots of such > corrections > > Our Github was here in case there is anything else of value to you > https://github.com/pdxosgeo/pdxbldgimport, and our wiki page was here: > https://wiki.openstreetmap.org/wiki/Portland,_OR_Bldg_import > > On Thu, Sep 10, 2020 at 6:31 PM Martin Machyna <[email protected]> wrote: > >> So to update this thread, I have integrated addresses from CT Open Data >> dataset and also updated the wiki page ( >> https://wiki.openstreetmap.org/wiki/Connecticut/Western_COG_Building_Import >> ) >> >> The whole dataset can be looked at: >> https://drive.google.com/file/d/10hUl09WSmK-I8h1hkMlELIGwDYsEziSe/view?usp=sharing >> For quick loading I also made a subset for one town (Westport): >> https://drive.google.com/file/d/1oMSbHXpPY5eLSHhGlikGy2uudyq3VyoV/view?usp=sharing >> >> If there is any issue, please let us know. >> >> On the side note of the CT Open Data buildings suitability for import I >> found a way how to simplify all buildings in an automated way in python. >> Here is a quick comparison of before/after >> https://files.slack.com/files-pri/T029HV94T-F01AD0FNFD3/simplified.png >> (you need to have Slack account) >> It doesn't look so bad and we could consider it for a next round of >> import. >> >> Just for a future reference in case someone would need to do the same, >> the python code is: >> >> import geopandas as gpd >> import pandas as pd >> from shapely import speedups >> speedups.enable() >> >> address = >> gpd.read_file("Connecticut_Buildings_with_Addresses_experimental.shp") >> simple = address.simplify(0.000005, preserve_topology=True) >> simple.to_file('Buildings-simplified.geojson', driver='GeoJSON') >> >> >> >> On Sat, Aug 29, 2020 at 6:50 PM <[email protected]> wrote: >> >>> I was going to look at the buildings too. I’ve used a tool in ArcGIS to >>> correct some pretty awful buildings, but I couldn’t download them either. >>> If there is no hurry, I’d check in again with the contact on Monday. It >>> would be nice to have the buildings with addresses on them. >>> >>> >>> >>> Joe >>> >>> >>> >>> *From:* Yury Yatsynovich <[email protected]> >>> *Sent:* Saturday, August 29, 2020 5:05 PM >>> *To:* Julien Lepiller <[email protected]> >>> *Cc:* [email protected]; [email protected] >>> *Subject:* Re: [Talk-us] [Imports] Import WestCOG building footprints >>> in south-west Connecticut >>> >>> >>> >>> Hi Julien, >>> >>> Unfortunately, I have limited knowledge on the data quality as I wasn't >>> able to download it (the server returns error). I let the CT point of >>> contact (Scott) know about the problem -- he mentioned in our communication >>> that he forwarded the issue to the tech support team, but I haven't heard >>> from them since then and I'm still unable to download it. >>> >>> >>> >>> On Sat, Aug 29, 2020, 4:57 PM Julien Lepiller <[email protected]> wrote: >>> >>> So, it's been a week since that last message. Do you think we should >>> import addresses and buildings at the same time? Should we import the >>> buildings first and care about addresses later? >>> >>> Yury, what are your thoughts about the data source quality? Do you >>> think it's a good idea to import from WestCOG and maybe rely on CT data >>> for the rest of CT? I tried playing with the data and I didn't see any >>> difference between drawing the buildings from scratch and having to >>> simplify and correct CT's data. >>> >>> Thanks! >>> >>> Le Sat, 22 Aug 2020 19:36:23 -0400, >>> Martin Machyna <[email protected]> a écrit : >>> >>> > Thank Julien for pushing this forward! >>> > >>> > yeah, I tried to get addresses from here: >>> > >>> http://geodata-ctmaps.opendata.arcgis.com/datasets/bfa7da83da384c2aa809882179369dc4_0/features/305004 >>> > and add them on top of the westCOG buildings. >>> > >>> > The data is a big mess because it's a join_table of like 30 different >>> > address databases. I lost a bit of motivation there, but I could have >>> > a look at it again. >>> > >>> > Martin >>> > >>> > On Sat, Aug 22, 2020 at 2:19 PM Julien Lepiller <[email protected]> >>> > wrote: >>> > >>> > > Le Sat, 22 Aug 2020 13:30:02 -0400, >>> > > Yury Yatsynovich <[email protected]> a écrit : >>> > > >>> > > > Hi Julien, >>> > > > The following communication that I've had recently with a CT >>> > > > official might be of interest to you: >>> > > > >>> > > > >>> > > >>> > > Oh, great! I think we already saw this data (I tried to contact them >>> > > too, but never got a reply :/). From what we saw (I think it was in >>> > > February?) the footprints have simplification issues (see >>> > > https://files.slack.com/files-pri/T029HV94T-FTDGDHXTM/image.png for >>> > > instance) where they are too detailed, not square enough, etc. Some >>> > > buildings also have holes in them, when there's none in the imagery. >>> > > >>> > > So I think it's too bad to be used directly, without a lot of manual >>> > > effort to simplify, square and redraw the shapes. However, the >>> > > address data is very interesting, so maybe we could extract from >>> > > it? Or we could use a separate dataset if they have addresses >>> > > separately. >>> > > >>> > > _______________________________________________ >>> > > Imports mailing list >>> > > [email protected] >>> > > https://lists.openstreetmap.org/listinfo/imports >>> > > >>> >>> >>> _______________________________________________ >>> Talk-us mailing list >>> [email protected] >>> https://lists.openstreetmap.org/listinfo/talk-us >>> >>> _______________________________________________ >>> Imports mailing list >>> [email protected] >>> https://lists.openstreetmap.org/listinfo/imports >>> >> _______________________________________________ >> Talk-us mailing list >> [email protected] >> https://lists.openstreetmap.org/listinfo/talk-us >> >
_______________________________________________ Imports mailing list [email protected] https://lists.openstreetmap.org/listinfo/imports
