I just converted them to geopackage layers and went from there. Doug
________________________________ From: gdal-dev <[email protected]> on behalf of Mike Flannigan <[email protected]> Sent: Thursday, July 29, 2021 1:28 PM To: Simon Eves <[email protected]> Cc: [email protected] <[email protected]> Subject: [EXTERNAL] Re: [gdal-dev] Large GeoJSONs and aborting file opening This email has been received from outside of DOI - Use caution before clicking on links, opening attachments, or responding. Yeah, there must be a way of breaking this up in Postgis or some other method. Maybe by county or city region, if that meets your workflow needs. I would break it up outside of QGIS, and then import the various geoj's into QGIS. I'm thinking these methods don't work on a geoj: https://www.reddit.com/r/QGIS/comments/mbljrw/render_visible_layers_only/<https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.reddit.com%2Fr%2FQGIS%2Fcomments%2Fmbljrw%2Frender_visible_layers_only%2F&data=04%7C01%7Cdoug_newcomb%40fws.gov%7C6ed70bff2fda471014ab08d952b9edc2%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637631781801562533%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=0NB3OrZA5ngYcz8b6QqRy%2Fen%2BAAIk9jc6Uu0j0Fo3CA%3D&reserved=0> https://gis.stackexchange.com/questions/111784/displaying-only-selected-features-on-map-in-qgis<https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgis.stackexchange.com%2Fquestions%2F111784%2Fdisplaying-only-selected-features-on-map-in-qgis&data=04%7C01%7Cdoug_newcomb%40fws.gov%7C6ed70bff2fda471014ab08d952b9edc2%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637631781801562533%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=CjG5lHGmA%2BK6plQuGhY2d2wZ0I97T7k2GR1SBM9i7nk%3D&reserved=0> Mike On 7/29/21 11:30 AM, Simon Eves wrote: The problematic one in this case is about 30GB, with ~5.9M features, of property parcels in Florida, each with polygons with 5-10 vertices and 57 (!) other columns. Below is the first feature as printed by ogrinfo. It appears to have originated as a Shapefile, which we have also converted to regular GeoJSON (the 30GB one), and linear GeoJSONL/Seq. The Shapefile version imports with no issues or obvious delays, with features flowing basically immediately, and the overall process taking about 7 minutes. The regular GeoJSON version spends over 20 minutes in the GDALOpenEx call, and another 10 minutes before features flow (haven't looked yet in what), after which it takes about the same 7 minutes. The GeoJSONL/Seq version spends about 12 minutes in GDALOpenEx and then the same 10 minutes, and then the same 7 minutes. Note that ogrinfo has the same initial delays (20 minutes and 12 minutes) before it prints anything. This is all with GDAL 3.2.2 on Ubuntu 20.04 on a quad i7 4.2 with 32GB and SSD. The 7 minutes of import is not the issue, and once features are flowing, our code is able to be aborted. The issue is the 20-30 minutes where it can't because it's (seemingly) stuck in GDAL calls. Simon ______________________________ OGRFeature(fl_parcels):0 CNTYNAME (String) = ESCAMBIA LINK (String) = 27-083S321301060003 PARCELID (String) = 083S321301060003 NPARNO (String) = 12-033-083S321301060003 DORUC (String) = 000 PAUC (String) = 1 PARUSEDESC (String) = VACANT RESIDENTIAL SPASS_CD (String) = (null) IMPROVVAL (Integer) = 0 LNDVAL (Integer64) = 1069 JV (Integer64) = 1069 JV_CHNG (Integer) = (null) JV_HMSTD (Integer) = (null) AV_SD (Integer64) = 1069 AV_NSD (Integer64) = 1069 AV_HMSTD (Integer) = (null) JV_CLASS_U (Integer) = (null) ONAME (String) = GRAF MABIE PARTNERSHIP OADDR1 (String) = 5544 BAKER RD OADDR2 (String) = (null) OCITY (String) = MILTON OSTATE (String) = FL OZIPCD (String) = 32570 PHYADDR1 (String) = (null) PHYCITY (String) = PERDIDO KEY PHYZIP (String) = 32507 SLEGAL (String) = LT 6 BLK 3 PERDIDO BAY COUNTRY ALTKEY (String) = 103002391 ACTYRBLT (Integer) = (null) EFFYRBLT (Integer) = (null) TOTLVGAREA (Integer) = (null) NOBULDNG (Integer) = (null) NORESUNTS (Integer) = (null) PARSPLT (String) = (null) LNDSQFOOT (Real) = 14610.000000000000000 CONSTCLASS (String) = (null) SALEPRC1 (Integer64) = (null) SALEYR1 (Integer) = (null) SALEMO1 (Integer) = (null) ORBOOK1 (String) = (null) ORPAGE1 (String) = (null) SALEPRC2 (Integer) = (null) SALEYR2 (Integer) = (null) SALEMO2 (Integer) = (null) NBRHDCD (String) = (null) PUBLICLND (String) = (null) TAXAUTHCD (String) = MSTU SEC (String) = 8 TWN (String) = 03S RNG (String) = (null) CENSUSBK (String) = 12033002604 SOURCEAGE (String) = ESCAMBIA COUNTY PROPERTY APPRAISER SOURCEDATE (Integer64) = 1506643200 LAT_DD (Real) = 30.334948776670700 LONG_DD (Real) = -87.417015732515793 MGRS (String) = 16RDU5991555975 ACRES (Real) = 0.335475404173654 EXMPT (String) = (null) LU_RES (String) = (null) LUCODE (String) = 000 GCID (Integer) = 3070217 DESCRIPT (String) = VACANT RESIDENTIAL FLAG (String) = (null) FGDLAQDATE (Integer64) = 1509494400 AUTOID (Integer) = 3070217 Shape_Leng (Real) = 161.443473611912992 Shape_Area (Real) = 1357.620793937390090 POLYGON ((-87.4169063602653 30.3347107536787,-87.4169237108049 30.3346997733855,-87.4172722303389 30.3349846323649,-87.41724423478 23 30.3350359296124,-87.4172384512691 30.3351985804435,-87.4171183385966 30.335195856325,-87.4169790313658 30.3350555432658,-87.4167 283286418 30.3348031641612,-87.4167416558679 30.3347974225575,-87.4167635326352 30.3347875319118,-87.4167852417644 30.3347773059899, -87.4168030952182 30.334768463082,-87.4168206972148 30.3347594525361,-87.4168382153925 30.3347501905331,-87.416855482113 30.33474067 7073,-87.4168725811955 30.3347309121558,-87.4168895964589 30.334720937691,-87.4169063602653 30.3347107536787)) On Thu, Jul 29, 2021 at 6:49 AM Mike Flannigan <[email protected]<mailto:[email protected]>> wrote: I would like to hear more about large GeoJSON files. How large are they? My GeoJSON files contain linear features only. My largest one is 50.2 MB with 1,230,000 newlines in it. Next biggest one is 12 MB with 280,000 newlines. These and about 140 other geojsons are open in the same project and I have no problems. In fact I converted from SHP to geojson 2 years ago because I used to have problems with SHP linear files. I use QGIS 3.16.8 on Linux Mint. Mike On 7/28/21 2:36 PM, [email protected]<mailto:[email protected]> wrote: > Date: Wed, 28 Jul 2021 12:22:12 -0700 > From: Simon Eves<[email protected]<mailto:[email protected]>> > To:[email protected]<mailto:to%[email protected]> > Subject: [gdal-dev] Large GeoJSONs and aborting file opening > Message-ID: > > <cajf0ktrsasksospv8tba+itb+tql_ui5y4n05wgldw_3gur...@mail.gmail.com<mailto:cajf0ktrsasksospv8tba%2bitb%[email protected]>> > Content-Type: text/plain; charset="utf-8" > > Dear All, > > I am aware that some improvements were made in the 2.3 timeframe with > regards to dealing with large GeoJSON files, although even in 3.2, it's > still very slow and memory hungry. > > Our system allows for aborting imports, but this only works reliably once > it has actually got to the stage of reading features from the file. With > the GeoJSON, it just sits in the GDALOpenEx call for ages. > > My question, therefore, is whether it might be practical to run the > GDALOpenEx in a separate thread with a future to return the resulting > handle, such that it could be monitored and killed if necessary? > > Mainly I would be concerned that killing the thread might trash some global > GDAL state that might then not be recoverable, or that the open relies on > some TLS for the process thread and therefore might not work properly. > > We're going to try it anyway, but opinions welcomed, thanks! > > Simon -- [http://www2.omnisci.com/l/298412/2018-09-18/3sqpg/298412/61753/OmniSci_Email_Header2.png]<https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.omnisci.com%2F&data=04%7C01%7Cdoug_newcomb%40fws.gov%7C6ed70bff2fda471014ab08d952b9edc2%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637631781801572491%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=XYs9TEZeQyJhogwTVfjzY2DY%2FDeey7Flbeorpvmh%2FGw%3D&reserved=0> Simon Eves Senior Graphics Engineer, Rendering Group 100 Montgomery St (5th Floor), San Francisco, CA 94104, USA Email: [email protected]<mailto:[email protected]> | Cell: +1 (415) 902-1996
_______________________________________________ gdal-dev mailing list [email protected] https://lists.osgeo.org/mailman/listinfo/gdal-dev
