Re: [Qgis-user] Convert huge shapefile to small size

[Nyall Dawson via Qgis-user]

On Fri, 29 Jul 2022 at 06:22, Madry, Scott via Qgis-user
 wrote:
>
> Thanks Chris, but I do think that this is a topic is worth considering by the 
> QGIS community, beyond the limitations of the archaeic shapefile format. We 
> are moving into an era of very big data, both raster and vector, and QGIS 
> should be able to meet this challenge. This will require some foresight and 
> planning, thus my raising the issue. Or not...

In truth QGIS is already well prepared for handling massive datasets.
(E.g. you'll see in recent versions how well it handles mammoth point
cloud formats!). Sure, there's further improvements which can be made,
but they're evolutionary rather than revolutionary!

Nyall


>
> Best,
>
> Scott
>
> On Jul 28, 2022, at 4:05 PM, chris hermansen  wrote:
>
> Scott and list,
>
> With respect I think this conversation is moving off the point, which was 
> related to a 5gb shapefile
>
> https://en.m.wikipedia.org/wiki/Shapefile
>
> Which is beyond the design limits of the Shapefile standards and perhaps 
> therefore the source of the problem that the OP has.
>
> Notably not a limitation of QGIS
>
> On Thu, Jul 28, 2022, 10:45 Madry, Scott via Qgis-user 
>  wrote:
>>
>> Dear colleagues,
>>
>> As we move more into the world of big data, IOE, data mining, and data 
>> analytics, dealing with larger and larger data sets will become required, 
>> and, eventually, the norm.
>>
>> GRASS GIS has made very good progress in this regard, optimizing the code to 
>> run on massively parallel computer architectures, with very good effect:
>>
>> https://grasswiki.osgeo.org/wiki/Supercomputing
>>
>> Perhaps it is time for QGIS to consider this in its future development 
>> plans? We will certainly be needing this in the future.
>>
>> And thanks to all the team who keep the QGIS train running, I am constantly 
>> amazed at the energy, effort, and availability of people to the broader 
>> community.
>>
>> Scott
>>
>> Scott Madry, Ph.D.
>> Research Associate Professor of Archaeology
>> The University of North Carolina at Chapel Hill
>>
>> Tel 1-919-448-4493
>> Email:mad...@email.unc.edu
>> https://scottmadry.web.unc.edu
>> Skype:   scott madry
>>
>>
>>
>> On Jul 28, 2022, at 10:14 AM, jhubbslist--- via Qgis-user 
>>  wrote:
>>
>> Past a point, trying to run a GIS app with very large data structures just 
>> doesn't work well. If you really need all the information you're carrying 
>> around for your analysis or whatever, you may need to reach for different 
>> tools or use the tools you have differently.
>>
>> I assume it's not so much the >5GiB of disk space that's the issue and that 
>> you've maxxed out the CPU, graphics, and disk I/O rate as much as is 
>> practical so mostly it's a matter of how long it takes maps etc. to paint 
>> onscreen. It may help you to move the heavy-lift onto PostgreSQL/PostGIS 
>> where you can make use of spatial indexing. Or, you can craft your 
>> operations the way you want them in QGIS but do the actual work with e.g. 
>> GDAL calls in Python. I spoke with someone a couple weeks ago whose 
>> particular GIS process worked better in GRASS than in QGIS, so that's 
>> something you might look into as well.
>>
>> On 7/28/22 7:52 AM, krishna Ayyala via Qgis-user wrote:
>>
>> dbf file itself is 5.1GB. Rest all other files are less than 500MB. It is 
>> the number of records which is huge. It has about 117,2100 points.
>>
>> On Thu, Jul 28, 2022 at 12:02 AM Bernd Vogelgesang via Qgis-user 
>>  wrote:
>>>
>>> how big is the dbf file of that shape? Maybe you can also drop some
>>> attributes.
>>>
>>> Am 28.07.22 um 03:46 schrieb krishna Ayyala via Qgis-user:
>>> > Hello,
>>> > I have a shapefile of 5GB in size. Is it possible to convert this
>>> > shapefile to a smaller size file? It can be any format, not
>>> > necessarily a shapefile. But, preferably a vector format. I tried to
>>> > convert it into tiles, but that didn't work as it was losing the
>>> > resolution. I am looking to convert this 5GB size file to about 500MB.
>>> >
>>> > Regards.
>>> >
>>> > ___
>>> > Qgis-user mailing list
>>> > Qgis-user@lists.osgeo.org
>>> > List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>>> > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>>> ___
>>> Qgis-user mailing list
>>> Qgis-user@lists.osgeo.org
>>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>>
>>
>> ___
>> Qgis-user mailing list
>> Qgis-user@lists.osgeo.org
>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>>
>>
>> ___
>> Qgis-user mailing list
>> Qgis-user@lists.osgeo.org
>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>> Unsubscribe: 

Re: [Qgis-user] Convert huge shapefile to small size

[Patrick Dunford via Qgis-user]

A shapefile is a vector files, as far as I know. I don't think any 
rasters are stored in this format.


If there is a large volume of data, it may have a great many records 
referencing a large number of features. Vector layers are usually made 
up of a number of points that together form one feature. So practically 
all the data in a vector layer is the coordinates of these points.


The only way of getting it smaller is reduce the number of points, 
reduce the number of features, or maybe there is a format that's compressed?


Tiles depend on the zoom level, you don't need all the resolution at the 
lower scales or zoom levels because the map is just cluttered, you only 
need them when you zoom in.


On 28/07/22 13:46, krishna Ayyala via Qgis-user wrote:

Hello,
I have a shapefile of 5GB in size. Is it possible to convert this 
shapefile to a smaller size file? It can be any format, not 
necessarily a shapefile. But, preferably a vector format. I tried to 
convert it into tiles, but that didn't work as it was losing the 
resolution. I am looking to convert this 5GB size file to about 500MB.


Regards.

___
Qgis-user mailing list
Qgis-user@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

___
Qgis-user mailing list
Qgis-user@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

Re: [Qgis-user] Convert huge shapefile to small size

[Madry, Scott via Qgis-user]

Thanks Chris, but I do think that this is a topic is worth considering by the 
QGIS community, beyond the limitations of the archaeic shapefile format. We are 
moving into an era of very big data, both raster and vector, and QGIS should be 
able to meet this challenge. This will require some foresight and planning, 
thus my raising the issue. Or not...

Best,

Scott

On Jul 28, 2022, at 4:05 PM, chris hermansen 
mailto:clherman...@gmail.com>> wrote:

Scott and list,

With respect I think this conversation is moving off the point, which was 
related to a 5gb shapefile

https://en.m.wikipedia.org/wiki/Shapefile

Which is beyond the design limits of the Shapefile standards and perhaps 
therefore the source of the problem that the OP has.

Notably not a limitation of QGIS

On Thu, Jul 28, 2022, 10:45 Madry, Scott via Qgis-user 
mailto:qgis-user@lists.osgeo.org>> wrote:
Dear colleagues,

As we move more into the world of big data, IOE, data mining, and data 
analytics, dealing with larger and larger data sets will become required, and, 
eventually, the norm.

GRASS GIS has made very good progress in this regard, optimizing the code to 
run on massively parallel computer architectures, with very good effect:

https://grasswiki.osgeo.org/wiki/Supercomputing

Perhaps it is time for QGIS to consider this in its future development plans? 
We will certainly be needing this in the future.

And thanks to all the team who keep the QGIS train running, I am constantly 
amazed at the energy, effort, and availability of people to the broader 
community.

Scott

Scott Madry, Ph.D.
Research Associate Professor of Archaeology
The University of North Carolina at Chapel Hill

Tel 1-919-448-4493
Email:mad...@email.unc.edu
https://scottmadry.web.unc.edu
Skype:   scott madry



On Jul 28, 2022, at 10:14 AM, jhubbslist--- via Qgis-user 
mailto:qgis-user@lists.osgeo.org>> wrote:

Past a point, trying to run a GIS app with very large data structures just 
doesn't work well. If you really need all the information you're carrying 
around for your analysis or whatever, you may need to reach for different tools 
or use the tools you have differently.

I assume it's not so much the >5GiB of disk space that's the issue and that 
you've maxxed out the CPU, graphics, and disk I/O rate as much as is practical 
so mostly it's a matter of how long it takes maps etc. to paint onscreen. It 
may help you to move the heavy-lift onto PostgreSQL/PostGIS where you can make 
use of spatial indexing. Or, you can craft your operations the way you want 
them in QGIS but do the actual work with e.g. GDAL calls in Python. I spoke 
with someone a couple weeks ago whose particular GIS process worked better in 
GRASS than in QGIS, so that's something you might look into as well.

On 7/28/22 7:52 AM, krishna Ayyala via Qgis-user wrote:
dbf file itself is 5.1GB. Rest all other files are less than 500MB. It is the 
number of records which is huge. It has about 117,2100 points.

On Thu, Jul 28, 2022 at 12:02 AM Bernd Vogelgesang via Qgis-user 
mailto:qgis-user@lists.osgeo.org>> wrote:
how big is the dbf file of that shape? Maybe you can also drop some
attributes.

Am 28.07.22 um 03:46 schrieb krishna Ayyala via Qgis-user:
> Hello,
> I have a shapefile of 5GB in size. Is it possible to convert this
> shapefile to a smaller size file? It can be any format, not
> necessarily a shapefile. But, preferably a vector format. I tried to
> convert it into tiles, but that didn't work as it was losing the
> resolution. I am looking to convert this 5GB size file to about 500MB.
>
> Regards.
>
> ___
> Qgis-user mailing list
> Qgis-user@lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
___
Qgis-user mailing list
Qgis-user@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user



___
Qgis-user mailing list
Qgis-user@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user



___
Qgis-user mailing list
Qgis-user@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

___
Qgis-user mailing list
Qgis-user@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

Re: [Qgis-user] Convert huge shapefile to small size

[Andrea Giudiceandrea via Qgis-user]

*chris hermansen*
/Thu Jul 28 13:05:08 PDT 2022/

With respect I think this conversation is moving off the point, which was
related to a 5gb shapefile

https://en.m.wikipedia.org/wiki/Shapefile

Which is beyond the design limits of the Shapefile standards and perhaps
therefore the source of the problem that the OP has.


Hi Chris,
The QGIS / GDAL/OGR implementation goes beyond the normal ESRI 
implementation limit ("2 GB size limit for any shapefile component file").


See https://gdal.org/drivers/vector/shapefile.html
***
Size Issues
Geometry: The Shapefile format explicitly uses 32bit offsets and so 
cannot go over 8GB (it actually uses 32bit offsets to 16bit words), but 
the OGR shapefile implementation has a limitation to 4GB.


Attributes: The dbf format does not have any offsets in it, so it can be 
arbitrarily large.

***

Best regards.

Andrea Giudiceandrea___
Qgis-user mailing list
Qgis-user@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

Re: [Qgis-user] Convert huge shapefile to small size

[chris hermansen via Qgis-user]

Scott and list,

With respect I think this conversation is moving off the point, which was
related to a 5gb shapefile

https://en.m.wikipedia.org/wiki/Shapefile

Which is beyond the design limits of the Shapefile standards and perhaps
therefore the source of the problem that the OP has.

Notably not a limitation of QGIS

On Thu, Jul 28, 2022, 10:45 Madry, Scott via Qgis-user <
qgis-user@lists.osgeo.org> wrote:

> Dear colleagues,
>
> As we move more into the world of big data, IOE, data mining, and data
> analytics, dealing with larger and larger data sets will become required,
> and, eventually, the norm.
>
> GRASS GIS has made very good progress in this regard, optimizing the code
> to run on massively parallel computer architectures, with very good effect:
>
> https://grasswiki.osgeo.org/wiki/Supercomputing
>
> Perhaps it is time for QGIS to consider this in its future development
> plans? We will certainly be needing this in the future.
>
> And thanks to all the team who keep the QGIS train running, I am
> constantly amazed at the energy, effort, and availability of people to the
> broader community.
>
> Scott
>
> Scott Madry, Ph.D.
> Research Associate Professor of Archaeology
> The University of North Carolina at Chapel Hill
>
> Tel 1-919-448-4493
> Email:mad...@email.unc.edu 
> https://scottmadry.web.unc.edu
> Skype:   scott madry
>
>
>
> On Jul 28, 2022, at 10:14 AM, jhubbslist--- via Qgis-user <
> qgis-user@lists.osgeo.org> wrote:
>
> Past a point, trying to run a GIS app with very large data structures just
> doesn't work well. If you really need all the information you're carrying
> around for your analysis or whatever, you may need to reach for different
> tools or use the tools you have differently.
>
> I assume it's not so much the >5GiB of disk space that's the issue and
> that you've maxxed out the CPU, graphics, and disk I/O rate as much as is
> practical so mostly it's a matter of how long it takes maps etc. to paint
> onscreen. It may help you to move the heavy-lift onto PostgreSQL/PostGIS
> where you can make use of spatial indexing. Or, you can craft your
> operations the way you want them in QGIS but do the actual work with e.g.
> GDAL calls in Python. I spoke with someone a couple weeks ago whose
> particular GIS process worked better in GRASS than in QGIS, so that's
> something you might look into as well.
>
> On 7/28/22 7:52 AM, krishna Ayyala via Qgis-user wrote:
>
> dbf file itself is 5.1GB. Rest all other files are less than 500MB. It is
> the number of records which is huge. It has about 117,2100 points.
>
> On Thu, Jul 28, 2022 at 12:02 AM Bernd Vogelgesang via Qgis-user <
> qgis-user@lists.osgeo.org> wrote:
>
>> how big is the dbf file of that shape? Maybe you can also drop some
>> attributes.
>>
>> Am 28.07.22 um 03:46 schrieb krishna Ayyala via Qgis-user:
>> > Hello,
>> > I have a shapefile of 5GB in size. Is it possible to convert this
>> > shapefile to a smaller size file? It can be any format, not
>> > necessarily a shapefile. But, preferably a vector format. I tried to
>> > convert it into tiles, but that didn't work as it was losing the
>> > resolution. I am looking to convert this 5GB size file to about 500MB.
>> >
>> > Regards.
>> >
>> > ___
>> > Qgis-user mailing list
>> > Qgis-user@lists.osgeo.org
>> > List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>> > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>> ___
>> Qgis-user mailing list
>> Qgis-user@lists.osgeo.org
>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>>
>
> ___
> Qgis-user mailing listqgis-u...@lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>
>
> ___
> Qgis-user mailing list
> Qgis-user@lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>
>
> ___
> Qgis-user mailing list
> Qgis-user@lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>
___
Qgis-user mailing list
Qgis-user@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

Re: [Qgis-user] Convert huge shapefile to small size

[Madry, Scott via Qgis-user]

Dear colleagues,

As we move more into the world of big data, IOE, data mining, and data 
analytics, dealing with larger and larger data sets will become required, and, 
eventually, the norm.

GRASS GIS has made very good progress in this regard, optimizing the code to 
run on massively parallel computer architectures, with very good effect:

https://grasswiki.osgeo.org/wiki/Supercomputing

Perhaps it is time for QGIS to consider this in its future development plans? 
We will certainly be needing this in the future.

And thanks to all the team who keep the QGIS train running, I am constantly 
amazed at the energy, effort, and availability of people to the broader 
community.

Scott

Scott Madry, Ph.D.
Research Associate Professor of Archaeology
The University of North Carolina at Chapel Hill

Tel 1-919-448-4493
Email:mad...@email.unc.edu
https://scottmadry.web.unc.edu
Skype:   scott madry



On Jul 28, 2022, at 10:14 AM, jhubbslist--- via Qgis-user 
mailto:qgis-user@lists.osgeo.org>> wrote:

Past a point, trying to run a GIS app with very large data structures just 
doesn't work well. If you really need all the information you're carrying 
around for your analysis or whatever, you may need to reach for different tools 
or use the tools you have differently.

I assume it's not so much the >5GiB of disk space that's the issue and that 
you've maxxed out the CPU, graphics, and disk I/O rate as much as is practical 
so mostly it's a matter of how long it takes maps etc. to paint onscreen. It 
may help you to move the heavy-lift onto PostgreSQL/PostGIS where you can make 
use of spatial indexing. Or, you can craft your operations the way you want 
them in QGIS but do the actual work with e.g. GDAL calls in Python. I spoke 
with someone a couple weeks ago whose particular GIS process worked better in 
GRASS than in QGIS, so that's something you might look into as well.

On 7/28/22 7:52 AM, krishna Ayyala via Qgis-user wrote:
dbf file itself is 5.1GB. Rest all other files are less than 500MB. It is the 
number of records which is huge. It has about 117,2100 points.

On Thu, Jul 28, 2022 at 12:02 AM Bernd Vogelgesang via Qgis-user 
mailto:qgis-user@lists.osgeo.org>> wrote:
how big is the dbf file of that shape? Maybe you can also drop some
attributes.

Am 28.07.22 um 03:46 schrieb krishna Ayyala via Qgis-user:
> Hello,
> I have a shapefile of 5GB in size. Is it possible to convert this
> shapefile to a smaller size file? It can be any format, not
> necessarily a shapefile. But, preferably a vector format. I tried to
> convert it into tiles, but that didn't work as it was losing the
> resolution. I am looking to convert this 5GB size file to about 500MB.
>
> Regards.
>
> ___
> Qgis-user mailing list
> Qgis-user@lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
___
Qgis-user mailing list
Qgis-user@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user



___
Qgis-user mailing list
Qgis-user@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user



___
Qgis-user mailing list
Qgis-user@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

___
Qgis-user mailing list
Qgis-user@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

Re: [Qgis-user] Convert huge shapefile to small size

[David Strip via Qgis-user]

  
  
On 7/28/2022 5:52 AM, krishna Ayyala via Qgis-user wrote:

  
  dbf file itself is 5.1GB. Rest all other files are
less than 500MB. It is the number of records which is huge. It
has about 117,2100 points.
  


Is that 117,200 points or 1,172,100 points? In either case, there's
almost certainly  something wrong with your file, as you have either
50K or 5K bytes per point, which is off by one or more orders of
magnitude.

For comparison, I created a set of 1M random points with 14
attributes inherited from the polygon they lie in. This includes 3
qlongong fields, 9 doubles,  and 3 strings with a total length of 40
chars. I saved this as a shapefile and the total size is ~300M.
  

___
Qgis-user mailing list
Qgis-user@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

Re: [Qgis-user] Convert huge shapefile to small size

[Richard Greenwood via Qgis-user]

You could export the shapefile and use *Layer Options > RESIZE=YES*. This
is the same as the ogr2ogr switch *-lco RESIZE=YES*, which reduces the
attribute column widths down to their minimum width required to store
the longest actual value in the data. A dbf file allocates space for the
full column width even if it's empty. So for example if you have a
column width of 254 characters but the longest string is only 12
characters, your dbf will have a lot of wasted space in it. Resizing
the column would write out a new dbf with a column width of 12 characters.

On Thu, Jul 28, 2022 at 8:15 AM jhubbslist--- via Qgis-user <
qgis-user@lists.osgeo.org> wrote:

> Past a point, trying to run a GIS app with very large data structures just
> doesn't work well. If you really need all the information you're carrying
> around for your analysis or whatever, you may need to reach for different
> tools or use the tools you have differently.
>
> I assume it's not so much the >5GiB of disk space that's the issue and
> that you've maxxed out the CPU, graphics, and disk I/O rate as much as is
> practical so mostly it's a matter of how long it takes maps etc. to paint
> onscreen. It may help you to move the heavy-lift onto PostgreSQL/PostGIS
> where you can make use of spatial indexing. Or, you can craft your
> operations the way you want them in QGIS but do the actual work with e.g.
> GDAL calls in Python. I spoke with someone a couple weeks ago whose
> particular GIS process worked better in GRASS than in QGIS, so that's
> something you might look into as well.
>
> On 7/28/22 7:52 AM, krishna Ayyala via Qgis-user wrote:
>
> dbf file itself is 5.1GB. Rest all other files are less than 500MB. It is
> the number of records which is huge. It has about 117,2100 points.
>
> On Thu, Jul 28, 2022 at 12:02 AM Bernd Vogelgesang via Qgis-user <
> qgis-user@lists.osgeo.org> wrote:
>
>> how big is the dbf file of that shape? Maybe you can also drop some
>> attributes.
>>
>> Am 28.07.22 um 03:46 schrieb krishna Ayyala via Qgis-user:
>> > Hello,
>> > I have a shapefile of 5GB in size. Is it possible to convert this
>> > shapefile to a smaller size file? It can be any format, not
>> > necessarily a shapefile. But, preferably a vector format. I tried to
>> > convert it into tiles, but that didn't work as it was losing the
>> > resolution. I am looking to convert this 5GB size file to about 500MB.
>> >
>> > Regards.
>> >
>> > ___
>> > Qgis-user mailing list
>> > Qgis-user@lists.osgeo.org
>> > List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>> > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>> ___
>> Qgis-user mailing list
>> Qgis-user@lists.osgeo.org
>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>>
>
> ___
> Qgis-user mailing listqgis-u...@lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>
>
> ___
> Qgis-user mailing list
> Qgis-user@lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>


-- 
Richard W. Greenwood
www.greenwoodmap.com
___
Qgis-user mailing list
Qgis-user@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

Re: [Qgis-user] Convert huge shapefile to small size

[jhubbslist--- via Qgis-user]

Past a point, trying to run a GIS app with very large data structures 
just doesn't work well. If you really need all the information you're 
carrying around for your analysis or whatever, you may need to reach for 
different tools or use the tools you have differently.


I assume it's not so much the >5GiB of disk space that's the issue and 
that you've maxxed out the CPU, graphics, and disk I/O rate as much as 
is practical so mostly it's a matter of how long it takes maps etc. to 
paint onscreen. It may help you to move the heavy-lift onto 
PostgreSQL/PostGIS where you can make use of spatial indexing. Or, you 
can craft your operations the way you want them in QGIS but do the 
actual work with e.g. GDAL calls in Python. I spoke with someone a 
couple weeks ago whose particular GIS process worked better in GRASS 
than in QGIS, so that's something you might look into as well.


On 7/28/22 7:52 AM, krishna Ayyala via Qgis-user wrote:
dbf file itself is 5.1GB. Rest all other files are less than 500MB. It 
is the number of records which is huge. It has about 117,2100 points.


On Thu, Jul 28, 2022 at 12:02 AM Bernd Vogelgesang via Qgis-user 
 wrote:


how big is the dbf file of that shape? Maybe you can also drop some
attributes.

Am 28.07.22 um 03:46 schrieb krishna Ayyala via Qgis-user:
> Hello,
> I have a shapefile of 5GB in size. Is it possible to convert this
> shapefile to a smaller size file? It can be any format, not
> necessarily a shapefile. But, preferably a vector format. I tried to
> convert it into tiles, but that didn't work as it was losing the
> resolution. I am looking to convert this 5GB size file to about
500MB.
>
> Regards.
>
> ___
> Qgis-user mailing list
> Qgis-user@lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
___
Qgis-user mailing list
Qgis-user@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user


___
Qgis-user mailing list
Qgis-user@lists.osgeo.org
List info:https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe:https://lists.osgeo.org/mailman/listinfo/qgis-user


___
Qgis-user mailing list
Qgis-user@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

Re: [Qgis-user] Convert huge shapefile to small size

[Bernd Vogelgesang via Qgis-user]

so, maybe you should really drop all unneccessary fields or maybe all but an id 
field, and then create a relation to the original database, but in any case I 
would convert that stuff away from ESRI-Shape to Geopackage beforehand, as is 
it is a pita.

On 28.07.22 13:52, krishna Ayyala  wrote:

dbf file itself is 5.1GB. Rest all other files are less than 500MB. It
is the number of records which is huge. It has about 117,2100 points.

On Thu, Jul 28, 2022 at 12:02 AM Bernd Vogelgesang via Qgis-user
mailto:qgis-user@lists.osgeo.org>> wrote:

how big is the dbf file of that shape? Maybe you can also drop some
attributes.

Am 28.07.22 um 03:46 schrieb krishna Ayyala via Qgis-user:
 > Hello,
 > I have a shapefile of 5GB in size. Is it possible to convert this
 > shapefile to a smaller size file? It can be any format, not
 > necessarily a shapefile. But, preferably a vector format. I tried to
 > convert it into tiles, but that didn't work as it was losing the
 > resolution. I am looking to convert this 5GB size file to about
500MB.
 >
 > Regards.
 >
 > ___
 > Qgis-user mailing list
 > Qgis-user@lists.osgeo.org 
 > List info: https://lists.osgeo.org/mailman/listinfo/qgis-user

 > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

___
Qgis-user mailing list
Qgis-user@lists.osgeo.org 
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user

Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user




___
Qgis-user mailing list
Qgis-user@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

Re: [Qgis-user] Convert huge shapefile to small size

[krishna Ayyala via Qgis-user]

dbf file itself is 5.1GB. Rest all other files are less than 500MB. It is
the number of records which is huge. It has about 117,2100 points.

On Thu, Jul 28, 2022 at 12:02 AM Bernd Vogelgesang via Qgis-user <
qgis-user@lists.osgeo.org> wrote:

> how big is the dbf file of that shape? Maybe you can also drop some
> attributes.
>
> Am 28.07.22 um 03:46 schrieb krishna Ayyala via Qgis-user:
> > Hello,
> > I have a shapefile of 5GB in size. Is it possible to convert this
> > shapefile to a smaller size file? It can be any format, not
> > necessarily a shapefile. But, preferably a vector format. I tried to
> > convert it into tiles, but that didn't work as it was losing the
> > resolution. I am looking to convert this 5GB size file to about 500MB.
> >
> > Regards.
> >
> > ___
> > Qgis-user mailing list
> > Qgis-user@lists.osgeo.org
> > List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
> ___
> Qgis-user mailing list
> Qgis-user@lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>
___
Qgis-user mailing list
Qgis-user@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

Re: [Qgis-user] Convert huge shapefile to small size

[Nicolas Cadieux via Qgis-user]

Hi,

There is a distinct possibility that the 5GB shape file is corrupt or that it 
will be if you manipulate it.  It’s too big for the normal shape file format.  
I would first save it as a geopackage then stat looking at why the file is so 
big.  

Unless shapes can be simplified without loosing resolution (keep a back up 
always), you might have to split the file using a field to make multiple files 
(example splitting a lake file by state).  You may look at the data types.  
Perhaps they can be simplified (ex int16 vs float64 for a filed that has no 
decimal values…

Try compressing the resulting file.

Nicolas Cadieux
https://gitlab.com/njacadieux

> Le 27 juill. 2022 à 22:47, krishna Ayyala via Qgis-user 
>  a écrit :
> 
> 
> Hello,
> I have a shapefile of 5GB in size. Is it possible to convert this shapefile 
> to a smaller size file? It can be any format, not necessarily a shapefile. 
> But, preferably a vector format. I tried to convert it into tiles, but that 
> didn't work as it was losing the resolution. I am looking to convert this 5GB 
> size file to about 500MB.
> 
> Regards.
> ___
> Qgis-user mailing list
> Qgis-user@lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
___
Qgis-user mailing list
Qgis-user@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

Re: [Qgis-user] Convert huge shapefile to small size

[Bernd Vogelgesang via Qgis-user]

how big is the dbf file of that shape? Maybe you can also drop some
attributes.

Am 28.07.22 um 03:46 schrieb krishna Ayyala via Qgis-user:

Hello,
I have a shapefile of 5GB in size. Is it possible to convert this
shapefile to a smaller size file? It can be any format, not
necessarily a shapefile. But, preferably a vector format. I tried to
convert it into tiles, but that didn't work as it was losing the
resolution. I am looking to convert this 5GB size file to about 500MB.

Regards.

___
Qgis-user mailing list
Qgis-user@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

___
Qgis-user mailing list
Qgis-user@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

[Qgis-user] Convert huge shapefile to small size

[krishna Ayyala via Qgis-user]

Hello,
I have a shapefile of 5GB in size. Is it possible to convert this shapefile
to a smaller size file? It can be any format, not necessarily a shapefile.
But, preferably a vector format. I tried to convert it into tiles, but that
didn't work as it was losing the resolution. I am looking to convert this
5GB size file to about 500MB.

Regards.
___
Qgis-user mailing list
Qgis-user@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user