Re: [Qgis-user] Search through folders and combine SHP files

2024-05-12 Thread Even Rouault via QGIS-User 

Hi,

The GDAL ogrtindex might be of good help to create an index of various 
vector sources: https://gdal.org/programs/ogrtindex.html


And you can possibly "gdalmanage identify -r some_path" to find all GDAL 
recognizable datasets: https://gdal.org/programs/gdalmanage.html


Even

Le 12/05/2024 à 23:41, Tony Shepherd (FarmMaps NZ) via QGIS-User a écrit :


Hi Nigel

Thank you, that works a treat, although it does fail if I try to merge 
a line feature at the same time as a polygon.  That is not a big deal 
and can get around that.


Your comment, “Then parse these through to the merge vector layer”.  I 
can’t see a way of parsing the filelist.txt to the tool. I am not 
experienced in using python.


Cheers

TIby

*From:*Nigel Berjak - S3 Technologies 
*Sent:* Friday, May 10, 2024 6:03 PM
*To:* Nyall Dawson 
*Cc:* Tony Shepherd (FarmMaps NZ) ; 
qgis-user@lists.osgeo.org

*Subject:* Re: [Qgis-user] Search through folders and combine SHP files

Hi

For your listing, although it would not provide the file type i.e. 
point, line, polygon etc, you could simply create a DOS batch command 
(dir/s *.shp > shapefilelist.txt) to locate all the file and generate 
a list of their locations. Then parse these through to the merge 
vector layer. If you are merging them all into a Geopackage, then file 
type would not be relevant and you could then sort them once they are 
in a single location. I don't think the Geopackage would have a 
limitation on the number of files and records in it.


Perhaps this can assist.

---

Regards,

Nigel Berjak
S3 Technologies
Geographic Information Systems & Large Format Printing specialists
T: +27 33 3423681
F: +27 86 6721242
E: ni...@s3.co.za <mailto:ni...@s3.co.za>
Website: http://www.S3.co.za <http://www.S3.co.za>

Please consider the environment before printing this email.

On 2024-05-10 04:06, Nyall Dawson via QGIS-User wrote:

On Fri, 10 May 2024 at 09:06, Tony Shepherd (FarmMaps NZ) via
QGIS-User mailto:qgis-user@lists.osgeo.org>> wrote:

Hi

I have loads, literally thousands, of SHP, GPKG, TAB files
located in loads of different folders and sub-folders.

Is there a plugin that will trawl through a set of folders,
copy say just the polygons from each file into a new file, and
populate a field with say the filename and folder the
polygon(s) came from?

Essentially combining a heap of files into one with some
metadata about where the data came from?

Why?  I have files from clients dating back about 20 years.
Clients sell properties.  Properties get subdivided.  New
clients come and see us, and often we don't realise we have
GIS info on file for that property from many years ago,
usually under a different name.  It would be handy to see a
polygon on the screen and easily go back to some old GIS data.

Maybe I am missing an easy tool to do this, but I can't see
anything obvious.

You could use the "Merge Vector Layers" tool from the Processing
toolbox, that does everything you want. The trickiest bit would be
getting a master list of ALL your shapefiles/etc you want to
combine and passing this to the tool. It's not going to be easy
from the GUI itself, as that only allows you to add files from a
single folder at once.

I'd run the tool on the files from a single folder, then checkout
the processing history dialog and copy the python command which
corresponds to what you just did. You could then adapt this
command to pass a complete list of all the files you want to
combine. (It's quite straightforward in Python to build a list of
files which recursively match a file pattern -- see eg
https://stackoverflow.com/a/2186565
<https://stackoverflow.com/a/2186565>)

Hope that helps!

Nyall

Cheers

Tony


--

*Tony Shepherd*

/  GeoSpatial Manager @ FarmMaps NZ & Photographer @ Shepherd
Photos/

/*Phone*/*–* 027 435 6193  | /*Website*/*–
*shepherdphotos.co.nz <http://shepherdphotos.co.nz/>

/*Email*/*– Maps*t...@farmmaps.nz <mailto:t...@farmmaps.nz>|
/*Email*/*– Photos* t...@shepherdphotos.co.nz
<mailto:t...@shepherdphotos.co.nz>

/*Facebook*/TonyShepherdPhotos
<https://www.facebook.com/TonyShepherdPhotos>

FM_logo1_small (Custom)

___
QGIS-User mailing list
QGIS-User@lists.osgeo.org <mailto:QGIS-User@lists.osgeo.org>
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
<https://lists.osgeo.org/mailman/listinfo/qgis-user>
Unsubscribe:
https://lists.osgeo

Re: [Qgis-user] Search through folders and combine SHP files

2024-05-12 Thread Tony Shepherd (FarmMaps NZ) via QGIS-User 
Hi Nigel

 

Thank you, that works a treat, although it does fail if I try to merge a line 
feature at the same time as a polygon.  That is not a big deal and can get 
around that.

 

Your comment, “Then parse these through to the merge vector layer”.  I can’t 
see a way of parsing the filelist.txt to the tool. I am not experienced in 
using python.

 

Cheers

TIby

 

 

 

From: Nigel Berjak - S3 Technologies mailto:ni...@s3.co.za> > 
Sent: Friday, May 10, 2024 6:03 PM
To: Nyall Dawson mailto:nyall.daw...@gmail.com> >
Cc: Tony Shepherd (FarmMaps NZ) mailto:t...@farmmaps.nz> >; 
qgis-user@lists.osgeo.org <mailto:qgis-user@lists.osgeo.org> 
Subject: Re: [Qgis-user] Search through folders and combine SHP files

 

Hi

For your listing, although it would not provide the file type i.e. point, line, 
polygon etc, you could simply create a DOS batch command (dir/s *.shp > 
shapefilelist.txt) to locate all the file and generate a list of their 
locations. Then parse these through to the merge vector layer. If you are 
merging them all into a Geopackage, then file type would not be relevant and 
you could then sort them once they are in a single location. I don't think the 
Geopackage would have a limitation on the number of files and records in it.

Perhaps this can assist.

---

Regards,

Nigel Berjak
S3 Technologies
Geographic Information Systems & Large Format Printing specialists
T: +27 33 3423681
F: +27 86 6721242
E:  <mailto:ni...@s3.co.za> ni...@s3.co.za
Website:  <http://www.S3.co.za> http://www.S3.co.za

Please consider the environment before printing this email.

 

On 2024-05-10 04:06, Nyall Dawson via QGIS-User wrote:

 

 

On Fri, 10 May 2024 at 09:06, Tony Shepherd (FarmMaps NZ) via QGIS-User < 
<mailto:qgis-user@lists.osgeo.org> qgis-user@lists.osgeo.org> wrote:

Hi

 

I have loads, literally thousands, of SHP, GPKG, TAB files located in loads of 
different folders and sub-folders.

 

Is there a plugin that will trawl through a set of folders, copy say just the 
polygons from each file into a new file, and populate a field with say the 
filename and folder the polygon(s) came from?

 

Essentially combining a heap of files into one with some metadata about where 
the data came from?

 

Why?  I have files from clients dating back about 20 years.  Clients sell 
properties.  Properties get subdivided.  New clients come and see us, and often 
we don't realise we have GIS info on file for that property from many years 
ago, usually under a different name.  It would be handy to see a polygon on the 
screen and easily go back to some old GIS data.

 

Maybe I am missing an easy tool to do this, but I can't see anything obvious.

 

You could use the "Merge Vector Layers" tool from the Processing toolbox, that 
does everything you want. The trickiest bit would be getting a master list of 
ALL your shapefiles/etc you want to combine and passing this to the tool. It's 
not going to be easy from the GUI itself, as that only allows you to add files 
from a single folder at once.

 

I'd run the tool on the files from a single folder, then checkout the 
processing history dialog and copy the python command which corresponds to what 
you just did. You could then adapt this command to pass a complete list of all 
the files you want to combine. (It's quite straightforward in Python to build a 
list of files which recursively match a file pattern -- see eg  
<https://stackoverflow.com/a/2186565> https://stackoverflow.com/a/2186565 )

 

Hope that helps!

Nyall

 

 

 

 

Cheers

Tony

 

--

Tony Shepherd

  GeoSpatial Manager @ FarmMaps NZ & Photographer @ Shepherd Photos

 

Phone – 027 435 6193  | Website –  <http://shepherdphotos.co.nz/> 
shepherdphotos.co.nz

Email – Maps  <mailto:t...@farmmaps.nz> t...@farmmaps.nz  |  Email – Photos   
<mailto:t...@shepherdphotos.co.nz> t...@shepherdphotos.co.nz

Facebook  <https://www.facebook.com/TonyShepherdPhotos> TonyShepherdPhotos

 

 



 

___
QGIS-User mailing list
 <mailto:QGIS-User@lists.osgeo.org> QGIS-User@lists.osgeo.org
List info:  <https://lists.osgeo.org/mailman/listinfo/qgis-user> 
https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe:  <https://lists.osgeo.org/mailman/listinfo/qgis-user> 
https://lists.osgeo.org/mailman/listinfo/qgis-user

 

___
QGIS-User mailing list
 <mailto:QGIS-User@lists.osgeo.org> QGIS-User@lists.osgeo.org
List info:  <https://lists.osgeo.org/mailman/listinfo/qgis-user> 
https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe:  <https://lists.osgeo.org/mailman/listinfo/qgis-user> 
https://lists.osgeo.org/mailman/listinfo/qgis-user

__

Re: [Qgis-user] Search through folders and combine SHP files

2024-05-10 Thread Nigel Berjak - S3 Technologies via QGIS-User 

Hi

For your listing, although it would not provide the file type i.e. 
point, line, polygon etc, you could simply create a DOS batch command 
(dir/s *.shp > shapefilelist.txt) to locate all the file and generate a 
list of their locations. Then parse these through to the merge vector 
layer. If you are merging them all into a Geopackage, then file type 
would not be relevant and you could then sort them once they are in a 
single location. I don't think the Geopackage would have a limitation on 
the number of files and records in it.


Perhaps this can assist.

---
Regards,

Nigel Berjak
S3 Technologies
Geographic Information Systems & Large Format Printing specialists
T: +27 33 3423681
F: +27 86 6721242
E: ni...@s3.co.za
Website: http://www.S3.co.za

Please consider the environment before printing this email.

On 2024-05-10 04:06, Nyall Dawson via QGIS-User wrote:

On Fri, 10 May 2024 at 09:06, Tony Shepherd (FarmMaps NZ) via QGIS-User 
 wrote:



Hi

I have loads, literally thousands, of SHP, GPKG, TAB files located in 
loads of different folders and sub-folders.


Is there a plugin that will trawl through a set of folders, copy say 
just the polygons from each file into a new file, and populate a field 
with say the filename and folder the polygon(s) came from?


Essentially combining a heap of files into one with some metadata 
about where the data came from?


Why?  I have files from clients dating back about 20 years.  Clients 
sell properties.  Properties get subdivided.  New clients come and see 
us, and often we don't realise we have GIS info on file for that 
property from many years ago, usually under a different name.  It 
would be handy to see a polygon on the screen and easily go back to 
some old GIS data.


Maybe I am missing an easy tool to do this, but I can't see anything 
obvious.


You could use the "Merge Vector Layers" tool from the Processing 
toolbox, that does everything you want. The trickiest bit would be 
getting a master list of ALL your shapefiles/etc you want to combine 
and passing this to the tool. It's not going to be easy from the GUI 
itself, as that only allows you to add files from a single folder at 
once.


I'd run the tool on the files from a single folder, then checkout the 
processing history dialog and copy the python command which corresponds 
to what you just did. You could then adapt this command to pass a 
complete list of all the files you want to combine. (It's quite 
straightforward in Python to build a list of files which recursively 
match a file pattern -- see eg https://stackoverflow.com/a/2186565 )


Hope that helps!
Nyall


Cheers

Tony

--

Tony Shepherd

_  GeoSpatial Manager @ FarmMaps NZ & Photographer @ Shepherd Photos_

_Phone_ - 027 435 6193  | _Website_ - shepherdphotos.co.nz [1]

_Email_ - Maps t...@farmmaps.nz  |  _Email_ - Photos  
t...@shepherdphotos.co.nz


_Facebook_ TonyShepherdPhotos [2]

___
QGIS-User mailing list
QGIS-User@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user


___
QGIS-User mailing list
QGIS-User@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user



Links:
--
[1] http://shepherdphotos.co.nz/
[2] https://www.facebook.com/TonyShepherdPhotos___
QGIS-User mailing list
QGIS-User@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

Re: [Qgis-user] Search through folders and combine SHP files

2024-05-09 Thread Nyall Dawson via QGIS-User 
On Fri, 10 May 2024 at 09:06, Tony Shepherd (FarmMaps NZ) via QGIS-User <
qgis-user@lists.osgeo.org> wrote:

> Hi
>
>
>
> I have loads, literally thousands, of SHP, GPKG, TAB files located in
> loads of different folders and sub-folders.
>
>
>
> Is there a plugin that will trawl through a set of folders, copy say just
> the polygons from each file into a new file, and populate a field with say
> the filename and folder the polygon(s) came from?
>
>
>
> Essentially combining a heap of files into one with some metadata about
> where the data came from?
>
>
>
> Why?  I have files from clients dating back about 20 years.  Clients sell
> properties.  Properties get subdivided.  New clients come and see us, and
> often we don’t realise we have GIS info on file for that property from many
> years ago, usually under a different name.  It would be handy to see a
> polygon on the screen and easily go back to some old GIS data.
>
>
>
> Maybe I am missing an easy tool to do this, but I can’t see anything
> obvious.
>

You could use the "Merge Vector Layers" tool from the Processing toolbox,
that does everything you want. The trickiest bit would be getting a master
list of ALL your shapefiles/etc you want to combine and passing this to the
tool. It's not going to be easy from the GUI itself, as that only allows
you to add files from a single folder at once.

I'd run the tool on the files from a single folder, then checkout the
processing history dialog and copy the python command which corresponds to
what you just did. You could then adapt this command to pass a complete
list of all the files you want to combine. (It's quite straightforward in
Python to build a list of files which recursively match a file pattern --
see eg https://stackoverflow.com/a/2186565 )

Hope that helps!
Nyall


>
>
>
>
> Cheers
>
> Tony
>
>
>
>
> --
>
> *Tony Shepherd*
>
> *  GeoSpatial Manager @ FarmMaps NZ & Photographer @ Shepherd Photos*
>
>
>
> *Phone* *–* 027 435 6193  | *Website – *shepherdphotos.co.nz
>
> *Email** – Maps* t...@farmmaps.nz  |  *Email – Photos*
> t...@shepherdphotos.co.nz
>
> *Facebook* TonyShepherdPhotos
> 
>
>
>
>
>
> [image: FM_logo1_small (Custom)]
>
>
> ___
> QGIS-User mailing list
> QGIS-User@lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>
___
QGIS-User mailing list
QGIS-User@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

Re: [Qgis-user] Search through folders and combine SHP files

2024-05-09 Thread chris hermansen via QGIS-User 
Tony and list,

On Thu, May 9, 2024, 16:06 Tony Shepherd (FarmMaps NZ) via QGIS-User <
qgis-user@lists.osgeo.org> wrote:

> Hi
>
>
>
> I have loads, literally thousands, of SHP, GPKG, TAB files located in
> loads of different folders and sub-folders.
>
>
>
> Is there a plugin that will trawl through a set of folders, copy say just
> the polygons from each file into a new file, and populate a field with say
> the filename and folder the polygon(s) came from?
>
>
>
> Essentially combining a heap of files into one with some metadata about
> where the data came from?
>
>
>
> Why?  I have files from clients dating back about 20 years.  Clients sell
> properties.  Properties get subdivided.  New clients come and see us, and
> often we don’t realise we have GIS info on file for that property from many
> years ago, usually under a different name.  It would be handy to see a
> polygon on the screen and easily go back to some old GIS data.
>
>
>
> Maybe I am missing an easy tool to do this, but I can’t see anything
> obvious.
>

Have you considered creating a PostGIS database, where each file imported
is in a table named for the source? You could also create a companion
metadata table.

You would script something like this with shp2pgsql

https://postgis.net/docs/using_postgis_dbmanagement.html

To load the spatial data and likely the psql client with CREATE TABLE  and
INSERT statements in an SQL script.

If you're on Linux or OS/X you would use the find command to create the
shell script. Probably something similar in Power Shell if you're on
Windows.

>
___
QGIS-User mailing list
QGIS-User@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

[Qgis-user] Search through folders and combine SHP files

2024-05-09 Thread Tony Shepherd (FarmMaps NZ) via QGIS-User 
Hi

 

I have loads, literally thousands, of SHP, GPKG, TAB files located in loads
of different folders and sub-folders.

 

Is there a plugin that will trawl through a set of folders, copy say just
the polygons from each file into a new file, and populate a field with say
the filename and folder the polygon(s) came from?

 

Essentially combining a heap of files into one with some metadata about
where the data came from?

 

Why?  I have files from clients dating back about 20 years.  Clients sell
properties.  Properties get subdivided.  New clients come and see us, and
often we don't realise we have GIS info on file for that property from many
years ago, usually under a different name.  It would be handy to see a
polygon on the screen and easily go back to some old GIS data.

 

Maybe I am missing an easy tool to do this, but I can't see anything
obvious.

 

 

Cheers

Tony

 


--

Tony Shepherd

  GeoSpatial Manager @ FarmMaps NZ & Photographer @ Shepherd Photos

 

Phone - 027 435 6193  | Website -  
shepherdphotos.co.nz

Email - Maps   t...@farmmaps.nz  |  Email - Photos
 t...@shepherdphotos.co.nz

Facebook   TonyShepherdPhotos

 

 



 

___
QGIS-User mailing list
QGIS-User@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user