Re: [Qgis-user] ctrl-c selection issue

[Tobias Wendorff]

Hi,

Am So, 14.01.2018, 02:23 schrieb Mike Flannigan:
>
> I would recommend not using Excel to modify CSV files.  That
> program has a reputation for changing numbers to dates and
> performing other modifications to the file contents that may
> not be wanted.

By the way: I'm wondering, why native Excel support didn't make
it into QGIS3. I think, FreeXL is integrated into GDAL/OGR, so
we could support it.

There was a plugin for QGIS2, I hope it will get ported to QGIS3.

Best regards,
Tobias

___
Qgis-user mailing list
Qgis-user@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

Re: [Qgis-user] ctrl-c selection issue

[Mike Flannigan]

Hi,

I would recommend not using Excel to modify CSV files.  That
program has a reputation for changing numbers to dates and
performing other modifications to the file contents that may
not be wanted.

Personally I use the Text::CSV module in Perl, but there are
many other methods to modify a CSV file without the changes that
Excel imposes on you.


Mike


On 1/9/2018 4:37 PM, qgis-user-requ...@lists.osgeo.org wrote:

Hi,
  


Using qgis 2.14.18 I have been working today with a large csv file (7.5 mln
objects). I just need selections of this database which I can then modify in
excel. So I opened the attribute table and made a selection based on a
specific value for an attribute, and copied that selection (about 30.000
objects) via the clipboard (ctrl-c) to a new Microsoft Excel file, then
saved the excel file and repeated the process for another selection. This
worked perfect for some time. However, some selections won't copy: when I
press the copy button the programs turn to 'not responding' and won't snap
out of it anymore.  Selection size when this happens is about the same
(about 20k to 35k objects).

  


What could be the problem? Could the data be corrupted? How can I check what
is wrong?

  


Thank you

Gr.

Jeroen


___
Qgis-user mailing list
Qgis-user@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

Re: [Qgis-user] ctrl-c selection issue

[Fernando M. Roxo da Motta]

On Wed, 10 Jan 2018 12:26:42 +0100, "Jeroen Hovens" <m...@groenebij.nl>
wrote:

Hi Jeroen,

  If you allow, I will suggest you two possible approach.  You didn't
specify your OS platform.  I tested on Linux but the first approach
should work anywhere.

  First one using qgis only.   Load your CSV file in qgis as a table
only, no geom.  At least I did not found one.   After loading save it
as a SQLite database (save as...).   It should sabe and load the
database as a table.   Unload the CSV file.

  At this point double click the sqlite layer and got to general tab.
There click on Query Builder button and use an expression like:

"Gem2017" = 310

You can just apply this selection or, if you wish, test it to get the
number of lines selected.   When you apply a selection like this your
layer is composed of only those selected one.   Just save it.  For
example save as a "MS Office Open XML spreadsheet", that is one of the
offered formats in my Qgis.

  After that you just change the desired value and so on.

  Another approach is pure shell approach.

  I am attaching to the message a shell script that will:

 1- Convert the CVS from DOS (CRLF) to Linux (LF) format
 2- Sort it by the 5th column (the Gem2017)
 3- Get the unique values for that column
 4- Iterate over the CSV file extracting all lines with
each unique Gem2017, saving it with the name:
Gem2017-.csv

  I think it is possible to bach convert them to Excel or LibreOffice
calc format, just did not looked at it.

  If you are using Windows, you can try this in a Cygwin or even the
Linux subsystem for Windows 10 (I never saw this last one, it is just a
guess).

  Hope this helps.


P.S. - If you prefer to deal with database it self, a SQL like:

   select * from pc6hnr20170801_gwb where gem2017 = 310;

   Will get you all entries with that value of gem2017.

> At the moment I make the selection using the Select by expression
> tool. I then choose one of the  Gem2017 values (each object is an
> address, and has one of 388 unique Gem217 values). I want to select
> all objects bwith one unique Gem2017 value. Making the selection is
> no problem, I can make any selection I want. Making the selection
> usually takes less then a minute. The problem is in copying specific
> selections to the clipboard or saving a selection as a new file.
> Copying a selection of about 20.000 objects from the 7,5 mln takes
> about a second. Unless the issue arises, then the copy process takes
> forever
> 
> I have no experience with Postgis or using databases. I tried saving
> the csv file as a sqlite, but then had no idea how to re-open that
> file. The dialogbox got me all confused ☹
> 
> 
> 
> 
> -Oorspronkelijk bericht-
> Van: Richard Duivenvoorde [mailto:rdmaili...@duif.net] 
> Verzonden: woensdag 10 januari 2018 10:21
> Aan: Jeroen Hovens <m...@groenebij.nl>; 'Ramon Andinach'
> <cust...@westnet.com.au>; 'qgis-user' <qgis-user@lists.osgeo.org>
> Onderwerp: Re: [Qgis-user] ctrl-c selection issue
> 
> 
> Hi Jeroen,
> 
> Loading a csv in QGIS is not optimal. I think csv does not have an
> index, so when you create a small selection, QGIS has to run over all
> those 7,5 mln object every time.
> 
> Not sure how you do your 'selecting'?
> Is it spatial or based on zip/postcode/buurtnr/wijknr?
> 
> I think I would either load it in Postgis (if available) or save as
> GeoPackage (better use QGIS 2.99 for that) or if all fails, create a
> shapefile for it. AND then create indexes on desired columns.
> 
> IF you are searching for strings like zipcodes in that list, I think
> tools like grep (available in OSGeo4W too!) work best:
> 
> Eg searching fro 3 zip codes is milliseconds work:
> 
> grep '2022ZJ\|2211ZE\|1016XX' pc6hnr20170801_gwb.csv
> 
> If all fails, give me a call.
> 
> Regards,
> 
> Richard Duivenvoorde
> 
> On 10-01-18 00:33, Jeroen Hovens wrote:
> > Hmm, I can’t use Edit > Paste As, because qgis does not finish the 
> > Edit  
> >> Copy command.  
> > 
> >  
> > 
> > The csv file is about 270 MB, 7,5 mln objects with 5 attributes, no 
> > x-y data.
> > 
> > If you want to recreate the issue, the file can be found here:
> > 
> > https://www.cbs.nl/nl-nl/maatwerk/2017/38/buurt-wijk-en-gemeente-2017-
> > voor-postcode-huisnummer
> > 
> > I run into this issue when selecting all objects with Attribute
> > Gem217 = 310
> > 
> > 355 also has this issue. 321 and 356 work without a problem.
> > 
> >  
> > 
> > I am using qgis here just as a means to chop the csv file into
> > smaller selections which I can then use to edit, add attributes and
> > data to and make

Re: [Qgis-user] ctrl-c selection issue

[Jeroen Hovens]

Update: Solution.

After saving the csv file as a shape-file and then loading the .dbf file into 
qgis everything worked as intended.

Although Qgis couldn’t create the .shp file and the .dbf file actually ‘grew’ 
from 227 MB (.cvs) to 9.4 GB (.dbf), I can now copy any selection without any 
problem.

 

And along the way I discovered the option Split Vector Layer, which is what I 
was doing, however I was doing it one at the time, instead of creating 388 new 
layers at once (because I need just a few).

And Split Vector Layer doesn’t work fort the .csv file or the .dbf by the way, 
it is probably missing some information.

 

So, still no idea what is going wrong in the .cvs file, but I have a workable 
solution.

Thank you all for your thoughts.

 

Gr.

Jeroen

 

Van: Saber Razmjooei [mailto:saber.razmjo...@lutraconsulting.co.uk] 
Verzonden: woensdag 10 januari 2018 12:33
Aan: Jeroen Hovens <m...@groenebij.nl>
CC: Richard  Duivenvoorde <rich...@duif.net>; Ramon Andinach 
<cust...@westnet.com.au>; qgis-user <qgis-user@lists.osgeo.org>
Onderwerp: Re: [Qgis-user] ctrl-c selection issue

 

Drag-n-drop the file on QGIS canvas!

 

Regards

Saber

 

On 10 January 2018 at 11:26, Jeroen Hovens <m...@groenebij.nl 
<mailto:m...@groenebij.nl> > wrote:

At the moment I make the selection using the Select by expression tool. I then 
choose one of the  Gem2017 values (each object is an address, and has one of 
388 unique Gem217 values). I want to select all objects bwith one unique 
Gem2017 value.
Making the selection is no problem, I can make any selection I want. Making the 
selection usually takes less then a minute.
The problem is in copying specific selections to the clipboard or saving a 
selection as a new file. Copying a selection of about 20.000 objects from the 
7,5 mln takes about a second. Unless the issue arises, then the copy process 
takes forever

I have no experience with Postgis or using databases. I tried saving the csv 
file as a sqlite, but then had no idea how to re-open that file. The dialogbox 
got me all confused ☹




-Oorspronkelijk bericht-
Van: Richard Duivenvoorde [mailto:rdmaili...@duif.net 
<mailto:rdmaili...@duif.net> ]
Verzonden: woensdag 10 januari 2018 10:21
Aan: Jeroen Hovens <m...@groenebij.nl <mailto:m...@groenebij.nl> >; 'Ramon 
Andinach' <cust...@westnet.com.au <mailto:cust...@westnet.com.au> >; 
'qgis-user' <qgis-user@lists.osgeo.org <mailto:qgis-user@lists.osgeo.org> >
Onderwerp: Re: [Qgis-user] ctrl-c selection issue



Hi Jeroen,

Loading a csv in QGIS is not optimal. I think csv does not have an index, so 
when you create a small selection, QGIS has to run over all those 7,5 mln 
object every time.

Not sure how you do your 'selecting'?
Is it spatial or based on zip/postcode/buurtnr/wijknr?

I think I would either load it in Postgis (if available) or save as GeoPackage 
(better use QGIS 2.99 for that) or if all fails, create a shapefile for it. AND 
then create indexes on desired columns.

IF you are searching for strings like zipcodes in that list, I think tools like 
grep (available in OSGeo4W too!) work best:

Eg searching fro 3 zip codes is milliseconds work:

grep '2022ZJ\|2211ZE\|1016XX' pc6hnr20170801_gwb.csv

If all fails, give me a call.

Regards,

Richard Duivenvoorde

On 10-01-18 00:33, Jeroen Hovens wrote:
> Hmm, I can’t use Edit > Paste As, because qgis does not finish the
> Edit
>> Copy command.
>
>
>
> The csv file is about 270 MB, 7,5 mln objects with 5 attributes, no
> x-y data.
>
> If you want to recreate the issue, the file can be found here:
>
> https://www.cbs.nl/nl-nl/maatwerk/2017/38/buurt-wijk-en-gemeente-2017-
> voor-postcode-huisnummer
>
> I run into this issue when selecting all objects with Attribute Gem217
> = 310
>
> 355 also has this issue. 321 and 356 work without a problem.
>
>
>
> I am using qgis here just as a means to chop the csv file into smaller
> selections which I can then use to edit, add attributes and data to
> and make maps of them.
>
>
>
> Gr.
>
> Jeroen
>
>
>
> *Van:* Qgis-user [mailto:qgis-user-boun...@lists.osgeo.org 
> <mailto:qgis-user-boun...@lists.osgeo.org> ] *Namens
> *Ramon Andinach
> *Verzonden:* dinsdag 9 januari 2018 23:35
> *Aan:* qgis-user <qgis-user@lists.osgeo.org 
> <mailto:qgis-user@lists.osgeo.org> >
> *Onderwerp:* Re: [Qgis-user] ctrl-c selection issue
>
>
>
> Just so we’ve got just about everything covered, does this happen for
> “Edit-> Paste As”?
>
>
>
> Also looking at the size of your original CSV, I’d go along with
> Jeroen, and try and import that csv into some other format first.
>
> If you can find a way of making the selection without going near the
> attribute table that will help too.
>
>
>
>
>
> Regards,

Re: [Qgis-user] ctrl-c selection issue

[Saber Razmjooei]

Drag-n-drop the file on QGIS canvas!

Regards
Saber

On 10 January 2018 at 11:26, Jeroen Hovens <m...@groenebij.nl> wrote:

> At the moment I make the selection using the Select by expression tool. I
> then choose one of the  Gem2017 values (each object is an address, and has
> one of 388 unique Gem217 values). I want to select all objects bwith one
> unique Gem2017 value.
> Making the selection is no problem, I can make any selection I want.
> Making the selection usually takes less then a minute.
> The problem is in copying specific selections to the clipboard or saving a
> selection as a new file. Copying a selection of about 20.000 objects from
> the 7,5 mln takes about a second. Unless the issue arises, then the copy
> process takes forever
>
> I have no experience with Postgis or using databases. I tried saving the
> csv file as a sqlite, but then had no idea how to re-open that file. The
> dialogbox got me all confused ☹
>
>
>
>
> -Oorspronkelijk bericht-
> Van: Richard Duivenvoorde [mailto:rdmaili...@duif.net]
> Verzonden: woensdag 10 januari 2018 10:21
> Aan: Jeroen Hovens <m...@groenebij.nl>; 'Ramon Andinach' <
> cust...@westnet.com.au>; 'qgis-user' <qgis-user@lists.osgeo.org>
> Onderwerp: Re: [Qgis-user] ctrl-c selection issue
>
>
> Hi Jeroen,
>
> Loading a csv in QGIS is not optimal. I think csv does not have an index,
> so when you create a small selection, QGIS has to run over all those 7,5
> mln object every time.
>
> Not sure how you do your 'selecting'?
> Is it spatial or based on zip/postcode/buurtnr/wijknr?
>
> I think I would either load it in Postgis (if available) or save as
> GeoPackage (better use QGIS 2.99 for that) or if all fails, create a
> shapefile for it. AND then create indexes on desired columns.
>
> IF you are searching for strings like zipcodes in that list, I think tools
> like grep (available in OSGeo4W too!) work best:
>
> Eg searching fro 3 zip codes is milliseconds work:
>
> grep '2022ZJ\|2211ZE\|1016XX' pc6hnr20170801_gwb.csv
>
> If all fails, give me a call.
>
> Regards,
>
> Richard Duivenvoorde
>
> On 10-01-18 00:33, Jeroen Hovens wrote:
> > Hmm, I can’t use Edit > Paste As, because qgis does not finish the
> > Edit
> >> Copy command.
> >
> >
> >
> > The csv file is about 270 MB, 7,5 mln objects with 5 attributes, no
> > x-y data.
> >
> > If you want to recreate the issue, the file can be found here:
> >
> > https://www.cbs.nl/nl-nl/maatwerk/2017/38/buurt-wijk-en-gemeente-2017-
> > voor-postcode-huisnummer
> >
> > I run into this issue when selecting all objects with Attribute Gem217
> > = 310
> >
> > 355 also has this issue. 321 and 356 work without a problem.
> >
> >
> >
> > I am using qgis here just as a means to chop the csv file into smaller
> > selections which I can then use to edit, add attributes and data to
> > and make maps of them.
> >
> >
> >
> > Gr.
> >
> > Jeroen
> >
> >
> >
> > *Van:* Qgis-user [mailto:qgis-user-boun...@lists.osgeo.org] *Namens
> > *Ramon Andinach
> > *Verzonden:* dinsdag 9 januari 2018 23:35
> > *Aan:* qgis-user <qgis-user@lists.osgeo.org>
> > *Onderwerp:* Re: [Qgis-user] ctrl-c selection issue
> >
> >
> >
> > Just so we’ve got just about everything covered, does this happen for
> > “Edit-> Paste As”?
> >
> >
> >
> > Also looking at the size of your original CSV, I’d go along with
> > Jeroen, and try and import that csv into some other format first.
> >
> > If you can find a way of making the selection without going near the
> > attribute table that will help too.
> >
> >
> >
> >
> >
> > Regards,
> >
> > Ramon.
> >
> > On 10 Jan 2018, at 06:26, Jeroen Hovens <m...@groenebij.nl
> > <mailto:m...@groenebij.nl>> wrote:
> >
> >
> >
> > Thanks for the ideas.
> >
> >
> >
> > The option to save the selection as new csv file as Ramon suggested,
> > doesn’t work; qgis goes into ‘not responding’ and no csv file is
> > added to the layer list. However, a csv file is made, lacking a
> > decent number of objects (first test 15095 of 18582 objects and
> > second test is 15179 of 18582 objects, so missing about 3,5k
> > objects). This csv file can be opened manually in qgis.
> >
> > And I checked: this options does work for selections where the
> > ctrl-c option also worked. Then the new cvs layer is added to the
> >

Re: [Qgis-user] ctrl-c selection issue

[Jeroen Hovens]

At the moment I make the selection using the Select by expression tool. I then 
choose one of the  Gem2017 values (each object is an address, and has one of 
388 unique Gem217 values). I want to select all objects bwith one unique 
Gem2017 value.
Making the selection is no problem, I can make any selection I want. Making the 
selection usually takes less then a minute.
The problem is in copying specific selections to the clipboard or saving a 
selection as a new file. Copying a selection of about 20.000 objects from the 
7,5 mln takes about a second. Unless the issue arises, then the copy process 
takes forever

I have no experience with Postgis or using databases. I tried saving the csv 
file as a sqlite, but then had no idea how to re-open that file. The dialogbox 
got me all confused ☹




-Oorspronkelijk bericht-
Van: Richard Duivenvoorde [mailto:rdmaili...@duif.net] 
Verzonden: woensdag 10 januari 2018 10:21
Aan: Jeroen Hovens <m...@groenebij.nl>; 'Ramon Andinach' 
<cust...@westnet.com.au>; 'qgis-user' <qgis-user@lists.osgeo.org>
Onderwerp: Re: [Qgis-user] ctrl-c selection issue


Hi Jeroen,

Loading a csv in QGIS is not optimal. I think csv does not have an index, so 
when you create a small selection, QGIS has to run over all those 7,5 mln 
object every time.

Not sure how you do your 'selecting'?
Is it spatial or based on zip/postcode/buurtnr/wijknr?

I think I would either load it in Postgis (if available) or save as GeoPackage 
(better use QGIS 2.99 for that) or if all fails, create a shapefile for it. AND 
then create indexes on desired columns.

IF you are searching for strings like zipcodes in that list, I think tools like 
grep (available in OSGeo4W too!) work best:

Eg searching fro 3 zip codes is milliseconds work:

grep '2022ZJ\|2211ZE\|1016XX' pc6hnr20170801_gwb.csv

If all fails, give me a call.

Regards,

Richard Duivenvoorde

On 10-01-18 00:33, Jeroen Hovens wrote:
> Hmm, I can’t use Edit > Paste As, because qgis does not finish the 
> Edit
>> Copy command.
> 
>  
> 
> The csv file is about 270 MB, 7,5 mln objects with 5 attributes, no 
> x-y data.
> 
> If you want to recreate the issue, the file can be found here:
> 
> https://www.cbs.nl/nl-nl/maatwerk/2017/38/buurt-wijk-en-gemeente-2017-
> voor-postcode-huisnummer
> 
> I run into this issue when selecting all objects with Attribute Gem217 
> = 310
> 
> 355 also has this issue. 321 and 356 work without a problem.
> 
>  
> 
> I am using qgis here just as a means to chop the csv file into smaller 
> selections which I can then use to edit, add attributes and data to 
> and make maps of them.
> 
>  
> 
> Gr.
> 
> Jeroen
> 
>  
> 
> *Van:* Qgis-user [mailto:qgis-user-boun...@lists.osgeo.org] *Namens 
> *Ramon Andinach
> *Verzonden:* dinsdag 9 januari 2018 23:35
> *Aan:* qgis-user <qgis-user@lists.osgeo.org>
> *Onderwerp:* Re: [Qgis-user] ctrl-c selection issue
> 
>  
> 
> Just so we’ve got just about everything covered, does this happen for 
> “Edit-> Paste As”?
> 
>  
> 
> Also looking at the size of your original CSV, I’d go along with 
> Jeroen, and try and import that csv into some other format first.
> 
> If you can find a way of making the selection without going near the 
> attribute table that will help too.
> 
>  
> 
>  
> 
> Regards,
> 
> Ramon.
> 
> On 10 Jan 2018, at 06:26, Jeroen Hovens <m...@groenebij.nl
> <mailto:m...@groenebij.nl>> wrote:
> 
>  
> 
> Thanks for the ideas.
> 
>  
> 
> The option to save the selection as new csv file as Ramon suggested,
> doesn’t work; qgis goes into ‘not responding’ and no csv file is
> added to the layer list. However, a csv file is made, lacking a
> decent number of objects (first test 15095 of 18582 objects and
> second test is 15179 of 18582 objects, so missing about 3,5k
> objects). This csv file can be opened manually in qgis.
> 
> And I checked: this options does work for selections where the
> ctrl-c option also worked. Then the new cvs layer is added to the
> layer list in no time.
> 
>  
> 
> Is there some sort of log file in qgis that might give clues to the
> ‘not responding?
> 
>  
> 
> @Kirk
> 
> I did close and reopen qgis multiple times, even shut down the pc
> entirely, but to no effect. Selections that did allow for copying
> earlier, still work: I can make that selection again and ctrl-c
> works fine. Seems to me this rules out memory-issues.
> 
>  
> 
> Gr.
> 
> Jeroen
> 
>  
> 
>  
> 
> *Van:* Qgis-user
> [mailto:qgis-user-boun...@lists.osgeo.org] *Namens *Kirk Schmidt
> *Ve

Re: [Qgis-user] ctrl-c selection issue

[Richard Duivenvoorde]

Hi Jeroen,

Loading a csv in QGIS is not optimal. I think csv does not have an
index, so when you create a small selection, QGIS has to run over all
those 7,5 mln object every time.

Not sure how you do your 'selecting'?
Is it spatial or based on zip/postcode/buurtnr/wijknr?

I think I would either load it in Postgis (if available) or save as
GeoPackage (better use QGIS 2.99 for that) or if all fails, create a
shapefile for it. AND then create indexes on desired columns.

IF you are searching for strings like zipcodes in that list, I think
tools like grep (available in OSGeo4W too!) work best:

Eg searching fro 3 zip codes is milliseconds work:

grep '2022ZJ\|2211ZE\|1016XX' pc6hnr20170801_gwb.csv

If all fails, give me a call.

Regards,

Richard Duivenvoorde

On 10-01-18 00:33, Jeroen Hovens wrote:
> Hmm, I can’t use Edit > Paste As, because qgis does not finish the Edit
>> Copy command.
> 
>  
> 
> The csv file is about 270 MB, 7,5 mln objects with 5 attributes, no x-y
> data.
> 
> If you want to recreate the issue, the file can be found here:
> 
> https://www.cbs.nl/nl-nl/maatwerk/2017/38/buurt-wijk-en-gemeente-2017-voor-postcode-huisnummer
> 
> I run into this issue when selecting all objects with Attribute Gem217 = 310
> 
> 355 also has this issue. 321 and 356 work without a problem.
> 
>  
> 
> I am using qgis here just as a means to chop the csv file into smaller
> selections which I can then use to edit, add attributes and data to and
> make maps of them.
> 
>  
> 
> Gr.
> 
> Jeroen
> 
>  
> 
> *Van:* Qgis-user [mailto:qgis-user-boun...@lists.osgeo.org] *Namens
> *Ramon Andinach
> *Verzonden:* dinsdag 9 januari 2018 23:35
> *Aan:* qgis-user <qgis-user@lists.osgeo.org>
> *Onderwerp:* Re: [Qgis-user] ctrl-c selection issue
> 
>  
> 
> Just so we’ve got just about everything covered, does this happen for
> “Edit-> Paste As”?
> 
>  
> 
> Also looking at the size of your original CSV, I’d go along with Jeroen,
> and try and import that csv into some other format first. 
> 
> If you can find a way of making the selection without going near the
> attribute table that will help too.
> 
>  
> 
>  
> 
> Regards,
> 
> Ramon.
> 
> On 10 Jan 2018, at 06:26, Jeroen Hovens <m...@groenebij.nl
> <mailto:m...@groenebij.nl>> wrote:
> 
>  
> 
> Thanks for the ideas.
> 
>  
> 
> The option to save the selection as new csv file as Ramon suggested,
> doesn’t work; qgis goes into ‘not responding’ and no csv file is
> added to the layer list. However, a csv file is made, lacking a
> decent number of objects (first test 15095 of 18582 objects and
> second test is 15179 of 18582 objects, so missing about 3,5k
> objects). This csv file can be opened manually in qgis.
> 
> And I checked: this options does work for selections where the
> ctrl-c option also worked. Then the new cvs layer is added to the
> layer list in no time.
> 
>  
> 
> Is there some sort of log file in qgis that might give clues to the
> ‘not responding?
> 
>  
> 
> @Kirk
> 
> I did close and reopen qgis multiple times, even shut down the pc
> entirely, but to no effect. Selections that did allow for copying
> earlier, still work: I can make that selection again and ctrl-c
> works fine. Seems to me this rules out memory-issues.
> 
>      
> 
> Gr.
> 
> Jeroen
> 
>  
> 
>  
> 
> *Van:* Qgis-user
> [mailto:qgis-user-boun...@lists.osgeo.org] *Namens *Kirk Schmidt
> *Verzonden:* dinsdag 9 januari 2018 22:20
> *Aan:* qgis-user@lists.osgeo.org <mailto:qgis-user@lists.osgeo.org>
> *Onderwerp:* Re: [Qgis-user] ctrl-c selection issue
> 
>  
> 
> After a few selections, you may have to close and reopen qgis to
> purge the memory as well. Importing a csv is memory intensive so
> another approach is to try saving as a spatial database (SQLite) or
> even a shapefile.  Once written, you can load the spatial file and
> you should have lots of memory to keep selecting points. 
> 
> 
> 
> Kirk
> 
> On 1/9/2018 4:58 PM, Ramon Andinach wrote:
> 
> I’m not sure what the problem is, but if I were doing that
> repeatedly, I’d probably use Layer -> Save As, tick the Save
> Only Selected Features box and save it to CSV.  Then muck about
> with excel later. 
> 
>  
> 
> This way ought to be a bit lighter on the memory. 
> 
>  
> 
>  
> 
> Ramon.
> 
> On 10 Jan 2018, at 04:25, Jeroen Hovens <m...

Re: [Qgis-user] ctrl-c selection issue

[Jeroen Hovens]

Hmm, I can’t use Edit > Paste As, because qgis does not finish the Edit > Copy 
command.

 

The csv file is about 270 MB, 7,5 mln objects with 5 attributes, no x-y data. 

If you want to recreate the issue, the file can be found here:

https://www.cbs.nl/nl-nl/maatwerk/2017/38/buurt-wijk-en-gemeente-2017-voor-postcode-huisnummer

I run into this issue when selecting all objects with Attribute Gem217 = 310

355 also has this issue. 321 and 356 work without a problem.

 

I am using qgis here just as a means to chop the csv file into smaller 
selections which I can then use to edit, add attributes and data to and make 
maps of them.

 

Gr.

Jeroen

 

Van: Qgis-user [mailto:qgis-user-boun...@lists.osgeo.org] Namens Ramon Andinach
Verzonden: dinsdag 9 januari 2018 23:35
Aan: qgis-user <qgis-user@lists.osgeo.org>
Onderwerp: Re: [Qgis-user] ctrl-c selection issue

 

Just so we’ve got just about everything covered, does this happen for “Edit-> 
Paste As”?

 

Also looking at the size of your original CSV, I’d go along with Jeroen, and 
try and import that csv into some other format first. 

If you can find a way of making the selection without going near the attribute 
table that will help too.

 

 

Regards,

Ramon.

On 10 Jan 2018, at 06:26, Jeroen Hovens <m...@groenebij.nl 
<mailto:m...@groenebij.nl> > wrote:

 

Thanks for the ideas.

 

The option to save the selection as new csv file as Ramon suggested, doesn’t 
work; qgis goes into ‘not responding’ and no csv file is added to the layer 
list. However, a csv file is made, lacking a decent number of objects (first 
test 15095 of 18582 objects and second test is 15179 of 18582 objects, so 
missing about 3,5k objects). This csv file can be opened manually in qgis.

And I checked: this options does work for selections where the ctrl-c option 
also worked. Then the new cvs layer is added to the layer list in no time.

 

Is there some sort of log file in qgis that might give clues to the ‘not 
responding?

 

@Kirk

I did close and reopen qgis multiple times, even shut down the pc entirely, but 
to no effect. Selections that did allow for copying earlier, still work: I can 
make that selection again and ctrl-c works fine. Seems to me this rules out 
memory-issues.

 

Gr.

Jeroen

 

 

Van: Qgis-user [mailto:qgis-user-boun...@lists.osgeo.org] Namens Kirk Schmidt
Verzonden: dinsdag 9 januari 2018 22:20
Aan: qgis-user@lists.osgeo.org <mailto:qgis-user@lists.osgeo.org> 
Onderwerp: Re: [Qgis-user] ctrl-c selection issue

 

After a few selections, you may have to close and reopen qgis to purge the 
memory as well. Importing a csv is memory intensive so another approach is to 
try saving as a spatial database (SQLite) or even a shapefile.  Once written, 
you can load the spatial file and you should have lots of memory to keep 
selecting points. 





Kirk

On 1/9/2018 4:58 PM, Ramon Andinach wrote:

I’m not sure what the problem is, but if I were doing that repeatedly, I’d 
probably use Layer -> Save As, tick the Save Only Selected Features box and 
save it to CSV.  Then muck about with excel later. 

 

This way ought to be a bit lighter on the memory. 

 

 

Ramon.

On 10 Jan 2018, at 04:25, Jeroen Hovens < <mailto:m...@groenebij.nl> 
m...@groenebij.nl> wrote:

 

Hi,

 

Using qgis 2.14.18 I have been working today with a large csv file (7.5 mln 
objects). I just need selections of this database which I can then modify in 
excel. So I opened the attribute table and made a selection based on a specific 
value for an attribute, and copied that selection (about 30.000 objects) via 
the clipboard (ctrl-c) to a new Microsoft Excel file, then saved the excel file 
and repeated the process for another selection. This worked perfect for some 
time. However, some selections won’t copy: when I press the copy button the 
programs turn to ‘not responding’ and won’t snap out of it anymore.  Selection 
size when this happens is about the same (about 20k to 35k objects).

 

What could be the problem? Could the data be corrupted? How can I check what is 
wrong?

 

Thank you

Gr.

Jeroen 

 

___
Qgis-user mailing list
 <mailto:Qgis-user@lists.osgeo.org> Qgis-user@lists.osgeo.org
List info:  <https://lists.osgeo.org/mailman/listinfo/qgis-user> 
https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe:  <https://lists.osgeo.org/mailman/listinfo/qgis-user> 
https://lists.osgeo.org/mailman/listinfo/qgis-user

 







___
Qgis-user mailing list
 <mailto:Qgis-user@lists.osgeo.org> Qgis-user@lists.osgeo.org
List info:  <https://lists.osgeo.org/mailman/listinfo/qgis-user> 
https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe:  <https://lists.osgeo.org/mailman/listinfo/qgis-user> 
https://lists.osgeo.org/mailman/listinfo/qgis-user






-- 
Kirk Schmidt, MScF, BScF, RPF
General Manager

Re: [Qgis-user] ctrl-c selection issue

[Ramon Andinach]

Just so we’ve got just about everything covered, does this happen for “Edit-> 
Paste As”?

Also looking at the size of your original CSV, I’d go along with Jeroen, and 
try and import that csv into some other format first. 
If you can find a way of making the selection without going near the attribute 
table that will help too.


Regards,
Ramon.
> On 10 Jan 2018, at 06:26, Jeroen Hovens <m...@groenebij.nl> wrote:
> 
> Thanks for the ideas.
>  
> The option to save the selection as new csv file as Ramon suggested, doesn’t 
> work; qgis goes into ‘not responding’ and no csv file is added to the layer 
> list. However, a csv file is made, lacking a decent number of objects (first 
> test 15095 of 18582 objects and second test is 15179 of 18582 objects, so 
> missing about 3,5k objects). This csv file can be opened manually in qgis.
> And I checked: this options does work for selections where the ctrl-c option 
> also worked. Then the new cvs layer is added to the layer list in no time.
>  
> Is there some sort of log file in qgis that might give clues to the ‘not 
> responding?
>  
> @Kirk
> I did close and reopen qgis multiple times, even shut down the pc entirely, 
> but to no effect. Selections that did allow for copying earlier, still work: 
> I can make that selection again and ctrl-c works fine. Seems to me this rules 
> out memory-issues.
>  
> Gr.
> Jeroen
>  
>  
> Van: Qgis-user [mailto:qgis-user-boun...@lists.osgeo.org] Namens Kirk Schmidt
> Verzonden: dinsdag 9 januari 2018 22:20
> Aan: qgis-user@lists.osgeo.org
> Onderwerp: Re: [Qgis-user] ctrl-c selection issue
>  
> After a few selections, you may have to close and reopen qgis to purge the 
> memory as well. Importing a csv is memory intensive so another approach is to 
> try saving as a spatial database (SQLite) or even a shapefile.  Once written, 
> you can load the spatial file and you should have lots of memory to keep 
> selecting points. 
> 
> 
> 
> Kirk
> 
> On 1/9/2018 4:58 PM, Ramon Andinach wrote:
>> I’m not sure what the problem is, but if I were doing that repeatedly, I’d 
>> probably use Layer -> Save As, tick the Save Only Selected Features box and 
>> save it to CSV.  Then muck about with excel later. 
>>  
>> This way ought to be a bit lighter on the memory. 
>>  
>>  
>> Ramon.
>>> On 10 Jan 2018, at 04:25, Jeroen Hovens <m...@groenebij.nl 
>>> <mailto:m...@groenebij.nl>> wrote:
>>>  
>>> Hi,
>>>  
>>> Using qgis 2.14.18 I have been working today with a large csv file (7.5 mln 
>>> objects). I just need selections of this database which I can then modify 
>>> in excel. So I opened the attribute table and made a selection based on a 
>>> specific value for an attribute, and copied that selection (about 30.000 
>>> objects) via the clipboard (ctrl-c) to a new Microsoft Excel file, then 
>>> saved the excel file and repeated the process for another selection. This 
>>> worked perfect for some time. However, some selections won’t copy: when I 
>>> press the copy button the programs turn to ‘not responding’ and won’t snap 
>>> out of it anymore.  Selection size when this happens is about the same 
>>> (about 20k to 35k objects).
>>>  
>>> What could be the problem? Could the data be corrupted? How can I check 
>>> what is wrong?
>>>  
>>> Thank you
>>> Gr.
>>> Jeroen 
>>>  
>>> ___
>>> Qgis-user mailing list
>>> Qgis-user@lists.osgeo.org <mailto:Qgis-user@lists.osgeo.org>
>>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user 
>>> <https://lists.osgeo.org/mailman/listinfo/qgis-user>
>>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user 
>>> <https://lists.osgeo.org/mailman/listinfo/qgis-user>
>>  
>> 
>> 
>> 
>> ___
>> Qgis-user mailing list
>> Qgis-user@lists.osgeo.org <mailto:Qgis-user@lists.osgeo.org>
>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user 
>> <https://lists.osgeo.org/mailman/listinfo/qgis-user>
>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user 
>> <https://lists.osgeo.org/mailman/listinfo/qgis-user>
> 
> -- 
> Kirk Schmidt, MScF, BScF, RPF
> General Manager
> Nortek Resource Solutions Inc.
> RR # 1
> Thorburn, NS
> B0K 1W0
> Tel (902) 922.3607
> Email: k...@nortekresources.com <mailto:k...@nortekresources.com>
> Web: www.nortekreources.com 
> <http://www.nortekreources.com/>___
> Qgis-user mailing list
> Qgis-user@lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

___
Qgis-user mailing list
Qgis-user@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

Re: [Qgis-user] ctrl-c selection issue

[Jeroen Hovens]

Thanks for the ideas.

 

The option to save the selection as new csv file as Ramon suggested, doesn’t 
work; qgis goes into ‘not responding’ and no csv file is added to the layer 
list. However, a csv file is made, lacking a decent number of objects (first 
test 15095 of 18582 objects and second test is 15179 of 18582 objects, so 
missing about 3,5k objects). This csv file can be opened manually in qgis.

And I checked: this options does work for selections where the ctrl-c option 
also worked. Then the new cvs layer is added to the layer list in no time.

 

Is there some sort of log file in qgis that might give clues to the ‘not 
responding?

 

@Kirk

I did close and reopen qgis multiple times, even shut down the pc entirely, but 
to no effect. Selections that did allow for copying earlier, still work: I can 
make that selection again and ctrl-c works fine. Seems to me this rules out 
memory-issues.

 

Gr.

Jeroen

 

 

Van: Qgis-user [mailto:qgis-user-boun...@lists.osgeo.org] Namens Kirk Schmidt
Verzonden: dinsdag 9 januari 2018 22:20
Aan: qgis-user@lists.osgeo.org
Onderwerp: Re: [Qgis-user] ctrl-c selection issue

 

After a few selections, you may have to close and reopen qgis to purge the 
memory as well. Importing a csv is memory intensive so another approach is to 
try saving as a spatial database (SQLite) or even a shapefile.  Once written, 
you can load the spatial file and you should have lots of memory to keep 
selecting points. 




Kirk

On 1/9/2018 4:58 PM, Ramon Andinach wrote:

I’m not sure what the problem is, but if I were doing that repeatedly, I’d 
probably use Layer -> Save As, tick the Save Only Selected Features box and 
save it to CSV.  Then muck about with excel later. 

 

This way ought to be a bit lighter on the memory. 

 

 

Ramon.

On 10 Jan 2018, at 04:25, Jeroen Hovens <m...@groenebij.nl 
<mailto:m...@groenebij.nl> > wrote:

 

Hi,

 

Using qgis 2.14.18 I have been working today with a large csv file (7.5 mln 
objects). I just need selections of this database which I can then modify in 
excel. So I opened the attribute table and made a selection based on a specific 
value for an attribute, and copied that selection (about 30.000 objects) via 
the clipboard (ctrl-c) to a new Microsoft Excel file, then saved the excel file 
and repeated the process for another selection. This worked perfect for some 
time. However, some selections won’t copy: when I press the copy button the 
programs turn to ‘not responding’ and won’t snap out of it anymore.  Selection 
size when this happens is about the same (about 20k to 35k objects).

 

What could be the problem? Could the data be corrupted? How can I check what is 
wrong?

 

Thank you

Gr.

Jeroen 

 

___
Qgis-user mailing list
Qgis-user@lists.osgeo.org <mailto:Qgis-user@lists.osgeo.org> 
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

 






___
Qgis-user mailing list
Qgis-user@lists.osgeo.org <mailto:Qgis-user@lists.osgeo.org> 
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user





-- 
Kirk Schmidt, MScF, BScF, RPF
General Manager
Nortek Resource Solutions Inc.
RR # 1
Thorburn, NS
B0K 1W0
Tel (902) 922.3607
Email: k...@nortekresources.com <mailto:k...@nortekresources.com> 
Web: www.nortekreources.com <http://www.nortekreources.com> 
___
Qgis-user mailing list
Qgis-user@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

Re: [Qgis-user] ctrl-c selection issue

[Kirk Schmidt]

After a few selections, you may have to close and reopen qgis to purge 
the memory as well. Importing a csv is memory intensive so another 
approach is to try saving as a spatial database (SQLite) or even a 
shapefile.  Once written, you can load the spatial file and you should 
have lots of memory to keep selecting points.


Kirk

On 1/9/2018 4:58 PM, Ramon Andinach wrote:
I’m not sure what the problem is, but if I were doing that repeatedly, 
I’d probably use Layer -> Save As, tick the Save Only Selected 
Features box and save it to CSV.  Then muck about with excel later.


This way ought to be a bit lighter on the memory.


Ramon.
On 10 Jan 2018, at 04:25, Jeroen Hovens > wrote:


Hi,
Using qgis 2.14.18 I have been working today with a large csv file 
(7.5 mln objects). I just need selections of this database which I 
can then modify in excel. So I opened the attribute table and made a 
selection based on a specific value for an attribute, and copied that 
selection (about 30.000 objects) via the clipboard (ctrl-c) to a new 
Microsoft Excel file, then saved the excel file and repeated the 
process for another selection. This worked perfect for some time. 
However, some selections won’t copy: when I press the copy button the 
programs turn to ‘not responding’ and won’t snap out of it anymore. 
 Selection size when this happens is about the same (about 20k to 35k 
objects).
What could be the problem? Could the data be corrupted? How can I 
check what is wrong?

Thank you
Gr.
Jeroen
___
Qgis-user mailing list
Qgis-user@lists.osgeo.org 
List info:https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe:https://lists.osgeo.org/mailman/listinfo/qgis-user




___
Qgis-user mailing list
Qgis-user@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user


--
Kirk Schmidt, MScF, BScF, RPF
General Manager
Nortek Resource Solutions Inc.
RR # 1
Thorburn, NS
B0K 1W0
Tel (902) 922.3607
Email: k...@nortekresources.com
Web: www.nortekreources.com

___
Qgis-user mailing list
Qgis-user@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

Re: [Qgis-user] ctrl-c selection issue

[Ramon Andinach]

I’m not sure what the problem is, but if I were doing that repeatedly, I’d 
probably use Layer -> Save As, tick the Save Only Selected Features box and 
save it to CSV.  Then muck about with excel later.

This way ought to be a bit lighter on the memory. 


Ramon.
> On 10 Jan 2018, at 04:25, Jeroen Hovens  wrote:
> 
> Hi,
>  
> Using qgis 2.14.18 I have been working today with a large csv file (7.5 mln 
> objects). I just need selections of this database which I can then modify in 
> excel. So I opened the attribute table and made a selection based on a 
> specific value for an attribute, and copied that selection (about 30.000 
> objects) via the clipboard (ctrl-c) to a new Microsoft Excel file, then saved 
> the excel file and repeated the process for another selection. This worked 
> perfect for some time. However, some selections won’t copy: when I press the 
> copy button the programs turn to ‘not responding’ and won’t snap out of it 
> anymore.  Selection size when this happens is about the same (about 20k to 
> 35k objects).
>  
> What could be the problem? Could the data be corrupted? How can I check what 
> is wrong?
>  
> Thank you
> Gr.
> Jeroen 
>  
> ___
> Qgis-user mailing list
> Qgis-user@lists.osgeo.org 
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user 
> 
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user 
> 
___
Qgis-user mailing list
Qgis-user@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user