Re: [Qgis-developer] Warning - shapefile corruption on 2.16/master and how to recover

[Luca Lanteri]

I've encountered the same problem when i try to split some polygons.

I don't really understand what was the problem, just some hint:

- it occours only on windows. The same shapefile works fine on Linux.
- it also affects qgis 2.14
- is not connected with topology error. The problem remain also after using
v.clean.

I hope this could be useful
Luca
Il 16/Set/2016 03:08, "Nyall Dawson"  ha scritto:

> Unfortunately a fairly nasty regression has slipped in to 2.16 and is
> still present on master. This results in shapefile corruption in
> certain circumstances.
>
> While I've hit this issue maybe 3 or 4 times in the last 2 months, I
> haven't been able to track down exactly what causes this. It seems
> related to using the split or reshape tool on a shapefile, then saving
> the changes and getting the errors "Cannot reopen datasource
> xxx.shp|layerid=0 in read-only mode" and "Data source is invalid
> (Unable to open xxx.shx or xxx.SHX.Try --config SHAPE_RESTORE_SHX true
> to restore or create it)" in the log.
>
> A full report is at http://hub.qgis.org/issues/15570
>
> Hopefully we can resolve this before the next round of releases. In
> the meantime, there's a workaround for recovering data:
>
> In the same folder as the shapefile there'll be additional files
> "xxx_packed.shp" and "xxx_packed.shx". Renaming the "xxx_packed.shx"
> file to "xxx.shx" allows the original shapefile to be reopened.
>
> Nyall
> ___
> Qgis-developer mailing list
> Qgis-developer@lists.osgeo.org
> List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Warning - shapefile corruption on 2.16/master and how to recover

[Nyall Dawson]

On 7 Oct 2016 12:28 AM, "Even Rouault"  wrote:
>
> Le vendredi 16 septembre 2016 03:08:37, Nyall Dawson a écrit :
> > Unfortunately a fairly nasty regression has slipped in to 2.16 and is
> > still present on master. This results in shapefile corruption in
> > certain circumstances.
>
> I've commited in GDAL trunk and 2.1 branch a fix that revises the way
repacking
> of shapefiles is done, so as to use in-place rewrite of the already
opened file
> rather than trying to remove the temporary copies on top of them, which
was
> prone to issues with files being locked elsewhere. From my tests this
solve the
> issues I could reproduce in http://hub.qgis.org/issues/15570 and
> http://hub.qgis.org/issues/15393 and I'm reasonably confident this is a
robust
> fix for other similar cases. Anyway, testing by others to confirm would be
> appreciated.

Even - thanks for all the hard work you've put into this fix. It's much
appreciated!

I'll give it some testing over the next week and will let you know if I hit
any issues.

Nyall

>
> --
> Spatialys - Geospatial professional services
> http://www.spatialys.com
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Warning - shapefile corruption on 2.16/master and how to recover

[Even Rouault]

Le samedi 17 septembre 2016 00:00:55, Jeff McKenna a écrit :
> On 2016-09-16 3:53 PM, Even Rouault wrote:
> > One of my hypothesis is some bad interaction with the "connection pool".
> > Could you (or anyone) open the attribute table before doing the edits,and
> > do the edits and commit them no more than 60 seconds afterwards ? I'm
> > not sure but this might perhaps also require the shapefile to be big
> > enough (several thousands of features).
> 
> Here was my next test:
> 
> - download OSM land polygons for the world, unsplit:
> http://data.openstreetmapdata.com/land-polygons-complete-4326.zip
> 424MB   568,635 features  <--- big test!
> 
> - add shapefile to QGIS 2.16.2 view
> 
> - open attribute table
> 
> - start editing
> 
> - after a while waiting for nodes to appear as QGIS works, message is
> displayed of "qgis-bin.exe has stopped working"

I've tried your file 2.16.3 in a Win7 VM with 4GB and trying to reproduce your 
actions but didn't get a crash. I managed painfully to move a node of the 
coastline of western part of France. Note that I used a 64 bit build, and I 
noticed that the memory consumption goes up to 3 GB when you enable the Node 
tool (goes down to 300 MB once you finish all editing actions), so if you use a 
32 bit build, a crash is not surprising. 
All the actions are super slow, particularly making a node active and moving 
it. There are clearly issues, with algorithms, data structures and memory use, 
when dealing with huge polygons.
I also noticed that there's apparently always a core busy until you finish 
editing.

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Warning - shapefile corruption on 2.16/master and how to recover

[Even Rouault]

Le vendredi 16 septembre 2016 03:08:37, Nyall Dawson a écrit :
> Unfortunately a fairly nasty regression has slipped in to 2.16 and is
> still present on master. This results in shapefile corruption in
> certain circumstances.

I've commited in GDAL trunk and 2.1 branch a fix that revises the way repacking 
of shapefiles is done, so as to use in-place rewrite of the already opened file 
rather than trying to remove the temporary copies on top of them, which was 
prone to issues with files being locked elsewhere. From my tests this solve the 
issues I could reproduce in http://hub.qgis.org/issues/15570 and 
http://hub.qgis.org/issues/15393 and I'm reasonably confident this is a robust 
fix for other similar cases. Anyway, testing by others to confirm would be 
appreciated.

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Warning - shapefile corruption on 2.16/master and how to recover

[Even Rouault]

Le mardi 20 septembre 2016 13:11:36, Giovanni Manghi a écrit :
> Hi all,
> 
> > One of my hypothesis is some bad interaction with the "connection pool".
> > Could you (or anyone) open the attribute table before doing the edits,and
> > do the edits and commit them no more than 60 seconds afterwards ?
> 
> this sounds familiar too... this is exactly how the editing workflow
> of geopackage layers gets broken
> 
> http://hub.qgis.org/issues/15351
> 
> is this just coincidence?

I'm going to work soon on this GPKG issue. On surface, it looks similar to the 
shapefile issue, but technically file locking and sqlite database locking use 
different mechanisms so the solutions will be likely different.

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Warning - shapefile corruption on 2.16/master and how to recover

[Giovanni Manghi]

Hi all,

> One of my hypothesis is some bad interaction with the "connection pool".
> Could you (or anyone) open the attribute table before doing the edits,and do
> the edits and commit them no more than 60 seconds afterwards ?

this sounds familiar too... this is exactly how the editing workflow
of geopackage layers gets broken

http://hub.qgis.org/issues/15351

is this just coincidence?

cheers

-- G --
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Warning - shapefile corruption on 2.16/master and how to recover

[Jeff McKenna]

On 2016-09-16 10:48 PM, Randal Hale wrote:

I just ran a test



Randy, can you try with the osm-lands shapefile that I linked to in my 
message?  I'm curious how it works on other systems.  This problem could 
just be on Windows it sounds like. 
http://data.openstreetmapdata.com/land-polygons-complete-4326.zip



-jeff



--
Jeff McKenna
MapServer Consulting and Training Services
http://www.gatewaygeomatics.com/






___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Warning - shapefile corruption on 2.16/master and how to recover

[Randal Hale]

I just ran a test

Ubuntu Linux
QGIS 2.16.2
Shapefile with 925,000+ polygons

I opened the attribute table which took a bit.
Started editing.
Split 5 polygons.
Saved.
Stopped editing.
Removed the layer.
Added it back - it didn't corrupt.

I'm wondering if I should have done something more deliberate like 
delete or add more polygons.


Randy




On 09/16/2016 06:00 PM, Jeff McKenna wrote:

On 2016-09-16 3:53 PM, Even Rouault wrote:


One of my hypothesis is some bad interaction with the "connection pool".
Could you (or anyone) open the attribute table before doing the 
edits,and do
the edits and commit them no more than 60 seconds afterwards ? I'm 
not sure
but this might perhaps also require the shapefile to be big enough 
(several

thousands of features).





Here was my next test:

- download OSM land polygons for the world, unsplit: 
http://data.openstreetmapdata.com/land-polygons-complete-4326.zip 
424MB   568,635 features  <--- big test!


- add shapefile to QGIS 2.16.2 view

- open attribute table

- start editing

- after a while waiting for nodes to appear as QGIS works, message is 
displayed of "qgis-bin.exe has stopped working"


With QGIS 2.14.0:

- exact same issue when starting editing of that layer

With QGIS 1.8.0:

- able to click start editing and no crash as nodes are displayed for 
world


- able to use "Split Features" and save edits

- seems to write changes to .dbf .shp .shx (to be honest, changes 
don't look like a valid feature split, hard to tell, but it seems a 
few more feature records were added)


You likely aren't happy that I tried with such large of a file, but, 
it was honestly the first big file that I found.



other test for fun:

- checked layer summary through ogrinfo
- indexed the .shp with the shptree utility (generates 22MB .qix index 
file, wow)
- created a small MapServer mapfile with only that layer, for world 
extents
- generated map image through MapServer, recording layer draw speed 
(cool!)
- tried to compare that speed, to QGIS 2.16.2 loading that same 
shapefile (doing both a same time)


Fun results!  Any guess how MapServer did?  :)

Honestly I didn't plan to check that, and that is not why I tried to 
help here, but when I saw QGIS having a hard time with this large 
file, on a 16MB RAM new machine, I thought 'hmm i wonder how MapServer 
would handle this huge file?'   For one of the previous WMS 
Benchmarking exercises at an old FOSS4G event I tested with a 1GB 
shapefile, and we had to fix the indexing for MapServer...so it is 
good to try to push limits once in a while.


sorry if i offend anyone with these tests, was just having some fun 
pushing MapServer as well.


Let me know if you need more tests.

-jeff




--

Randal Hale
North River Geographic Systems, Inc
http://www.northrivergeographic.com
423.653.3611 rjh...@northrivergeographic.com
twitter:rjhale http://about.me/rjhale
http://www.northrivergeographic.com/introduction-to-quantum-gis
https://www.facebook.com/NRGSInc

___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Warning - shapefile corruption on 2.16/master and how to recover

[Jeff McKenna]

On 2016-09-16 7:00 PM, Jeff McKenna wrote:

On 2016-09-16 3:53 PM, Even Rouault wrote:


One of my hypothesis is some bad interaction with the "connection pool".
Could you (or anyone) open the attribute table before doing the
edits,and do
the edits and commit them no more than 60 seconds afterwards ? I'm not
sure
but this might perhaps also require the shapefile to be big enough
(several
thousands of features).





Here was my next test:

- download OSM land polygons for the world, unsplit:
http://data.openstreetmapdata.com/land-polygons-complete-4326.zip
424MB   568,635 features  <--- big test!

- add shapefile to QGIS 2.16.2 view

- open attribute table

- start editing

- after a while waiting for nodes to appear as QGIS works, message is
displayed of "qgis-bin.exe has stopped working"

With QGIS 2.14.0:

- exact same issue when starting editing of that layer

With QGIS 1.8.0:

- able to click start editing and no crash as nodes are displayed for world

- able to use "Split Features" and save edits

- seems to write changes to .dbf .shp .shx (to be honest, changes don't
look like a valid feature split, hard to tell, but it seems a few more
feature records were added)

You likely aren't happy that I tried with such large of a file, but, it
was honestly the first big file that I found.


other test for fun:

- checked layer summary through ogrinfo
- indexed the .shp with the shptree utility (generates 22MB .qix index
file, wow)
- created a small MapServer mapfile with only that layer, for world extents
- generated map image through MapServer, recording layer draw speed (cool!)
- tried to compare that speed, to QGIS 2.16.2 loading that same
shapefile (doing both a same time)

Fun results!  Any guess how MapServer did?  :)

Honestly I didn't plan to check that, and that is not why I tried to
help here, but when I saw QGIS having a hard time with this large file,
on a 16MB RAM new machine, I thought 'hmm i wonder how MapServer would
handle this huge file?'   For one of the previous WMS Benchmarking
exercises at an old FOSS4G event I tested with a 1GB shapefile, and we
had to fix the indexing for MapServer...so it is good to try to push
limits once in a while.

sorry if i offend anyone with these tests, was just having some fun
pushing MapServer as well.

Let me know if you need more tests.

-jeff




it would be fun, yet make no sense to technical people, to compare 
MapServer WFS output or WFS-Transaction requests, versus other tools 
like QGIS editing.  But that is apples vs oranges and could never be 
agreed upon by development teams (or any sane person).


Anyway, I was just having some fun pushing MapServer.

I hope those tests are somewhat useful!

-jeff


___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Warning - shapefile corruption on 2.16/master and how to recover

[Jeff McKenna]

On 2016-09-16 3:53 PM, Even Rouault wrote:


One of my hypothesis is some bad interaction with the "connection pool".
Could you (or anyone) open the attribute table before doing the edits,and do
the edits and commit them no more than 60 seconds afterwards ? I'm not sure
but this might perhaps also require the shapefile to be big enough (several
thousands of features).





Here was my next test:

- download OSM land polygons for the world, unsplit: 
http://data.openstreetmapdata.com/land-polygons-complete-4326.zip 
424MB   568,635 features  <--- big test!


- add shapefile to QGIS 2.16.2 view

- open attribute table

- start editing

- after a while waiting for nodes to appear as QGIS works, message is 
displayed of "qgis-bin.exe has stopped working"


With QGIS 2.14.0:

- exact same issue when starting editing of that layer

With QGIS 1.8.0:

- able to click start editing and no crash as nodes are displayed for world

- able to use "Split Features" and save edits

- seems to write changes to .dbf .shp .shx (to be honest, changes don't 
look like a valid feature split, hard to tell, but it seems a few more 
feature records were added)


You likely aren't happy that I tried with such large of a file, but, it 
was honestly the first big file that I found.



other test for fun:

- checked layer summary through ogrinfo
- indexed the .shp with the shptree utility (generates 22MB .qix index 
file, wow)

- created a small MapServer mapfile with only that layer, for world extents
- generated map image through MapServer, recording layer draw speed (cool!)
- tried to compare that speed, to QGIS 2.16.2 loading that same 
shapefile (doing both a same time)


Fun results!  Any guess how MapServer did?  :)

Honestly I didn't plan to check that, and that is not why I tried to 
help here, but when I saw QGIS having a hard time with this large file, 
on a 16MB RAM new machine, I thought 'hmm i wonder how MapServer would 
handle this huge file?'   For one of the previous WMS Benchmarking 
exercises at an old FOSS4G event I tested with a 1GB shapefile, and we 
had to fix the indexing for MapServer...so it is good to try to push 
limits once in a while.


sorry if i offend anyone with these tests, was just having some fun 
pushing MapServer as well.


Let me know if you need more tests.

-jeff


--
Jeff McKenna
MapServer Consulting and Training Services
http://www.gatewaygeomatics.com/





___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Warning - shapefile corruption on 2.16/master and how to recover

[Jeff McKenna]

On 2016-09-16 3:53 PM, Even Rouault wrote:


One of my hypothesis is some bad interaction with the "connection pool".
Could you (or anyone) open the attribute table before doing the edits,and do
the edits and commit them no more than 60 seconds afterwards ? I'm not sure
but this might perhaps also require the shapefile to be big enough (several
thousands of features).




hmm, still works with 1000 features.  Will look for a bigger test.

-jeff



___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Warning - shapefile corruption on 2.16/master and how to recover

[Even Rouault]

Le vendredi 16 septembre 2016 20:27:05, Jeff McKenna a écrit :
> I just tried to reproduce with QGIS 2.16.2 on Windows 10, I had no
> problems using the "Split Features" tool on a shapefile, and saving. I
> notice that the .dbf .shp .shx were modified properly.
> 
> (I did have a brain lapse to remember that the Split Features tool lives
> in the /edit menu ha)

One of my hypothesis is some bad interaction with the "connection pool".
Could you (or anyone) open the attribute table before doing the edits,and do 
the edits and commit them no more than 60 seconds afterwards ? I'm not sure 
but this might perhaps also require the shapefile to be big enough (several 
thousands of features). 

> 
> -jeff
> 
> > Unfortunately a fairly nasty regression has slipped in to 2.16 and is
> > still present on master. This results in shapefile corruption in
> > certain circumstances.
> > 
> > While I've hit this issue maybe 3 or 4 times in the last 2 months, I
> > haven't been able to track down exactly what causes this. It seems
> > related to using the split or reshape tool on a shapefile, then saving
> > the changes and getting the errors "Cannot reopen datasource
> > xxx.shp|layerid=0 in read-only mode" and "Data source is invalid
> > (Unable to open xxx.shx or xxx.SHX.Try --config SHAPE_RESTORE_SHX true
> > to restore or create it)" in the log.
> > 
> > A full report is at http://hub.qgis.org/issues/15570
> > 
> > Hopefully we can resolve this before the next round of releases. In
> > the meantime, there's a workaround for recovering data:
> > 
> > In the same folder as the shapefile there'll be additional files
> > "xxx_packed.shp" and "xxx_packed.shx". Renaming the "xxx_packed.shx"
> > file to "xxx.shx" allows the original shapefile to be reopened.
> > 
> > Nyall
> > ___
> > Qgis-developer mailing list
> > Qgis-developer@lists.osgeo.org
> > List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
> > Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer
> 
> ___
> Qgis-developer mailing list
> Qgis-developer@lists.osgeo.org
> List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Warning - shapefile corruption on 2.16/master and how to recover

[Jeff McKenna]

I just tried to reproduce with QGIS 2.16.2 on Windows 10, I had no 
problems using the "Split Features" tool on a shapefile, and saving. I 
notice that the .dbf .shp .shx were modified properly.


(I did have a brain lapse to remember that the Split Features tool lives 
in the /edit menu ha)


-jeff



--
Jeff McKenna
MapServer Consulting and Training Services
http://www.gatewaygeomatics.com/



On 2016-09-15 10:08 PM, Nyall Dawson wrote:

Unfortunately a fairly nasty regression has slipped in to 2.16 and is
still present on master. This results in shapefile corruption in
certain circumstances.

While I've hit this issue maybe 3 or 4 times in the last 2 months, I
haven't been able to track down exactly what causes this. It seems
related to using the split or reshape tool on a shapefile, then saving
the changes and getting the errors "Cannot reopen datasource
xxx.shp|layerid=0 in read-only mode" and "Data source is invalid
(Unable to open xxx.shx or xxx.SHX.Try --config SHAPE_RESTORE_SHX true
to restore or create it)" in the log.

A full report is at http://hub.qgis.org/issues/15570

Hopefully we can resolve this before the next round of releases. In
the meantime, there's a workaround for recovering data:

In the same folder as the shapefile there'll be additional files
"xxx_packed.shp" and "xxx_packed.shx". Renaming the "xxx_packed.shx"
file to "xxx.shx" allows the original shapefile to be reopened.

Nyall
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer




___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Warning - shapefile corruption on 2.16/master and how to recover

[Lene Fischer]

Hi Nyall,
I have been using 2.16 in a 3 week summer class. We have been using these 
functions, and not seen the described errors.
We use OS : Windows 7 and 8 - the MAC users was using 2.14

Is it on Linux you get the errors ?
Regards


Lene Fischer

-Oprindelig meddelelse-
Fra: Qgis-developer [mailto:qgis-developer-boun...@lists.osgeo.org] På vegne af 
Nyall Dawson
Sendt: 16. september 2016 03:09
Til: qgis-developer
Emne: [Qgis-developer] Warning - shapefile corruption on 2.16/master and how to 
recover

Unfortunately a fairly nasty regression has slipped in to 2.16 and is still 
present on master. This results in shapefile corruption in certain 
circumstances.

While I've hit this issue maybe 3 or 4 times in the last 2 months, I haven't 
been able to track down exactly what causes this. It seems related to using the 
split or reshape tool on a shapefile, then saving the changes and getting the 
errors "Cannot reopen datasource
xxx.shp|layerid=0 in read-only mode" and "Data source is invalid (Unable to 
open xxx.shx or xxx.SHX.Try --config SHAPE_RESTORE_SHX true to restore or 
create it)" in the log.

A full report is at http://hub.qgis.org/issues/15570

Hopefully we can resolve this before the next round of releases. In the 
meantime, there's a workaround for recovering data:

In the same folder as the shapefile there'll be additional files 
"xxx_packed.shp" and "xxx_packed.shx". Renaming the "xxx_packed.shx"
file to "xxx.shx" allows the original shapefile to be reopened.

Nyall
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Warning - shapefile corruption on 2.16/master and how to recover

[Geo DrinX]

Hi Nyall,


2016-09-16 3:08 GMT+02:00 Nyall Dawson :

Unfortunately a fairly nasty regression has slipped in to 2.16 and is
> still present on master.


perhaps, the name "NODEB" stands for  "--NODEBug"   ?   :(



> This results in shapefile corruption in
> certain circumstances.
>

well,  but today  a SHP is not so important   ;)

Serioulsly,  do you have some  plugin active ?   Do you tested with
"--NOPLUGINS"  option active ?
Do you can try renaming  .qgis2   directory,  so that we are sure  QGIS  do
no use it ?
And what is the result ?

Please, let us know.


Regards

Roberto
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Warning - shapefile corruption on 2.16/master and how to recover

[Saber Razmjooei]

There was another report of Shapefile corruption earlier with 2.14 which I 
thought it could be side effects of the solution for #11007:
http://hub.qgis.org/issues/15407

Cheers,
Saber

-Original Message-
From: Qgis-developer [mailto:qgis-developer-boun...@lists.osgeo.org] On Behalf 
Of Giovanni Manghi
Sent: 16 September 2016 09:55
To: qgis-developer@lists.osgeo.org
Subject: Re: [Qgis-developer] Warning - shapefile corruption on 2.16/master and 
how to recover

Hi all,


> While I've hit this issue maybe 3 or 4 times in the last 2 months, I 
> haven't been able to track down exactly what causes this. It seems 
> related to using the split or reshape tool on a shapefile, then saving 
> the changes and getting the errors "Cannot reopen datasource
> xxx.shp|layerid=0 in read-only mode" and "Data source is invalid 
> (Unable to open xxx.shx or xxx.SHX.Try --config SHAPE_RESTORE_SHX true 
> to restore or create it)" in the log.
>
> A full report is at http://hub.qgis.org/issues/15570


geez... this unfortunately sounds familiar... remember all the issues with 
deleted features and shapefiles opened in software other than QGIS?

Can this be something related to that?

cheers

-- G --
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer


--
This email and any files transmitted with it are confidential and intended 
solely for the use of the individual or entity to whom they are addressed.
If you have received this email in error please notify the system manager. This 
message contains confidential information and is intended only for the
individual named. If you are not the named addressee you should not 
disseminate, distribute or copy this e-mail. Please notify the sender 
immediately
by e-mail if you have received this e-mail by mistake and delete this e-mail 
from your system. If you are not the intended recipient you are notified
that disclosing, copying, distributing or taking any action in reliance on the 
contents of this information is strictly prohibited.

Whilst reasonable care has been taken to avoid virus transmission, no 
responsibility for viruses is taken and it is your responsibility to carry out
such checks as you feel appropriate.

If this email contains a quote or offer to sell products, carry out work or 
perform services then our standard terms and conditions (which can be found at 
http://www.lutraconsulting.co.uk/downloads/Lutra%20Consulting%20Standard%20Terms%20and%20Conditions.pdf
 shall apply unless explicitly stated otherwise.

Saber Razmjooei and Peter Wells trading as Lutra Consulting.
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Warning - shapefile corruption on 2.16/master and how to recover

[Giovanni Manghi]

Hi all,


> While I've hit this issue maybe 3 or 4 times in the last 2 months, I
> haven't been able to track down exactly what causes this. It seems
> related to using the split or reshape tool on a shapefile, then saving
> the changes and getting the errors "Cannot reopen datasource
> xxx.shp|layerid=0 in read-only mode" and "Data source is invalid
> (Unable to open xxx.shx or xxx.SHX.Try --config SHAPE_RESTORE_SHX true
> to restore or create it)" in the log.
>
> A full report is at http://hub.qgis.org/issues/15570


geez... this unfortunately sounds familiar... remember all the issues
with deleted features and shapefiles opened in software other than
QGIS?

Can this be something related to that?

cheers

-- G --
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Warning - shapefile corruption on 2.16/master and how to recover

[Nyall Dawson]

On 16 September 2016 at 16:16, Lene Fischer  wrote:
> Hi Nyall,
> I have been using 2.16 in a 3 week summer class. We have been using these 
> functions, and not seen the described errors.
> We use OS : Windows 7 and 8 - the MAC users was using 2.14
>
> Is it on Linux you get the errors ?

No - it's on Windows. And I can't reproduce it everytime, but it has
happened multiple times.

Nyall


> Regards
>
>
> Lene Fischer
>
> -Oprindelig meddelelse-
> Fra: Qgis-developer [mailto:qgis-developer-boun...@lists.osgeo.org] På vegne 
> af Nyall Dawson
> Sendt: 16. september 2016 03:09
> Til: qgis-developer
> Emne: [Qgis-developer] Warning - shapefile corruption on 2.16/master and how 
> to recover
>
> Unfortunately a fairly nasty regression has slipped in to 2.16 and is still 
> present on master. This results in shapefile corruption in certain 
> circumstances.
>
> While I've hit this issue maybe 3 or 4 times in the last 2 months, I haven't 
> been able to track down exactly what causes this. It seems related to using 
> the split or reshape tool on a shapefile, then saving the changes and getting 
> the errors "Cannot reopen datasource
> xxx.shp|layerid=0 in read-only mode" and "Data source is invalid (Unable to 
> open xxx.shx or xxx.SHX.Try --config SHAPE_RESTORE_SHX true to restore or 
> create it)" in the log.
>
> A full report is at http://hub.qgis.org/issues/15570
>
> Hopefully we can resolve this before the next round of releases. In the 
> meantime, there's a workaround for recovering data:
>
> In the same folder as the shapefile there'll be additional files 
> "xxx_packed.shp" and "xxx_packed.shx". Renaming the "xxx_packed.shx"
> file to "xxx.shx" allows the original shapefile to be reopened.
>
> Nyall
> ___
> Qgis-developer mailing list
> Qgis-developer@lists.osgeo.org
> List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer