[sqlite] Disk-image malformed

[Klaas V]

Dear fellow users and developers of SQLite,

sqlite> create table dual(dummy blob);
sqlite> insert into dual values ('SQLite version 3.8.11.1 2015-07-29 20:00:57');
sqlite> create table z4usm1 as select (unicode(substr(type,2,1)) - 97) / 4 "zk",
   ...>name "zn", rootpage "zp", tbl_name "zt" from 
sqlite_master; 
Error: database disk image is malformed

-- relevant part of .dbinfo
number of tables:1
number of indexes:   0
number of triggers:  0
number of views: 0
schema size: 29

I found a work-around by creating a view (actually my first intention, because 
that is what it actually is), but 
  if you really want to create a table this way how to achieve that?

sqlite> pragma page_size=512;
sqlite> create table dual(dummy blob);
sqlite> insert into dual values ('SQLite version 3.8.11.1 2015-07-29 20:00:57');
sqlite> create view z4usm0 as select (unicode(substr(type,2,1)) - 97) / 4 "zk",
   ...>name "zn", rootpage "zp", tbl_name "zt" from 
sqlite_master; 
sqlite> .he on
sqlite> select * from z4usm0;
zk|zn|zp|zt
0|dual|2|dual
2|z4usm0|0|z4usm0

-- relevant part of .dbinfo
number of tables:1
number of indexes:   0
number of triggers:  0
number of views: 1
schema size: 182

sqlite> .sh ls -lg m?.sq?
-rw-r--r--  1 staff  1024  9 Set 11:20 m0.sqb
-rw-r--r--  1 staff  2048  9 Set 11:10 m1.sqb


Another question: When can I see type 'meta' (would result in zk=1) [ exists 
according to https://www.sqlite.org/cli.html#fileio ] ?


Kind regards |?Cordiali saluti | Vriendelijke groeten | Freundliche Gr?sse,
Klaas `Z4us` V ?- OrcID -0001-7190-2544

Re: [sqlite] disk image malformed

[dd]

I do Integrity check every often.


On Sun, Dec 9, 2012 at 5:07 PM, Simon Slavin  wrote:

>
> On 9 Dec 2012, at 12:44pm, dd  wrote:
>
> > Can I read the database when sqlite throws disk io or image malformed or
> > other critical errors first time. I cannot simulate these issues on my
> > machine. So, I need your suggestion.
>
> You should not continue to work with the database in your own application
> on customer computers.
>
> I would recommend that you copy the database to your own computer and open
> it with the sqlite3 shell tool:
>
> 
>
> you will find a copy on every Mac in '/usr/bin/sqlite3'.
>
> You can immediately run
>
> 
>
> and find out whether the error means that the file itself is corrupt.  If
> the file fails the integrity check then the file is corrupt.  if it
> doesn't, it means that something went wrong (software or hardware) on your
> customer's computer but that this didn't cause anything bad to be written
> to the database file.
>
> Simon.
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] disk image malformed

[Simon Slavin]

On 9 Dec 2012, at 12:44pm, dd  wrote:

> Can I read the database when sqlite throws disk io or image malformed or
> other critical errors first time. I cannot simulate these issues on my
> machine. So, I need your suggestion.

You should not continue to work with the database in your own application on 
customer computers.

I would recommend that you copy the database to your own computer and open it 
with the sqlite3 shell tool:



you will find a copy on every Mac in '/usr/bin/sqlite3'.

You can immediately run 



and find out whether the error means that the file itself is corrupt.  If the 
file fails the integrity check then the file is corrupt.  if it doesn't, it 
means that something went wrong (software or hardware) on your customer's 
computer but that this didn't cause anything bad to be written to the database 
file.

Simon.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] disk image malformed

[Kees Nuyt]

On Sun, 9 Dec 2012 16:44:13 +0400, dd  wrote:

> Can I read the database when sqlite throws disk io or image malformed or
> other critical errors first time. I cannot simulate these issues on my
> machine. So, I need your suggestion.

Here are a few suggestions to simulate (or rather, to force) corruption:
http://sqlite.org/lockingv3.html#how_to_corrupt

You may need http://sqlite.org/fileformat2.html for more insight.

-- 
Groet, Cordialement, Pozdrawiam, Regards,

Kees Nuyt

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] disk image malformed

[dd]

Can I read the database when sqlite throws disk io or image malformed or
other critical errors first time. I cannot simulate these issues on my
machine. So, I need your suggestion.


On Sun, Dec 9, 2012 at 4:40 PM, dd  wrote:

> Thank you Simon Slavin.
>
>
> On Sun, Dec 9, 2012 at 4:27 PM, Simon Slavin  wrote:
>
>>
>> On 9 Dec 2012, at 12:10pm, dd  wrote:
>>
>> > I have code in C++. Right now, I got this issue on Mac.
>>
>> Oh.  Then you don't have to worry about Windows codepages.  All your text
>> is already in Unicode.
>>
>> >>> you still have faulty hardware or software
>> >
>> > faulty software means? (OS or sqlite version)
>>
>> The software you wrote.  Your C++ code.  You may be writing to memory
>> that SQLite is using.  You may not be allocating or disposing of memory
>> correctly.  The C programming language is very bad at allowing you to do
>> all sorts of dangerous things.  Every C programmer makes mistakes like this
>> from time to time.
>>
>> Make sure that you are checking the return values of all your calls to
>> the SQLite API, reporting any unexpected values and immediately halting
>> your application.  That is the easiest way to detect problems as soon as
>> they happen.  If you don't do this then your application may continue to
>> use the database, writing corrupted memory to the database file.
>>
>> Simon.
>> ___
>> sqlite-users mailing list
>> sqlite-users@sqlite.org
>> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>>
>
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] disk image malformed

[dd]

Thank you Simon Slavin.


On Sun, Dec 9, 2012 at 4:27 PM, Simon Slavin  wrote:

>
> On 9 Dec 2012, at 12:10pm, dd  wrote:
>
> > I have code in C++. Right now, I got this issue on Mac.
>
> Oh.  Then you don't have to worry about Windows codepages.  All your text
> is already in Unicode.
>
> >>> you still have faulty hardware or software
> >
> > faulty software means? (OS or sqlite version)
>
> The software you wrote.  Your C++ code.  You may be writing to memory that
> SQLite is using.  You may not be allocating or disposing of memory
> correctly.  The C programming language is very bad at allowing you to do
> all sorts of dangerous things.  Every C programmer makes mistakes like this
> from time to time.
>
> Make sure that you are checking the return values of all your calls to the
> SQLite API, reporting any unexpected values and immediately halting your
> application.  That is the easiest way to detect problems as soon as they
> happen.  If you don't do this then your application may continue to use the
> database, writing corrupted memory to the database file.
>
> Simon.
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] disk image malformed

[Simon Slavin]

On 9 Dec 2012, at 12:10pm, dd  wrote:

> I have code in C++. Right now, I got this issue on Mac.

Oh.  Then you don't have to worry about Windows codepages.  All your text is 
already in Unicode.

>>> you still have faulty hardware or software
> 
> faulty software means? (OS or sqlite version)

The software you wrote.  Your C++ code.  You may be writing to memory that 
SQLite is using.  You may not be allocating or disposing of memory correctly.  
The C programming language is very bad at allowing you to do all sorts of 
dangerous things.  Every C programmer makes mistakes like this from time to 
time.

Make sure that you are checking the return values of all your calls to the 
SQLite API, reporting any unexpected values and immediately halting your 
application.  That is the easiest way to detect problems as soon as they 
happen.  If you don't do this then your application may continue to use the 
database, writing corrupted memory to the database file.

Simon.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] disk image malformed

[dd]

I have code in C++. Right now, I got this issue on Mac.

>>you still have faulty hardware or software

faulty software means? (OS or sqlite version)




On Sun, Dec 9, 2012 at 3:37 PM, Simon Slavin  wrote:

>
> On 9 Dec 2012, at 7:34am, dd  wrote:
>
> > I have 10 databases. One database got disk io error, later it's used by
> > application without proper handling, so it's corrupted.
> >
> > After some days, one more database also corrupted without io error.
> Sqlite
> > returned error code 11. It's fully corrupted. Are these corruptions
> depend
> > on other database integrity?
>
> Corruption may spread from one database to another if you have a corrupt
> database open at the same time as an uncorrupt database.  It is rare but it
> can happen.  If you are no longer opening corrupt database files and are
> still getting more corruption you still have faulty hardware or software.
>
> > I am using this database for "schengen countries" customers. Can sqlite
> > handle these characters(special)?
>
> SQLite handles text in UTF-8 or UTF-16 format.  If you are storing data in
> a database relying on other Windows codepages then you will get strange
> results.  If you are allowing non-Roman characters and rely on your user's
> codepage when handling text, use a routine like 'WideCharToMultiByte':
>
> 
>
> to convert the data to UTF-8 or UTF-16 before storing it in the SQLite
> database.  Actually it's easier to convert your text to Unicode immediately
> you receive it from your user.  You do not have to use the above routine, I
> provide it just as an example.
>
> I can't help more without knowing what OS and programming language you're
> using but maybe someone else can.
>
> Simon.
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] disk image malformed

[Simon Slavin]

On 9 Dec 2012, at 7:34am, dd  wrote:

> I have 10 databases. One database got disk io error, later it's used by
> application without proper handling, so it's corrupted.
> 
> After some days, one more database also corrupted without io error. Sqlite
> returned error code 11. It's fully corrupted. Are these corruptions depend
> on other database integrity?

Corruption may spread from one database to another if you have a corrupt 
database open at the same time as an uncorrupt database.  It is rare but it can 
happen.  If you are no longer opening corrupt database files and are still 
getting more corruption you still have faulty hardware or software.

> I am using this database for "schengen countries" customers. Can sqlite
> handle these characters(special)?

SQLite handles text in UTF-8 or UTF-16 format.  If you are storing data in a 
database relying on other Windows codepages then you will get strange results.  
If you are allowing non-Roman characters and rely on your user's codepage when 
handling text, use a routine like 'WideCharToMultiByte':



to convert the data to UTF-8 or UTF-16 before storing it in the SQLite 
database.  Actually it's easier to convert your text to Unicode immediately you 
receive it from your user.  You do not have to use the above routine, I provide 
it just as an example.

I can't help more without knowing what OS and programming language you're using 
but maybe someone else can.

Simon.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] disk image malformed

[dd]

I have 10 databases. One database got disk io error, later it's used by
application without proper handling, so it's corrupted.

After some days, one more database also corrupted without io error. Sqlite
returned error code 11. It's fully corrupted. Are these corruptions depend
on other database integrity?

I am using this database for "schengen countries" customers. Can sqlite
handle these characters(special)?


On Fri, Dec 7, 2012 at 5:51 PM, Simon Slavin  wrote:

>
> On 7 Dec 2012, at 1:36pm, dd  wrote:
>
> > Thank you Simon.
> >
> > So I have to stop using database when error between 1 to 26.
>
> You have to stop when you get any code you do not have specific logic to
> handle.  Here is the list of result codes:
>
> 
>
> You can expect 0 from anything, and 100 and 101 from _step() calls.  You
> may have written your own software which will cause or handle various
> errors (e.g. 4, 9, 19).  Anything you haven't written your own code to
> handle is a sign that your program needs to immediately stop until you have
> figured out what caused the error.
>
> You might find it useful to look at the extended result codes if you don't
> understand what's causing an error:
>
> 
> 
>
> > If I use new database connection (sqlite3*) for every query, will it
> cause
> > disk io error?
>
> It will not create them or prevent them, it will just make your software
> slower.  That result code is what happens when your operating system
> reports an error, which it may do for anything from broken hardware, a full
> hard disk, or trying to edit a file without high enough permissions.  I
> don't know what's causing the SQLITE_IOERR that you have seen.
>
> > I am encrypting database with secret key. I changed page size also. Can
> > sqlite handle encrypted database and different page sizes?
>
> Each database has only one page size.  You can have as many databases open
> at the same time as you want.  They do not all have to have the same page
> size.
>
> Simon.
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] disk image malformed

[Simon Slavin]

On 7 Dec 2012, at 1:36pm, dd  wrote:

> Thank you Simon.
> 
> So I have to stop using database when error between 1 to 26.

You have to stop when you get any code you do not have specific logic to 
handle.  Here is the list of result codes:



You can expect 0 from anything, and 100 and 101 from _step() calls.  You may 
have written your own software which will cause or handle various errors (e.g. 
4, 9, 19).  Anything you haven't written your own code to handle is a sign that 
your program needs to immediately stop until you have figured out what caused 
the error.

You might find it useful to look at the extended result codes if you don't 
understand what's causing an error:




> If I use new database connection (sqlite3*) for every query, will it cause
> disk io error?

It will not create them or prevent them, it will just make your software 
slower.  That result code is what happens when your operating system reports an 
error, which it may do for anything from broken hardware, a full hard disk, or 
trying to edit a file without high enough permissions.  I don't know what's 
causing the SQLITE_IOERR that you have seen.

> I am encrypting database with secret key. I changed page size also. Can
> sqlite handle encrypted database and different page sizes?

Each database has only one page size.  You can have as many databases open at 
the same time as you want.  They do not all have to have the same page size.

Simon.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] disk image malformed

[dd]

Thank you Simon.

So I have to stop using database when error between 1 to 26.

If I use new database connection (sqlite3*) for every query, will it cause
disk io error?

I am encrypting database with secret key. I changed page size also. Can
sqlite handle encrypted database and different page sizes?


On Fri, Dec 7, 2012 at 2:06 AM, Simon Slavin  wrote:

>
> On 6 Dec 2012, at 7:57pm, dd  wrote:
>
> > It's checking with OK, BUSY, 101 and some other errors. But not disk io,
> > image malformed errors. How to handle these errors in run time? these
> > databases have critical information about customers. We cannot stop
> > application. Any inputs?
>
> Proceeding after anything but SQLITE_OK, and the special result codes
> expected when you do sqlite3_step() will only make more problems.  The
> first time you get an error like DISK IO you have to stop.  Anything
> written to the database after that will probably fail, but if it doesn't
> fail it'll probably just corrupt the database file.
>
> But it's extremely unusual for users of SQLite to see errors 10 or 11.
>  They almost always indicates faulty hardware or that some part of your
> operating system has become corrupt (which might itself be the result of
> faulty hardware).
>
> Simon.
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] disk image malformed

[sqdev sqdev]

The several cases of corrupted databases i mentioned all resulted from a
newly created database.
I do not reuse repaired databases with my application.

The corrupted databases came from different computers and different users.


2012/12/6 Simon Slavin 

>
> On 6 Dec 2012, at 7:57pm, dd  wrote:
>
> > It's checking with OK, BUSY, 101 and some other errors. But not disk io,
> > image malformed errors. How to handle these errors in run time? these
> > databases have critical information about customers. We cannot stop
> > application. Any inputs?
>
> Proceeding after anything but SQLITE_OK, and the special result codes
> expected when you do sqlite3_step() will only make more problems.  The
> first time you get an error like DISK IO you have to stop.  Anything
> written to the database after that will probably fail, but if it doesn't
> fail it'll probably just corrupt the database file.
>
> But it's extremely unusual for users of SQLite to see errors 10 or 11.
>  They almost always indicates faulty hardware or that some part of your
> operating system has become corrupt (which might itself be the result of
> faulty hardware).
>
> Simon.
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] disk image malformed (Durga D)

[Klaas V]

If you made a backup in one of following ways yes:

1. SQLite's Internal command .backup 
2. cp (or copy) command of your operating system's command line.


> On Thu, Dec 6, 2012  Durga D  wrote:

Is there anyway to recover?


> On Thu, Dec 6, 2012 at 10:23 PM, Richard Hipp  wrote:

>
> > Hi,
> >
> >  Is it possible to corrupt a single table among 10 tables in a database?
> >
> >  Is it possible to corrupt some records among millions of records in a
> > table?
> >
>
> Yes.  Yes.
>
 

Cordiali saluti/Vriendelijke groeten/Kind regards,
Klaas V

http://innocentisart.net
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] disk image malformed

[Simon Slavin]

On 6 Dec 2012, at 7:57pm, dd  wrote:

> It's checking with OK, BUSY, 101 and some other errors. But not disk io,
> image malformed errors. How to handle these errors in run time? these
> databases have critical information about customers. We cannot stop
> application. Any inputs?

Proceeding after anything but SQLITE_OK, and the special result codes expected 
when you do sqlite3_step() will only make more problems.  The first time you 
get an error like DISK IO you have to stop.  Anything written to the database 
after that will probably fail, but if it doesn't fail it'll probably just 
corrupt the database file.

But it's extremely unusual for users of SQLite to see errors 10 or 11.  They 
almost always indicates faulty hardware or that some part of your operating 
system has become corrupt (which might itself be the result of faulty hardware).

Simon.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] disk image malformed

[dd]

Databases integrity fine when application starts. These critical errors are
introduced in run time. I cannot take backup of these databases also.
space(memory) problem.


On Fri, Dec 7, 2012 at 1:27 AM, dd  wrote:

> It's checking with OK, BUSY, 101 and some other errors. But not disk io,
> image malformed errors. How to handle these errors in run time? these
> databases have critical information about customers. We cannot stop
> application. Any inputs?
>
>
> On Fri, Dec 7, 2012 at 1:20 AM, Simon Slavin  wrote:
>
>>
>> On 6 Dec 2012, at 7:47pm, dd  wrote:
>>
>> > No. I replaced with fresh database. I want to prevent these errors in
>> our
>> > customers place for future purpose.
>>
>> Run the integrity check every so often and see if the problem occurs
>> again.  Make sure that your program checks the result returned by every
>> SQLite API call to make sure it is SQLITE_OK.
>>
>> Simon.
>> ___
>> sqlite-users mailing list
>> sqlite-users@sqlite.org
>> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>>
>
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] disk image malformed

[dd]

It's checking with OK, BUSY, 101 and some other errors. But not disk io,
image malformed errors. How to handle these errors in run time? these
databases have critical information about customers. We cannot stop
application. Any inputs?


On Fri, Dec 7, 2012 at 1:20 AM, Simon Slavin  wrote:

>
> On 6 Dec 2012, at 7:47pm, dd  wrote:
>
> > No. I replaced with fresh database. I want to prevent these errors in our
> > customers place for future purpose.
>
> Run the integrity check every so often and see if the problem occurs
> again.  Make sure that your program checks the result returned by every
> SQLite API call to make sure it is SQLITE_OK.
>
> Simon.
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] disk image malformed

[Simon Slavin]

On 6 Dec 2012, at 7:47pm, dd  wrote:

> No. I replaced with fresh database. I want to prevent these errors in our
> customers place for future purpose.

Run the integrity check every so often and see if the problem occurs again.  
Make sure that your program checks the result returned by every SQLite API call 
to make sure it is SQLITE_OK.

Simon.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] disk image malformed

[dd]

No. I replaced with fresh database. I want to prevent these errors in our
customers place for future purpose.




On Fri, Dec 7, 2012 at 1:09 AM, Simon Slavin  wrote:

>
> On 6 Dec 2012, at 7:30pm, dd  wrote:
>
> > My multithreaded application do all read/write operations. For each
> query,
> > it opens database connection and execute query then close connection.
> This
> > application executes plenty of records within a second. In this scenario,
> > Application got disk io error within couple of hours with new database.
>
> Are you still using the database which the shell tool reports as corrupt ?
>
> Simon.
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] disk image malformed

[Simon Slavin]

On 6 Dec 2012, at 7:30pm, dd  wrote:

> My multithreaded application do all read/write operations. For each query,
> it opens database connection and execute query then close connection. This
> application executes plenty of records within a second. In this scenario,
> Application got disk io error within couple of hours with new database.

Are you still using the database which the shell tool reports as corrupt ?

Simon.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] disk image malformed

[dd]

I understand your points Simon.

Let me explain scenario where I got this disk io error then my database got
corrupted.

My multithreaded application do all read/write operations. For each query,
it opens database connection and execute query then close connection. This
application executes plenty of records within a second. In this scenario,
Application got disk io error within couple of hours with new database.
(even space available on that machine). Application didn't handle this
error. So application continued execution. After that, disk image malformed
errors are thrown by sqlite for some records only. There was no other
errors before disk io error. I suspect, after disk io error only these
errors are introduced.

Any ideas?


On Fri, Dec 7, 2012 at 12:39 AM, dd  wrote:

> No Robert. It's on same machine -- multiple threads within process,
> TRUNCATE mode.
>
>
> On Fri, Dec 7, 2012 at 12:38 AM, Robert Myers wrote:
>
>> One thing I haven't seen anyone ask yet - are you putting this on a
>> network drive?
>> On 12/6/2012 10:52 AM, Durga D wrote:
>> > Hi,
>> >
>> >   Is it possible to corrupt a single table among 10 tables in a
>> database?
>> >
>> >   Is it possible to corrupt some records among millions of records in a
>> > table?
>> >
>> > Best Regards,
>> > ___
>> > sqlite-users mailing list
>> > sqlite-users@sqlite.org
>> > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>>
>> ___
>> sqlite-users mailing list
>> sqlite-users@sqlite.org
>> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>>
>
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] disk image malformed

[dd]

No Robert. It's on same machine -- multiple threads within process,
TRUNCATE mode.


On Fri, Dec 7, 2012 at 12:38 AM, Robert Myers  wrote:

> One thing I haven't seen anyone ask yet - are you putting this on a
> network drive?
> On 12/6/2012 10:52 AM, Durga D wrote:
> > Hi,
> >
> >   Is it possible to corrupt a single table among 10 tables in a database?
> >
> >   Is it possible to corrupt some records among millions of records in a
> > table?
> >
> > Best Regards,
> > ___
> > sqlite-users mailing list
> > sqlite-users@sqlite.org
> > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] disk image malformed

[Robert Myers]

One thing I haven't seen anyone ask yet - are you putting this on a
network drive?
On 12/6/2012 10:52 AM, Durga D wrote:
> Hi,
>
>   Is it possible to corrupt a single table among 10 tables in a database?
>
>   Is it possible to corrupt some records among millions of records in a
> table?
>
> Best Regards,
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] disk image malformed

[Simon Slavin]

On 6 Dec 2012, at 6:51pm, Durga D  wrote:

> Does disk io error is related to hardware issue? i mean, fsync() dint get
> success on unix. so, this error may be thrown by sqlite. Is it?

Disk IO is sometimes caused by a hardware issue, but it can be caused by 
trusting data which is already in a corrupt database.  If you are trying to use 
a corrupt database you can get all sorts of error messages at any time, because 
SQLite makes the assumption that the database is not corrupt.

To make up an imaginary example, there may be data in your corrupt database 
that says the next piece of a table is in block -999.  The software then issues 
the instruction "read block -999" to your file system which says "Don't be 
stupid, there is no such thing as block -999." and issues a DISK IO ERROR.  So 
here you have a genuine correct DISK IO ERROR which is actually caused by the 
corrupt database and not a hardware problem.  But this is not definitely what's 
causing your message.  Perhaps the database is asking to read a block which has 
a good block number, and you have a genuine hardware error caused by a faulty 
hard disk.

So as I wrote before we cannot tell what is wrong with your system while you 
are still using a corrupt database file.  It is only the first error which is 
useful.  After that you can get all sorts of error messages but they tell us 
nothing.  Only when you stop trying to use your corrupt database can we help 
you.

Alternatively, if you are willing to pay lots of money, you can send a copy of 
the corrupt database to hwaci and pay a human who understands how SQLite works 
to rescue all the rescuable data from it.

Simon.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] disk image malformed

[Durga D]

Does disk io error is related to hardware issue? i mean, fsync() dint get
success on unix. so, this error may be thrown by sqlite. Is it?


On Fri, Dec 7, 2012 at 12:00 AM, Simon Slavin  wrote:

>
> On 6 Dec 2012, at 6:13pm, Durga D  wrote:
>
> > I suspect, linking among pages will  be broken in sqlite database when
> disk
> > io error thrown by sqlite. Is it?
>
> This is one form of database corruption.  But there are others.  Sometimes
> it's just a few strange characters inside a string.  Other times the
> corruption is in the first page of the database and is so bad that the
> database file cannot even be opened successfully.
>
> Simon.
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] disk image malformed

[Simon Slavin]

On 6 Dec 2012, at 6:49pm, sqdev sqdev  wrote:

> Please have a look at the post I mentioned:
> I ran "PRAGMA integrity_check" in the shell tool (SQLite CLI) and got
> errors, but still, they don't tell me what caused the corruption.

I did read it.  The shell tool can't tell you what the nature of the corruption 
is.  There are no automated tools which do that.  It has to be done by a human 
who has deep knowledge of how SQLite works.

As I wrote before: stop using the corrupted database.  Until you have done that 
we can't help you.

Simon.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] disk image malformed

[sqdev sqdev]

Please have a look at the post I mentioned:
I ran "PRAGMA integrity_check" in the shell tool (SQLite CLI) and got
errors, but still, they don't tell me what caused the corruption.



2012/12/6 Simon Slavin 

>
> On 6 Dec 2012, at 6:38pm, sqdev sqdev  wrote:
>
> > Maybe this post might be of interest here:
> >
> http://sqlite.1065341.n5.nabble.com/malformed-database-schema-td65610.html
> >
> > I had several corrupted databases the last few days. Unfortunately, I
> > haven't found out the cause yet.
>
> The answer, as always, is to try to open it in the shell tool.
>
> Simon.
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] disk image malformed

[Simon Slavin]

On 6 Dec 2012, at 6:38pm, sqdev sqdev  wrote:

> Maybe this post might be of interest here:
> http://sqlite.1065341.n5.nabble.com/malformed-database-schema-td65610.html
> 
> I had several corrupted databases the last few days. Unfortunately, I
> haven't found out the cause yet.

The answer, as always, is to try to open it in the shell tool.

Simon.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] disk image malformed

[sqdev sqdev]

Maybe this post might be of interest here:
http://sqlite.1065341.n5.nabble.com/malformed-database-schema-td65610.html

I had several corrupted databases the last few days. Unfortunately, I
haven't found out the cause yet.


2012/12/6 Durga D 

> I will run integrity check.
>
> >>I would like to know, why sqlite throws disk io error while inserting
>  records...
>
> I suspect, linking among pages will  be broken in sqlite database when disk
> io error thrown by sqlite. Is it?
>
>
> On Thu, Dec 6, 2012 at 11:30 PM, Simon Slavin 
> wrote:
>
> >
> > On 6 Dec 2012, at 5:23pm, Durga D  wrote:
> >
> > > Is it (corruption) related error code 10? (disk io)
> > >
> > > I would like to know, why sqlite throws disk io error while inserting
> > > records...
> >
> > It doesn't matter.  Run an in integrity check:
> >
> > 
> >
> > If the integrity check reports a problem, rescue the data you can.  Throw
> > away the corrupt database and make a new one.
> >
> > If the integrity check doesn't report a problem, your error is very
> > unlikely to be related to a corrupt database.  Check your hardware and
> your
> > software.
> >
> > Simon.
> > ___
> > sqlite-users mailing list
> > sqlite-users@sqlite.org
> > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
> >
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] disk image malformed

[Simon Slavin]

On 6 Dec 2012, at 6:13pm, Durga D  wrote:

> I suspect, linking among pages will  be broken in sqlite database when disk
> io error thrown by sqlite. Is it?

This is one form of database corruption.  But there are others.  Sometimes it's 
just a few strange characters inside a string.  Other times the corruption is 
in the first page of the database and is so bad that the database file cannot 
even be opened successfully.

Simon.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] disk image malformed

[Durga D]

I will run integrity check.

>>I would like to know, why sqlite throws disk io error while inserting
 records...

I suspect, linking among pages will  be broken in sqlite database when disk
io error thrown by sqlite. Is it?


On Thu, Dec 6, 2012 at 11:30 PM, Simon Slavin  wrote:

>
> On 6 Dec 2012, at 5:23pm, Durga D  wrote:
>
> > Is it (corruption) related error code 10? (disk io)
> >
> > I would like to know, why sqlite throws disk io error while inserting
> > records...
>
> It doesn't matter.  Run an in integrity check:
>
> 
>
> If the integrity check reports a problem, rescue the data you can.  Throw
> away the corrupt database and make a new one.
>
> If the integrity check doesn't report a problem, your error is very
> unlikely to be related to a corrupt database.  Check your hardware and your
> software.
>
> Simon.
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] disk image malformed

[Simon Slavin]

On 6 Dec 2012, at 5:23pm, Durga D  wrote:

> Is it (corruption) related error code 10? (disk io)
> 
> I would like to know, why sqlite throws disk io error while inserting
> records...

It doesn't matter.  Run an in integrity check:



If the integrity check reports a problem, rescue the data you can.  Throw away 
the corrupt database and make a new one.

If the integrity check doesn't report a problem, your error is very unlikely to 
be related to a corrupt database.  Check your hardware and your software.

Simon.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] disk image malformed

[Black, Michael (IS)]

Can you try doing table copies using select with offset and limit?

See where limit crashes the system and then offset past it perhaps?

Just a guess on my part as one possibility.

Michael D. Black
Senior Scientist
Advanced Analytics Directorate
Advanced GEOINT Solutions Operating Unit
Northrop Grumman Information Systems


From: sqlite-users-boun...@sqlite.org [sqlite-users-boun...@sqlite.org] on 
behalf of Durga D [durga.d...@gmail.com]
Sent: Thursday, December 06, 2012 11:04 AM
To: General Discussion of SQLite Database
Subject: EXT :Re: [sqlite] disk image malformed

i mean in second case. some records corrupted but not all. How to recover
this?


On Thu, Dec 6, 2012 at 10:27 PM, Durga D <durga.d...@gmail.com> wrote:

> Is there anyway to recover?
>
>
> On Thu, Dec 6, 2012 at 10:23 PM, Richard Hipp <d...@sqlite.org> wrote:
>
>> On Thu, Dec 6, 2012 at 11:52 AM, Durga D <durga.d...@gmail.com> wrote:
>>
>> > Hi,
>> >
>> >   Is it possible to corrupt a single table among 10 tables in a
>> database?
>> >
>> >   Is it possible to corrupt some records among millions of records in a
>> > table?
>> >
>>
>> Yes.  Yes.
>>
>>
>>
>> >
>> > Best Regards,
>> > ___
>> > sqlite-users mailing list
>> > sqlite-users@sqlite.org
>> > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>> >
>>
>>
>>
>> --
>> D. Richard Hipp
>> d...@sqlite.org
>> ___
>> sqlite-users mailing list
>> sqlite-users@sqlite.org
>> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>>
>
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] disk image malformed

[Durga D]

Is it (corruption) related error code 10? (disk io)

I would like to know, why sqlite throws disk io error while inserting
records...


On Thu, Dec 6, 2012 at 10:43 PM, Simon Slavin  wrote:

>
> On 6 Dec 2012, at 5:04pm, Durga D  wrote:
>
> > i mean in second case. some records corrupted but not all. How to recover
> > this?
>
> There's no fixed answer.  It depends on which parts of the file are
> corrupt, and what they had written to them.  If you used the shell tool on
> the database and dumped all the data using .dump you might get perfectly
> usable file with all data you could just .read back in.  On the other hand
> the corruption might be in a pointer that caused the .dump routine to
> crash, or go around in circles listing the same rows again and again.
>
> Simon.
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] disk image malformed

[Simon Slavin]

On 6 Dec 2012, at 5:04pm, Durga D  wrote:

> i mean in second case. some records corrupted but not all. How to recover
> this?

There's no fixed answer.  It depends on which parts of the file are corrupt, 
and what they had written to them.  If you used the shell tool on the database 
and dumped all the data using .dump you might get perfectly usable file with 
all data you could just .read back in.  On the other hand the corruption might 
be in a pointer that caused the .dump routine to crash, or go around in circles 
listing the same rows again and again.

Simon.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] disk image malformed

[Durga D]

i mean in second case. some records corrupted but not all. How to recover
this?


On Thu, Dec 6, 2012 at 10:27 PM, Durga D  wrote:

> Is there anyway to recover?
>
>
> On Thu, Dec 6, 2012 at 10:23 PM, Richard Hipp  wrote:
>
>> On Thu, Dec 6, 2012 at 11:52 AM, Durga D  wrote:
>>
>> > Hi,
>> >
>> >   Is it possible to corrupt a single table among 10 tables in a
>> database?
>> >
>> >   Is it possible to corrupt some records among millions of records in a
>> > table?
>> >
>>
>> Yes.  Yes.
>>
>>
>>
>> >
>> > Best Regards,
>> > ___
>> > sqlite-users mailing list
>> > sqlite-users@sqlite.org
>> > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>> >
>>
>>
>>
>> --
>> D. Richard Hipp
>> d...@sqlite.org
>> ___
>> sqlite-users mailing list
>> sqlite-users@sqlite.org
>> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>>
>
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] disk image malformed

[Durga D]

Is there anyway to recover?


On Thu, Dec 6, 2012 at 10:23 PM, Richard Hipp  wrote:

> On Thu, Dec 6, 2012 at 11:52 AM, Durga D  wrote:
>
> > Hi,
> >
> >   Is it possible to corrupt a single table among 10 tables in a database?
> >
> >   Is it possible to corrupt some records among millions of records in a
> > table?
> >
>
> Yes.  Yes.
>
>
>
> >
> > Best Regards,
> > ___
> > sqlite-users mailing list
> > sqlite-users@sqlite.org
> > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
> >
>
>
>
> --
> D. Richard Hipp
> d...@sqlite.org
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] disk image malformed

[Richard Hipp]

On Thu, Dec 6, 2012 at 11:52 AM, Durga D  wrote:

> Hi,
>
>   Is it possible to corrupt a single table among 10 tables in a database?
>
>   Is it possible to corrupt some records among millions of records in a
> table?
>

Yes.  Yes.



>
> Best Regards,
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>



-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] disk image malformed

[Durga D]

Hi,

  Is it possible to corrupt a single table among 10 tables in a database?

  Is it possible to corrupt some records among millions of records in a
table?

Best Regards,
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] disk image malformed with no system crash

[Claudio Cilloni]

Hi All.

I'm new to Sqlite, and I'm getting some rare and random 'database disk image 
is malformed' error. I have no system crashes, power failures or filesystem 
corruption. My machine is a Linux Debian, ext3 filesystem running on a flash 
disk. I'm using Sqlite 3.6.19 through Python 2.6.2.

I'm trying to get a repeatable case of this problem but, till now, with no 
luck. I even made a stress-test but 1000 times the activity created by my 
application didn't get me any errors.

My Sqlite database is created with auto_vacuum = full (if this was 
important...)

Could you suggest me something to do? The damaged database file could be 
analyzed to get ideas of what is going wrong?

Thank you.
Claudio
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] disk image malformed

[Kevin Schmeichel]

  I recently reported a similar error.  After
upgrading to sqlite3, I noticed that I would
occassionally get the "database disk image is
malformed" errors using the library code.  I just
closed/reopened the db, retried the query and always
had success on the retry.  No idea why - D. Richard
Hipp says that it's not possible for a database to be
temporarily corrupt.

  I've also sometimes observed a "some_table doesn't
exist" error.  Closing/reopening and retrying the
query also always works.

  I think that it's a good idea to have a wrapper
around the sqlite library that will retry queries a
few times if they don't return SQLITE_OK.  This is
especially useful in the cases when the db is busy but
the busy handler isn't called (because apparently that
would cause deadlock).

Kevin

--- Leander Gillard <[EMAIL PROTECTED]> wrote:

> Hello,
> 
> Just wondering why I get this error when trying to
> open the DB with the Binary interface. It has
> actually worked on the odd occasion.
> 
>  sqlite l_main.dbl
> Unable to open database "l_main.dbl": database disk
> image is malformed
> 
> 
> Regards,
> 
> 
> Leander


__
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

[sqlite] disk image malformed

[Leander Gillard]

Hello,

Just wondering why I get this error when trying to open the DB with the Binary 
interface. It has actually worked on the odd occasion.

 sqlite l_main.dbl
Unable to open database "l_main.dbl": database disk image is malformed


Regards,


Leander