Re: [sqlite] Database corrupted 28 billion bytes

2015-01-17 Thread Richard Hipp
On 1/16/15, MayW wrote: > Pages on the freelist (per header) 2144 0.008% > Pages on the freelist (calculated) 3344382 12.1% The fact that the freelist size as reported by the header is different from the actual freelist size is troubling. A VACUUM shou

Re: [sqlite] Database corrupted 28 billion bytes

2015-01-17 Thread MayW
Thank you very much! /** Disk-Space Utilization Report For junk.db3 Page size in bytes 1024 Pages in the whole file (measured) 27606264 Pages in the whole file (calculated).. 24264026 Pages that store data..

Re: [sqlite] Database corrupted 28 billion bytes

2015-01-16 Thread Richard Hipp
On 1/16/15, MayW wrote: > sqlite3_analyzer.exe ended with 16 INSERT statements showing in my Windows 7 > CMD console screen: > The interesting information was in the part that scrolled off the top of you console. I suggest you rerun the command, directing output into a file: sqlite3_analyz

Re: [sqlite] Database corrupted 28 billion bytes

2015-01-16 Thread Richard Hipp
On 1/16/15, MayW wrote: > Found table that was huge, it was named MyTable. > It was created with: > Create Table MyTable(comment); > > select max(rowid),* from MyTable; > 80002 "This","is"," 4" > > Drop table Mytable; > Vacuum. > It went down to a little over 3,000,000 bytes. > > I

Re: [sqlite] Database corrupted 28 billion bytes

2015-01-16 Thread MayW
sqlite3_analyzer.exe ended with 16 INSERT statements showing in my Windows 7 CMD console screen: They look similar to this one: INSERT INTO space_used VALUES(‘Deductions’,’Deductions’,0,0,0,0,0,0,16,0,0,1024); I remember helping someone try to get a UNION statement correct a few months ago. Th

Re: [sqlite] Database corrupted 28 billion bytes

2015-01-16 Thread MayW
Last statement 2 statements were: insert into space_used values(‘MyTable,”MyTable,0,728251738,704277138,588,0,0,26,285986,23974601,0,34710955,683321040,0,12,24842841088); COMMIT; - May -- View this message in context: http://sqlite.1065341.n5.nabble.com/Database-corrupted-28-billion-bytes-

Re: [sqlite] Database corrupted 28 billion bytes

2015-01-16 Thread MayW
>What is the output from the following: > PRAGMA page_size; > PRAGMA journal_mode; > PRAGMA freelist_count; > PRAGMA page_count; 1024 delete 0 27606264 The sqlite3_analyzer,exe is running. - May -- View this message in context: http://sqlite.1065

Re: [sqlite] Database corrupted 28 billion bytes

2015-01-16 Thread MayW
Found table that was huge, it was named MyTable. It was created with: Create Table MyTable(comment); select max(rowid),* from MyTable; 80002 "This","is"," 4" Drop table Mytable; Vacuum. It went down to a little over 3,000,000 bytes. I'm going to include a filesize routine withi

Re: [sqlite] Database corrupted 28 billion bytes

2015-01-16 Thread Simon Slavin
On 16 Jan 2015, at 11:18am, MikeD wrote: > Just terminated sqlite3_analyzer. You can let it run. Overnight if need be. Its CPU usage will never increase much past what you've already seen and memory usage shouldn't be excessive. Simon. ___ sqlite-

Re: [sqlite] Database corrupted 28 billion bytes

2015-01-16 Thread Richard Hipp
On 1/16/15, MikeD wrote: > I have a database that has become 28,268,814,336 bytes so > downloaded the sqlite3_analyzer and it has been running for over > 15-minutes. > > Task manager shows sqlite3_analyzer.exe using 13% and the memory stays > steady at 23,768K. > 19 handles, 1 thread(s). > > The d

Re: [sqlite] Database corrupted 28 billion bytes

2015-01-16 Thread Richard Hipp
On 1/16/15, MikeD wrote: > I have a database that has become 28,268,814,336 bytes so > downloaded the sqlite3_analyzer and it has been running for over > 15-minutes. > > Task manager shows sqlite3_analyzer.exe using 13% and the memory stays > steady at 23,768K. > 19 handles, 1 thread(s). > > The d

Re: [sqlite] Database corrupted 28 billion bytes

2015-01-16 Thread Stephan Beal
On Fri, Jan 16, 2015 at 12:18 PM, MikeD wrote: > I have a database that has become 28,268,814,336 bytes so > downloaded the sqlite3_analyzer and it has been running for over > 15-minutes. > ... > The database is still working. What should I be doing? > How about continue to use it? Why do you f

Re: [sqlite] Database Corrupted!

2013-05-28 Thread Simon Slavin
On 28 May 2013, at 8:30pm, Richard Hipp wrote: > On Tue, May 28, 2013 at 3:17 PM, Ryan Johnson > wrote: > >> Cheap (aka counterfeit) flash drives are notorious for advertizing more >> space to the OS than they actually have, and so at some point writes start >> to silently erase data that was w

Re: [sqlite] Database Corrupted!

2013-05-28 Thread Richard Hipp
On Tue, May 28, 2013 at 3:17 PM, Ryan Johnson wrote: > Cheap (aka counterfeit) flash drives are notorious for advertizing more > space to the OS than they actually have, and so at some point writes start > to silently erase data that was written earlier. I've heard the same thing, except that th

Re: [sqlite] Database Corrupted!

2013-05-28 Thread Ryan Johnson
On 27/05/2013 9:40 PM, Woody Wu wrote: On Mon, May 27, 2013 at 04:31:25PM +0100, Simon Slavin wrote: On 27 May 2013, at 4:22pm, Woody Wu wrote: If Yaffs2 is the cause, how can I write an effective test to exposure it? Do you have an opportunity to format the same drive in a different format

Re: [sqlite] Database Corrupted!

2013-05-28 Thread Woody Wu
On Tue, May 28, 2013 at 06:31:47AM -0500, Thanumalayan Sankaranarayana Pillai wrote: > Woody, this mailing list might not be the best place to discuss problems > with YAFFS2. Saying that, a simple test could be to almost fully fill the > YAFFS2 partition with a bunch of files, then read those file

Re: [sqlite] Database Corrupted!

2013-05-28 Thread Thanumalayan Sankaranarayana Pillai
Woody, this mailing list might not be the best place to discuss problems with YAFFS2. Saying that, a simple test could be to almost fully fill the YAFFS2 partition with a bunch of files, then read those files and make sure the files have the data they are supposed to have. Files should have sensibl

Re: [sqlite] Database Corrupted!

2013-05-27 Thread Woody Wu
On Tue, May 28, 2013 at 03:08:55AM +0100, Simon Slavin wrote: > > On 28 May 2013, at 2:37am, Woody Wu wrote: > > > How do you guys think about this: if NAND has an > > IO problem, Yaffs2 should recover it or forward the error to > > applications, right? > > Arguably. The file system can send a

Re: [sqlite] Database Corrupted!

2013-05-27 Thread Simon Slavin
On 28 May 2013, at 2:37am, Woody Wu wrote: > How do you guys think about this: if NAND has an > IO problem, Yaffs2 should recover it or forward the error to > applications, right? Arguably. The file system can send an error back to the application. If something does that to SQLite3 SQLite3 w

Re: [sqlite] Database Corrupted!

2013-05-27 Thread Woody Wu
On Mon, May 27, 2013 at 04:31:25PM +0100, Simon Slavin wrote: > > On 27 May 2013, at 4:22pm, Woody Wu wrote: > > > If Yaffs2 is the cause, how can I write an effective test to exposure it? > > Do you have an opportunity to format the same drive in a different format ? > I'm not telling you to

Re: [sqlite] Database Corrupted!

2013-05-27 Thread Woody Wu
On Mon, May 27, 2013 at 10:59:25AM -0500, Thanumalayan Sankaranarayana Pillai wrote: > Hi Woody, > > If the log messages that you see are "chunk nnn not erased", it might > probably be an error between your NAND device and YAFFS2 (according to a > couple of Google searches). > > Ref: http://osdi

Re: [sqlite] Database Corrupted!

2013-05-27 Thread Woody Wu
On Mon, May 27, 2013 at 11:33:08AM -0400, Richard Hipp wrote: > On Mon, May 27, 2013 at 11:22 AM, Woody Wu wrote: > > > Richard, > > > > If Yaffs2 is the cause, how can I write an effective test to exposure it? > > > > Do you really need to prove that yaffs2 is at fault? Try this experiment: >

Re: [sqlite] Database Corrupted!

2013-05-27 Thread Woody Wu
On Mon, May 27, 2013 at 04:31:25PM +0100, Simon Slavin wrote: > > On 27 May 2013, at 4:22pm, Woody Wu wrote: > > > If Yaffs2 is the cause, how can I write an effective test to exposure it? > > Do you have an opportunity to format the same drive in a different > format ? I'm not telling you to

Re: [sqlite] Database Corrupted!

2013-05-27 Thread Thanumalayan Sankaranarayana Pillai
Hi Woody, If the log messages that you see are "chunk nnn not erased", it might probably be an error between your NAND device and YAFFS2 (according to a couple of Google searches). Ref: http://osdir.com/ml/linux.file-systems.yaffs/2006-09/msg00033.html -- Thanu On Mon, May 27, 2013 at 10:22 AM

Re: [sqlite] Database Corrupted!

2013-05-27 Thread Richard Hipp
On Mon, May 27, 2013 at 11:31 AM, Simon Slavin wrote: > > > On one hand, I am not > > sure if this is caused by sqlite or Yaffs2 itself, on the other hand, I > > also cannot prove this really means bad things since the program at that > > moment was still running fine. > > The chances that this

Re: [sqlite] Database Corrupted!

2013-05-27 Thread Richard Hipp
On Mon, May 27, 2013 at 11:22 AM, Woody Wu wrote: > Richard, > > If Yaffs2 is the cause, how can I write an effective test to exposure it? > Do you really need to prove that yaffs2 is at fault? Try this experiment: mention that you might be having problems with yaffs2 to anybody else who has do

Re: [sqlite] Database Corrupted!

2013-05-27 Thread Simon Slavin
On 27 May 2013, at 4:22pm, Woody Wu wrote: > If Yaffs2 is the cause, how can I write an effective test to exposure it? Do you have an opportunity to format the same drive in a different format ? I'm not telling you to change your long-term practises, just to try a different format for testin

Re: [sqlite] Database Corrupted!

2013-05-27 Thread Woody Wu
Richard, If Yaffs2 is the cause, how can I write an effective test to exposure it? I straced my test sqlite test program and wanted to understand its IO behavior pattern. The difficulty is that I cannot run strace on my ARM target too long since the log will fill the limited memory, but if I don

Re: [sqlite] Database Corrupted!

2013-05-27 Thread Richard Hipp
On Mon, May 27, 2013 at 8:28 AM, Simon Slavin wrote: > > On 27 May 2013, at 1:25pm, Clemens Ladisch wrote: > > > Woody Wu wrote: > >> I have a testing code, attached in this email, if continuously run it > for > >> 20 - 40 hours, the sqlite database will be corrupted. > >> > >> The application i

Re: [sqlite] Database Corrupted!

2013-05-27 Thread Simon Slavin
On 27 May 2013, at 1:25pm, Clemens Ladisch wrote: > Woody Wu wrote: >> I have a testing code, attached in this email, if continuously run it for >> 20 - 40 hours, the sqlite database will be corrupted. >> >> The application is running on an ARM Linux system with Yaffs2 filesystem >> on NAND fla

Re: [sqlite] Database Corrupted!

2013-05-27 Thread Clemens Ladisch
Woody Wu wrote: > I have a testing code, attached in this email, if continuously run it for > 20 - 40 hours, the sqlite database will be corrupted. > > The application is running on an ARM Linux system with Yaffs2 filesystem > on NAND flashes. I'd guess that the flash is not very reliable. Does t

Re: [sqlite] Database Corrupted!

2013-05-27 Thread Woody Wu
On Mon, May 27, 2013 at 11:08:57AM +0800, Woody Wu wrote: > Hi, List > > Probably this is another case of database corrupted. I read the > documents about this topic and think I did not make same mistakes > described in that 'how to corrupt ...' documentation. > > I have a testing code, attached

Re: [sqlite] Database corrupted

2010-12-02 Thread Black, Michael (IS)
And your version is. Is your app a stand-alone you can share? If you 've discovered such a bug the community would be MUCH appreciative if you could share so it can be fixed (or at least identified to allay concerns we all may have over thread safety). Michael D. Black Senior Scien