Re: [sqlite] SQLite bug report - large databases only - 'database or disk is full'

2009-12-12 Thread pirx

Apologies for not answering earlier, I* was away.
I've never used Process Monitor - the whole setup you describing is new to me - 
but I will try to do something about it in the coming week. It has to be a 
background project for me - I do not use SQLite in production. 




-Original Message-
From: Filip Navara <filip.nav...@gmail.com>
To: General Discussion of SQLite Database <sqlite-users@sqlite.org>
Sent: Thu, Dec 10, 2009 1:35 am
Subject: Re: [sqlite] SQLite bug report - large databases only - 'database or 
disk is full'


On Mon, Dec 7, 2009 at 10:21 PM,  <p...@mail.com> wrote:

 SQLite bug report

 Summary:
 --

 error message:

   Error: near line 2: database or disk is full

 It happens with plenty of disk space available and with 'unlimited' database 
ize.
 It does not happen on all systems. It does not happen on small databases.

I run your scripts with SQLite 3.6.21 on Windows 7 64-bit on NTFS
rive and unfortunately I run out of disk space before the problem
anifested. Would it be possible for you to setup Process Monitor with
ilter on the database path (just the path, so both journal and the
ain database file are in the log) and history depth set to 1 million
the lowest value) and then capture the file accesses during the run
f the reproduction scripts? It is possible to save and export the
ata then for further analysis and hopefully it will give a clue on
hy it happens.
Best regards,
ilip Navara
__
qlite-users mailing list
qlite-us...@sqlite.org
ttp://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] SQLite bug report - large databases only - 'database or disk is full'

2009-12-10 Thread Filip Navara
On Mon, Dec 7, 2009 at 10:21 PM,   wrote:
>
> SQLite bug report
>
> Summary:
> --
>
> error message:
>
>       Error: near line 2: database or disk is full
>
> It happens with plenty of disk space available and with 'unlimited' database 
> size.
> It does not happen on all systems. It does not happen on small databases.
>

I run your scripts with SQLite 3.6.21 on Windows 7 64-bit on NTFS
drive and unfortunately I run out of disk space before the problem
manifested. Would it be possible for you to setup Process Monitor with
filter on the database path (just the path, so both journal and the
main database file are in the log) and history depth set to 1 million
(the lowest value) and then capture the file accesses during the run
of the reproduction scripts? It is possible to save and export the
data then for further analysis and hopefully it will give a clue on
why it happens.

Best regards,
Filip Navara
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] SQLite bug report - large databases only - 'database or disk is full'

2009-12-08 Thread Filip Navara
On Mon, Dec 7, 2009 at 10:21 PM,   wrote:
>
> SQLite bug report
>
> Summary:
> --
>
> error message:
>
>       Error: near line 2: database or disk is full
>
> It happens with plenty of disk space available and with 'unlimited' database 
> size.
> It does not happen on all systems. It does not happen on small databases.
>
>
> Details:
> --
>
>
> The error occurs on
> - windows server 2003 SP2 64bit, windows server 2009 64bit
> - windows vista, windows 7 rc1, windows 7 - all 64-bit
>
> The error does _not_ occur on
> - windows server 2003 SP2 32bit, windows xp sp3 32 bit

Just a short note, we experience very similar problem with our program
and the symptoms match exactly. So far we were unable to reproduce it
reliably on our systems, but we have plenty of bug reports from
production systems. I will post more details soon.

Our database files have page size 4Kb and all of the affected systems
are running Windows. We use incremental vacuum mode and the file sizes
are commonly over 1 Gb large, file system is NTFS (afaik).

Best regards,
Filip Navara
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


[sqlite] SQLite bug report - large databases only - 'database or disk is full'

2009-12-07 Thread pirx

SQLite bug report
 
Summary:
--

error message:

   Error: near line 2: database or disk is full

It happens with plenty of disk space available and with 'unlimited' database 
size.
It does not happen on all systems. It does not happen on small databases.

 
Details:
--


The error occurs on 
- windows server 2003 SP2 64bit, windows server 2009 64bit
- windows vista, windows 7 rc1, windows 7 - all 64-bit

The error does _not_ occur on 
- windows server 2003 SP2 32bit, windows xp sp3 32 bit

SQLite3 versions affected: all the ones released in 2009, possibly all the 
earlier ones, too.


The bug was 'distilled' with the reproduction described below. 



Reproduction
--

We (1) create a table in the database, (2) create an input file; (3) import the 
input file into the table mulltiple times.

Using the files attached below the error did occur during the third/fourth 
import, at database size around 22-28gb.



Additional information and observations


1. This is not a new bug - it's been around for the last year, probably two - 
but it was difficult to spot and isolate.

2. The problem is 'touchy' - with different random seed in the generated table 
the problem may occur at 45gb db size instead of 20-30gb. When I tried to use, 
as input data, the same line repeated millions of times - the problem did not 
occur at all - tests were aborted by me at 160-260 gb database size.

3.The problem does _not_ depend _purely_ on the data being imported. The 
reproduction script loads the same data set multiple times and fails on the 
third/fourth time - so it is the database size which triggers it.

4. When the input file is smaller (1gb instead of 8gb), the problem still 
occurs in the 20-30gb database size range - so there is nothing magical about 
the number of imports.

5. The test database here is created with pragma page_size = 32768. The same 
error message occurs for other page sizes and at various cache sizes. 

6. In some other tests (not using to the scripts here, with different data) the 
import was good (about 20gb size) but an attempt to create an index on the 
imported data resulted in the same error diagnostic - after about 30mins of 
running.

7. This is not an SQLITE3 problem - when using .NET wrapper for SQLite or using 
any of the admin tools - the problems occurs in the same area, even though 
messages are not reported the same way by these tools [and I am getting 
sometimes 'database image in malformed' after such tools].

 
 


 
Scripts to reproduce the bug
--

 
++
== awk program to produce the table - save as: t1b.w
BEGIN {
srand(13) # assure all tests have the same data
for (i=0;i inpb.txt
 
echo - load table with data sets the first 10 times
time /T
for %%i in (1,2,3,4,5,6,7,8,9,10) do c:\apps\sqlite3 tstb.db http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users