[sqlite] Segfault during FTS index creation from huge data

2015-04-29 Thread Dan Kennedy
On 04/29/2015 01:55 AM, Scott Robison wrote: > On Tue, Apr 28, 2015 at 7:08 AM, Hick Gunter wrote: > >> Getting "NoMem" sounds very much like a memory leak somewhere, with the >> most likely place being your own application, followed by the wrapper you >> are using, the FTS code and lastly the

[sqlite] Segfault during FTS index creation from huge data

2015-04-29 Thread Dan Kennedy
On 04/29/2015 02:03 AM, Dan Kennedy wrote: > On 04/29/2015 01:55 AM, Scott Robison wrote: >> On Tue, Apr 28, 2015 at 7:08 AM, Hick Gunter wrote: >> >>> Getting "NoMem" sounds very much like a memory leak somewhere, with the >>> most likely place being your own application, followed by the >>>

[sqlite] SELECT performance vs PRIMARY KEY/UNIQUE constraints

2015-04-29 Thread Nicolas Boullis
Hi, On Mon, Apr 20, 2015 at 06:25:21AM -0400, Richard Hipp wrote: > > Please send us: > > (1) The output of "SELECT sqlite_source_id();" > > (2) The complete text of you SELECT statement. > > (3) The output of the ".fullschema" command from the most recent > version of the sqlite3.exe

[sqlite] Segfault during FTS index creation from huge data

2015-04-29 Thread Artem
Another one tip about this situation: when I make query like INSERT INTO test_fts(test_fts) VALUES('rebuild'); for the table with 100 000 000 rows - it works fine! When 1 000 000 000 - out of memory. So, the table and the query are correct. > How about trying the sqlite3.exe command line utility.

[sqlite] Segfault during FTS index creation from huge data

2015-04-29 Thread Tim Streater
On 28 Apr 2015 at 23:14, Artem wrote: >> How about trying the sqlite3.exe command line utility. put your >> sql for that operation in a text file, launch the program, open >> the database, then read in the sql file with the .read command. >> If the error occurs, then possibly sqlite3. if not

[sqlite] Segfault during FTS index creation from huge data

2015-04-29 Thread Artem
, Tim. ?? ?? 29 ?? 2015 ?., 1:21:00: > On 28 Apr 2015 at 23:14, Artem wrote: >>> How about trying the sqlite3.exe command line utility. put your >>> sql for that operation in a text file, launch the program, open >>> the database, then read in the sql file with the .read

[sqlite] Segfault during FTS index creation from huge data

2015-04-29 Thread Simon Slavin
On 28 Apr 2015, at 11:49pm, Scott Robison wrote: > I never saw a segfault in my case, though I never tried anything on any > posix style system. It was strictly Windows. Windows doesn't call it segfault. It will report an access violation or (very occasionally) a page fault. Simon.

[sqlite] Segfault during FTS index creation from huge data

2015-04-29 Thread Dan Kennedy
On 04/29/2015 05:27 AM, Artem wrote: > , Tim. > > ?? ?? 29 ?? 2015 ?., 1:21:00: > >> On 28 Apr 2015 at 23:14, Artem wrote: How about trying the sqlite3.exe command line utility. put your sql for that operation in a text file, launch the program, open the

[sqlite] Segfault during FTS index creation from huge data

2015-04-29 Thread Eduardo Morras
On Tue, 28 Apr 2015 16:49:46 -0600 Scott Robison wrote: > On Tue, Apr 28, 2015 at 4:27 PM, Artem wrote: > > > > That's not a segfault, though, is it. > > > > When I did the same in linux version of SQLite - I saw > > the "Segmentation Fault" error. > > > > I never saw a segfault in my case,

[sqlite] Segfault during FTS index creation from huge data

2015-04-29 Thread Scott Robison
Win 7 64 bit system, building & testing 32 & 64 bit versions of our app. On Apr 29, 2015 1:34 AM, "Eduardo Morras" wrote: > On Tue, 28 Apr 2015 16:49:46 -0600 > Scott Robison wrote: > > > On Tue, Apr 28, 2015 at 4:27 PM, Artem wrote: > > > > > > That's not a segfault, though, is it. > > > > >

[sqlite] Segfault during FTS index creation from huge data

2015-04-29 Thread Artem
So, is that a real bug of SQLIte and how to fix it? I really need to create huge FTS-indexes like that, I have 32GB of memory for that. Many thanks for your attention. > On 04/29/2015 05:27 AM, Artem wrote: >> , Tim. >> >> ?? ?? 29 ?? 2015 ?., 1:21:00: >> >>> On 28 Apr 2015

[sqlite] Segfault during FTS index creation from huge data

2015-04-29 Thread Scott Robison
Personally I don't see it as a bug. A limitation, yes. A different algorithm that requires less ram would remove / change the limit. I'll be trying some sample data tomorrow (if time permits) to see if I can come up with any ideas. I described an approach in a previous email where I divided my

[sqlite] Segfault during FTS index creation from huge data

2015-04-29 Thread Simon Slavin
On 29 Apr 2015, at 9:21am, Scott Robison wrote: > Personally I don't see it as a bug. A limitation, yes. A different > algorithm that requires less ram would remove / change the limit. Anything inside the SQLite code which causes a Segfault is a bug. Assuming that you're not hacking your OS

[sqlite] Segfault during FTS index creation from huge data

2015-04-29 Thread Artem
> Personally I don't see it as a bug. A limitation, yes. A different > algorithm that requires less ram would remove / change the limit. > I'll be trying some sample data tomorrow (if time permits) to see if I can > come up with any ideas. > I described an approach in a previous email where I

[sqlite] Segfault during FTS index creation from huge data

2015-04-29 Thread Scott Robison
On windows, malloc returns null if the allocation fails. Sqlite detects this and returns an error. On linux, malloc may return a non null yet invalid pointer and only fail when the memory is accessed because it wasn't really available. If Sqlite is not at fault when posix APIs lie about file

[sqlite] Segfault during FTS index creation from huge data

2015-04-29 Thread Dan Kennedy
On 04/29/2015 03:39 PM, Scott Robison wrote: > On windows, malloc returns null if the allocation fails. Sqlite detects > this and returns an error. > > On linux, malloc may return a non null yet invalid pointer and only fail > when the memory is accessed because it wasn't really available. That

[sqlite] Segfault during FTS index creation from huge data

2015-04-29 Thread Scott Robison
I created multiple FTS virtual tables and inserted data into specific tables based on ID. The FTS tables used the content option to specify the text was stored elsewhere. Something like "insert into FTSBUCKET# (...) ..." where # was computed from a rowid to spread the load around. This may not be

[sqlite] Segfault during FTS index creation from huge data

2015-04-29 Thread Scott Robison
On Apr 29, 2015 2:55 AM, "Dan Kennedy" wrote: > > On 04/29/2015 03:39 PM, Scott Robison wrote: >> >> On windows, malloc returns null if the allocation fails. Sqlite detects >> this and returns an error. >> >> On linux, malloc may return a non null yet invalid pointer and only fail >> when the

[sqlite] Segfault during FTS index creation from huge data

2015-04-29 Thread Simon Slavin
On 29 Apr 2015, at 9:39am, Scott Robison wrote: > On windows, malloc returns null if the allocation fails. Sqlite detects > this and returns an error. > > On linux, malloc may return a non null yet invalid pointer and only fail > when the memory is accessed because it wasn't really available.

[sqlite] Changing WAL mode for a transaction

2015-04-29 Thread Navaneeth K N
Hi Dan, > On 27-Apr-2015, at 11:34 am, Dan Kennedy wrote: > > On 04/27/2015 12:55 PM, Navaneeth K N wrote: >> Hi Dan, >> >> >>> On 27-Apr-2015, at 10:52 am, Dan Kennedy wrote: >>> >>> On 04/26/2015 07:01 PM, Navaneeth K N wrote: Hello, My application runs the following

[sqlite] Segfault during FTS index creation from huge data

2015-04-29 Thread Artem
> On Apr 29, 2015 2:55 AM, "Dan Kennedy" wrote: >> >> On 04/29/2015 03:39 PM, Scott Robison wrote: >>> >>> On windows, malloc returns null if the allocation fails. Sqlite detects >>> this and returns an error. >>> >>> On linux, malloc may return a non null yet invalid pointer and only fail >>>

[sqlite] Segfault during FTS index creation from huge data

2015-04-29 Thread Dominique Pellé
Simon Slavin wrote: > On 29 Apr 2015, at 9:39am, Scott Robison wrote: > >> On windows, malloc returns null if the allocation fails. Sqlite detects >> this and returns an error. >> >> On linux, malloc may return a non null yet invalid pointer and only fail >> when the memory is accessed because

[sqlite] Changing WAL mode for a transaction

2015-04-29 Thread Dan Kennedy
On 04/29/2015 04:28 PM, Navaneeth K N wrote: > Hi Dan, > > >> On 27-Apr-2015, at 11:34 am, Dan Kennedy wrote: >> >> On 04/27/2015 12:55 PM, Navaneeth K N wrote: >>> Hi Dan, >>> >>> On 27-Apr-2015, at 10:52 am, Dan Kennedy wrote: On 04/26/2015 07:01 PM, Navaneeth K N wrote: >

[sqlite] Segfault during FTS index creation from huge data

2015-04-29 Thread Eduardo Morras
On Wed, 29 Apr 2015 12:00:02 +0200 Dominique Pell? wrote: > I'd suggest to try reproducing the problem on Linux after disabling > memory overcommit (at least temporarily for the experimentation): > > * If it returns SQLITE_NOMEM without segfault, then it's not a bug > but a limitation of

[sqlite] Odd results with ICU extensions?

2015-04-29 Thread Philip Warner
Just came back to check on this, but there were no replies. Is there a better contact for the ICU extensions? On 14/04/2015 2:43 PM, Philip Warner wrote: > I've been using SqlCipher (built locally with ICU extensions) and can do the > following: > > sqlite> select

[sqlite] DB Browser for SQLite 3.6.0 released

2015-04-29 Thread Dominique Devienne
On Tue, Apr 28, 2015 at 12:44 AM, wrote: > Quick FYI. We've just released DB Browser for SQLite, version 3.6.0. > I tried pragma threads (Win64 build), and it returns nothing (unlike pragma page_size for example), while sqlite3.exe returns 0 (the default). Doesn't support worker threads for

[sqlite] how are the SQLITE_MAX_WORKER_THREADS and SQLITE_DEFAULT_WORKER_THREADS settings used to create multiple threads for a query ?

2015-04-29 Thread Jason Vas Dias
I am trying to make use of SQLite's multi-threading capabilities to create a temporary table from a query .Currently, I have not had enough time to wait for SQLite to create the temporary table ( I tried for over 2 hours) , but I can see from the /proc/${pid}/task listing for the sqlite3 $pid

[sqlite] Segfault during FTS index creation from huge data

2015-04-29 Thread Tim Streater
On 29 Apr 2015 at 00:06, Simon Slavin wrote: > On 28 Apr 2015, at 11:49pm, Scott Robison wrote: > >> I never saw a segfault in my case, though I never tried anything on any >> posix style system. It was strictly Windows. > > Windows doesn't call it segfault. It will report an access violation

[sqlite] how are the SQLITE_MAX_WORKER_THREADS and SQLITE_DEFAULT_WORKER_THREADS settings used to create multiple threads for a query ?

2015-04-29 Thread Simon Slavin
On 29 Apr 2015, at 3:22pm, Jason Vas Dias wrote: > My question is simply how to get sqlite3 to use multiple threads to create the > temporary table from a query If you are working on a database on disk (rather than one in memory), and most of your time is spend storing data in the

[sqlite] Segfault during FTS index creation from huge data

2015-04-29 Thread Scott Robison
On Wed, Apr 29, 2015 at 2:55 AM, Dan Kennedy wrote: > > That we're getting a segfault instead of SQLITE_NOMEM is not an SQLite bug. > > But that SQLite is requesting a ridiculously large allocation (assuming that is what is happening) is less than ideal as well though. Even if it's not

[sqlite] SQLite and Scientific Computing: Arrays and Complex Numbers

2015-04-29 Thread James K. Lowden
On Tue, 28 Apr 2015 09:24:56 +0200 Dominique Devienne wrote: > On Tue, Apr 28, 2015 at 4:16 AM, James K. Lowden > wrote: > > > A major hurdle is the memory model: because array-programming > > libraries normally mandate the data be in contiguous memory, > > there's a cost to converting to/from

[sqlite] Segfault during FTS index creation from huge data

2015-04-29 Thread James K. Lowden
On Wed, 29 Apr 2015 02:39:50 -0600 Scott Robison wrote: > On linux, malloc may return a non null yet invalid pointer and only > fail when the memory is accessed because it wasn't really available. Citation needed. I believe SIGSEGV is possible with anonymous mmap and overcommitted memory.

[sqlite] Segfault during FTS index creation from huge data

2015-04-29 Thread James K. Lowden
On Wed, 29 Apr 2015 01:27:03 +0300 Artem wrote: > >> Error: near line 1: out of memory > > > That's not a segfault, though, is it. > > When I did the same in linux version of SQLite - I saw > the "Segmentation Fault" error. The above message indicates that SQLite trapped an errno of ENOMEM

[sqlite] Segfault during FTS index creation from huge data

2015-04-29 Thread Scott Robison
On Wed, Apr 29, 2015 at 8:04 PM, James K. Lowden wrote: > On Wed, 29 Apr 2015 02:39:50 -0600 > Scott Robison wrote: > > > On linux, malloc may return a non null yet invalid pointer and only > > fail when the memory is accessed because it wasn't really available. > > Citation needed. I believe

[sqlite] Segfault during FTS index creation from huge data

2015-04-29 Thread Scott Robison
On Wed, Apr 29, 2015 at 8:23 PM, Scott Robison wrote: > > 1. From http://linux.die.net/man/3/malloc > > By default, Linux follows an optimistic memory allocation strategy. This >> means that when *malloc*() returns non-NULL there is no guarantee that >> the memory really is available. In case it