[sqlite] Segfault during FTS index creation from huge data

On 05/19/2015 05:54 PM, Artem wrote: >> On 05/19/2015 03:35 PM, Artem wrote: >>> Hi! >>> >>> And what about result of our conversation? >>> Can developers increase this limitations for using all memory that user >>> have? >> Hi Artem, >> The conclusion was that although the first problem encountere

[sqlite] Segfault during FTS index creation from huge data

On 05/19/2015 03:35 PM, Artem wrote: > Hi! > > And what about result of our conversation? > Can developers increase this limitations for using all memory that user > have? Hi Artem, The conclusion was that although the first problem encountered is the massive allocation that FTS tries to make, f

[sqlite] Segfault during FTS index creation from huge data

> On 05/19/2015 03:35 PM, Artem wrote: >> Hi! >> >> And what about result of our conversation? >> Can developers increase this limitations for using all memory that user >> have? > Hi Artem, > The conclusion was that although the first problem encountered is the > massive allocation that FTS tri

[sqlite] Segfault during FTS index creation from huge data

> On 19 May 2015, at 9:35am, Artem wrote: >> Can developers increase this limitations for using all memory that user >> have? > The answer to this depends on which operating system you are > running and how it is configured. The answers cannot be changed by changing > a setting in SQLite. Pl

[sqlite] Segfault during FTS index creation from huge data

Hi! And what about result of our conversation? Can developers increase this limitations for using all memory that user have? > One, you should remove sqlite-users at sqlite.org from your To list. I keep > bouncing email when I reply to you. Not a big deal, just an FYI. > Two: > On Sun, May 3, 2

[sqlite] Segfault during FTS index creation from huge data

On 19 May 2015, at 9:35am, Artem wrote: > Can developers increase this limitations for using all memory that user > have? The answer to this depends on which operating system you are running and how it is configured. The answers cannot be changed by changing a setting in SQLite. Simon.

[sqlite] Segfault during FTS index creation from huge data

On Thu, 30 Apr 2015 12:47:57 -0600 Scott Robison wrote: > Perhaps you are correct and "sigsegv" is not the literal signal that > is triggered in this case. I don't care, really. The fact is that an > apparently valid pointer was returned from a memory allocation > function yet can result in an in

[sqlite] Segfault during FTS index creation from huge data

One, you should remove sqlite-users at sqlite.org from your To list. I keep bouncing email when I reply to you. Not a big deal, just an FYI. Two: On Sun, May 3, 2015 at 2:13 PM, James K. Lowden wrote: > On Thu, 30 Apr 2015 12:47:57 -0600 > Scott Robison wrote: > > > Perhaps you are correct and

[sqlite] Segfault during FTS index creation from huge data

On 04/30/2015 02:08 PM, Dominique Pell? wrote: > Dan Kennedy wrote: > >> On 04/30/2015 07:41 AM, Scott Robison wrote: >>> >>> I wrote up some test code today that definitely forces a SQLITE_NOMEM >>> condition, and there are no leaks (though there are a couple really large >>> outstanding allocati

[sqlite] Segfault during FTS index creation from huge data

On Wed, 29 Apr 2015 20:29:07 -0600 Scott Robison wrote: > > That code can fail on a system configured to overcommit memory. By > > that standard, the pointer is invalid. > > > > Accidentally sent before I was finished. In any case, by "invalid > pointer" I did not mean to imply "it returns a bit

[sqlite] Segfault during FTS index creation from huge data

On 04/30/2015 07:41 AM, Scott Robison wrote: > > I wrote up some test code today that definitely forces a SQLITE_NOMEM > condition, and there are no leaks (though there are a couple really large > outstanding allocations to force an error without needing to insert > gigabytes of data). > > In think

[sqlite] Segfault during FTS index creation from huge data

On Thu, Apr 30, 2015 at 11:42 AM, James K. Lowden wrote: > On Wed, 29 Apr 2015 20:29:07 -0600 > Scott Robison wrote: > > > > That code can fail on a system configured to overcommit memory. By > > > that standard, the pointer is invalid. > > > > > > > Accidentally sent before I was finished. In a

[sqlite] Segfault during FTS index creation from huge data

On Fri, 3 Apr 2015 18:16:18 +0300 Artem wrote: > Hi! > > The situation is like that. There?s a SQLite database with around 3 > billion records. Each record consists of a certain CHAR field and > several other additional fields with different types. The file size > is approx. 340 gb. The maximum

[sqlite] Segfault during FTS index creation from huge data

This makes my head spin. ridiculous that an over commit even exists. any slight performance gain is totally nullified by a crashing program. - Scott Doctor scott at scottdoctor.com - On 4/30/2015 11:47 AM, Scott Robison wrote: > On Thu, Apr 30, 2015 at 11

[sqlite] Segfault during FTS index creation from huge data

On Apr 29, 2015 11:50 PM, "Dan Kennedy" wrote: > > On 04/30/2015 07:41 AM, Scott Robison wrote: >> >> >> I wrote up some test code today that definitely forces a SQLITE_NOMEM >> condition, and there are no leaks (though there are a couple really large >> outstanding allocations to force an error w

[sqlite] Segfault during FTS index creation from huge data

Dan Kennedy wrote: > On 04/30/2015 07:41 AM, Scott Robison wrote: >> >> >> I wrote up some test code today that definitely forces a SQLITE_NOMEM >> condition, and there are no leaks (though there are a couple really large >> outstanding allocations to force an error without needing to insert >> g

[sqlite] Segfault during FTS index creation from huge data

On Wed, 29 Apr 2015 01:27:03 +0300 Artem wrote: > >> Error: near line 1: out of memory > > > That's not a segfault, though, is it. > > When I did the same in linux version of SQLite - I saw > the "Segmentation Fault" error. The above message indicates that SQLite trapped an errno of ENOMEM aft

[sqlite] Segfault during FTS index creation from huge data

On Wed, 29 Apr 2015 02:39:50 -0600 Scott Robison wrote: > On linux, malloc may return a non null yet invalid pointer and only > fail when the memory is accessed because it wasn't really available. Citation needed. I believe SIGSEGV is possible with anonymous mmap and overcommitted memory. ISTR

[sqlite] Segfault during FTS index creation from huge data

On Wed, Apr 29, 2015 at 8:23 PM, Scott Robison wrote: > > 1. From http://linux.die.net/man/3/malloc > > By default, Linux follows an optimistic memory allocation strategy. This >> means that when *malloc*() returns non-NULL there is no guarantee that >> the memory really is available. In case it

[sqlite] Segfault during FTS index creation from huge data

On Wed, Apr 29, 2015 at 8:04 PM, James K. Lowden wrote: > On Wed, 29 Apr 2015 02:39:50 -0600 > Scott Robison wrote: > > > On linux, malloc may return a non null yet invalid pointer and only > > fail when the memory is accessed because it wasn't really available. > > Citation needed. I believe S

[sqlite] Segfault during FTS index creation from huge data

On Wed, Apr 29, 2015 at 2:55 AM, Dan Kennedy wrote: > > That we're getting a segfault instead of SQLITE_NOMEM is not an SQLite bug. > > But that SQLite is requesting a ridiculously large allocation (assuming that is what is happening) is less than ideal as well though. Even if it's not technically

[sqlite] Segfault during FTS index creation from huge data

On 29 Apr 2015 at 00:06, Simon Slavin wrote: > On 28 Apr 2015, at 11:49pm, Scott Robison wrote: > >> I never saw a segfault in my case, though I never tried anything on any >> posix style system. It was strictly Windows. > > Windows doesn't call it segfault. It will report an access violation

[sqlite] Segfault during FTS index creation from huge data

On 04/29/2015 03:39 PM, Scott Robison wrote: > On windows, malloc returns null if the allocation fails. Sqlite detects > this and returns an error. > > On linux, malloc may return a non null yet invalid pointer and only fail > when the memory is accessed because it wasn't really available. That we

[sqlite] Segfault during FTS index creation from huge data

On Wed, 29 Apr 2015 12:00:02 +0200 Dominique Pell? wrote: > I'd suggest to try reproducing the problem on Linux after disabling > memory overcommit (at least temporarily for the experimentation): > > * If it returns SQLITE_NOMEM without segfault, then it's not a bug > but a limitation of SQLi

[sqlite] Segfault during FTS index creation from huge data

On 04/29/2015 05:27 AM, Artem wrote: > , Tim. > > ?? ?? 29 ?? 2015 ?., 1:21:00: > >> On 28 Apr 2015 at 23:14, Artem wrote: How about trying the sqlite3.exe command line utility. put your sql for that operation in a text file, launch the program, open the database

[sqlite] Segfault during FTS index creation from huge data

> On Apr 29, 2015 2:55 AM, "Dan Kennedy" wrote: >> >> On 04/29/2015 03:39 PM, Scott Robison wrote: >>> >>> On windows, malloc returns null if the allocation fails. Sqlite detects >>> this and returns an error. >>> >>> On linux, malloc may return a non null yet invalid pointer and only fail >>> whe

[sqlite] Segfault during FTS index creation from huge data

Simon Slavin wrote: > On 29 Apr 2015, at 9:39am, Scott Robison wrote: > >> On windows, malloc returns null if the allocation fails. Sqlite detects >> this and returns an error. >> >> On linux, malloc may return a non null yet invalid pointer and only fail >> when the memory is accessed because i

[sqlite] Segfault during FTS index creation from huge data

> Personally I don't see it as a bug. A limitation, yes. A different > algorithm that requires less ram would remove / change the limit. > I'll be trying some sample data tomorrow (if time permits) to see if I can > come up with any ideas. > I described an approach in a previous email where I di

[sqlite] Segfault during FTS index creation from huge data

So, is that a real bug of SQLIte and how to fix it? I really need to create huge FTS-indexes like that, I have 32GB of memory for that. Many thanks for your attention. > On 04/29/2015 05:27 AM, Artem wrote: >> , Tim. >> >> ?? ?? 29 ?? 2015 ?., 1:21:00: >> >>> On 28 Apr 2015 at

[sqlite] Segfault during FTS index creation from huge data

On 29 Apr 2015, at 9:39am, Scott Robison wrote: > On windows, malloc returns null if the allocation fails. Sqlite detects > this and returns an error. > > On linux, malloc may return a non null yet invalid pointer and only fail > when the memory is accessed because it wasn't really available. >

[sqlite] Segfault during FTS index creation from huge data

On Tue, 28 Apr 2015 16:49:46 -0600 Scott Robison wrote: > On Tue, Apr 28, 2015 at 4:27 PM, Artem wrote: > > > > That's not a segfault, though, is it. > > > > When I did the same in linux version of SQLite - I saw > > the "Segmentation Fault" error. > > > > I never saw a segfault in my case, th

[sqlite] Segfault during FTS index creation from huge data

On 29 Apr 2015, at 9:21am, Scott Robison wrote: > Personally I don't see it as a bug. A limitation, yes. A different > algorithm that requires less ram would remove / change the limit. Anything inside the SQLite code which causes a Segfault is a bug. Assuming that you're not hacking your OS o

[sqlite] Segfault during FTS index creation from huge data

On Apr 29, 2015 2:55 AM, "Dan Kennedy" wrote: > > On 04/29/2015 03:39 PM, Scott Robison wrote: >> >> On windows, malloc returns null if the allocation fails. Sqlite detects >> this and returns an error. >> >> On linux, malloc may return a non null yet invalid pointer and only fail >> when the memo

[sqlite] Segfault during FTS index creation from huge data

I created multiple FTS virtual tables and inserted data into specific tables based on ID. The FTS tables used the content option to specify the text was stored elsewhere. Something like "insert into FTSBUCKET# (...) ..." where # was computed from a rowid to spread the load around. This may not be

[sqlite] Segfault during FTS index creation from huge data

On windows, malloc returns null if the allocation fails. Sqlite detects this and returns an error. On linux, malloc may return a non null yet invalid pointer and only fail when the memory is accessed because it wasn't really available. If Sqlite is not at fault when posix APIs lie about file lock

[sqlite] Segfault during FTS index creation from huge data

Personally I don't see it as a bug. A limitation, yes. A different algorithm that requires less ram would remove / change the limit. I'll be trying some sample data tomorrow (if time permits) to see if I can come up with any ideas. I described an approach in a previous email where I divided my F

[sqlite] Segfault during FTS index creation from huge data

On 04/29/2015 02:03 AM, Dan Kennedy wrote: > On 04/29/2015 01:55 AM, Scott Robison wrote: >> On Tue, Apr 28, 2015 at 7:08 AM, Hick Gunter wrote: >> >>> Getting "NoMem" sounds very much like a memory leak somewhere, with the >>> most likely place being your own application, followed by the >>> wra

[sqlite] Segfault during FTS index creation from huge data

On 04/29/2015 01:55 AM, Scott Robison wrote: > On Tue, Apr 28, 2015 at 7:08 AM, Hick Gunter wrote: > >> Getting "NoMem" sounds very much like a memory leak somewhere, with the >> most likely place being your own application, followed by the wrapper you >> are using, the FTS code and lastly the SQL

[sqlite] Segfault during FTS index creation from huge data

Win 7 64 bit system, building & testing 32 & 64 bit versions of our app. On Apr 29, 2015 1:34 AM, "Eduardo Morras" wrote: > On Tue, 28 Apr 2015 16:49:46 -0600 > Scott Robison wrote: > > > On Tue, Apr 28, 2015 at 4:27 PM, Artem wrote: > > > > > > That's not a segfault, though, is it. > > > > > >

[sqlite] Segfault during FTS index creation from huge data

, Tim. ?? ?? 29 ?? 2015 ?., 1:21:00: > On 28 Apr 2015 at 23:14, Artem wrote: >>> How about trying the sqlite3.exe command line utility. put your >>> sql for that operation in a text file, launch the program, open >>> the database, then read in the sql file with the .read

[sqlite] Segfault during FTS index creation from huge data

ave swapped out SQLite versions and even operating >>> systems without eliminating the problem, it seems rational to look >>> into the parts that have remained the same. >>> Maybe you could run a test on Linux under the control of valgrind >>> and have its memcheck too

[sqlite] Segfault during FTS index creation from huge data

On 28 Apr 2015, at 11:49pm, Scott Robison wrote: > I never saw a segfault in my case, though I never tried anything on any > posix style system. It was strictly Windows. Windows doesn't call it segfault. It will report an access violation or (very occasionally) a page fault. Simon.

[sqlite] Segfault during FTS index creation from huge data

On 28 Apr 2015 at 23:14, Artem wrote: >> How about trying the sqlite3.exe command line utility. put your >> sql for that operation in a text file, launch the program, open >> the database, then read in the sql file with the .read command. >> If the error occurs, then possibly sqlite3. if not

[sqlite] Segfault during FTS index creation from huge data

) To: General Discussion of SQLite Database Subject: Re: [sqlite] Segfault during FTS index creation from huge data No, I'm sure that is not a problem in my software, it exactly error of the SQLite library. My software is very simple - it creates simple connection to the database with con

[sqlite] Segfault during FTS index creation from huge data

Discussion of SQLite Database Subject: Re: [sqlite] Segfault during FTS index creation from huge data No, I'm sure that is not a problem in my software, it exactly error of the SQLite library. My software is very simple - it creates simple connection to the database with connection string like

[sqlite] Segfault during FTS index creation from huge data

On Tue, Apr 28, 2015 at 5:06 PM, Simon Slavin wrote: > > On 28 Apr 2015, at 11:49pm, Scott Robison wrote: > > > I never saw a segfault in my case, though I never tried anything on any > > posix style system. It was strictly Windows. > > Windows doesn't call it segfault. It will report an access

[sqlite] Segfault during FTS index creation from huge data

On Tue, Apr 28, 2015 at 4:27 PM, Artem wrote: > > That's not a segfault, though, is it. > > When I did the same in linux version of SQLite - I saw > the "Segmentation Fault" error. > I never saw a segfault in my case, though I never tried anything on any posix style system. It was strictly Windo

[sqlite] Segfault during FTS index creation from huge data

run a test on Linux under the control of valgrind > and have its memcheck tool take a look. > -Urspr?ngliche Nachricht- > Von: Artem [mailto:devspec at yandex.ru] > Gesendet: Dienstag, 28. April 2015 14:36 > An: General Discussion of SQLite Database > Betreff: Re: [sqlite] Se

[sqlite] Segfault during FTS index creation from huge data

> On 04/03/2015 10:16 PM, Artem wrote: >> Hi! >> >> The situation is like that. There?s a SQLite database with around 3 billion >> records. Each record consists of a certain CHAR field and several other >> additional fields with different types. The file size is approx. 340 gb. The >> maximum co

[sqlite] Segfault during FTS index creation from huge data

On Tue, Apr 28, 2015 at 1:05 PM, Dan Kennedy wrote: > I just took a quick glance at the FTS code. As I said, it has been a couple >>> years, but this looks like the malloc that was failing for me at the >>> time: >>> http://www.sqlite.org/cgi/src/artifact/81f9ed55ad586148?ln=2473 >>> >> >> That o

[sqlite] Segfault during FTS index creation from huge data

ult during FTS index creation from huge data > On 04/03/2015 10:16 PM, Artem wrote: >> Hi! >> >> The situation is like that. There?s a SQLite database with around 3 billion >> records. Each record consists of a certain CHAR field and several other >> additional fi

[sqlite] Segfault during FTS index creation from huge data

On Tue, Apr 28, 2015 at 1:03 PM, Dan Kennedy wrote: > On 04/29/2015 01:55 AM, Scott Robison wrote: > >> I just took a quick glance at the FTS code. As I said, it has been a >> couple >> years, but this looks like the malloc that was failing for me at the time: >> http://www.sqlite.org/cgi/src/art

[sqlite] Segfault during FTS index creation from huge data

On Tue, Apr 28, 2015 at 7:08 AM, Hick Gunter wrote: > Getting "NoMem" sounds very much like a memory leak somewhere, with the > most likely place being your own application, followed by the wrapper you > are using, the FTS code and lastly the SQLite core. Lastly because the > SQLite core is exten

[sqlite] Segfault during FTS index creation from huge data

ould run a test on Linux under the control of valgrind >> and have its memcheck tool take a look. >> -Urspr?ngliche Nachricht- >> Von: Artem [mailto:devspec at yandex.ru] >> Gesendet: Dienstag, 28. April 2015 14:36 >> An: General Discussion of SQLite Database &

[sqlite] Segfault during FTS index creation from huge data

On 04/03/2015 10:16 PM, Artem wrote: > Hi! > > The situation is like that. There?s a SQLite database with around 3 billion > records. Each record consists of a certain CHAR field and several other > additional fields with different types. The file size is approx. 340 gb. The > maximum content le

[sqlite] Segfault during FTS index creation from huge data

Hi! The situation is like that. There?s a SQLite database with around 3 billion records. Each record consists of a certain CHAR field and several other additional fields with different types. The file size is approx. 340 gb. The maximum content length in the doc field is 256 symbols, the conten

[sqlite] Segfault during FTS index creation from huge data

Artem wrote: ...snip... > SQLite is functioning for about 4 hours, after which Segmentation Fault error > occurs inevitably. ...snip... You did not provide any stack dump, so I don't think that SQLite developers can do anything to help you. Anyway, it's more likely that the bug is in your appli