On 05/19/2015 05:54 PM, Artem wrote:
>> On 05/19/2015 03:35 PM, Artem wrote:
>>> Hi!
>>>
>>> And what about result of our conversation?
>>> Can developers increase this limitations for using all memory that user
>>> have?
>> Hi Artem,
>> The conclusion was that although the first problem
On 05/19/2015 03:35 PM, Artem wrote:
> Hi!
>
> And what about result of our conversation?
> Can developers increase this limitations for using all memory that user
> have?
Hi Artem,
The conclusion was that although the first problem encountered is the
massive allocation that FTS tries to make,
> On 05/19/2015 03:35 PM, Artem wrote:
>> Hi!
>>
>> And what about result of our conversation?
>> Can developers increase this limitations for using all memory that user
>> have?
> Hi Artem,
> The conclusion was that although the first problem encountered is the
> massive allocation that FTS
> On 19 May 2015, at 9:35am, Artem wrote:
>> Can developers increase this limitations for using all memory that user
>> have?
> The answer to this depends on which operating system you are
> running and how it is configured. The answers cannot be changed by changing
> a setting in SQLite.
Hi!
And what about result of our conversation?
Can developers increase this limitations for using all memory that user
have?
> One, you should remove sqlite-users at sqlite.org from your To list. I keep
> bouncing email when I reply to you. Not a big deal, just an FYI.
> Two:
> On Sun, May 3,
On 19 May 2015, at 9:35am, Artem wrote:
> Can developers increase this limitations for using all memory that user
> have?
The answer to this depends on which operating system you are running and how it
is configured. The answers cannot be changed by changing a setting in SQLite.
Simon.
On Thu, 30 Apr 2015 12:47:57 -0600
Scott Robison wrote:
> Perhaps you are correct and "sigsegv" is not the literal signal that
> is triggered in this case. I don't care, really. The fact is that an
> apparently valid pointer was returned from a memory allocation
> function yet can result in an
One, you should remove sqlite-users at sqlite.org from your To list. I keep
bouncing email when I reply to you. Not a big deal, just an FYI.
Two:
On Sun, May 3, 2015 at 2:13 PM, James K. Lowden
wrote:
> On Thu, 30 Apr 2015 12:47:57 -0600
> Scott Robison wrote:
>
> > Perhaps you are correct
On 04/30/2015 02:08 PM, Dominique Pell? wrote:
> Dan Kennedy wrote:
>
>> On 04/30/2015 07:41 AM, Scott Robison wrote:
>>>
>>> I wrote up some test code today that definitely forces a SQLITE_NOMEM
>>> condition, and there are no leaks (though there are a couple really large
>>> outstanding
On Wed, 29 Apr 2015 20:29:07 -0600
Scott Robison wrote:
> > That code can fail on a system configured to overcommit memory. By
> > that standard, the pointer is invalid.
> >
>
> Accidentally sent before I was finished. In any case, by "invalid
> pointer" I did not mean to imply "it returns a
On 04/30/2015 07:41 AM, Scott Robison wrote:
>
> I wrote up some test code today that definitely forces a SQLITE_NOMEM
> condition, and there are no leaks (though there are a couple really large
> outstanding allocations to force an error without needing to insert
> gigabytes of data).
>
> In
On Thu, Apr 30, 2015 at 11:42 AM, James K. Lowden
wrote:
> On Wed, 29 Apr 2015 20:29:07 -0600
> Scott Robison wrote:
>
> > > That code can fail on a system configured to overcommit memory. By
> > > that standard, the pointer is invalid.
> > >
> >
> > Accidentally sent before I was finished. In
On Fri, 3 Apr 2015 18:16:18 +0300
Artem wrote:
> Hi!
>
> The situation is like that. There?s a SQLite database with around 3
> billion records. Each record consists of a certain CHAR field and
> several other additional fields with different types. The file size
> is approx. 340 gb. The maximum
This makes my head spin. ridiculous that an over commit even exists. any
slight performance gain is totally nullified by a crashing program.
-
Scott Doctor
scott at scottdoctor.com
-
On 4/30/2015 11:47 AM, Scott Robison wrote:
> On Thu, Apr 30, 2015 at
On Apr 29, 2015 11:50 PM, "Dan Kennedy" wrote:
>
> On 04/30/2015 07:41 AM, Scott Robison wrote:
>>
>>
>> I wrote up some test code today that definitely forces a SQLITE_NOMEM
>> condition, and there are no leaks (though there are a couple really large
>> outstanding allocations to force an error
Dan Kennedy wrote:
> On 04/30/2015 07:41 AM, Scott Robison wrote:
>>
>>
>> I wrote up some test code today that definitely forces a SQLITE_NOMEM
>> condition, and there are no leaks (though there are a couple really large
>> outstanding allocations to force an error without needing to insert
>>
On Wed, 29 Apr 2015 01:27:03 +0300
Artem wrote:
> >> Error: near line 1: out of memory
>
> > That's not a segfault, though, is it.
>
> When I did the same in linux version of SQLite - I saw
> the "Segmentation Fault" error.
The above message indicates that SQLite trapped an errno of ENOMEM
On Wed, 29 Apr 2015 02:39:50 -0600
Scott Robison wrote:
> On linux, malloc may return a non null yet invalid pointer and only
> fail when the memory is accessed because it wasn't really available.
Citation needed. I believe SIGSEGV is possible with anonymous mmap and
overcommitted memory.
On Wed, Apr 29, 2015 at 8:23 PM, Scott Robison
wrote:
>
> 1. From http://linux.die.net/man/3/malloc
>
> By default, Linux follows an optimistic memory allocation strategy. This
>> means that when *malloc*() returns non-NULL there is no guarantee that
>> the memory really is available. In case it
On Wed, Apr 29, 2015 at 8:04 PM, James K. Lowden
wrote:
> On Wed, 29 Apr 2015 02:39:50 -0600
> Scott Robison wrote:
>
> > On linux, malloc may return a non null yet invalid pointer and only
> > fail when the memory is accessed because it wasn't really available.
>
> Citation needed. I believe
On Wed, Apr 29, 2015 at 2:55 AM, Dan Kennedy wrote:
>
> That we're getting a segfault instead of SQLITE_NOMEM is not an SQLite
bug.
>
> But that SQLite is requesting a ridiculously large allocation (assuming
that is what is happening) is less than ideal as well though. Even if it's
not
On 29 Apr 2015 at 00:06, Simon Slavin wrote:
> On 28 Apr 2015, at 11:49pm, Scott Robison wrote:
>
>> I never saw a segfault in my case, though I never tried anything on any
>> posix style system. It was strictly Windows.
>
> Windows doesn't call it segfault. It will report an access violation
On 04/29/2015 03:39 PM, Scott Robison wrote:
> On windows, malloc returns null if the allocation fails. Sqlite detects
> this and returns an error.
>
> On linux, malloc may return a non null yet invalid pointer and only fail
> when the memory is accessed because it wasn't really available.
That
On Wed, 29 Apr 2015 12:00:02 +0200
Dominique Pell? wrote:
> I'd suggest to try reproducing the problem on Linux after disabling
> memory overcommit (at least temporarily for the experimentation):
>
> * If it returns SQLITE_NOMEM without segfault, then it's not a bug
> but a limitation of
On 04/29/2015 05:27 AM, Artem wrote:
> , Tim.
>
> ?? ?? 29 ?? 2015 ?., 1:21:00:
>
>> On 28 Apr 2015 at 23:14, Artem wrote:
How about trying the sqlite3.exe command line utility. put your
sql for that operation in a text file, launch the program, open
the
> On Apr 29, 2015 2:55 AM, "Dan Kennedy" wrote:
>>
>> On 04/29/2015 03:39 PM, Scott Robison wrote:
>>>
>>> On windows, malloc returns null if the allocation fails. Sqlite detects
>>> this and returns an error.
>>>
>>> On linux, malloc may return a non null yet invalid pointer and only fail
>>>
Simon Slavin wrote:
> On 29 Apr 2015, at 9:39am, Scott Robison wrote:
>
>> On windows, malloc returns null if the allocation fails. Sqlite detects
>> this and returns an error.
>>
>> On linux, malloc may return a non null yet invalid pointer and only fail
>> when the memory is accessed because
> Personally I don't see it as a bug. A limitation, yes. A different
> algorithm that requires less ram would remove / change the limit.
> I'll be trying some sample data tomorrow (if time permits) to see if I can
> come up with any ideas.
> I described an approach in a previous email where I
So, is that a real bug of SQLIte and how to fix it?
I really need to create huge FTS-indexes like that,
I have 32GB of memory for that.
Many thanks for your attention.
> On 04/29/2015 05:27 AM, Artem wrote:
>> , Tim.
>>
>> ?? ?? 29 ?? 2015 ?., 1:21:00:
>>
>>> On 28 Apr 2015
On 29 Apr 2015, at 9:39am, Scott Robison wrote:
> On windows, malloc returns null if the allocation fails. Sqlite detects
> this and returns an error.
>
> On linux, malloc may return a non null yet invalid pointer and only fail
> when the memory is accessed because it wasn't really available.
On Tue, 28 Apr 2015 16:49:46 -0600
Scott Robison wrote:
> On Tue, Apr 28, 2015 at 4:27 PM, Artem wrote:
>
> > > That's not a segfault, though, is it.
> >
> > When I did the same in linux version of SQLite - I saw
> > the "Segmentation Fault" error.
> >
>
> I never saw a segfault in my case,
On 29 Apr 2015, at 9:21am, Scott Robison wrote:
> Personally I don't see it as a bug. A limitation, yes. A different
> algorithm that requires less ram would remove / change the limit.
Anything inside the SQLite code which causes a Segfault is a bug. Assuming
that you're not hacking your OS
On Apr 29, 2015 2:55 AM, "Dan Kennedy" wrote:
>
> On 04/29/2015 03:39 PM, Scott Robison wrote:
>>
>> On windows, malloc returns null if the allocation fails. Sqlite detects
>> this and returns an error.
>>
>> On linux, malloc may return a non null yet invalid pointer and only fail
>> when the
I created multiple FTS virtual tables and inserted data into specific
tables based on ID. The FTS tables used the content option to specify the
text was stored elsewhere. Something like "insert into FTSBUCKET# (...)
..." where # was computed from a rowid to spread the load around.
This may not be
On windows, malloc returns null if the allocation fails. Sqlite detects
this and returns an error.
On linux, malloc may return a non null yet invalid pointer and only fail
when the memory is accessed because it wasn't really available.
If Sqlite is not at fault when posix APIs lie about file
Personally I don't see it as a bug. A limitation, yes. A different
algorithm that requires less ram would remove / change the limit.
I'll be trying some sample data tomorrow (if time permits) to see if I can
come up with any ideas.
I described an approach in a previous email where I divided my
On 04/29/2015 02:03 AM, Dan Kennedy wrote:
> On 04/29/2015 01:55 AM, Scott Robison wrote:
>> On Tue, Apr 28, 2015 at 7:08 AM, Hick Gunter wrote:
>>
>>> Getting "NoMem" sounds very much like a memory leak somewhere, with the
>>> most likely place being your own application, followed by the
>>>
On 04/29/2015 01:55 AM, Scott Robison wrote:
> On Tue, Apr 28, 2015 at 7:08 AM, Hick Gunter wrote:
>
>> Getting "NoMem" sounds very much like a memory leak somewhere, with the
>> most likely place being your own application, followed by the wrapper you
>> are using, the FTS code and lastly the
Win 7 64 bit system, building & testing 32 & 64 bit versions of our app.
On Apr 29, 2015 1:34 AM, "Eduardo Morras" wrote:
> On Tue, 28 Apr 2015 16:49:46 -0600
> Scott Robison wrote:
>
> > On Tue, Apr 28, 2015 at 4:27 PM, Artem wrote:
> >
> > > > That's not a segfault, though, is it.
> > >
> >
, Tim.
?? ?? 29 ?? 2015 ?., 1:21:00:
> On 28 Apr 2015 at 23:14, Artem wrote:
>>> How about trying the sqlite3.exe command line utility. put your
>>> sql for that operation in a text file, launch the program, open
>>> the database, then read in the sql file with the .read
t;> systems without eliminating the problem, it seems rational to look
>>> into the parts that have remained the same.
>>> Maybe you could run a test on Linux under the control of valgrind
>>> and have its memcheck tool take a look.
>>> -Urspr?ngliche Nachricht
On 28 Apr 2015, at 11:49pm, Scott Robison wrote:
> I never saw a segfault in my case, though I never tried anything on any
> posix style system. It was strictly Windows.
Windows doesn't call it segfault. It will report an access violation or (very
occasionally) a page fault.
Simon.
On 28 Apr 2015 at 23:14, Artem wrote:
>> How about trying the sqlite3.exe command line utility. put your
>> sql for that operation in a text file, launch the program, open
>> the database, then read in the sql file with the .read command.
>> If the error occurs, then possibly sqlite3. if not
(GMT+00:00)
To: General Discussion of SQLite Database
Subject: Re: [sqlite] Segfault during FTS index creation from huge data
No, I'm sure that is not a problem in my software, it exactly
error of the SQLite library. My software is very simple - it creates
simple connection to the database with co
0)
To: General Discussion of SQLite Database
Subject: Re: [sqlite] Segfault during FTS index creation from huge data
No, I'm sure that is not a problem in my software, it exactly
error of the SQLite library. My software is very simple - it creates
simple connection to the database with connection s
On Tue, Apr 28, 2015 at 5:06 PM, Simon Slavin wrote:
>
> On 28 Apr 2015, at 11:49pm, Scott Robison wrote:
>
> > I never saw a segfault in my case, though I never tried anything on any
> > posix style system. It was strictly Windows.
>
> Windows doesn't call it segfault. It will report an
On Tue, Apr 28, 2015 at 4:27 PM, Artem wrote:
> > That's not a segfault, though, is it.
>
> When I did the same in linux version of SQLite - I saw
> the "Segmentation Fault" error.
>
I never saw a segfault in my case, though I never tried anything on any
posix style system. It was strictly
control of valgrind
> and have its memcheck tool take a look.
> -Urspr?ngliche Nachricht-
> Von: Artem [mailto:devspec at yandex.ru]
> Gesendet: Dienstag, 28. April 2015 14:36
> An: General Discussion of SQLite Database
> Betreff: Re: [sqlite] Segfault during FTS index c
> On 04/03/2015 10:16 PM, Artem wrote:
>> Hi!
>>
>> The situation is like that. There?s a SQLite database with around 3 billion
>> records. Each record consists of a certain CHAR field and several other
>> additional fields with different types. The file size is approx. 340 gb. The
>> maximum
On Tue, Apr 28, 2015 at 1:05 PM, Dan Kennedy wrote:
> I just took a quick glance at the FTS code. As I said, it has been a couple
>>> years, but this looks like the malloc that was failing for me at the
>>> time:
>>> http://www.sqlite.org/cgi/src/artifact/81f9ed55ad586148?ln=2473
>>>
>>
>> That
ult during FTS index creation from huge data
> On 04/03/2015 10:16 PM, Artem wrote:
>> Hi!
>>
>> The situation is like that. There?s a SQLite database with around 3 billion
>> records. Each record consists of a certain CHAR field and several other
>> additional
On Tue, Apr 28, 2015 at 1:03 PM, Dan Kennedy wrote:
> On 04/29/2015 01:55 AM, Scott Robison wrote:
>
>> I just took a quick glance at the FTS code. As I said, it has been a
>> couple
>> years, but this looks like the malloc that was failing for me at the time:
>>
On Tue, Apr 28, 2015 at 7:08 AM, Hick Gunter wrote:
> Getting "NoMem" sounds very much like a memory leak somewhere, with the
> most likely place being your own application, followed by the wrapper you
> are using, the FTS code and lastly the SQLite core. Lastly because the
> SQLite core is
der the control of valgrind
>> and have its memcheck tool take a look.
>> -----Urspr?ngliche Nachricht-
>> Von: Artem [mailto:devspec at yandex.ru]
>> Gesendet: Dienstag, 28. April 2015 14:36
>> An: General Discussion of SQLite Database
>> Betreff: Re: [sqlite]
On 04/03/2015 10:16 PM, Artem wrote:
> Hi!
>
> The situation is like that. There?s a SQLite database with around 3 billion
> records. Each record consists of a certain CHAR field and several other
> additional fields with different types. The file size is approx. 340 gb. The
> maximum content
Hi!
The situation is like that. There?s a SQLite database with around 3 billion
records. Each record consists of a certain CHAR field and several other
additional fields with different types. The file size is approx. 340 gb. The
maximum content length in the doc field is 256 symbols, the
Artem wrote:
...snip...
> SQLite is functioning for about 4 hours, after which Segmentation Fault error
> occurs inevitably.
...snip...
You did not provide any stack dump, so I don't think that SQLite developers
can do anything to help you. Anyway, it's more likely that the bug is in
your
57 matches
Mail list logo