Hi!

The situation is like that. There?s a SQLite database with around 3 billion 
records. Each record consists of a certain CHAR field and several other 
additional fields with different types. The file size is approx. 340 gb. The 
maximum content length in the doc field is 256 symbols, the content is in 
Russian.

I?m trying to create a full-text index, but it results in a Segmentation Fault 
error. I?ve been trying to create it in different possible ways, both under 
Windows (with SQLite Expert and my own .NET software, including one with x64 
architecture) and Linux (both Ubuntu and Centos). I?ve even compiled sqlite 
from the sources, having included necessary flags for FTS3 and FTS4, but every 
time I get one and the same error. 

I?ve tried two options:
- creating a contentless FTS4, when content is stored in a regular table, and 
FTS-table contains only index (create virtual table docs_fts using 
fts4(content='docs'... )
- creating a full-fledged FTS table from a regular one (insert into docs_fts 
select doc... from docs;)

SQLite is functioning for about 4 hours, after which Segmentation Fault error 
occurs inevitably. 
There?re no NULL fields in the database. 

I?ve worked with 3 different SQLite versions, including the latest one, 
available on the website. I started trying to create the full-text index of the 
base shortly after it was created and filled; no other activity, apart from 
filling the base with data, was conveyed. It has only one docs table, that for 
sure doesn?t contain any NULL values. 
I also had an idea that Reindex and Vacuum might have done something wrong, 
however Reindex happens instantly and Vacuum works rather slowly, but 
successfully. 

In short, my actions are:

CREATE TABLE docs ([id] INTEGER PRIMARY KEY AUTOINCREMENT, [doc] CHAR... (other 
fields here)
(here?s the process of filling the base with the data, which are downloaded by 
means of my own script from text files.)

CREATE VIRTUAL TABLE docs_fts using fts4 (content='docs', doc... (other fields 
here)
(here?s an attempt of INSERT INTO docs_fts(docs_fts) VALUES ('rebuild') to 
rebuild contentless FTS index)

or

CREATE VIRTUAL TABLE docs_fts using fts4 (doc... (other fields here)
(here?s an attempt of INSERT INTO docs_fts SELECT doc... from docs;)

or

CREATE VIRTUAL TABLE docs_fts using fts3 (doc... (other fields here)
(here?s an attempt of INSERT INTO docs_fts SELECT doc... from docs;)

For each attempt I?ve been making a new copy of the source file, because I 
suspected that the base could have got broken after Segmentation Fault. I even 
changed the ram-cards, in case if memory was the problem.

But every time I get one and the same result - Segmentation Fault error.

So, can you please pay your attention to this problem and fix it ASAP?

I can send you a file if you need.

Thank you.

Reply via email to