Hi!
The situation is like that. There?s a SQLite database with around 3 billion
records. Each record consists of a certain CHAR field and several other
additional fields with different types. The file size is approx. 340 gb. The
maximum content length in the doc field is 256 symbols, the content is in
Russian.
I?m trying to create a full-text index, but it results in a Segmentation Fault
error. I?ve been trying to create it in different possible ways, both under
Windows (with SQLite Expert and my own .NET software, including one with x64
architecture) and Linux (both Ubuntu and Centos). I?ve even compiled sqlite
from the sources, having included necessary flags for FTS3 and FTS4, but every
time I get one and the same error.
I?ve tried two options:
- creating a contentless FTS4, when content is stored in a regular table, and
FTS-table contains only index (create virtual table docs_fts using
fts4(content='docs'... )
- creating a full-fledged FTS table from a regular one (insert into docs_fts
select doc... from docs;)
SQLite is functioning for about 4 hours, after which Segmentation Fault error
occurs inevitably.
There?re no NULL fields in the database.
I?ve worked with 3 different SQLite versions, including the latest one,
available on the website. I started trying to create the full-text index of the
base shortly after it was created and filled; no other activity, apart from
filling the base with data, was conveyed. It has only one docs table, that for
sure doesn?t contain any NULL values.
I also had an idea that Reindex and Vacuum might have done something wrong,
however Reindex happens instantly and Vacuum works rather slowly, but
successfully.
In short, my actions are:
CREATE TABLE docs ([id] INTEGER PRIMARY KEY AUTOINCREMENT, [doc] CHAR... (other
fields here)
(here?s the process of filling the base with the data, which are downloaded by
means of my own script from text files.)
CREATE VIRTUAL TABLE docs_fts using fts4 (content='docs', doc... (other fields
here)
(here?s an attempt of INSERT INTO docs_fts(docs_fts) VALUES ('rebuild') to
rebuild contentless FTS index)
or
CREATE VIRTUAL TABLE docs_fts using fts4 (doc... (other fields here)
(here?s an attempt of INSERT INTO docs_fts SELECT doc... from docs;)
or
CREATE VIRTUAL TABLE docs_fts using fts3 (doc... (other fields here)
(here?s an attempt of INSERT INTO docs_fts SELECT doc... from docs;)
For each attempt I?ve been making a new copy of the source file, because I
suspected that the base could have got broken after Segmentation Fault. I even
changed the ram-cards, in case if memory was the problem.
But every time I get one and the same result - Segmentation Fault error.
So, can you please pay your attention to this problem and fix it ASAP?
I can send you a file if you need.
Thank you.
