On 04/03/2015 10:16 PM, Artem wrote:
> Hi!
>
> The situation is like that. There?s a SQLite database with around 3 billion
> records. Each record consists of a certain CHAR field and several other
> additional fields with different types. The file size is approx. 340 gb. The
> maximum content length in the doc field is 256 symbols, the content is in
> Russian.
>
> I?m trying to create a full-text index, but it results in a Segmentation
> Fault error. I?ve been trying to create it in different possible ways, both
> under Windows (with SQLite Expert and my own .NET software, including one
> with x64 architecture) and Linux (both Ubuntu and Centos). I?ve even compiled
> sqlite from the sources, having included necessary flags for FTS3 and FTS4,
> but every time I get one and the same error.
This does sound like a real problem, but one that might be difficult to
track down.
Are you able to get us a stack trace of the crash? Ideally one from a
build with compiler options "-g -DSQLITE_DEBUG" set.
Thanks,
Dan.
>
> I?ve tried two options:
> - creating a contentless FTS4, when content is stored in a regular table, and
> FTS-table contains only index (create virtual table docs_fts using
> fts4(content='docs'... )
> - creating a full-fledged FTS table from a regular one (insert into docs_fts
> select doc... from docs;)
>
> SQLite is functioning for about 4 hours, after which Segmentation Fault error
> occurs inevitably.
> There?re no NULL fields in the database.
>
> I?ve worked with 3 different SQLite versions, including the latest one,
> available on the website. I started trying to create the full-text index of
> the base shortly after it was created and filled; no other activity, apart
> from filling the base with data, was conveyed. It has only one docs table,
> that for sure doesn?t contain any NULL values.
> I also had an idea that Reindex and Vacuum might have done something wrong,
> however Reindex happens instantly and Vacuum works rather slowly, but
> successfully.
>
> In short, my actions are:
>
> CREATE TABLE docs ([id] INTEGER PRIMARY KEY AUTOINCREMENT, [doc] CHAR...
> (other fields here)
> (here?s the process of filling the base with the data, which are downloaded
> by means of my own script from text files.)
>
> CREATE VIRTUAL TABLE docs_fts using fts4 (content='docs', doc... (other
> fields here)
> (here?s an attempt of INSERT INTO docs_fts(docs_fts) VALUES ('rebuild') to
> rebuild contentless FTS index)
>
> or
>
> CREATE VIRTUAL TABLE docs_fts using fts4 (doc... (other fields here)
> (here?s an attempt of INSERT INTO docs_fts SELECT doc... from docs;)
>
> or
>
> CREATE VIRTUAL TABLE docs_fts using fts3 (doc... (other fields here)
> (here?s an attempt of INSERT INTO docs_fts SELECT doc... from docs;)
>
> For each attempt I?ve been making a new copy of the source file, because I
> suspected that the base could have got broken after Segmentation Fault. I
> even changed the ram-cards, in case if memory was the problem.
>
> But every time I get one and the same result - Segmentation Fault error.
>
> So, can you please pay your attention to this problem and fix it ASAP?
>
> I can send you a file if you need.
>
> Thank you.
>
> _______________________________________________
> sqlite-users mailing list
> sqlite-users at mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
