On 04/03/2015 10:16 PM, Artem wrote: > Hi! > > The situation is like that. There?s a SQLite database with around 3 billion > records. Each record consists of a certain CHAR field and several other > additional fields with different types. The file size is approx. 340 gb. The > maximum content length in the doc field is 256 symbols, the content is in > Russian. > > I?m trying to create a full-text index, but it results in a Segmentation > Fault error. I?ve been trying to create it in different possible ways, both > under Windows (with SQLite Expert and my own .NET software, including one > with x64 architecture) and Linux (both Ubuntu and Centos). I?ve even compiled > sqlite from the sources, having included necessary flags for FTS3 and FTS4, > but every time I get one and the same error.
This does sound like a real problem, but one that might be difficult to track down. Are you able to get us a stack trace of the crash? Ideally one from a build with compiler options "-g -DSQLITE_DEBUG" set. Thanks, Dan. > > I?ve tried two options: > - creating a contentless FTS4, when content is stored in a regular table, and > FTS-table contains only index (create virtual table docs_fts using > fts4(content='docs'... ) > - creating a full-fledged FTS table from a regular one (insert into docs_fts > select doc... from docs;) > > SQLite is functioning for about 4 hours, after which Segmentation Fault error > occurs inevitably. > There?re no NULL fields in the database. > > I?ve worked with 3 different SQLite versions, including the latest one, > available on the website. I started trying to create the full-text index of > the base shortly after it was created and filled; no other activity, apart > from filling the base with data, was conveyed. It has only one docs table, > that for sure doesn?t contain any NULL values. > I also had an idea that Reindex and Vacuum might have done something wrong, > however Reindex happens instantly and Vacuum works rather slowly, but > successfully. > > In short, my actions are: > > CREATE TABLE docs ([id] INTEGER PRIMARY KEY AUTOINCREMENT, [doc] CHAR... > (other fields here) > (here?s the process of filling the base with the data, which are downloaded > by means of my own script from text files.) > > CREATE VIRTUAL TABLE docs_fts using fts4 (content='docs', doc... (other > fields here) > (here?s an attempt of INSERT INTO docs_fts(docs_fts) VALUES ('rebuild') to > rebuild contentless FTS index) > > or > > CREATE VIRTUAL TABLE docs_fts using fts4 (doc... (other fields here) > (here?s an attempt of INSERT INTO docs_fts SELECT doc... from docs;) > > or > > CREATE VIRTUAL TABLE docs_fts using fts3 (doc... (other fields here) > (here?s an attempt of INSERT INTO docs_fts SELECT doc... from docs;) > > For each attempt I?ve been making a new copy of the source file, because I > suspected that the base could have got broken after Segmentation Fault. I > even changed the ram-cards, in case if memory was the problem. > > But every time I get one and the same result - Segmentation Fault error. > > So, can you please pay your attention to this problem and fix it ASAP? > > I can send you a file if you need. > > Thank you. > > _______________________________________________ > sqlite-users mailing list > sqlite-users at mailinglists.sqlite.org > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users