I don't know if it will actually cause problems, but is the "?command.Dispose()" needed? Doesn't the "using" handle disposing?
Graham Sent from Samsung Mobile -------- Original message -------- From: Artem <devs...@yandex.ru> Date: 28/04/2015 14:29 (GMT+00:00) To: General Discussion of SQLite Database <sqlite-users at mailinglists.sqlite.org> Subject: Re: [sqlite] Segfault during FTS index creation from huge data No, I'm sure that is not a problem in my software, it exactly error of the SQLite library. My software is very simple - it creates simple connection to the database with connection string like "Data Source={0};New=false;Journal Mode=Off;Synchronous=Off;FailIfMissing=True" and executes a query like INSERT INTO test_fts(test_fts) VALUES('rebuild'); and that is all. I'm pretty sure because I got exactly the same error in SQLite Expert Professional - popular? desktop? sqlite-management? software, that uses another sqlite driver. P.S. Source code of my function: Using conn As New SQLiteConnection(String.Format("Data Source={0};New=false;Journal Mode=Off;Synchronous=Off;FailIfMissing=True", "f:\Suggests\suggests.db")) ??? conn.Open() ??? Using command = conn.CreateCommand ??????? command.CommandText = "INSERT INTO suggests_fts(suggests_fts) VALUES('rebuild');" ??????? command.ExecuteNonQuery() ??????? command.Dispose() ??? End Using ??? conn.Close() End Using P.S. I can send the database to someone who can try, 19 GB in rar-archive. > Getting "NoMem" sounds very much like a memory leak somewhere, with > the most likely place being your own application, followed by the > wrapper you are using, the FTS code and lastly the SQLite core. > Lastly because the SQLite core is extensively tested with an > explicit emphasis on not leaking memory (or other resources) in the > first place and secondly recovering gracefully from memory allocation > failures. > Also, since you have swapped out SQLite versions and even operating > systems without eliminating the problem, it seems rational to look > into the parts that have remained the same. > Maybe you could run a test on Linux under the control of valgrind > and have its memcheck tool take a look. > -----Urspr?ngliche Nachricht----- > Von: Artem [mailto:devspec at yandex.ru] > Gesendet: Dienstag, 28. April 2015 14:36 > An: General Discussion of SQLite Database > Betreff: Re: [sqlite] Segfault during FTS index creation from huge data >> On 04/03/2015 10:16 PM, Artem wrote: >>> Hi! >>> >>> The situation is like that. There?s a SQLite database with around 3 billion >>> records. Each record consists of a certain CHAR field and several other >>> additional fields with different types. The file size is approx. 340 gb. >>> The maximum content length in the doc field is 256 symbols, the content is >>> in Russian. >>> >>> I?m trying to create a full-text index, but it results in a Segmentation >>> Fault error. I?ve been trying to create it in different possible ways, both >>> under Windows (with SQLite Expert and my own .NET software, including one >>> with x64 architecture) and Linux (both Ubuntu and Centos). I?ve even >>> compiled sqlite from the sources, having included necessary flags for FTS3 >>> and FTS4, but every time I get one and the same error. >> This does sound like a real problem, but one that might be difficult >> to track down. >> Are you able to get us a stack trace of the crash? Ideally one from a >> build with compiler options "-g -DSQLITE_DEBUG" set. >> Thanks, >> Dan. > Hi, Dan. Now I can't to do this because I haven't Linux on my PC. > But I tried to create FTS table again (now it was another database with > 1 350 000 000 rows, smaller than before). And I got the same error (out of > memory) on function: > internal override SQLiteErrorCode Reset(SQLiteStatement stmt) > in file SQLite3.cs > It returns System.Data.SQLite.SQLiteErrorCode.NoMem. > I home it helps. > P.S.? It? is? latest? version? of SQLite.Net compiled in Visual Studio 2012. >>> >>> I?ve tried two options: >>> - creating a contentless FTS4, when content is stored in a regular >>> table, and FTS-table contains only index (create virtual table >>> docs_fts using fts4(content='docs'... ) >>> - creating a full-fledged FTS table from a regular one (insert into >>> docs_fts select doc... from docs;) >>> >>> SQLite is functioning for about 4 hours, after which Segmentation Fault >>> error occurs inevitably. >>> There?re no NULL fields in the database. >>> >>> I?ve worked with 3 different SQLite versions, including the latest one, >>> available on the website. I started trying to create the full-text index of >>> the base shortly after it was created and filled; no other activity, apart >>> from filling the base with data, was conveyed. It has only one docs table, >>> that for sure doesn?t contain any NULL values. >>> I also had an idea that Reindex and Vacuum might have done something wrong, >>> however Reindex happens instantly and Vacuum works rather slowly, but >>> successfully. >>> >>> In short, my actions are: >>> >>> CREATE TABLE docs ([id] INTEGER PRIMARY KEY AUTOINCREMENT, [doc] >>> CHAR... (other fields here) (here?s the process of filling the base >>> with the data, which are downloaded by means of my own script from >>> text files.) >>> >>> CREATE VIRTUAL TABLE docs_fts using fts4 (content='docs', doc... >>> (other fields here) (here?s an attempt of INSERT INTO >>> docs_fts(docs_fts) VALUES ('rebuild') to rebuild contentless FTS >>> index) >>> >>> or >>> >>> CREATE VIRTUAL TABLE docs_fts using fts4 (doc... (other fields here) >>> (here?s an attempt of INSERT INTO docs_fts SELECT doc... from docs;) >>> >>> or >>> >>> CREATE VIRTUAL TABLE docs_fts using fts3 (doc... (other fields here) >>> (here?s an attempt of INSERT INTO docs_fts SELECT doc... from docs;) >>> >>> For each attempt I?ve been making a new copy of the source file, because I >>> suspected that the base could have got broken after Segmentation Fault. I >>> even changed the ram-cards, in case if memory was the problem. >>> >>> But every time I get one and the same result - Segmentation Fault error. >>> >>> So, can you please pay your attention to this problem and fix it ASAP? >>> >>> I can send you a file if you need. >>> >>> Thank you. >>> >>> _______________________________________________ >>> sqlite-users mailing list >>> sqlite-users at mailinglists.sqlite.org >>> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users >> _______________________________________________ >> sqlite-users mailing list >> sqlite-users at mailinglists.sqlite.org >> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users > _______________________________________________ > sqlite-users mailing list > sqlite-users at mailinglists.sqlite.org > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users > ___________________________________________ >? Gunter Hick > Software Engineer > Scientific Games International GmbH > FN 157284 a, HG Wien > Klitschgasse 2-4, A-1130 Vienna, Austria > Tel: +43 1 80100 0 > E-Mail: hick at scigames.at > This communication (including any attachments) is intended for the > use of the intended recipient(s) only and may contain information > that is confidential, privileged or legally protected. Any > unauthorized use or dissemination of this communication is strictly > prohibited. If you have received this communication in error, please > immediately notify the sender by return e-mail message and delete > all copies of the original communication. Thank you for your cooperation. > _______________________________________________ > sqlite-users mailing list > sqlite-users at mailinglists.sqlite.org > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users -- ? ?????????, Artem????????????????????????? mailto:devspec at yandex.ru _______________________________________________ sqlite-users mailing list sqlite-users at mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users