[sqlite] Segfault during FTS index creation from huge data
Hi!
The situation is like that. There?s a SQLite database with around 3 billion
records. Each record consists of a certain CHAR field and several other
additional fields with different types. The file size is approx. 340 gb. The
maximum content length in the doc field is 256 symbols, the content is in
Russian.
I?m trying to create a full-text index, but it results in a Segmentation Fault
error. I?ve been trying to create it in different possible ways, both under
Windows (with SQLite Expert and my own .NET software, including one with x64
architecture) and Linux (both Ubuntu and Centos). I?ve even compiled sqlite
from the sources, having included necessary flags for FTS3 and FTS4, but every
time I get one and the same error.
I?ve tried two options:
- creating a contentless FTS4, when content is stored in a regular table, and
FTS-table contains only index (create virtual table docs_fts using
fts4(content='docs'... )
- creating a full-fledged FTS table from a regular one (insert into docs_fts
select doc... from docs;)
SQLite is functioning for about 4 hours, after which Segmentation Fault error
occurs inevitably.
There?re no NULL fields in the database.
I?ve worked with 3 different SQLite versions, including the latest one,
available on the website. I started trying to create the full-text index of the
base shortly after it was created and filled; no other activity, apart from
filling the base with data, was conveyed. It has only one docs table, that for
sure doesn?t contain any NULL values.
I also had an idea that Reindex and Vacuum might have done something wrong,
however Reindex happens instantly and Vacuum works rather slowly, but
successfully.
In short, my actions are:
CREATE TABLE docs ([id] INTEGER PRIMARY KEY AUTOINCREMENT, [doc] CHAR... (other
fields here)
(here?s the process of filling the base with the data, which are downloaded by
means of my own script from text files.)
CREATE VIRTUAL TABLE docs_fts using fts4 (content='docs', doc... (other fields
here)
(here?s an attempt of INSERT INTO docs_fts(docs_fts) VALUES ('rebuild') to
rebuild contentless FTS index)
or
CREATE VIRTUAL TABLE docs_fts using fts4 (doc... (other fields here)
(here?s an attempt of INSERT INTO docs_fts SELECT doc... from docs;)
or
CREATE VIRTUAL TABLE docs_fts using fts3 (doc... (other fields here)
(here?s an attempt of INSERT INTO docs_fts SELECT doc... from docs;)
For each attempt I?ve been making a new copy of the source file, because I
suspected that the base could have got broken after Segmentation Fault. I even
changed the ram-cards, in case if memory was the problem.
But every time I get one and the same result - Segmentation Fault error.
So, can you please pay your attention to this problem and fix it ASAP?
I can send you a file if you need.
Thank you.
[sqlite] Segfault during FTS index creation from huge data
Artem wrote: ...snip... > SQLite is functioning for about 4 hours, after which Segmentation Fault error > occurs inevitably. ...snip... You did not provide any stack dump, so I don't think that SQLite developers can do anything to help you. Anyway, it's more likely that the bug is in your application. Since you use Linux, you should try to recompile and link with -fsanitize=address -g -O0 -fno-omit-frame-pointer -fsanitize option is available in recent versions of gcc or clang. It will likely tell you where the bug is. Regards Dominique
[sqlite] NtFlushBuffersFileEx for SQLITE_SYNC_DATAONLY on Windows ?
ping This email may contain confidential and privileged information. Any unauthorized use is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. From: Howard Kapustein Sent: Monday, March 9, 2015 10:10 AM To: 'sqlite-users at mailinglists.sqlite.org' Subject: NtFlushBuffersFileEx for SQLITE_SYNC_DATAONLY on Windows ? Has anyone considered supporting SQLITE_SYNC_DATAONLY on Windows using NtFlushBuffersFileEx? http://msdn.microsoft.com/en-us/library/windows/hardware/hh967720(v=vs.85).aspx >When the SQLITE_SYNC_DATAONLY flag is used, it means that the sync operation >only needs to flush data to mass storage. Inode information need not be >flushed. Am I missing something in SQLite's expectations? Isn't this just flush the file's data/content but not the associated metadata (filesize, etc Windows inode-equivalent) i.e. NtFlushBuffersFileEx(...,flags=FLUSH_FLAGS_FILE_DATA_ONLY,...)? It looks like it's just a change in winSync() to replace rc = osFlushFileBuffers(pFile->h); with something like IO_STATUS_BLOCK ioStatusBlock; ... int isDataOnly = (flags_SYNC_DATAONLY); ... if (isDataOnly) rc = osFlushFileBuffers(pFile->h); else rc = HRESULT_FROM_NT(NtFlushBuffersFileEx(pFile->h, FLUSH_FLAGS_FILE_DATA_ONLY, ); Would a new system call be used to optionally support sync-dataonly semantics? Or would this be handled via a new compile-time option and #ifdef in winSync? - Howard
[sqlite] Windows 8.x security requirements / SafeSEHCheck - NXCheck - DBCheck
Thank you very much for the DLL and this very interesting discussion. SQLite passes the the certification test with this version. I don't have my hands on an XP box, to test if this version runs there, but I can ship a different version for XP if that is the tall to pay. Notice that SQLite is not the only dll with problems, many libraries like Cairo graphics and pacparser for instance do not comply yet. But then like any industry, security is something we have to deal with. Best regards and thanks again. Maarten MOSTERT 28 Av Alphonse Denis 83400 Hy?res, France +33 676411296 [ http://stakepoint.com/ ]( http://stakepoint.com/ ) > "Random Coder" | > On Thu, Apr 2, 2015 at 10:58 AM, wrote: > > > > This are basically compiler switches as far as I understand. > > > > I am not a C programmer, so also quite difficult for me to link the library > statically. > > These tests basically map to the /SAFESEH, /DYNAMICBASE, and /NXCOMPAT > linker flags. Though, as I understand it, passing these tests is > recommended, and not required for Windows 8 certification. > > I have a dump of a build of sqlite from a recent internal build. I'm > not connected with the Sqlite project, I just use it, and happen to > turn on these flags in my build environment for other reasons. > > https://bucket-1f8c9d86.s3.amazonaws.com/sqlite3.zip > > You can either use my DLL, or build your own version of the DLL with > these options enabled. > > I'd recommend the SQLite team turn them on for the version of the DLL > they distribute, but I'm honestly not sure if there are negative side > effects to doing so. > ___ > sqlite-users mailing list > sqlite-users at mailinglists.sqlite.org > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users >
[sqlite] Windows 8.x security requirements / SafeSEHCheck - NXCheck - DBCheck
> -Original Message- > From: sqlite-users-bounces at mailinglists.sqlite.org [mailto:sqlite-users- > bounces at mailinglists.sqlite.org] On Behalf Of Keith Medcalf > Sent: vrijdag 3 april 2015 01:51 > To: General Discussion of SQLite Database > Subject: Re: [sqlite] Windows 8.x security requirements / SafeSEHCheck - > NXCheck - DBCheck > > > BTW, I have verified that these options all work as described and the options > are recognized and processed properly by Windows, and that BinScope is > happy: > > Failed checks > d:\source\sqlite\sqlite3.dll - SafeSEHCheck ( FAIL ) > > Passed checks > d:\source\sqlite\sqlite3.dll - NXCheck ( PASS ) This makes all the allocated memory and stack not-executable > d:\source\sqlite\sqlite3.dll - DBCheck ( PASS ) > (Note, position independent code (PIC) is by definition loadable at any base. > Microsoft is just several decades behind in generating position independent > code.) And this makes it possible to load the DLL above 2 GB, which by default isn't enabled because too much existing legacy code still casts pointers to integers and back.. Another opt-in flag in the header, to allow existing programs to still work on bigger machines. SafeSEH is a similar flag for Structured Exception Handling. Without this flag the stack is unwound on exceptions by using debugger tricks and information stored on the stack itself. When this flag is set the compiler promises to provide all this information in a parsable format somewhere in the binary. This improves performance and reliability on these exceptions. But as far as I know Sqlite doesn't use structured exception handling at all. Perhaps MingW supports a flag of just telling that, if it doesn't support creating these tables. (In Windows x86_64 these tables are always required, so this flag doesn't apply there) Windows 8.1 and Windows 10 will add another feature like this where all callback function pointers are stored in a central table for verification if a passed callback function is really to the intended code. (As far as I know this is currently only supported by internal and pre-release versions of the MSVC compiler) Bert
[sqlite] NtFlushBuffersFileEx for SQLITE_SYNC_DATAONLY onWindows ?
Howard Kapustein wrote: > > Has anyone considered supporting SQLITE_SYNC_DATAONLY on Windows using > NtFlushBuffersFileEx? > http://msdn.microsoft.com/en-us/library/windows/hardware/hh967720(v=vs.85).a spx > Our experience with various Unix flavors teaches us that data-only sync does not have a performance advantage over a full sync. And the Nt*() interfaces may cause portability problems for things like WinCE. -- Joe Mistachkin
[sqlite] Version 3.8.9 is in testing
On 4/2/15, Richard Hipp wrote: > SQLite version 3.8.9 is now in testing. When the status board at > https://www.sqlite.org/checklists/3080900/index goes all green, we > will cut the release. The release has been postponed due to an obscure problem that testing revealed on OS-X. The problem is related to file locking and only seems to occur on OS-X. (See the importance of cross-platform testing!) We will need to chase this problem to ground and then restart the testing effort from the beginning. So this is likely to mean a long delay before the 3.8.9 release. -- D. Richard Hipp drh at sqlite.org
