[Tracker] Fwd: PSA: If you get DB update errors when updating Tracker to version 1.8.0 from a pre FTS5 version, read this

[Jose Arroyo]

Hello everyone,

This might be a little long so here is a tl;dr

Tl;dr: If you are having problems with old Tracker DB files (from before
the update to FTS 5), check if the SQLite version Tracker is linking
against was compiled with the flag SQLITE_OMIT_AUTOVACUUM. You can check
this by with the following SQL query: "PRAGMA auto_vacuum". If the reply is
NULL it means SQLite was compiled with this flag, if the reply is 0 it
means that it wasn't.

Building SQLite *without* SQLITE_OMIT_AUTOVACUUM *might* solve your issues.
I don't really understand exactly *why* just yet.
There is also an upcoming patch by Carlos G that *might* solve your issue,
which will avoid creating the fts5 table multiple times.

** end of tl;dr

So here is what happened to me, most of this was discussed in the IRC
Tracker channel. I use a slightly modified version of Tracker and while
testing the latest stable version (1.8.0 and sqlite 3.12.2) against some
old DB files I ran into some trouble.

The tracker-store process would raise an abort signal systematically when
started using these files. I got the following error every time:

Tracker:ERROR:tracker-fts.c:202:tracker_fts_create_table: code should not
be reached

What was going on was that these old DB files were created with Tracker
1.3.2 and SQLite 3.8.4.1. This meant that there were 2 ontology files that
had been modified since (nco.ontology and nmo.ontology) and that these
files used the old FTS3/4. Each of these two files would trigger a call to
tracker_fts_alter_table() during the initialization of tracker-store. Each
of these calls would then call tracker_fts_create_table() so that the new
fts5 table is created. In my case, everything went well the first time this
function was called. However, when the second ontology file triggered a
call to tracker_fts_create_table(), the first sqlite3_exec() call returned
SQLITE_CORRUPT, which made tracker exit through g_assert_not_reached().

I suspected that this behaviour was caused by my tracker modifications so I
set to try and reproduce this behaviour using upstream tracker versions. I
was able to partially reproduce this behaviour reliably. The only
difference is that now I got the SQLITE_CORRUPT error a little further down
the executing, after tracker_fts_create_table() returned. Precisely, it
happened in https://git.gnome.org/browse/tracker/tree/src/libtracker-
fts/tracker-fts.c#n226 . This meant that the fts5_tmp was never renamed
correctly to just fts5.

To reproduce this problem I was able to narrow it down to the following
steps:


I tried using 3 different SQLite versions, 3.8.4.1, 3.12.2 and 3.14.2.
- Build autoconf sqlite version using the following command:
CFLAGS="-DSQLITE_OMIT_AUTOVACUUM" ./configure  --enable-shared
--disable-static --enable-threadsafe --enable-fts5 ; make
(If you build an SQLite version that include fts5 support, ensure that
extension loading is activated, I believe it is by default unless it is
built with SQLITE_OMIT_LOAD_EXTENSION)

- Build tracker version tagged as 1.3.2 linking against the sqlite version
you just compiled. This ensures that you'll build a version where fts5
wasn't supported yet by tracker and also that you'll use an ontology where
at least 2 files differ from the current upstream ontology (nco and nmo). I
tried both using the default config options and the ones I normally use, it
didn't seem to make any difference.

- Start the tracker-store process so that a new empty Database is created,
the stop the process. Save all these DB files (meta.db, onologies.gvdb,
etc...) so you can reuse them later.

- Build tracker from the upstream 1.8.0 branch linking against the sqlite
version you just compiled. You can try building a different version, it
doesn't really matter as long as you use the SQLITE_OMIT_AUTOVACUUM flag.

- Ensuring the the old DB files are where Tracker expects them to be, start
the tracker-store (export TRACKER_VERBOSITY=3 so you can see the errors).



In my environment, following these steps is enough to cause an
SQLITE_CORRUPT error in https://git.gnome.org/browse/
tracker/tree/src/libtracker-fts/tracker-fts.c#n226 the second time
tracker_fts_alter_table() is called.

If I build the same SQLite autoconf version without the
SQLITE_OMIT_AUTOVACUUM I don't get this error any more. Carlos G tried
running tracker against the empty  DB files I had generated and he didn't
get any errors either.

In my initial environment, I built SQLite without this flag as well and my
initial error (Tracker:ERROR:tracker-fts.c:202:tracker_fts_create_table:
code should not be reached) was gone as well.


Now, all this seems very fishy for various reasons. By default auto_vacuum
is set to 0 and in Tracker it is set explicitly to 0 on start-up. Building
SQLite with the SQLITE_OMIT_AUTOVACUUM flag and using this with Tracker
shouldn't really have an impact. The SQLite documentation states "If this
option is defined, the library cannot create or write to databases that
support auto_vacuum 

[Tracker] Fwd: tracker 1.10.0

[Carlos Garnacho]

-- Forwarded message --
From: Carlos Garnacho 
Date: Mon, Sep 19, 2016 at 5:10 PM
Subject: tracker 1.10.0
To: FTP Releases 


About Tracker
=

Tracker is a semantic data storage for desktop and mobile devices.
Tracker uses W3C standards for RDF ontologies using Nepomuk with
SPARQL to query and update the data.

Tracker is a central repository of user information, that provides two
big benefits for the user; shared data between applications and
information which is relational to other information (for example:
mixing contacts with files, locations, activities and etc.).

News


Translations: da, el, en_GB


ChangeLog
=
https://download.gnome.org/sources/tracker/1.10/tracker-1.10.0.changes  (315)

Download

https://download.gnome.org/sources/tracker/1.10/tracker-1.10.0.tar.xz (4.76M)
  sha256sum: df95b4a1e7de442f66d1097b725dd3cdd739862f491453fc7d7b1f88181a12fb
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list