Re: [Tracker] Moving Tracker discussions from tracker-list@gnome.org to discourse.gnome.org

[Carlos Garnacho]

Hi!,

On Tue, Aug 27, 2019 at 2:34 PM Sam Thursfield via tracker-list <
tracker-list@gnome.org> wrote:

> Hi all,
> The GNOME project now has a self-hosted instance of the Discourse forum
> software, running at https://discourse.gnome.org/.
>
> The GTK+ mailing lists have already closed and all discussion has moved to
> Discourse, and they have seen a significant increase in participation as a
> result.
>
> At the Tracker BoF in GUADEC we discussed moving to Discourse and everyone
> present thought it was a good idea, so I'm going to look at how we can
> migrate Tracker.
>
> Basically we need to create a 'Tracker' tag on the discourse instance, and
> close this list and cause it to respond to requests with a notice that
> discussion is now on discourse.
>
> Let me know your thoughts!
>

Just FTR, I'm totally on board with this. On one hand I personally suck at
email, and OTOH this list does not get to see a lot of activity (most of it
lately were release announces, and they even dropped on the floor at some
point, that is my bad).

I hope using Discourse would improve perception, not only from the people
interested in Tracker specifically, but also from other users of the GNOME
platform infrastructure in general.

Cheers,
  Carlos
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

Re: [Tracker] GSOC

[Carlos Garnacho]

Hi Mahmoud,

On Tue, May 7, 2019 at 12:26 AM Mahmoud Nagy via tracker-list
 wrote:
>
> I wasn't selected from tracker as gsoc member, if possible can you provide 
> some feedback to learn more from the experience.

I'm afraid that is not possible... The explanation is tragically
mundane: I missed google's deadline, somehow I marked it for tomorrow
in my calendar.

This is surely not any comforting, it is my fault and I profoundly
apologize. I hope that this doesn't deter your trajectory in Free
software, and that we will see more contributions from you.

Regards,
  Carlos
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

[Tracker] Tracker 2.2.0 released

[Carlos Garnacho]

NEW in 2.2.0 - 2019-02-20
=

  * Multiple memory leak and corruption fixes
  * Bumped glib minimum version to 2.46, it already was in practical terms
  * Test suite improvements
  * Restore log domain

Translations: ca, cs, da, kk, lt, pt_BR, sv, zh_TW

  Highlighted changes since 2.1.x:
  

  * New SPARQL parser, able to generate SQL that is generally more readable
and at places performs better. Multiple buglets fixed in the process
  * Much improved support of SPARQL1.1 features and syntax that was missing:
- Property paths: Allowing to match connectivity between two resources
  by an arbitrary length path. There is a number of supported operators
  (alternative, sequence, oneOrMany, ...) that can be combined, e.g:

  SELECT ?s ?p { ?s ^(nfo:belongsToContainer*)/(nie:url|nie:title) ?p }

  Only the negated path operator (!) is not supported at the moment.

- Support for fully unrestricted queries, eg:

  SELECT ?s ?p ?o { ?s ?p ?o } ORDER BY ?o ?p ?s

  Queries with unrestricted predicate (?p in the example above) were
  just supported in a very restricted set of situations. All those
  limitations are gone.
- MINUS allows subtracting the solutions that match the given triples
  template, eg:

  SELECT ?s { ?s a nfo:Media } MINUS { ?s a nfo:MusicPiece }

  * Support for prepared statements. TrackerSparqlStatement can be built
with SELECT queries containing (custom) ~var syntax, and updating
their values before obtaining a cursor.
  * Many tests were added, and Tracker is generally much better tested thanks
to CI.
  * tracker-store now automatically shuts down on inactivity.

  NOTE TO PACKAGERS: This release might trigger ABI checks, as
TrackerSparqlConnectionClass struct size grew. This object can only
be subclassed within Tracker and instantiated through Tracker API, so
the change is inconsequential.

Download

https://download.gnome.org/sources/tracker/2.2/tracker-2.2.0.tar.xz (2.67M)
  sha256sum: 6da6c4abd2341a5f2748e504590ef719b8c6fcc78f033d11fb22dab79b8f2306
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

[Tracker] Tracker-miners 2.2.0 released

[Carlos Garnacho]

NEW in 2.2.0 - 2019-02-20
=

  * Disable guarantee_metadata by default. It was the case on autotools
  * Stop merging translations to schema files
  * Test suite improvements
  * Meson build improvements

Translations: cs, da, de, es, gl, hu, id, lt, pl, pt_BR, ro, sl, sv, tr, zh_TW

  Highlighted changes since 2.1.x:
  

  * The functionality of tracker-miner-apps has been adopted by
tracker-miner-fs/tracker-extract.
  * All usage of deprecated TrackerSparqlBuilder is gone.

Download

https://download.gnome.org/sources/tracker-miners/2.2/tracker-miners-2.2.0.tar.xz
(2.77M)
  sha256sum: 4a3953d8e549212a73edfb7e795108f4c03d91692560c8ed96c65d84b6ab4e01
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

[Tracker] tracker-miners 2.2.0-alpha1

[Carlos Garnacho]

(Sending on behalf of install-module@, due to ftpadmin failures)

About Tracker Miners


Tracker is a semantic data storage for desktop and mobile devices.
Tracker uses W3C standards for RDF ontologies using Nepomuk with
SPARQL to query and update the data.

Tracker is a central repository of user information, that provides two
big benefits for the user; shared data between applications and
information which is relational to other information (for example:
mixing contacts with files, locations, activities and etc.).

News


  * The functionality of tracker-miner-apps has been adopted by
tracker-miner-fs/tracker-extract.
  * Updated tracker-miner-fs and tracker-miner-rss to use TrackerResource
  * Support for building through autotools has been removed.
  * Plugged several leaks
  * Other many build and code cleanups and fixes

Translations: ru, sk, sr

Download
===
https://download.gnome.org/sources/tracker-miners/2.2/tracker-miners-2.2.0-alpha1.tar.xz

sha256sum: 5562b7c2fcdc562c962e6f234baaee2a4b0294f8a59c2e326f025e8e72a47f86
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

[Tracker] tracker 2.2.0-alpha1

[Carlos Garnacho]

(Sending this on behalf of install-module@, due to ftpadmin failures)

About Tracker
=

Tracker is a semantic data storage for desktop and mobile devices.
Tracker uses W3C standards for RDF ontologies using Nepomuk with
SPARQL to query and update the data.

Tracker is a central repository of user information, that provides two
big benefits for the user; shared data between applications and
information which is relational to other information (for example:
mixing contacts with files, locations, activities and etc.).

News


  * New SPARQL parser, able to support more 1.1 features and generating
friendlier SQL at places. There is initial support for property
paths (/ and ^), and other missing 1.1 syntax (MINUS, SHA384, ...).
More improvements are expected to happen in the future thanks to this.
  * Support for prepared statements. TrackerSparqlStatement can be built
with SELECT queries containing (custom) ~var syntax, and updating
their values before obtaining a cursor.
  * Added global libtracker-sparql call to change the used DBus connection
at runtime.
  * Made tracker-store to automatically shutdown when unneeded.
  * Fixed ontology updates to work with behavioral changes in
sqlite >=3.25.
  * Support for building through autotools has been removed.
  * Other many build and code cleanups and fixes

Translations: pl, sl, sr

Download
===
https://download.gnome.org/sources/tracker/2.2/tracker-2.2.0-alpha1.tar.xz
 sha256sum: 794ea7bb54c6e61c97097b9641ddb748e55d2152fca1ef9d4d145c882c86753b
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

Re: [Tracker] tracker-store takes a long time to terminate

[Carlos Garnacho]

Hi!,

On Mon, Jan 22, 2018 at 6:07 AM, Will S  wrote:
> I have noticed that tracker-store takes a long time to terminate.
> tracker-store is started by systemd when I log in. When I try to log out,
> systemd hangs 90 seconds waiting for the stop job on tracker-store.service
> to time out. I tried starting starting tracker-store on the command line
> myself and sending it SIGTERM from another terminal. After some amount of
> time (at least five minutes), tracker-store did finish shutting down
> cleanly. Besides the annoyance of the time out on log out, tracker also has
> to do a CPU intensive integrity check the next time I log in because systemd
> kills the process before it shuts down cleanly.
>
> Could there be something wrong with my configuration, or could this be a
> bug? What could I try to avoid this? I am using tracker 2.0.2 on Arch Linux.

This would be a bug. I traced this to the VACUUM that was added sort
of recently, which turned out to be a bit too eager.

I pushed 
https://git.gnome.org/browse/tracker/commit/?id=57235eb302194554feb077019d9c69a4bbd8214f
so it kicks in only when the database gets exceedingly large.

Thanks,
  Carlos
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

[Tracker] Fwd: tracker 2.0.3

[Carlos Garnacho]

-- Forwarded message --
From: Carlos Garnacho <install-mod...@master.gnome.org>
Date: Wed, Feb 7, 2018 at 1:15 AM
Subject: tracker 2.0.3
To: FTP Releases <ftp-release-l...@gnome.org>


About Tracker
=

Tracker is a semantic data storage for desktop and mobile devices.
Tracker uses W3C standards for RDF ontologies using Nepomuk with
SPARQL to query and update the data.

Tracker is a central repository of user information, that provides two
big benefits for the user; shared data between applications and
information which is relational to other information (for example:
mixing contacts with files, locations, activities and etc.).

News


  * build: Improvements in meson support.
  * build: Remove stale dependencies after Tracker miners split
  * tests: Many fixes to functional tests
  * tests: Remove old checks for maemo-specific features
  * libtracker-miner: Small code improvements.
  * libtracker-sparql: use gint32 to unpack 'i' GVariant format

Translations: is, nb


ChangeLog
=
https://download.gnome.org/sources/tracker/2.0/tracker-2.0.3.changes  (7.51K)

Download

https://download.gnome.org/sources/tracker/2.0/tracker-2.0.3.tar.xz (2.50M)
  sha256sum: 5a2fb274c128ec67a920944937b5147ceaf5db16fef6691ea22c4cb841e20580
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

[Tracker] Fwd: tracker-miners 2.0.4

[Carlos Garnacho]

-- Forwarded message --
From: Carlos Garnacho <install-mod...@master.gnome.org>
Date: Wed, Feb 7, 2018 at 1:15 AM
Subject: tracker-miners 2.0.4
To: FTP Releases <ftp-release-l...@gnome.org>


About Tracker Miners


Tracker is a semantic data storage for desktop and mobile devices.
Tracker uses W3C standards for RDF ontologies using Nepomuk with
SPARQL to query and update the data.

Tracker is a central repository of user information, that provides two
big benefits for the user; shared data between applications and
information which is relational to other information (for example:
mixing contacts with files, locations, activities and etc.).

News


  * build: Allow building tracker repo as a meson subproject
  * libtracker-common: Rename to libtracker-miners-common
  * libtracker-miners-common: Whitelist arm_fadvise64_64, getegid and
getegid32 syscalls
  * tracker-extract: Add GExiv2-based extractor module for RAW files
  * tracker-extract: Blacklist gstreamer modules via plugin instead of
via feature
  * tracker-extract: Blacklist video4linux2 gstreamer plugin
  * tracker-extract: Use enumerations for EXIF values
  * tracker-extract: Fix image pixel density conversions
  * tracker-miner-fs: Avoid setting rdf:types on empty files
  * meson: dependency check fixes

Translations: nb


ChangeLog
=
https://download.gnome.org/sources/tracker-miners/2.0/tracker-miners-2.0.4.changes
 (7.86K)

Download

https://download.gnome.org/sources/tracker-miners/2.0/tracker-miners-2.0.4.tar.xz
(3.48M)
  sha256sum: 4626646167e405be5f8b3f831c82cfcdf86ba20d3598fca41ea8c4a8b8a8e956
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

Re: [Tracker] tracker over activities menu

[Carlos Garnacho]

Hi!,

On Tue, Jan 2, 2018 at 2:53 PM, Herr Oswald  wrote:
> Hello,
>
> I'm on ubuntu 17.04 GNOME.
>
> Using tracker-needle, everything is fine. But when I'm using GNOME's
> activities menu, I only get results from my home directory, not the
> sub-directories, allthough they all are in the settings (tracker-
> preferences). And only file names, no contents.
>
> Is it true that the activities menu should call tracker results as well
> as tracker-needle? - And in the case - what's going wrong on my
> machine?

That doesn't happen directly. Gnome-shell uses the "search provider"
dbus interfaces, nautilus implements one such search provider for the
shell, which does use Tracker as per code in upstream.

Now, even though Ubuntu allows installing Tracker, they went through
great lengths to avoid a mandatory dependency on it. As a result, all
Tracker code in Nautilus has been patched away by Ubuntu, which made
the search experience within nautilus/overview below par.

Hope that clears things up!

Cheers,
  Carlos
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

Re: [Tracker] tracker constantly crawling

[Carlos Garnacho]

Hi Brian!

What tracker version is this?

On Thu, Nov 30, 2017 at 3:55 AM, Brian J. Murrell  wrote:
> Is it just because /home/brian/Pictures is on an NFS server that
> tracker just loops, constantly crawling it?
>
> 29 Nov 2017, 20:04:04:1%  File System - Crawling recursively 
> directory 'file:///home/brian/Pictures'
> 29 Nov 2017, 20:10:14:1%  File System - Crawling recursively 
> directory 'file:///home/brian/Pictures'
> 29 Nov 2017, 20:16:32:1%  File System - Crawling recursively 
> directory 'file:///home/brian/Pictures'
> 29 Nov 2017, 20:22:38:1%  File System - Crawling recursively 
> directory 'file:///home/brian/Pictures'
> 29 Nov 2017, 20:29:01:1%  File System - Crawling recursively 
> directory 'file:///home/brian/Pictures'
> 29 Nov 2017, 20:35:16:1%  File System - Crawling recursively 
> directory 'file:///home/brian/Pictures'
> 29 Nov 2017, 20:41:30:1%  File System - Crawling recursively 
> directory 'file:///home/brian/Pictures'
> 29 Nov 2017, 20:48:59:1%  File System - Crawling recursively 
> directory 'file:///home/brian/Pictures'
> 29 Nov 2017, 20:55:06:1%  File System - Crawling recursively 
> directory 'file:///home/brian/Pictures'
> 29 Nov 2017, 21:01:34:1%  File System - Crawling recursively 
> directory 'file:///home/brian/Pictures'
> 29 Nov 2017, 21:07:52:1%  File System - Crawling recursively 
> directory 'file:///home/brian/Pictures'
> 29 Nov 2017, 21:14:06:1%  File System - Crawling recursively 
> directory 'file:///home/brian/Pictures'
> 29 Nov 2017, 21:21:32:1%  File System - Crawling recursively 
> directory 'file:///home/brian/Pictures'
> 29 Nov 2017, 21:29:10:1%  File System - Crawling recursively 
> directory 'file:///home/brian/Pictures'
> 29 Nov 2017, 21:36:35:1%  File System - Crawling recursively 
> directory 'file:///home/brian/Pictures'
>
> Is there really nothing better we can do except just constantly crawl
> NFS shares?  Is FAM no longer used?

We don't constantly crawl anything, at least on purpose :). Tracker
does no special treatment of NFS mounts, and fortunately it seems your
setup avoids all the scary file locking issues with the db file. In
the case there is no monitoring (disabled, run out of handles, ...)
tracker-miner-fs would simply crawl it once on startup to check mtimes
and reindex the content updated since the last index.

And as it's been pointed out, Tracker uses glib file monitors, which
have  a FAM implementation. However, the limits are checked on Tracker
by looking up the file monitor created on $HOME. That may result on
Tracker thinking it's got a lot more handles available than what FAM
can offer. I don't know if it's related to this bug or not.

Back to why it's stuck in that state... do you see cpu activity or
anything in logs from tracker-miner-fs or tracker-store? if you do,
there's maybe something tricking tracker-miner-fs into looping across
the FS. If you see tracker-miner-fs sitting idle it's perhaps the NFS
server is tripping and making tracker-miner-fs block on a call.

Please file a bug, and we get going from there.

Cheers,
  Carlos
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

[Tracker] Fwd: tracker 2.0.2

[Carlos Garnacho]

-- Forwarded message --
From: Carlos Garnacho <install-mod...@master.gnome.org>
Date: Wed, Nov 15, 2017 at 12:03 AM
Subject: tracker 2.0.2
To: FTP Releases <ftp-release-l...@gnome.org>


About Tracker
=

Tracker is a semantic data storage for desktop and mobile devices.
Tracker uses W3C standards for RDF ontologies using Nepomuk with
SPARQL to query and update the data.

Tracker is a central repository of user information, that provides two
big benefits for the user; shared data between applications and
information which is relational to other information (for example:
mixing contacts with files, locations, activities and etc.).

News


  * tests: Cleanups and Coverity fixes. A testsuite for libtracker-miner's
TrackerMinerFS object was added.
  * meson: Many small improvements.
  * libtracker-common: Preparation work to be able to build
tracker/tracker-miners as a bundle.
  * libtracker-direct: Implement update_array_async()
  * libtracker-miner: Multiple cleanups and code simplifications.
  * libtracker-miner: Properly honor lack of CHECK_MTIME flag, resulting
on faster startup times if it's not set.

Translations: ca@valencia


ChangeLog
=
https://download.gnome.org/sources/tracker/2.0/tracker-2.0.2.changes  (13.1K)

Download

https://download.gnome.org/sources/tracker/2.0/tracker-2.0.2.tar.xz (2.76M)
  sha256sum: ece71a56c29151a76fc1b6e43c15dd1b657b37162dc948fa2487faf5ddb47fda
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

Re: [Tracker] Tracker-CRITICAL

[Carlos Garnacho]

Hi Chris,

On Fri, Sep 22, 2017 at 3:48 AM, Chris  wrote:
> Tracker version 1.10.0 - I've run tracker daemon -k which killed all
> the processes but if the below continues I'd just as soon permanently
> disable it. How can that easily be done?

You can remove the dbus service files from /etc/xdg/autostart.

>
> Sep 21 20:00:04 localhost gnome-session[3498]: (tracker-miner-fs:3745):
> Tracker-CRITICAL **:   (Sparql buffer) Error in task 0
> (file:///home/chris/Downloads) of the array-update: UNIQUE constraint
> failed: nie:DataObject.nie:url (strerror of errno (not necessarily
> related): Resource temporarily unavailable)
> Sep 21 20:00:04 localhost gnome-session[3498]: (tracker-miner-fs:3745):
> Tracker-CRITICAL **: Could not execute sparql: UNIQUE constraint
> failed: nie:DataObject.nie:url (strerror of errno (not necessarily
> related): Resource temporarily unavailable)
>
> Hundreds of the above lines are written to syslog whenever I do a
> restart.

There is https://bugzilla.gnome.org/show_bug.cgi?id=786132 trying to
cover all conditions that might lead to this misbehavior. Feel free to
chime in there, it would be great that you provided the output of
"tracker info ~/Downloads" so that I have a better understanding of
why does this happen to you.

Cheers,
  Carlos
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

[Tracker] Fwd: tracker 2.0.0

[Carlos Garnacho]

Let's rejoice! One thing I forgot to mention in the notes... meson
build still can't build from tarballs, autotools remains in parallel
for obvious reasons.


-- Forwarded message --
From: Carlos Garnacho <install-mod...@master.gnome.org>
Date: Tue, Sep 12, 2017 at 12:56 PM
Subject: tracker 2.0.0
To: FTP Releases <ftp-release-l...@gnome.org>


About Tracker
=

Tracker is a semantic data storage for desktop and mobile devices.
Tracker uses W3C standards for RDF ontologies using Nepomuk with
SPARQL to query and update the data.

Tracker is a central repository of user information, that provides two
big benefits for the user; shared data between applications and
information which is relational to other information (for example:
mixing contacts with files, locations, activities and etc.).

News


  * Tracker shall from now on use semantic versioning
  * Drop --all from "tracker status" subcommand, it is the default
behavior now.
  * TrackerDecorator internal operations are now cancelled on shutdown
  * Add cancellable argument to sync libtracker-control call
  * Build fixes and minor cleanups

  Translations: ca, da, eu, fi, fur, it, ko, lt, lv, nl, pt_BR, sk, sv, zh_TW

  Overview of changes since 1.12:

  * Tracker core (store, libraries) and miners have been split, this source
tree contains the former.
  * Added support for domain ontologies, these can be used to set up private
databases managed and populated by a standalone set of daemons. It was
devised for sandboxing environments like Flatpak, but can be used
independently.
  * Added API to manage application-private databases with (optional) custom
ontologies. The application is in charge of content population as well as
queries.
No daemons are required for this usecase, so this becomes a SPARQL
endpoint equivalent to SQLite.
  * libtracker-miner objects can now be used on the application side in
combination with application-private endpoints. An additional object
has been added to make TrackerMiner implementations exported through
DBus.
  * The filesystem TrackerMiner base implementation has received some
notable speedups.
  * Running multiple concurrent SELECT queries is now lock-free on most
situations.
  * Meson build instructions have been added.


ChangeLog
=
https://download.gnome.org/sources/tracker/2.0/tracker-2.0.0.changes  (3.09K)

Download

https://download.gnome.org/sources/tracker/2.0/tracker-2.0.0.tar.xz (2.76M)
  sha256sum: efc3af5b44461ec938af0622fe8eacbf0c7abbadce32ae55df28c9d4b4b1ff6c
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

[Tracker] Fwd: tracker 1.12.2

[Carlos Garnacho]

-- Forwarded message --
From: Carlos Garnacho <install-mod...@master.gnome.org>
Date: Mon, Aug 7, 2017 at 6:36 PM
Subject: tracker 1.12.2
To: FTP Releases <ftp-release-l...@gnome.org>


About Tracker
=

Tracker is a semantic data storage for desktop and mobile devices.
Tracker uses W3C standards for RDF ontologies using Nepomuk with
SPARQL to query and update the data.

Tracker is a central repository of user information, that provides two
big benefits for the user; shared data between applications and
information which is relational to other information (for example:
mixing contacts with files, locations, activities and etc.).

News


  * Sqlite 3.20 introduced incompatible changes in the way FTS5 can be
extended, compile/runtime checks were added to adapt to the new
safer way if Sqlite >= 3.20.0 is found.
  * Fix TrackerDBInterface being reused with no FTS set up.


ChangeLog
=
https://download.gnome.org/sources/tracker/1.12/tracker-1.12.2.changes  (835)

Download

https://download.gnome.org/sources/tracker/1.12/tracker-1.12.2.tar.xz (4.81M)
  sha256sum: ebeb42ef982d0e45c8b8eea8440dcd1c06cd04c7974440a2012942552882bffd
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

[Tracker] wip/carlosg/resource-leak-fix merged

[Carlos Garnacho]

Hi!,

I've just been merged wip/carlosg/resource-leak-fix on master. For
some background, the URNs generated in the Resource table had no clear
lifetime, so Tracker has been leaking those "on purpose" for quite
some time now.

After a clumsy try that had to be reverted, some months ago I tried to
address this using referential integrity across tracker tables. It
turned out correct, but too slow on certain situations (eg. deletes)
as checking that a resource could be deleted had to hit many columns
without an index.

Philip OTOH suggested using a refcount and garbage collection
approach, which is what this branch implemented. I opted for managing
the refcount through triggers that increment or decrement the refcount
in the Resources table, all these extra updates happen on rowid
matches, so it's significantly faster and more O(1)y than the previous
approach.

This branch has been lingering for some weeks now, but I've been
testing it since with no hiccups, and also caters for updating
existing databases. I think it's in good shape for merging before the
freeze :).

Cheers,
   Carlos
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

[Tracker] Heads up: tracker has been split in two

[Carlos Garnacho]

(cross-posting to tracker and distributor-list MLs, please stick to
the one you consider most appropriate on replies)

Hi all,

After a massive cleanup in tracker, the miners have been finally split
from the Tracker core, they can now be found at:

https://git.gnome.org/browse/tracker-miners

The core is composed of all libraries plus tracker-store, and can
still be found at:

https://git.gnome.org/browse/tracker

This new tracker-miners is to be built and packaged separately. I will
add a pkgconfig file before the release scheduled for early next week
so direct consumers of data extracted by these miners can depend on
it.

In the mean time, jhbuild and gnome-continuous have been updated.

Cheers,
  Carlos
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

[Tracker] Fwd: tracker 1.99.1

[Carlos Garnacho]

A little bit closer!


-- Forwarded message --
From: Carlos Garnacho <install-mod...@master.gnome.org>
Date: Wed, Jul 19, 2017 at 3:09 PM
Subject: tracker 1.99.1
To: FTP Releases <ftp-release-l...@gnome.org>


About Tracker
=

Tracker is a semantic data storage for desktop and mobile devices.
Tracker uses W3C standards for RDF ontologies using Nepomuk with
SPARQL to query and update the data.

Tracker is a central repository of user information, that provides two
big benefits for the user; shared data between applications and
information which is relational to other information (for example:
mixing contacts with files, locations, activities and etc.).

News


  WARNING: This is unstable development towards 2.0. There are API and
   ABI incompatibilities that might affect you.

  * Notable speedups to tracker-miner-fs, main loop overhead was greatly
reduced by processing elements in batches. Indexing has been observed
to be up to 2x faster, and startup on an indexed and up-to-date
filesystem up to 3x.
  * More notable speedups to tracker-miner-fs startup, this applies only
to filesystems where the number of indexed folders exceed the amount
of inotify handles. Inotify monitoring is temporarily disabled during
filesystem mtime checks, resulting in up to 4x faster startup. (In
addition to the previous point).
  * Refurbished the allocation scheme for underlying DB interfaces. The
benefit is twofold, this makes TrackerSparqlConnections truly isolated
instances, and results in much reduced mutex contention on stress
situations.
  * Dropped deprecated API to get direct/bus connections. Use
tracker_sparql_connection_get().
  * Deprecated TrackerSparqlBuilder. Use TrackerResource.
  * Added tracker_sparql_connection_get_namespace_manager() to fetch the
namespaces as per the ontology of the connection.
  * Dropped support for non-standard SPARQL syntax "AS var", the right
syntax is "AS ?var", defined in SPARQL1.1 and accepted by Tracker
for a long time.
  * Added tracker:title-order() sparql function, only meant to be used
in "ORDER BY" clauses. It drops the common articles at the beginning
of the given variable for sorting purposes.
  * Fix shutdown issues on tracker-store introduced in 1.99.0. No more
spurious integrity checks on startup.
  * Misc code and build fixes.

Translations: fur, id, sk


ChangeLog
=
https://download.gnome.org/sources/tracker/1.99/tracker-1.99.1.changes  (11.7K)

Download

https://download.gnome.org/sources/tracker/1.99/tracker-1.99.1.tar.xz (4.66M)
  sha256sum: 4ae8eb7d94a8515be7c6a5e314b8ac422381c3820387222fd2bf22dfd9e02770
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

Re: [Tracker] domain-ontologies, continued

[Carlos Garnacho]

Hey :)

A general update for everyone, I do consider this ready for merging
and I think I'll do early next week unless I see glaring bugs. All
tests pass, tracker does still behave as it ever used to by default,
there's some documentation, and I put the branch to practice in:

https://people.gnome.org/~carlosg/gnome-news.flatpak

If you want to try just do:
flatpak --user install --bundle ./gnome-news.flatpak
flatpak run org.gnome.News

You will see in ps:
$ ps -ef |grep tracker
carlos   26673  1404  0 18:08 ?00:00:00 bwrap --args 13
/app/libexec/tracker-miner-rss -d org.gnome.News
carlos   26680 26673  0 18:08 ?00:00:00 bwrap --args 13
/app/libexec/tracker-miner-rss -d org.gnome.News
carlos   26681 26680  0 18:08 ?00:00:00
/app/libexec/tracker-miner-rss -d org.gnome.News
carlos   26687  1404  0 18:08 ?00:00:00 bwrap --args 13
/app/libexec/tracker-store -d org.gnome.News
carlos   26694 26687  0 18:08 ?00:00:00 bwrap --args 13
/app/libexec/tracker-store -d org.gnome.News
carlos   26695 26694  0 18:08 ?00:00:00
/app/libexec/tracker-store -d org.gnome.News
...

which are the sandboxed tracker processes spawned to handle the
gnome-news data. gnome-news requirements outside the sandbox can be
made to be quite minimal, eg. no r/w access to the user homedir
whatsoever.


On Sun, Jun 11, 2017 at 7:19 PM, Philip Van Hoof <phi...@codeminded.be> wrote:
> On Tue, 2017-06-06 at 20:42 +0200, Carlos Garnacho wrote:
>> Hey hey,
>>
>> During the past week I've been continuing the stuff that Philip
>> started on wip/domain-ontologies. I pushed my current progress on
>> wip/carlosg/domain-ontologies:
>>
>> https://git.gnome.org//browse/tracker/log/?h=wip/carlosg/domain-ontologies
>
> Great! I'll keep an eye on the upcoming changing. Maybe I can soon help
> out here and there.
>
>> So far it's shaping up nicely, private databases are possible there
>> through the public tracker_sparql_connection_local_new(_async) calls,
>> xsd/dc/rdfs/nrl/nao are the are loaded from GResource and are the base
>> ontology, the local connection will run a dedicated thread for
>> updates, in a very similar fashion to tracker-store itself.
>
> Wow, nice progression!

Thanks :), there was enough peer pressure already :p, flatpak has been
a thing already for a couple of years and Tracker was an important
missing piece. People just has been allowing sandboxed apps to talk to
org.freedesktop.Tracker1 which is far too coarse.

>
>> My topmost
>> items in the todo now are:
>
>> - TrackerDataManager (and many other subsystems) is still a singleton,
>> which doesn't play nicely if you can now create multiple connections
>> that require it differently. I'm halfway into having it be an
>> object/struct so each connection can get its own instance.
>
> Aha, I was at first thinking or planning to simply start multiple
> tracker-store processes. But maybe when TrackerDataManager isn't a
> singleton anymore then one tracker-store could host multiple databases.

Yeah, I've actually gone that way wrt tracker-store. You can get
separate tracker-store processes each managing its own locations. It
is indeed feasible with this work to have a single tracker-store
process exporting several dbus objects, it's just a combination I
don't see many gains with. You have to teach everything else to talk
with multiple objects, and doesn't bring many benefits to multiprocess
approach.

But FWIW I actually finished this item. I don't think it's as
important from the service pov as it is from the API one, mainly for
not having it limited to one single global TrackerSparqlConnection for
everything. Clients or plugins may feasibly create multiple local r/w
connections, although the traditional tracker-store backed one is
still unique.

>
> And, more importantly, a single client's process could connect to
> multiple different metadata graph stores.
>
> Because it's probably inexcusable to have to expect from a client
> developer to split code up in multiple processes, to connect to multiple
> graph stores.
>
>> - Lots of documentation need to be written around this: how to create
>> new ontologies, data migration concerns, dos and don'ts, ...
>
> Yup. But that's good. Lot's of work for the open source youth ;-)
>
>> - Even if some apps could take advantage of private databases, for
>> some scenarios we do need to make it possible running a standalone set
>> of tracker dbus services for private use. I'm still unclear on how to
>> make it most transparent to apps, probably through libtracker-control
>> API.
>
> nod. Encode a per-application and/or per domain UUID in the DBus path of
> the objects?

Everything fell more in place with the separate processes aproach,
tracker processes take over the o

Re: [Tracker] Removing Maemo support upstream

[Carlos Garnacho]

Hi!,

On Tue, May 23, 2017 at 12:30 PM, Sam Thursfield  wrote:
> Hi all
> There's a bunch of references to Maemo in the Tracker source code.
> Specifically we have:
>
>   * the 91-maemo ontology -- only referenced in some special casing in
> the applications miner
>   * the 39-mto (Maemo transfer ontology) -- unused in core Tracker code
>   * the 40-mlo (Maemo location ontology) -- unused in core Tracker code
>   * the 41-mfo (Maemo feeds ontology)  -- unused in core Tracker code
>   * the 92-slo (Simplified Maemo location ontology) -- still widely used

Let's keep these, they might be used elsewhere.

>   * the --enable-maemo config flag
>   * the create-tests-xml.py and create-tests-aegis.py scripts in
> tests/functional-tests
>   * some Maemo-specific functional tests
>   * some special casing in the applications miner
>   * the Maemo userguides miner

But I'll be delighted to see all this go :). it's worth pointing out
that --enable-maemo seems just used in order to install the 91-maemo
ontology, but the miner-apps stuff is under a HAVE_MEEGOTOUCH, so it
is possible to have a miner-apps using an ontology that is not
installed. This, and code that we can't possibly test (eg. the stuff
under HAVE_MEEGOTOUCH defines) makes us look bad IMO :(.

>
> The Maemo-specific bits of the functional tests are getting in my way
> quite a bit, I don't have any way of testing out that the
> scratchbox-based code paths actually work so I don't think we should
> be carrying them upstream.
>
> The ontologies and the user-guides miner don't cause me many issues,
> but it might make sense to remove these as well since we're not using
> them in the core of Tracker at all.
>
> The last release of Maemo or Meego appears to be 5 years ago and I
> don't know that more are planned.
>
> Mer / Sailfish OS still uses Tracker and have a fork at
> . They're building with
> --enable-maemo plus an additional --enable-nemo flag.
>
> I've CC'ed Andrew who seems to maintain that -- would you be happy to
> carry the Maemo specific bits that you're using as downstream patches
> from now on?

I think most stuff can be just implemented using tracker libraries and
maintained standalone, no need to patch tracker.

>
> Is anyone else using these ontologies and code paths that I wasn't aware of?

Did we get enough silence already? :p

Cheers,
  Carlos
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

Re: [Tracker] Extending indexed file types

[Carlos Garnacho]

Hi Bjorn,

On Tue, Jun 6, 2017 at 11:28 AM, Björn Johansson
 wrote:
> Hi, I just installed the latest Tracker from the ubuntu standard repo.
>
> Seems to be working fine but my Jupyter notebooks does not seem to be
> indexed.
> Is this expected?

Yes, Tracker has no specific knowledge of these files.

>
> Jupyter notebooks are a kind of JSON text files containing code and text
> (http://jupyter.org/).
>
> Can I extend Tracker to also index these files? They are popular in the
> scientific computing community.

We do accept patches :). As Debarshi pointed out, there's quite a few
examples in src/tracker-extract in the tracker git tree.

But those Tracker extract modules rely on specific enough mimetypes,
could be worth checking first whether shared-mime-info knows about
jupyter specifically, eg. doing:

gio info $FILE |grep content-type

Cheers,
  Carlos
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

[Tracker] domain-ontologies, continued

[Carlos Garnacho]

Hey hey,

During the past week I've been continuing the stuff that Philip
started on wip/domain-ontologies. I pushed my current progress on
wip/carlosg/domain-ontologies:

https://git.gnome.org//browse/tracker/log/?h=wip/carlosg/domain-ontologies

So far it's shaping up nicely, private databases are possible there
through the public tracker_sparql_connection_local_new(_async) calls,
xsd/dc/rdfs/nrl/nao are the are loaded from GResource and are the base
ontology, the local connection will run a dedicated thread for
updates, in a very similar fashion to tracker-store itself. My topmost
items in the todo now are:

- TrackerDataManager (and many other subsystems) is still a singleton,
which doesn't play nicely if you can now create multiple connections
that require it differently. I'm halfway into having it be an
object/struct so each connection can get its own instance.

- Lots of documentation need to be written around this: how to create
new ontologies, data migration concerns, dos and don'ts, ...

- Even if some apps could take advantage of private databases, for
some scenarios we do need to make it possible running a standalone set
of tracker dbus services for private use. I'm still unclear on how to
make it most transparent to apps, probably through libtracker-control
API.

There's of course more items for the longer term, but all tests pass
with no functional changes, so seems good enough for an update :).

And btw, I still think it makes sense to tag tracker-next as 2.0, and
use the opportunity to switch to semver, I do hope it plays out and
reduces some maintenance burden in maintaining multiple versions.

Cheers,
  Carlos
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

Re: [Tracker] Meson build instructions for Tracker

[Carlos Garnacho]

Hey :),

On Sun, May 21, 2017 at 11:35 PM, Sam Thursfield <sss...@gmail.com> wrote:
> On 5/21/17, Carlos Garnacho <carl...@gnome.org> wrote:
>> On Thu, Mar 30, 2017 at 11:54 PM, Sam Thursfield <sss...@gmail.com> wrote:
>> FWIW, the meson step fails here because I lack the gee library, which
>> we don't need anymore :), those bits can be removed from the root dir
>> and utils/tracker-resdump meson.build files.
>
> OK, done.
>
>>> You can run the test suite:
>>>
>>> ninja-build test
>>
>> Hmm, amusingly "ninja test" fails on some tests in weird ways here. I
>> must investigate further, but won't make this hold the merge.
>
> Interesting .. I see some failures as well. I'll see if I can solve
> any. The functional tests mostly fail but that's been the case anyway,
> it's just that previously they weren't run unless you manually did
> `make functional-test`.

Right, most failing tests are the functional ones. I'm more baffled at
the tracker-steroids test timing out here. Running it manually leaves
it stuck awaiting for the reply of the first update, while the
autotools generated one just succeeds.

Hopefully functional tests will get better when we have
domain-ontologies in place :).

>
>> As long as autotools config is preserved so far, I think this can be
>> merged soon so it gets more coverage through jhbuild/whatnot. At a
>> later point we can definitely consider going meson-only :).
> ...
>>
>> Please let's get this merged ASAP :), I think it looks and works good
>> enough that we get further coverage.
>
> I've pushed a new 'meson-final' branch with everything in single
> commit; I can merge this tomorrow if you're happy with it.

Go go go! I guess we'll have to do
https://mail.gnome.org/archives/desktop-devel-list/2017-April/msg00091.html
before the next release, but I can go ahead after you merge it.

Cheers,
  Carlos
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

Re: [Tracker] Meson build instructions for Tracker

[Carlos Garnacho]

Hi!,

On Thu, Mar 30, 2017 at 11:54 PM, Sam Thursfield  wrote:
> Hello
> I've been working on build instructions for Tracker using Meson.
> They are now pretty much ready for use!
>
> You can find them in the branch wip/sam/meson:
> https://git.gnome.org/browse/tracker/commit/?h=wip/sam/meson
>
> To use, you need to install Meson, which can either be done from Pip
> (pip3 install --user meson), from a distro package (but beware that it
> might be old), or you can run it right from Git
> (git://github.com/mesonbuild/meson).
>
> Then you do something like this:
>
> mkdir build
> cd build
> meson ..
> ninja-build

FWIW, the meson step fails here because I lack the gee library, which
we don't need anymore :), those bits can be removed from the root dir
and utils/tracker-resdump meson.build files.

>
> You can run the test suite:
>
> ninja-build test

Hmm, amusingly "ninja test" fails on some tests in weird ways here. I
must investigate further, but won't make this hold the merge.

>
> And `ninja-build install` to install of course.
>
> Meson accepts standard arguments like --prefix, but to change the
> Tracker-specific options you need to run mesonconf. Run it in the build
> directory with no arguments to see all the available options, and pass
> `-D option=value` to set something.
>
> I've compared an install of this with an equivalent Autotools build. So
> I'm confident there aren't major regressions, but it does require more
> testing. Help with that is appreciated!

As long as autotools config is preserved so far, I think this can be
merged soon so it gets more coverage through jhbuild/whatnot. At a
later point we can definitely consider going meson-only :).

>
> Here are a few remaining issues:
>
>   * There's no `make dist` equivalent. We can use `git archive` to
> produce tarballs, but these won't have the .c files generated by
> valac. That said, shipping the generated .c files does make the
> Vala preprocessor useless so it would be good if we can stop,
> but that could break things for downstreams unexpectedly.

I agree that shipping generated files is a bad idea, there's very
little scenarios where I can see this helping (perhaps embedded?), and
at that point I don't think additionally compiling vala is a big leap.

I agree with Michael in that reproducible builds are a vala problem
and we just tape over it. Introducing issues here is not quite ideal,
but there's also chances we 1) don't hit the code paths that might
result in flipped C code lines, or 2) we expose new places where this
happens, so I'd say it's more beneficial in the long term to just do
it.

>
>   * The Firefox, Thunderbird, Evolution and Nautilus plugins don't
> have Meson build rules. I'm not sure if any of these are actually
> still used, we can easily fix that if they are.

Sounds like a good time to kill them all :). They were also gone in my
past effort to divide tracker stuff (which would be nice to try again
after this is merged).

>
>   * Not every single configure flag has an equivalent in meson_options.txt

I see you've mainly killed tracker-extract toggles that had an "auto"
fallback, makes sense.

>
> Meson is still kinda beta quality but it's under active development, the
> maintainers are very helpful and responsive and the build is literally
> twice as fast than with Autotools. Plus the output is a lot clearer so all the
> compile warnings are there.
>
> If you try this, please let me know any issues and I'll be happy try and 
> triage
> and hopefully fix them.

Please let's get this merged ASAP :), I think it looks and works good
enough that we get further coverage.

Thanks,
  Carlos
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

Re: [Tracker] Tracker search does not give results when no content is indexed (only file names)

[Carlos Garnacho]

Hi!,

On Wed, Apr 19, 2017 at 7:10 AM, Petko Ditchev  wrote:
> Hello everyone,
> I'm trying to use tracker to only locate files (in GNOME), I don't
> really search by content for anything. So I set the "Index content of
> files found" option to false. I'm writing here instead of filing a bug,
> since I don't know if that's expected behaviour or there's something
> wrong in my setup.
>  Some details : OS - Manjaro (stable) , DE - GNOME, tracker - 1.10.5
> "tracker search a" does not give any results; "tracker search -f" lists
> the files "indexed" ; "tracker status" says there are a few thousand
> files indexed, and no active indexing (as expected) . I did a "tracker
> reset -r" and reindexing, but the result is the same. Searching by
> content works btw.

If "tracker search a" is not just an example, yes, that's expected.
There's a couple of things affecting full-text search:

1) There's a minimum length for the search term, set to 3 characters
2) There is a blacklist of far too common words (so called "stop
words") that are ignored for indexing and search purposes, basically
because they make database size explode and searching for those would
match virtually anything. I mean words like a/the/one/... you can see
all stop words at
https://git.gnome.org/browse/tracker/tree/src/libtracker-common/stop-words

If that is not the case and search fails even with more long/concrete
search terms that you know should match, please do file a bug.

Cheers,
  Carlos

>
> Any help would be greatly appreciated,
> Petko
> ___
> tracker-list mailing list
> tracker-list@gnome.org
> https://mail.gnome.org/mailman/listinfo/tracker-list
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

Re: [Tracker] TRACKER-Critical

[Carlos Garnacho]

Hi Jose!

On Sun, Apr 16, 2017 at 9:58 AM, Jose Arroyo  wrote:
> Hello Chris,
>
> It's been a while since I've dived into Tracker so my comments may not be
> terribly useful.
>
>> It seems that whenever I reboot my Ubuntu 16.04.2LTS box I get a huge
>> hourly syslog snippet with what I've put pastebin:
>
>
>  At a first glance, those traces seem to show that Tracker is trying to
> re-index everything after a reboot. I think that this may be triggered by
> some config option but it may be something else.
>
> In any case, the "UNIQUE constraint failed: nie:DataObject.nie:url" error
> means that the SPARQL query that is used to update the file's entry in
> Tracker's DB is trying to insert a second url the the file's entry as if it
> wasn't there. It then fails because there is a constraint to a single url
> per file.
>
> Then you have the logs of this kind: "(tracker-miner-fs:3725):
> Tracker-WARNING **: File
> 'file:///home/chris/Downloads/fetchmail-6.4.0.beta2/trio' has been
> reenqueued more than 2 times. It will not be indexed." The way the Tracker
> DB is organized, an element is not inserted in the DB unless its parent is
> already inserted in the DB. So if for some reason, file://foo/bar is being
> queued for insertion before file://foo, it'll be re-queued for insertion
> after file://foo is inserted.
>
> However, if for some reason, the insertion of file://foo fails, this
> mechanism will start looping. There is a counter to prevent these infinite
> loops which is why you get these logs that say that item X won't be indexed
> because it has already been reenqueued too many times.
>
> This whole behavior is somewhat harmless and, as Sam says, might be fixed by
> reindexing the whole thing from scratch. In any case, what I think is going
> on boils down to 3 things:
> - Something is triggering a re-indexation of already indexed directories on
> reboots (does it also happen if you just stop and start the tracker miner
> fs?). Is it limited only to your "Downloads" directory?
> - Something makes Tracker to try and index found files as if they weren't
> already present in the DB (maybe something wrong with the file last modified
> check/comparison and the held values in Tracker's internal file system
> cache?). This causes the SPARQL insert queries to fail consistently.
> - The reenqueue mechanism triggers the reinsertion of many directories,
> which continue to fail consistently and thus output a large amount of error
> logs.
>
> Btw, for the Tracker maintainers, there has been for some time a missing
> g_object_unref() when a file to be reenqueued is dropped
> (https://git.gnome.org/browse/tracker/tree/src/libtracker-miner/tracker-miner-fs.c#n2134).
> They are ref'd when item_reenqueue() is called, but they are never unref'd
> if the reenqueue'ing isn't actually done. This causes those files to get
> permanently stuck in Tracker's filesystem cache.

Oops, you are right. IIRC you brought this up over IRC but then went
forgotten... I've just pushed
https://git.gnome.org/browse/tracker/commit/?id=900636b2bfe5b91175e521a4acd2296216eb52b0
fixing this.

Thanks for the heads up!
  Carlos
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

[Tracker] Fwd: tracker 1.12.0

[Carlos Garnacho]

-- Forwarded message --
From: Carlos Garnacho <install-mod...@master.gnome.org>
Date: Mon, Mar 20, 2017 at 2:28 PM
Subject: tracker 1.12.0
To: FTP Releases <ftp-release-l...@gnome.org>


About Tracker
=

Tracker is a semantic data storage for desktop and mobile devices.
Tracker uses W3C standards for RDF ontologies using Nepomuk with
SPARQL to query and update the data.

Tracker is a central repository of user information, that provides two
big benefits for the user; shared data between applications and
information which is relational to other information (for example:
mixing contacts with files, locations, activities and etc.).

News


  * Multiple compile warning fixes
  * Fix compilation on older vala

  Overview of changes between 1.10 and 1.12:

  * The extractors are now sandboxed
  * Small improvements towards full sparql 1.1 compliance
  * Many fixes for Coverity warnings
  * Thread contention in direct-access tracker clients has been
eliminated, concurrent queries are now significantly faster.
  * Many small fixes all over the place.

Translations: da, id, it, ko, lt, lv


ChangeLog
=
https://download.gnome.org/sources/tracker/1.12/tracker-1.12.0.changes  (1.54K)

Download

https://download.gnome.org/sources/tracker/1.12/tracker-1.12.0.tar.xz (4.83M)
  sha256sum: 83193dd8c05e3e8b05fa8f1cfb80964fa78f7aeb86f3302fa4be6ec7e6ff596f
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

Re: [Tracker] Could not create FTS delete statement:

[Carlos Garnacho]

Hi Chris,

On Thu, Mar 2, 2017 at 3:24 PM, Chris  wrote:
> Could someone be kind enough to tell me what this warning means? The
> complete line reads:
>
> Mar  2 08:20:18 localhost org.freedesktop.Tracker1[3934]: (tracker-
> store:4352): Tracker-WARNING **: Could not create FTS delete statement:
> table fts5 has no column named nco:hobby
> Mar  2 08:20:18 localhost org.freedesktop.Tracker1[3934]: (tracker-
> store:4352): Tracker-WARNING **: Could not create FTS delete statement:
> table fts5 has no column named nco:hobby

This started appearing in some systems due to a glitch in the change
from fts4 to fts5, tracker had to trigger a change in full-text
indexed fields, and nco:hobby was the innocent victim.

But it does seem like the file ~/.cache/tracker/ontologies.gvdb
(containing a cache of the current ontology) didn't get updated
properly in some cases, which means that tracker still deals with that
property as if it were fts-indexed, while it's no longer the case.

>
> I'm very sure it's harmless but as one who looks at his syslog closely
> its one of those petty annoyances that bites into my OCD that I know
> shouldn't be there.

You can get rid of it by doing:
tracker daemon -t
rm ~/.cache/tracker/ontologies.gvdb
tracker daemon -s

That should get that file rebuilt from scratch, containing the
up-to-date ontology. I would have liked to do something from the
Tracker side, but that requires yet another change in FTS fields, and
those happen rarely...

Cheers,
  Carlos
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

Re: [Tracker] (tracker-extract:3603): dconf-CRITICAL

[Carlos Garnacho]

Hi Chris,

On Tue, Feb 28, 2017 at 3:50 PM, Chris <cpoll...@embarqmail.com> wrote:
> On Tue, 2017-02-28 at 11:23 +0100, Carlos Garnacho wrote:
>> Hi,
>>
>> On Mon, Feb 27, 2017 at 6:52 PM, Chris <cpoll...@embarqmail.com>
>> wrote:
>> >
>> > The complete notation to my syslog that I'm seeing every two
>> > minutes
>> > every hour, every day is:
>> >
>> > Feb 27 11:44:33 localhost tracker-extract.desktop[3603]: (tracker-
>> > extract:3603): dconf-CRITICAL **: unable to create file
>> > '/run/user/1000/dconf/user': Permission denied.  dconf will not
>> > work
>> > properly.
>> > Feb 27 11:46:42 localhost tracker-extract.desktop[3603]: message
>> > repeated 2 times: [ (tracker-extract:3603): dconf-CRITICAL **:
>> > unable
>> > to create file '/run/user/1000/dconf/user': Permission
>> > denied.  dconf
>> > will not work properly.]
>>
>> I've noticed this warning across reported logs since the
>> tracker-extract process was sandboxed. I suspect this is related to a
>> sandboxed thread accessing gsettings, although it's most clearly not
>> tracker itself, since all tracker-extract settings are read on
>> startup
>> and in-memory representation is accessible readonly to the extractor
>> threads.
>>
>> So this could be some tracker module that is trying to poke GSettings
>> underneath, but I'm entirely unclear which. If anyone can get me
>> backtrace of that specific warning (eg. setting up
>> G_DEBUG=fatal-criticals in tracker-extract environment), that'd be
>> appreciated.
>>
>> That said, the warning should be harmless, at least if I'm right.
>>
>> Cheers,
>>   Carlos
>>
> Hi Carlos, someone suggested, I think it was on the list, that an X
> process is changing the permission of /run/user/1000/dconf/user every 2
> minutes. I find that very hard to believe that could happen especially
> since the only process I have that continuously wakes every 2 minutes
> if fetchmail and it's not an X process. I can open Gnome terminal and
> see that the timestamp -rw--- 1 chris chris 2 Feb 28 08:42 user is
> the same as when I go to check it with the ls -l command. IOW I'm
> beginning to agree with you now that I don't think the permissions are
> being changed but tracker thinks they are. Does that sound about
> right?

That is precisely what I think is happening. Inside the sandboxed
bits, open() with write permissions is disallowed altogether, I think
something is attempting to read settings inside the sandbox which in
turn tries to open that file with readwrite permission. I've only seen
this warning recently in recent bug reports, and always tied to
tracker-extract.

On second read of gsettings/dconf code, it seems possible that file
descriptors are reloaded behind our back, so it sounds plausible that
it is tracker-extract after all, I'm doing a patch that I think should
fix these warnings.

>
> Since I'm retired and don't have much else to do except tend to my
> plants and go to doctors appointments at the VA I'll give setting up
> the backtrace you need a try if you can point me to a good how-to.

I'll try to make the patch available upstream asap, so hopefully
Ubuntu will pick it up soon. If you anyway want to get into this
trouble :), here's some steps:

1) you need gdb installed, and hopefully tracker debuginfo packages so
the backtrace has more sense
2) in a terminal do: tracker daemon -t
3) in another terminal do:
   G_DEBUG=fatal-criticals gdb /usr/lib/tracker/tracker-extract
4) in the gdb prompt, do 'r' (shortcut for "run"), tracker-extract
will run as usual
5) back in the first terminal, do: tracker daemon -s
6) get on with your normal activity
7) when the warning is hit (assuming it's the first one you'll get),
gdb will be back to its prompt, you can do "t a a bt" to get the
backtrace for all threads (what I ask for), or you can tell it to
continue with 'c'.

Please do file a bug to bugzilla.gnome.org with this information.

Cheers,
  Carlos
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

Re: [Tracker] (tracker-extract:3603): dconf-CRITICAL

[Carlos Garnacho]

Hi,

On Mon, Feb 27, 2017 at 6:52 PM, Chris  wrote:
> The complete notation to my syslog that I'm seeing every two minutes
> every hour, every day is:
>
> Feb 27 11:44:33 localhost tracker-extract.desktop[3603]: (tracker-
> extract:3603): dconf-CRITICAL **: unable to create file
> '/run/user/1000/dconf/user': Permission denied.  dconf will not work
> properly.
> Feb 27 11:46:42 localhost tracker-extract.desktop[3603]: message
> repeated 2 times: [ (tracker-extract:3603): dconf-CRITICAL **: unable
> to create file '/run/user/1000/dconf/user': Permission denied.  dconf
> will not work properly.]

I've noticed this warning across reported logs since the
tracker-extract process was sandboxed. I suspect this is related to a
sandboxed thread accessing gsettings, although it's most clearly not
tracker itself, since all tracker-extract settings are read on startup
and in-memory representation is accessible readonly to the extractor
threads.

So this could be some tracker module that is trying to poke GSettings
underneath, but I'm entirely unclear which. If anyone can get me
backtrace of that specific warning (eg. setting up
G_DEBUG=fatal-criticals in tracker-extract environment), that'd be
appreciated.

That said, the warning should be harmless, at least if I'm right.

Cheers,
  Carlos
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

Re: [Tracker] ubuntu-gnome/+bug/1653392

[Carlos Garnacho]

Hi,

On Wed, Feb 22, 2017 at 6:54 PM, Chris <cpoll...@embarqmail.com> wrote:
> On Wed, 2017-02-22 at 11:09 +0100, Carlos Garnacho wrote:
>> Hi Chris,
>>
>> On Wed, Feb 22, 2017 at 12:30 AM, Chris <cpoll...@embarqmail.com>
>> wrote:
>> >
>> > I'm sure hoping I'm sending this to the correct list. If not I
>>
>> It is :)
>>
>> >
>> > sincerely apologize. I've been experiencing this - tracker-store
>> > crashed with SIGSEGV in g_slice_alloc()  almost daily since
>> > 12/31/2016.
>> > It's listed here - https://bugs.launchpad.net/ubuntu-gnome/+bug/165
>> > 3392
>> >  still shown as new and assigned to nobody. Here is some version
>> > information:
>>
>> Strange, that link doesn't work here, just says "page not found".
>> Anyway, SIGSEGV when allocating memory is a pretty good indicator of
>> memory corruption. Valgrind could be of help there, you can run
>> tracker-store through valgrind with:
>>
>> killall -15 tracker-store; valgrind --leak-check=full --num-
>> callers=25
>> --log-file=~/not-indexed-folder/tracker-store-valgrind.txt
>> /usr/lib/tracker/tracker-store
>>
>> This will make tracker-store real slow (and all re-indexing process
>> in
>> chain effect), but it should catch and log invalid frees/writes into
>> that log file.
>>
>> Oh, also make sure you have debug packages for tracker, I forgot what
>> the ubuntu way is for that.
>>
>> >
>> >
>> > Distributor ID: Ubuntu
>> > Description:Ubuntu 16.04.2 LTS
>> > Release:16.04
>> > Codename:   xenial
>> >
>> > chris@localhost:~$ apt-cache policy tracker
>> > tracker:
>> >   Installed: 1.8.3-0ubuntu0~xenial1
>> >   Candidate: 1.8.3-0ubuntu0~xenial1
>> >   Version table:
>> >  *** 1.8.3-0ubuntu0~xenial1 500
>> > 500 http://ppa.launchpad.net/gnome3-team/gnome3-staging/ubu
>> > ntu
>> > xenial/main amd64 Packages
>>
>> Hmm, however, 1.8.x is slightly old now, you should perhaps try first
>> with the 1.10.x stable branch which has been out for a few months
>> now.
>> Although I tbh don't remember recent tracker-store memory corruption
>> bugs, nor having seen anything like that fixed recently.
>>
>> If after updating the bug still persists, please file a bug at
>> https://bugzilla.gnome.org/enter_bug.cgi?product=tracker with the
>> valgrind info and we'll take it from there.
>>
>> Cheers,
>>   Carlos
>>
> Carlos, an update to tracker 1.8.3-0ubuntu0~xenial2 was put out a bit
> ago, I installed
>
> tracker-gui amd64 1.8.3-0ubuntu0~xenial2
> tracker-miner-fs amd64 1.8.3-0ubuntu0~xenial2
> gir1.2-tracker-1.0 amd64 1.8.3-0ubuntu0~xenial2
> tracker-extract amd64 1.8.3-0ubuntu0~xenial2
> libtracker-miner-1.0-0 amd64 1.8.3-0ubuntu0~xenial2
> tracker amd64 1.8.3-0ubuntu0~xenial2
> libtracker-control-1.0-0 amd64 1.8.3-0ubuntu0~xenial2
> libtracker-sparql-1.0-0 amd64 1.8.3-0ubuntu0~xenial2
>
> They all came from - http://ppa.launchpad.net/gnome3-team/gnome3-stagin
> g/ubuntu xenial/main amd64
>
> Since the bug has a habit of popping up daily I'll see what happens

Thanks! I provided the fix in that package after I was granted access
to the bug you filed and saw the logs. It is also in tracker git now
fwiw.

I'm however surprised that you get through these error paths, please
keep an eye on "Transaction rollback failed" warnings from
tracker-store, as I wouldn't really expect these to happen, it
shouldn't crash anymore though.

Cheers,
  Carlos
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

Re: [Tracker] Discarding broken metadata from miners

[Carlos Garnacho]

Hey :),

On Fri, Feb 24, 2017 at 11:19 AM, Debarshi Ray  wrote:
> Hey,
>
> Every now and then, there is a bug or regression in one of miners
> which leads to broken metadata being inserted into the database.  For
> example, here is a low impact one:
> https://bugzilla.gnome.org/show_bug.cgi?id=767472#c48
>
> However, the breakage can be more serious and visibly impact
> applications. For example, this one:
> https://bugzilla.gnome.org/show_bug.cgi?id=776723
>
> It broke various fields in gnome-photos' properties dialog, and due to
> the wrong nfo:orientation values images were no longer correctly
> oriented.
>
> Bugs happen. Such is life. What is the recommended way to deal with
> these situations. So far I have been telling users to hard reset their
> database and restart the miners using the command line. I am afraid
> that isn't an elegant solution.

tracker reset --file is friendlier :), but I do agree. We already keep
several files in ~/.cache/tracker to ensure that the database itself
is up-to-date, the easy way would be adding a fuck-up counter for
miner-fs (and thus tracker-extract) to force reindex and thus do the
same maintenance with the database content.

Ideally, this would have a better granularity, if a bug only affects
image files, we shouldn't need reindexing everything from scratch. I
wonder if we could apply the same approach than we have for the FTS
tokenizer: keeping the most recent commit ID affecting it, and
checking it against a file, whenever it changes in the user setup, an
update is due.

However, the git files to track are rather varying and spread, there's
tracker-extract-*.c files, there's tracker-resource.c, there's eg.
tracker-xmp.c wherever it applies,... I guess that can get under
control soon.

>
> Do the Tracker miners version the metadata that they insert into the
> database? Or, is it possible to programmatically discard metadata
> coming from a certain miner and force a reindex?

There's no versioning... For dropping full miner data, I'd wish we
supported the DROP GRAPH syntax, all filesystem miners in tracker
share the same TRACKER_OWN_GRAPH_URN define.

This however could be open coded as:
"DELETE WHERE { GRAPH <" TRACKER_OWN_GRAPH "> { ?u a rdfs:Resource }}"

That should leave a clean slate for miners, still maybe a bit too clean :).

>
> In gnome-online-miners (those are the out-of-tree miners used by
> gnome-documents/photos to index online accounts advertised by
> gnome-online-accounts), we handle this by having each miner tag their
> insertions with nie:version (grep for 'version' in
> src/gom-miner.c). Whenever a bug that could have inserted broken
> metadata is fixed, we bump the miner version. When the user installs
> the updated miner, it will automatically purge the old metadata and
> re-index.
>
> So, any suggestions? Thoughts?

For data maintenance, I suggest you look into inserting g-o-m data
into its own graph, version management is more open to discussion. The
approach you picked seems indeed the nepomuk-y way, although I'm not
sure how much of a great argument that is nowadays :).

Cheers,
  Carlos
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

Re: [Tracker] ubuntu-gnome/+bug/1653392

[Carlos Garnacho]

Hi Chris,

On Wed, Feb 22, 2017 at 12:30 AM, Chris  wrote:
> I'm sure hoping I'm sending this to the correct list. If not I

It is :)

> sincerely apologize. I've been experiencing this - tracker-store
> crashed with SIGSEGV in g_slice_alloc()  almost daily since 12/31/2016.
> It's listed here - https://bugs.launchpad.net/ubuntu-gnome/+bug/1653392
>  still shown as new and assigned to nobody. Here is some version
> information:

Strange, that link doesn't work here, just says "page not found".
Anyway, SIGSEGV when allocating memory is a pretty good indicator of
memory corruption. Valgrind could be of help there, you can run
tracker-store through valgrind with:

killall -15 tracker-store; valgrind --leak-check=full --num-callers=25
--log-file=~/not-indexed-folder/tracker-store-valgrind.txt
/usr/lib/tracker/tracker-store

This will make tracker-store real slow (and all re-indexing process in
chain effect), but it should catch and log invalid frees/writes into
that log file.

Oh, also make sure you have debug packages for tracker, I forgot what
the ubuntu way is for that.

>
> Distributor ID: Ubuntu
> Description:Ubuntu 16.04.2 LTS
> Release:16.04
> Codename:   xenial
>
> chris@localhost:~$ apt-cache policy tracker
> tracker:
>   Installed: 1.8.3-0ubuntu0~xenial1
>   Candidate: 1.8.3-0ubuntu0~xenial1
>   Version table:
>  *** 1.8.3-0ubuntu0~xenial1 500
> 500 http://ppa.launchpad.net/gnome3-team/gnome3-staging/ubuntu
> xenial/main amd64 Packages

Hmm, however, 1.8.x is slightly old now, you should perhaps try first
with the 1.10.x stable branch which has been out for a few months now.
Although I tbh don't remember recent tracker-store memory corruption
bugs, nor having seen anything like that fixed recently.

If after updating the bug still persists, please file a bug at
https://bugzilla.gnome.org/enter_bug.cgi?product=tracker with the
valgrind info and we'll take it from there.

Cheers,
  Carlos
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

[Tracker] Fwd: tracker 1.8.3

[Carlos Garnacho]

-- Forwarded message --
From: Carlos Garnacho <install-mod...@master.gnome.org>
Date: Thu, Jan 19, 2017 at 12:27 PM
Subject: tracker 1.8.3
To: FTP Releases <ftp-release-l...@gnome.org>


About Tracker
=

Tracker is a semantic data storage for desktop and mobile devices.
Tracker uses W3C standards for RDF ontologies using Nepomuk with
SPARQL to query and update the data.

Tracker is a central repository of user information, that provides two
big benefits for the user; shared data between applications and
information which is relational to other information (for example:
mixing contacts with files, locations, activities and etc.).

News


  * tracker-extract: Whitelist multiple inocuous syscalls that were
reported to raise false positives in the extraction sandbox.


ChangeLog
=
https://download.gnome.org/sources/tracker/1.8/tracker-1.8.3.changes  (880)

Download

https://download.gnome.org/sources/tracker/1.8/tracker-1.8.3.tar.xz (4.75M)
  sha256sum: 9bbf8c8525b3a1496716a350bc50ba06af5e880a4386506ca3e45d6779065c42
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

[Tracker] Fwd: tracker 1.10.4

[Carlos Garnacho]

-- Forwarded message --
From: Carlos Garnacho <install-mod...@master.gnome.org>
Date: Thu, Jan 19, 2017 at 12:26 PM
Subject: tracker 1.10.4
To: FTP Releases <ftp-release-l...@gnome.org>


About Tracker
=

Tracker is a semantic data storage for desktop and mobile devices.
Tracker uses W3C standards for RDF ontologies using Nepomuk with
SPARQL to query and update the data.

Tracker is a central repository of user information, that provides two
big benefits for the user; shared data between applications and
information which is relational to other information (for example:
mixing contacts with files, locations, activities and etc.).

News


  * tracker-extract: Whitelist multiple inocuous syscalls that were
reported to raise false positives in the extraction sandbox.
  * Fixed tracker-extract insertion of pre-defined resources
  * Fixed TrackerResource SPARQL generation of rdfs:Resource properties
with cardinality>1


ChangeLog
=
https://download.gnome.org/sources/tracker/1.10/tracker-1.10.4.changes  (2.13K)

Download

https://download.gnome.org/sources/tracker/1.10/tracker-1.10.4.tar.xz (4.80M)
  sha256sum: 70e779e55bf580615a66db43ae45d081e1810cc11bbfcdd0a6339f506e0adf9d
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

[Tracker] Fwd: tracker 1.11.3

[Carlos Garnacho]

-- Forwarded message --
From: Carlos Garnacho <install-mod...@master.gnome.org>
Date: Wed, Jan 18, 2017 at 1:38 PM
Subject: tracker 1.11.3
To: FTP Releases <ftp-release-l...@gnome.org>


About Tracker
=

Tracker is a semantic data storage for desktop and mobile devices.
Tracker uses W3C standards for RDF ontologies using Nepomuk with
SPARQL to query and update the data.

Tracker is a central repository of user information, that provides two
big benefits for the user; shared data between applications and
information which is relational to other information (for example:
mixing contacts with files, locations, activities and etc.).

News


  * tracker-extract: Whitelist multiple inocuous syscalls that were
reported to raise false positives in the extraction sandbox.
  * Make libseccomp dependency only mandatory on Linux
  * Fix several leaks and Coverity warnings
  * Fixed tracker-extract insertion of pre-defined resources
  * Fixed TrackerResource SPARQL generation of rdfs:Resource properties
with cardinality>1


ChangeLog
=
https://download.gnome.org/sources/tracker/1.11/tracker-1.11.3.changes  (6.95K)

Download

https://download.gnome.org/sources/tracker/1.11/tracker-1.11.3.tar.xz (4.82M)
  sha256sum: 730fefc7a41997d83039ed7272b08e570b48ba58b73c92ba176532866914171a
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

Re: [Tracker] Incomplete results + lack of integration in Gnome for contents searching

[Carlos Garnacho]

Hi Jean-Christophe!,

On Thu, Dec 22, 2016 at 9:19 AM, Jean-Christophe Baptiste
 wrote:
> Hello,
>
> I started to use Tracker intensively as a replacement for Recoll. It is nice 
> and better integrated to Gnome, but the results are still less complete.
>
>
> 1) For some reason, some files are correctly indexed but not shown in the 
> result.
>
> See my screenshot enclosed :
>
> - the settings show paths that are indexed.
> - I search a term, with a limit of 100 files,
> - The expected file (and around ten others) does not show in the results
> - Checking the indexation manually on this file shows that the indexation 
> worked.
>
> What is going on ? Shall I open a bug report or am I missing something ?

That looks indeed strange, most probably down to tokenization of this
document text having gone wrong somehow. Quickly checking here, I see
Tracker doesn't handle too well the the d'match french form, the other
appearances should still result in a positive match though.

Please file a bug and we take it there. One first question for that
bug, does looking for any other terms inside these documents result in
match? or are these files entirely invisible to full-text search?

>
>
> 2) I have another question : tracker-needle works but is quite dated. Is 
> there a plan for a better GUI ?

Not from the Tracker side... tracker-needle was merely a showcase app,
it helped Tracker gain momentum, but nowadays the focus is in making
Tracker a better platform, and let the apps do the talking.

It also matches better the original Tracker vision, the purpose to
centralize the data was to make it widely available to applications,
those may have more contextual UI and search than a generic app can
have.

... And well, sure, lack of manpower is also involved. I'd however
advocate to make it a standalone project, as I'm actually looking into
splitting/sanitizing Tracker, tracker-needle will probably keep aging
in a separate repo unless someone picks it up.

>
> The integration in Nautilus is suprisingly not good, and it is also 
> disappointed to see that it is not used by gnome-shell in the activity view.

Nautilus implements search in 2 ways:
1) A fast path, based on Tracker. The query performed however chooses
to restrict to filename searches. And obviously, only works inside
indexed directories.
2) A slow path that works pretty much like find, and gets deeper in
the filesystem, again purely filename based, as there's no indexed
contents.

I think there are plans to show snippets in nautilus search, which
implies search within file content, this would bring nautilus search
on par with the "tracker search" tool. This is however something to
talk on #nautilus, or their bugzilla, as it's a matter of how do they
use Tracker.

On top of that, gnome-shell has these "search providers" as the
extension point. Nautilus implements one, which uses the fast path
above and hence has the same limitations.

>
> I am aware of a gnome-shell extension, but it does not work on recent 
> releases and the code looks a bit as a hack.
>
> Why not using Tracker as a search provider by default for file contents ?

gnome-shell search providers make sense with a backing application to
jump into when you click on the results. With the right tweaks
(changing the nautilus query so it doesn't do filename matching is
removing one line of code), nautilus has got everything to be that
application for generic file search.

Cheers,
  Carlos
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

Re: [Tracker] Referential Integrity (Was: Re: Fwd: tracker 1.11.2)

[Carlos Garnacho]

Hey :),

On Wed, Dec 21, 2016 at 2:49 PM, Philip Van Hoof <phi...@codeminded.be> wrote:
> On Wed, 2016-12-21 at 14:29 +0100, Carlos Garnacho wrote:


>
>> I need testing with further nodes though, as I'd expect things to get
>> linearly worse with data, both in size with the additional indexes and
>> performance w/o indexes. I kinda expected we'd have to pick our poison
>> here though :).
>
>
> Might be that just counting references and incrementing them in a column
> in the Resource table is cheaper than letting SQLite do all the extra
> required indexes?

Yeah, could be... the workload is spread between insertions and
deletes then, but at least the extra updates would poke already
indexed columns. A downside is that it also makes the harvesting of
broken triples rather manual.

>
> But right. Without testing, it's hard to know. And if SQLite does it for
> us with CASCADING Delete and RESTRICT: that's less code for us write.

That is my line of thought, the more we get the database to work for
us the better. Anyhow, I'll try to play with your explicit refcounting
idea, could be made to be automatically maintained with triggers.

Cheers,
  Carlos
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

[Tracker] Referential Integrity (Was: Re: Fwd: tracker 1.11.2)

[Carlos Garnacho]

Hey Philip :),

On Sun, Dec 11, 2016 at 9:45 PM, Philip Van Hoof <phi...@codeminded.be> wrote:
> On Sun, 2016-12-11 at 17:07 +0100, Carlos Garnacho wrote:
>
> [cut]
>
>> >> - Getting as close to supporting the full sparql 1.1 spec as possible
>> >> in libtracker-data:
>> >>   * property paths: last weekend got halfway with this \o/
>> >>   * graph management: for DROP GRAPH I think triggers will perform
>> >
>> > Did ever something happen to cleaning up anonymous nodes of deleted
>> > subjects/context, and or do reference counting on them (and clear them
>> > once they reach zero references)?
>> >
>> > If not then we are still leaking those in the db afaik. We always wanted
>> > to do something about that.
>>
>> Yeah, we still do leak those... I remember you/Jürg suggested setting
>> a foreign key with ON DELETE RESTRICT action from the various tables'
>> ID rows to the Resource table, so the cleaning up the Resource table
>> would fail for the still referenced nodes. I got as far as seeing
>> that:
>
>> - Performance would be just fine for the common ops, the trigger would
>> only run when trying to delete the parent key in the Resource table,
>> which is the once-in-a-while operation, modifications on the cols
>> setting the foreign key would be just as fast as they're now.
>
> nod
>
>> - It however wouldn't fix alone the other issue I saw happening before
>> the revert (graph URNs being deleted). I had a patch around that added
>> a Graph table, so IDs in the Resource table were ensured to be in
>> either rdfs:Resource or the Graph table. That already helped with
>> identifying and not deleting the graph URNs during garbage collection,
>> and seems useful for graph management, but I think I can't just add
>> the same RESTRICT action as CLEAR/DROP GRAPH will want pretty much the
>> opposite.
>
> Personally think some sort of reference counting will be needed for
> anonymous nodes references by different graphs..

It's not as much reference counting as it is a "weak pointer" what we
want here, no? i.e. we want to know whether the reference count
dropped to 0 anywhere else, and drop the last bit of info in that
case.

And that seems precisely the info we can obtain from foreign keys with
RESTRICT action, deleting the parent key would fail unless it's no
longer referenced.

>
>> - I also wondered if it's more desirable, or allowed by the sparql
>> spec, that we actually garbage collect the inconsistent nodes. IMO not
>> leaving this type of data coherence up to miners/apps being educated
>> when deleting would be a win, but I've only seen mentions in the
>> sparql spec about impls being free to drop empty graphs, nothing about
>> triples with no longer bound elements.
>
> I also don't think it's a problem to rid ourselves of orphaned anonymous
> nodes.
>
> Without a graph to be owned by, they can't be referenced other than by
> their uuid anyway.

Right, good that you confirm :). I've been playing further with
foreign keys, and I think there are two possible ways to properly
maintain the Resource table:

1) Doing immediate deletes, requires:
   a) all rowid columns in class tables to set a foreign key on
Resource.ID with ON DELETE CASCADE
   b) all property columns of type "resource" to set a foreign key on
Resource.ID with ON DELETE SET NULL
   c) Triggering deletion on the Resource table when a resource is to
be deleted.

2) Doing deferred deletes, requires:
   a) all rowid columns in class tables to set a foreign key on
Resource.ID with ON DELETE RESTRICT
   b) all property columns of type "resource" to set a foreign key on
Resource.ID with ON DELETE RESTRICT
   c) running a maintenance task that tries to harvest no longer used
resources, which past our initial heuristics (eg. checking it's not in
rdfs:Resource table) would still fail if referenced somewhere.

(FWIW, I'm omitting *:graph-id columns management)

#1 allows this broken node harvesting to happen immediately, while #2
still relies on the user being educated enough for the harvesting to
be effective. It might be an option though to use ON DELETE CASCADE on
2.b so the references pointing to this node are cut, and we get the
good effects of #1 when nodes are harvested.

But both fail pretty bad on step b... there's many checks going on in
child tables each time a Resource table row is deleted, and those
happen on columns that don't have an index. sqlite docs [1] recommend
that an index on the child table column(s) holding the foreign key is
created.

Given our current ontology, that means adding some 140 extra indexes,
as each one requires at least an extra page [2], that already means
~0.5MB extra on a da

[Tracker] Fwd: tracker 1.10.3

[Carlos Garnacho]

-- Forwarded message --
From: Carlos Garnacho <install-mod...@master.gnome.org>
Date: Fri, Dec 16, 2016 at 1:53 PM
Subject: tracker 1.10.3
To: FTP Releases <ftp-release-l...@gnome.org>


About Tracker
=

Tracker is a semantic data storage for desktop and mobile devices.
Tracker uses W3C standards for RDF ontologies using Nepomuk with
SPARQL to query and update the data.

Tracker is a central repository of user information, that provides two
big benefits for the user; shared data between applications and
information which is relational to other information (for example:
mixing contacts with files, locations, activities and etc.).

News


  * tracker-extract: Whitelisted further syscalls in the sandbox. False
positives were being triggered in i686 platforms, plus other syscalls
that have been missed in 1.10.2.


ChangeLog
=
https://download.gnome.org/sources/tracker/1.10/tracker-1.10.3.changes  (867)

Download

https://download.gnome.org/sources/tracker/1.10/tracker-1.10.3.tar.xz (4.80M)
  sha256sum: 6bbdad0a1b1346a364f0c371ceca2ef32827e7ad0e266e79ca0ebe5e9b1d900d
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

[Tracker] Fwd: tracker 1.8.2

[Carlos Garnacho]

Another round of stable releases... it seems we sandboxed too tight
for some specific arches/data, so now tracker-extract "crashes"
repeatedly for some. These releases should fix it.

Cheers,
  Carlos


-- Forwarded message ------
From: Carlos Garnacho <install-mod...@master.gnome.org>
Date: Fri, Dec 16, 2016 at 1:53 PM
Subject: tracker 1.8.2
To: FTP Releases <ftp-release-l...@gnome.org>


About Tracker
=

Tracker is a semantic data storage for desktop and mobile devices.
Tracker uses W3C standards for RDF ontologies using Nepomuk with
SPARQL to query and update the data.

Tracker is a central repository of user information, that provides two
big benefits for the user; shared data between applications and
information which is relational to other information (for example:
mixing contacts with files, locations, activities and etc.).

News


  * tracker-extract: Whitelisted further syscalls in the sandbox. False
positives were being triggered in i686 platforms, plus other syscalls
that have been missed in 1.8.1.


ChangeLog
=
https://download.gnome.org/sources/tracker/1.8/tracker-1.8.2.changes  (866)

Download

https://download.gnome.org/sources/tracker/1.8/tracker-1.8.2.tar.xz (4.76M)
  sha256sum: b93efbf077901315433a26b822ed6fb47a2592ee9d0fc7e054fe39a9968fa451
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

Re: [Tracker] Fwd: tracker 1.11.2

[Carlos Garnacho]

Hey Philip,

On Sat, Dec 10, 2016 at 4:57 PM, Philip Van Hoof <phi...@codeminded.be> wrote:
> On Sat, 2016-12-10 at 16:36 +0100, Carlos Garnacho wrote:
>
>
> [cut]
>
>> I however think going from "odd minor number is unstable" to semver
>> versioning with minor!=0 is going to be confusing...
>
> Yes, changing it in the middle of a major release number is perhaps not
> the brightest idea. True.
>
>>  I will suggest the following plan:
>>
>> - Finish this 1.11.x/1.12 cycle
>> - Sticking to 1.12.x for as long it's needed while
>> - Gearing to a Tracker 2.x that switches to using semver approach to
>> communicate API/ABI changes.
>
> This makes a lot of sense to me.

Cool :), seeing that you/Sam agree, sounds like a plan.

>
>> If communicated properly, that'd at least allow us to have a single
>> last/current stable release to care about most often, and the window
>> to accumulate backwards compatible changes can be made variable based
>> on urgency (situations like this don't come up often but you never
>> know...),
>
> Well, yes. Usually are security bugfixes just patch increments. But in
> this case you solved the situation by adding sandboxing as a feature.
>
> And then semver states that 'you added or changed functionality and or
> APIs but didn't break backwards API compatibility' = minor increment.

Right, this would indeed be a reason for breaking the schedule and
releasing, or data loss situations, etc. I however hope those don't
appear often :).

>
>> I wouldn't mind that this unstable window is still made to
>> roughly match the 6 months gnome schedule though, and maybe release
>> 2.[x+1] with whatever might accumulate in that period.
>
> Sure. I would however not withhold from doing interim releases, and
> assign one of those to the 6 month gnome release.
>
> Not being a monorepo or monolithic architecture liker, I also don't
> think that having a cadans dictated by something like gnome is
> necessarily a good idea. But that clearly is just an opinion.
>
> I think every project should be independent.

It is mostly down to convenience, I eg. remember when gtk+ had these 9
to 12 month cycles (in the 2.x days) which was quite messy for
applications, some new features would go virtually unused until the
next gnome cycle, and close enough to gtk+ freezes when they actually
were (and bugs were reported). This incoordination ended up being
unpleasant on both sides.

I get your point, Tracker is not a project used only by gnome, but
it's the community it's always worked closest with, and it's also a
community who's "invested" a lot in it (as in making Tracker a pillar
for several core apps). Its 6 months schedule might not be fit for
everyone, but it's IMHO reliable and short enough.

>
>> How does that sound to you?
>
> But yes, makes sense.
>
>> I'll also take the opportunity to introduce to the ML the "roadmap"
>> that's been shaping up in my head for 2.x:
>>
>> - Getting as close to supporting the full sparql 1.1 spec as possible
>> in libtracker-data:
>>   * property paths: last weekend got halfway with this \o/
>>   * graph management: for DROP GRAPH I think triggers will perform
>
> Did ever something happen to cleaning up anonymous nodes of deleted
> subjects/context, and or do reference counting on them (and clear them
> once they reach zero references)?
>
> If not then we are still leaking those in the db afaik. We always wanted
> to do something about that.

Yeah, we still do leak those... I remember you/Jürg suggested setting
a foreign key with ON DELETE RESTRICT action from the various tables'
ID rows to the Resource table, so the cleaning up the Resource table
would fail for the still referenced nodes. I got as far as seeing
that:

- Performance would be just fine for the common ops, the trigger would
only run when trying to delete the parent key in the Resource table,
which is the once-in-a-while operation, modifications on the cols
setting the foreign key would be just as fast as they're now.

- It however wouldn't fix alone the other issue I saw happening before
the revert (graph URNs being deleted). I had a patch around that added
a Graph table, so IDs in the Resource table were ensured to be in
either rdfs:Resource or the Graph table. That already helped with
identifying and not deleting the graph URNs during garbage collection,
and seems useful for graph management, but I think I can't just add
the same RESTRICT action as CLEAR/DROP GRAPH will want pretty much the
opposite.

- I also wondered if it's more desirable, or allowed by the sparql
spec, that we actually garbage collect the inconsistent nodes. IMO not
leaving this type of data coherence up to miners/apps being educated
when deletin

Re: [Tracker] Fwd: tracker 1.11.2

[Carlos Garnacho]

Hi Philip!,

On Fri, Dec 9, 2016 at 11:46 PM, Philip Van Hoof <phi...@codeminded.be> wrote:
> On Fri, 2016-12-09 at 00:44 +0100, Carlos Garnacho wrote:
>
> Hello Carlos,
>
> NO criticism on the content of the releases and the technical
> improvements, nor on the communication on the security bug and panic of
> certain people.

:)

>
> Just some remarks on versioning here.
>
>>   * tracker-extract: Sandbox extractor threads. Filesystem and network
>> access are limited to being read and local only.
>
> As a semver fan myself, I wonder why you didn't start 1.12.0 instead of
> calling this update 1.11.2?

Although following its own version numbers, Tracker has been following
lately the gnome schedule. I came to think lately (most nominally,
when rolling this last bunch of tarballs), that the gnome schedule is
indeed too fast paced for Tracker, while we've most often respected
backwards compatibility quite thoroughly. eg. there's distros shipping
1.8 because that's what came out with gnome 3.20, while 1.10 is a
qualitatively better drop-in replacement.

I however think going from "odd minor number is unstable" to semver
versioning with minor!=0 is going to be confusing... I will suggest
the following plan:

- Finish this 1.11.x/1.12 cycle
- Sticking to 1.12.x for as long it's needed while
- Gearing to a Tracker 2.x that switches to using semver approach to
communicate API/ABI changes.

If communicated properly, that'd at least allow us to have a single
last/current stable release to care about most often, and the window
to accumulate backwards compatible changes can be made variable based
on urgency (situations like this don't come up often but you never
know...), I wouldn't mind that this unstable window is still made to
roughly match the 6 months gnome schedule though, and maybe release
2.[x+1] with whatever might accumulate in that period.

How does that sound to you?

I'll also take the opportunity to introduce to the ML the "roadmap"
that's been shaping up in my head for 2.x:

- Getting as close to supporting the full sparql 1.1 spec as possible
in libtracker-data:
  * property paths: last weekend got halfway with this \o/
  * graph management: for DROP GRAPH I think triggers will perform
just fine, CREATE is also easy, for LOAD/MOVE/ADD it looks like we can
unroll into specific updates.
  * the VALUES clause
  * the MINUS filter
  * CONSTRUCT/ASK/DESCRIBE
  * Removing or limiting the extensions we've gained on the way and
are addressed in 1.1 (eg. accepting "AS var" as good, property paths
greatly reduce the need for our property functions,... other
extensions like FTS must stay of course)

- Double checking ontology migration code, ensure it can handle weird
ontology changes more or less elegantly.

- Library-fying tracker-store, and separating ontology for good, so
eg. an irc client wanting to store conversation logs privately can eg.
do:

connection_manager = tracker_open (".../.cache/my-app/private-store",
"/usr/share/ontologies/nepomuk/nmo.ontology", cancellable, );

And it gets a local database with just the relevant classes from the
specified ontology. This might need a hardcoded basis though,
xsd/rdf/nrl/dc/tracker at least. As long as that is properly
documented I'm fine with it.

- And of course, keep dbus-based implementations around, I guess we
can't move too far from nepomuk there, as it's already the implicit
contract between miners and all the surrounding ecosystem.

However would be great to have the tracker-store executable be more
generic, so you can make it claim a different dbus name, write to a
different location, construct the database using a different
ontology...

Does this sound sensible? matches reasonably with your own "Tracker as
generic SPARQL endpoint" ideas?

>
> Sandboxing sounds to me like a new feature. Not just a bugfix.
>
> As Tracker gets used in ever increasing use-cases, ie. not only
> desktops, I think our users want a clear version numbering policy. The
> semver rules offer just that, and are compatible with API and ABI
> changes and packaging formats like Debian and RedHat's.
>
>>   * tracker-miner-fs: Fixed high CPU use when receiving many writeback
>> notifications at once.
>>   * tracker-extract, libtracker-sparql, libtracker-miner: plug leaks
>>   * tests: cleanups and improvements
>
> Those two are worthy of a 1.11.2, yes. So I would have released a 1.12.0
> containing the sandboxing, and a 1.11.2 with these two.
>
>> Translations: hu
>
> 1.11.2
>
>
> Now, I understand that 1.10.1 was in need of a upgrade too, and
> incrementing its version number to 1.11.0 would have conflicted with
> existing releases under 1.11.x.
>
> What to do in that case, in my opinion, is to release a 1.12.0 coming
> from 1.10.1 (instead of 1.10.2). And a 1.13.0 com

[Tracker] Fwd: tracker 1.11.2

[Carlos Garnacho]

-- Forwarded message --
From: Carlos Garnacho <install-mod...@master.gnome.org>
Date: Thu, Dec 8, 2016 at 7:43 PM
Subject: tracker 1.11.2
To: FTP Releases <ftp-release-l...@gnome.org>


About Tracker
=

Tracker is a semantic data storage for desktop and mobile devices.
Tracker uses W3C standards for RDF ontologies using Nepomuk with
SPARQL to query and update the data.

Tracker is a central repository of user information, that provides two
big benefits for the user; shared data between applications and
information which is relational to other information (for example:
mixing contacts with files, locations, activities and etc.).

News


  * tracker-extract: Sandbox extractor threads. Filesystem and network
access are limited to being read and local only.
  * tracker-miner-fs: Fixed high CPU use when receiving many writeback
notifications at once.
  * tracker-extract, libtracker-sparql, libtracker-miner: plug leaks
  * tests: cleanups and improvements

Translations: hu


ChangeLog
=
https://download.gnome.org/sources/tracker/1.11/tracker-1.11.2.changes  (5.21K)

Download

https://download.gnome.org/sources/tracker/1.11/tracker-1.11.2.tar.xz (4.82M)
  sha256sum: 6df39329c6381cf11f58c600e4732baf6f4a181ff7ad1adf2cca738e740e9303
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

[Tracker] Fwd: tracker 1.10.2

[Carlos Garnacho]

-- Forwarded message --
From: Carlos Garnacho <install-mod...@master.gnome.org>
Date: Thu, Dec 8, 2016 at 7:43 PM
Subject: tracker 1.10.2
To: FTP Releases <ftp-release-l...@gnome.org>


About Tracker
=

Tracker is a semantic data storage for desktop and mobile devices.
Tracker uses W3C standards for RDF ontologies using Nepomuk with
SPARQL to query and update the data.

Tracker is a central repository of user information, that provides two
big benefits for the user; shared data between applications and
information which is relational to other information (for example:
mixing contacts with files, locations, activities and etc.).

News


  * tracker-extract: Sandbox extractor threads. Filesystem and network
access are limited to being read and local only.
  * libtracker-sparql: Fix compile on C++ compilers
  * tracker-extract: Use CUE info as a last resort on FLACs
  * tracker-extract: Minor improvements on albumartist extraction
  * libtracker-data: Handle overflows on libicu-based normalization

Translations: nb, zh_CN


ChangeLog
=
https://download.gnome.org/sources/tracker/1.10/tracker-1.10.2.changes  (4.42K)

Download

https://download.gnome.org/sources/tracker/1.10/tracker-1.10.2.tar.xz (4.80M)
  sha256sum: 6f4dace58429bb7155f1d89d6a159a4f4d3356f28192ade283f69eb81157a124
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

[Tracker] Fwd: tracker 1.8.1

[Carlos Garnacho]

-- Forwarded message --
From: Carlos Garnacho <install-mod...@master.gnome.org>
Date: Thu, Dec 8, 2016 at 7:44 PM
Subject: tracker 1.8.1
To: FTP Releases <ftp-release-l...@gnome.org>


About Tracker
=

Tracker is a semantic data storage for desktop and mobile devices.
Tracker uses W3C standards for RDF ontologies using Nepomuk with
SPARQL to query and update the data.

Tracker is a central repository of user information, that provides two
big benefits for the user; shared data between applications and
information which is relational to other information (for example:
mixing contacts with files, locations, activities and etc.).

News


  * tracker-extract: Sandbox extractor threads. Filesystem and network
access are limited to being read and local only.


ChangeLog
=
https://download.gnome.org/sources/tracker/1.8/tracker-1.8.1.changes  (5.46K)

Download

https://download.gnome.org/sources/tracker/1.8/tracker-1.8.1.tar.xz (4.75M)
  sha256sum: e3ed4cb384486ebc086adfad68b5d25f8b0424eb6eb1aca2252a508b757fbe51
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

Re: [Tracker] Tracker as a security risks

[Carlos Garnacho]

Hi,

On Mon, Dec 5, 2016 at 3:01 PM, Hanno Böck  wrote:
> On Mon, 5 Dec 2016 13:44:40 +
> Sam Thursfield  wrote:
>
>> The design of Tracker takes the risks into account. Metadata
>> extraction is isolated in its own process (tracker-extract) which can
>> crash without (theoretically) causing any harm.
>
> I don't see how that helps against security vulnerabilities.
>
> Having an isolated process probably helps in a way that a crash won't
> cause the whole tracker service to malfunction. Thus parsing broken
> files won't cause a service disruption. But as long as this process
> runs with normal user rights this doesn't protect in a security sense.
>
>> > I think there needs to be a wider discussion about this and the
>> > fundamental design choices done here need to be questioned.
>>
>> What questions do you have in particular?
>
> Quite frankly, I don't claim to have all the answers here, that's why I
> formulated it in an open "needs discussion" way.
>
> I think sandboxing the tracker parser (which you already indicated
> in your mail) is probably the most reasonable way to go forward. This
> isn't exactly my area of expertise, so I can't comment on which
> technique here is most promising.

It indeed sounds possible to lift extraction into a separate process
with limited access to the filesystem, we essentially need to pass an
fd to mmap() and an output one to receive sparql back. There's just
two things to consider:

- The extraction process sometimes needs access to misc files (eg. CUE
files, XMP sidecar files, ...), those might be passed along too, but
then we need detecting those cases beforehand.

- Ideally we wouldn't spawn one process per file being extracted,
although if we go to defcon 1 level of paranoia, that's probably what
should happen.

Anyway, this goes IMHO too much on the technical side for this ML, we
already have https://bugzilla.gnome.org/show_bug.cgi?id=764786 filed
to Tracker, and it's already high in my list for fixing on 1.12, feel
free to join there.

And I should add... Tracker is not alone here, if it's not Tracker
stumbling on infected content, with varying but still rather low
levels of interaction it may be a thumbnailer, a previewer like sushi,
or the web browser itself streaming content which hit this. So there's
more places in need of further isolation when dealing with untrusted
content.

And still, the chain is only as strong as its weakest link, as soon as
there is anything opening that file with wide enough permissions to
cause any harm, you're essentially screwed. This might sound like an
argument to running every app through flatpak, although I think the
long term answer always is "fix the vulnerability!".

>
>
> The other issue I think is that the quality of huge parts of the foss
> ecosystem needs to be improved. The good news here is that we got some
> powerful tools in terms of fuzzing (afl, libfuzzer) and memory safety
> bug detection (asan) in the past years. Ideally all free software devs
> should be aware of those tools and use them in their development
> process. I'm trying to help here where I can, see e.g. also my recent
> post on this list [1]. If our libraries would be better tested we could
> be more comfortable feeding it with untrusted inputs.

I agree some more active prevention would be positive, sounds like
something to tackle in the libraries dealing with file formats though,
Tracker is a strawman here, in the sense that filesystem extraction
it's only exploitable through its tracker-extract's modules, and those
are for the most part implemented using external libraries.

Cheers,
  Carlos
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

[Tracker] Fwd: tracker 1.11.1

[Carlos Garnacho]

Codenamed "I hate mondays"


-- Forwarded message ------
From: Carlos Garnacho <install-mod...@master.gnome.org>
Date: Mon, Nov 21, 2016 at 4:28 PM
Subject: tracker 1.11.1
To: FTP Releases <ftp-release-l...@gnome.org>


About Tracker
=

Tracker is a semantic data storage for desktop and mobile devices.
Tracker uses W3C standards for RDF ontologies using Nepomuk with
SPARQL to query and update the data.

Tracker is a central repository of user information, that provides two
big benefits for the user; shared data between applications and
information which is relational to other information (for example:
mixing contacts with files, locations, activities and etc.).

News


  Brown paper bag release, revert BIND() fix as it breaks other legit
  cases.


ChangeLog
=
https://download.gnome.org/sources/tracker/1.11/tracker-1.11.1.changes  (341)

Download

https://download.gnome.org/sources/tracker/1.11/tracker-1.11.1.tar.xz (4.81M)
  sha256sum: 112af88103153d383cf12cbec208db6b8c2d8e99086f78ee8782dbb967aac8ea
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

[Tracker] Fwd: tracker 1.11.0

[Carlos Garnacho]

It seems ftpadmin didn't pick the news this time, just for completeness:

NEW in 1.11.0 - 2016-11-21
==

  * libtracker-sparql: Added TrackerNotifier, helper object to receive
notifications of changes to the Tracker database. All users of the
GraphUpdated DBus signal are recommended to switch to it.
  * libtracker-sparql: Added client-side support for HTTP SPARQL
endpoints.
  * libtracker-direct: Much reduced mutex contention during threaded/async
queries on the direct access backend.
  * libtracker-sparql: Using BIND() after OPTIONAL{} now works properly
  * tracker-extract: Many improvements to music extraction, better labeling
of albums, nmm:albumArtist metadata is more faithful to the file
metadata.
  * libtracker-data: Fixed possible overflows in tracker:normalize/unaccent
  * Other fixes and cleanups.

Translations: ca, cs, de, es, eu, hu, lt, fur, pl, sv, zh_CN


-- Forwarded message --
From: Carlos Garnacho <install-mod...@master.gnome.org>
Date: Mon, Nov 21, 2016 at 2:06 PM
Subject: tracker 1.11.0
To: FTP Releases <ftp-release-l...@gnome.org>


About Tracker
=

Tracker is a semantic data storage for desktop and mobile devices.
Tracker uses W3C standards for RDF ontologies using Nepomuk with
SPARQL to query and update the data.

Tracker is a central repository of user information, that provides two
big benefits for the user; shared data between applications and
information which is relational to other information (for example:
mixing contacts with files, locations, activities and etc.).

ChangeLog
=

2016-11-21  Carlos Garnacho  <carl...@gnome.org>

Release 1.11.0

libtracker-data: Wrap BIND argument with () in SQL
Constructing the query as "SELECT $bind * FROM (...)" may trigger
sqlite parser overflows if the BIND form turns out to be too complex,
Doing the query as "SELECT ($bind) * FROM (...)" seems to be more
friendly to the parser.

This error was seen in gnome-music search, where the BIND() argument
is something like:

BIND((IF(STRSTARTS(?title_lower, "the "), SUBSTR(?title_lower, 5),
 IF(STRSTARTS(?title_lower, "a "), SUBSTR(?title_lower, 3),
IF(STRSTARTS(?title_lower, "an "), SUBSTR(?title_lower, 4),
   fn:replace(fn:lower-case(?title_lower),
  "^[
!\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~]+|[
!\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~]+$", "")
  )
   )
)
 ) AS ?title_collation)

Which, despite the fn:replace regex argument being passed as a host
parameter, seemed too much to sqlite.

libtracker-sparql: Distcheck fix

2016-11-21  Marek Cernocky  <marek_cerno...@conel.cz>

Updated Czech translation

2016-11-20  Carlos Garnacho  <carl...@gnome.org>

libtracker-sparql: Use internal headers
That's not guaranteed to be there yet at compile time, so it
randomly breaks build.

libtracker-miner: Use TrackerNotifier on TrackerDecorator
Remove the GraphUpdated handler in favor of TrackerNotifier,
it's been made to be a good fit here, and the porting results
in a nice code removal.

Given that the RDF types list is readwrite in TrackerDecorator,
but construct-only in TrackerNotifier, this means we need to
cater for changes in the RDF types list, just by creating another
TrackerNotifier and dropping the older one.

https://bugzilla.gnome.org/show_bug.cgi?id=773028

libtracker-sparql: Add TrackerNotifier to subscribe to change
notifications
This object abstracts GraphUpdated, and is now the recommended method
to receive notifications upon changes in Tracker data. After creation,
the caller may connect to the ::events signal. This signal packs all
events received in one go, so as to avoid signal emission overhead on
mass changes.

The TrackerNotifier behavior can be slightly modified through the
flags passed during instantiation. Eg. it can be requested to query
some data (urn, location) upfront, although the API is
tracker:id centric
otherwise.

https://bugzilla.gnome.org/show_bug.cgi?id=773028

tracker: Add remote connection support to "tracker sparql" CLI tool
The -r/--remote switch can be used to specify the base url to be used
in queries.

https://bugzilla.gnome.org/show_bug.cgi?id=773031

libtracker-remote: Add support for application/sparql-results+xml
As specified in https://www.w3.org/TR/rdf-sparql-XMLres/. This cursor
implementation is able to read the XML expected under that content
type.

  

[Tracker] Plans to split Tracker 1.12 (Was: Re: The Big Rip)

[Carlos Garnacho]

Hi Sam/all,

(CCing distributor-list as suggested by Sam)

On Sun, Oct 30, 2016 at 2:19 PM, Sam Thursfield <sss...@gmail.com> wrote:
> Hi!
>
> On 10/28/16, Carlos Garnacho <carl...@gnome.org> wrote:
>> It is well known that Tracker has been accumulating lots of code and
>> features over the years, with greatly varying states of maintenance.
>> There's been earlier talks in this ML about splitting the whole thing
>> into more palatable chunks, some refactoring happened towards making
>> this easier, but it was never accomplished.
>>
>> So I'd say now is just as a good time as any to finally do this, I've
>> pushed (so far) the following WIP branches I worked on the last couple
>> of evenings:
>>
>> - wip/split/core : contains Tracker "core", most notably
>> tracker-store, ontologies, libtracker-sparql, libtracker-miner,
>> libtracker-control and the tracker CLI tool.
>> - wip/split/miner-fs : contains everything around FS miner,
>> tracker-miner-fs, tracker-extract and tracker-writeback.
>> wip/split/rss: contains tracker-miner-rss only.
>>
>> The three branches pass distcheck and build stuff into separate
>> tarballs, so (with some rebasing) they're a suitable starting point
>> for standalone repos.
>
> I think we need to be careful about doing this in a way that's useful
> for everyone, and only dong it if there's a good reason. So it would
> be great if some Tracker users & packagers could comment here.

Indeed, I started this thread hoping to get a handful of "please
do/don't"s. There's a lot going on here and it's probably going to
cause some frustration somewhere, I hope for the best in the long
term.

>
> It might be worth cc'ing the distributor-list to get wider feedback.
>
> What are the use cases of the separated tracker-core and
> tracker-miner-fs repos? I mean, in what situations will someone want
> to use one without the other?

It's true that the most prominently used info from tracker is that
coming from tracker-miner-fs and tracker-extract. Although IMHO that's
far from being the sole purpose, currently Tracker is at its very core
a Nepomuk database.

IMO there's mainly one argument: modularity. Right now Tracker kind of
tries to be an ecosystem by itself, but also attempts not to impose
everything everywhere, so there's this maze of configure script
options which on one hand
adds a combinatorial explosion of code paths, and on the other hand
allows each distributor/packager the choice to pick their own
combination (eg. debian already splits into several packages IIRC, in
others having "tracker" installed is no guarantee that you'll have
/usr/libexec/tracker-miner-rss, while for others is, same for
tracker-preferences/needle).

This way, we avoid --disable-* toggles turning off entire
daemons/modules, provide a clear guideline about how tracker is
structured, and defer the "pick your own pieces" part to either/both
the user or the depends/recommends rules in packages depending on data
from specific miners.

>
>> Now, the bad news, things are broken, unmaintained or IMHO should be
>> deleted:
>>
>> - tracker-preferences: I'm a bit opinionated on this one,
>> tracker-preferences is a mixed bag of store and miner-fs
>> configuration, and having a GUI to configure a daemon is too 2000's
>> (not just the UI itself...). I think some of its functionality could
>> be moved to the tracker CLI tool, or just be left to DE integration
>> (like gnome search preferences).
>
> It should really be left to desktop environment configuration, indeed...
>
> The existing code is no doubt a good starting point for a
> gnome-control-centre panel. Perhaps we should have some kind of
> tracker-example-apps repo for the existing preferences code?

gnome-control-center's search panel has some minimal tracker
configuration already :) (clicking the gear button) , the only thing
I'm perhaps missing there is configuring the indexing of external
drives, the rest of the stuff in tracker-preferences is bells and
whistles to me.

TBH, I don't see this bunch of code gluing gtkbuilder and gsettings
together that much worthy to preserve...

>
> It's basically all miner-fs config, I don't see any store
> configuration options there (which seems correct, there's nothing
> useful for end-users to configure about the store).

AFAIR it's just a couple low-level full-text search toggles, I'm
perfectly fine with pointing people to dconf-editor for these tbh...

>
>> - tracker-needle: This is barely maintained, IMHO we're beyond the
>> times that we needed a standalone/demo GUI, other apps out there make
>> a better work at making Tracker look nice. That said, I know other
>> people saw this useful, so if anyo

[Tracker] The Big Rip

[Carlos Garnacho]

Hi all,

It is well known that Tracker has been accumulating lots of code and
features over the years, with greatly varying states of maintenance.
There's been earlier talks in this ML about splitting the whole thing
into more palatable chunks, some refactoring happened towards making
this easier, but it was never accomplished.

So I'd say now is just as a good time as any to finally do this, I've
pushed (so far) the following WIP branches I worked on the last couple
of evenings:

- wip/split/core : contains Tracker "core", most notably
tracker-store, ontologies, libtracker-sparql, libtracker-miner,
libtracker-control and the tracker CLI tool.
- wip/split/miner-fs : contains everything around FS miner,
tracker-miner-fs, tracker-extract and tracker-writeback.
wip/split/rss: contains tracker-miner-rss only.

The three branches pass distcheck and build stuff into separate
tarballs, so (with some rebasing) they're a suitable starting point
for standalone repos.

Now, the bad news, things are broken, unmaintained or IMHO should be deleted:

- tracker-preferences: I'm a bit opinionated on this one,
tracker-preferences is a mixed bag of store and miner-fs
configuration, and having a GUI to configure a daemon is too 2000's
(not just the UI itself...). I think some of its functionality could
be moved to the tracker CLI tool, or just be left to DE integration
(like gnome search preferences).

- tracker-needle: This is barely maintained, IMHO we're beyond the
times that we needed a standalone/demo GUI, other apps out there make
a better work at making Tracker look nice. That said, I know other
people saw this useful, so if anyone is willing to maintain it, I'd
gladly set up other branch/repo for it.

- tracker-miner-apps: AFAIK this is unused since the maemo times, I'd
say to send it to the chopper, unless anyone wants to resuscitate it.

- tracker-miner-user-guides: ditto.

- evolution plugin/miner: it's been broken for ages. Again, if anyone
cares enough to pick this up and fix it...

- firefox and thunderbird plugins: They are basically unmaintained,
and AFAIR thunderbird was even blacklisted due to stability issues.

- nautilus tagging extension: Another one I have a hard time caring...
If this feature is as desirable, should be implemented in nautilus
itself (and most likely with better looking and more integrated
results). I'll probably split it, make a release, and never revisit
again. As above, maintainers welcome.

If anyone feels like adopting one of these items, please reply or come
over to #tracker. Speak soon or be ready to rescue a bunch of code
from git history :).

So the split seemed quite painless (the cuts were mostly clear thanks
to earlier refactors), although I'm sure that some closer analysis
will reveal more stuff to remove on each repo (esp. libtracker-common
seems like a good candidate).

The only oddity I see is that the tracker CLI tool (residing in the
core) pokes virtually everywhere, it would be nicer to make the
"tracker" command a shallow interface to subcommands, that are
installed by either tracker core, miner-fs, or whatever.

In the future, we would want the ontology out of Tracker core, but
probably at the time we can claim tracker-store is a generic SPARQL
endpoint :)

Comments? Maintainers? I won't rush this much, but would certainly
want to tackle this before the branches become too hard to
rebase/merge.

Cheers,
  Carlos
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

[Tracker] Master open for 1.11.x. Request for reviews

[Carlos Garnacho]

Hey all,

Just the mandatory heads up that master is open again for development.
I'll use the opportunity to request review on 2 new features:

- Add libtracker-notify to wrap GraphUpdated.
  https://bugzilla.gnome.org/show_bug.cgi?id=773028
- Add client-side support for HTTP sparql endpoints
  https://bugzilla.gnome.org/show_bug.cgi?id=773031

The latter is quite accessory ATM, I did it as a playground to
Wikidata query service, which doesn't seem ready for prime time yet,
but the patches make it functional through tracker, which is cool.

I consider the former feature more important though, this is something
we've usually deferred to clients, but that plus bad documentation [1]
has lead to implementations not getting all details right (it's hard
to start with... see for example [2]). This IMO brings bad perception
on Tracker even if it's not necessarily at fault.

My hope is that all the scattered GraphUpdated callback
implementations will switch to using TrackerNotifier over time, so
bugs are fixed in a centralized place and tracker-side performance
issues can be discarded/fixed with sysprof/perf/etc.

Cheers,
  Carlos

[1] If we can consider
https://wiki.gnome.org/Projects/Tracker/Documentation/SignalsOnChanges
, and examples/libtracker-sparql/class-signal.c as such.
[2] https://bugzilla.gnome.org/show_bug.cgi?id=764419#c3 or
https://bugzilla.gnome.org/show_bug.cgi?id=757326#c3
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

[Tracker] Fwd: tracker 1.10.1

[Carlos Garnacho]

-- Forwarded message --
From: Carlos Garnacho <install-mod...@master.gnome.org>
Date: Fri, Oct 14, 2016 at 12:54 PM
Subject: tracker 1.10.1
To: FTP Releases <ftp-release-l...@gnome.org>


About Tracker
=

Tracker is a semantic data storage for desktop and mobile devices.
Tracker uses W3C standards for RDF ontologies using Nepomuk with
SPARQL to query and update the data.

Tracker is a central repository of user information, that provides two
big benefits for the user; shared data between applications and
information which is relational to other information (for example:
mixing contacts with files, locations, activities and etc.).

News


  * Tracker-extract:
- Fixed FD leak in flac extractor
- Fixes to tag parsing in flac extractor
- Memory leak fixes in libav extractor
  * Libtracker-sparl:
- Fixes to tracker:uri-is-descendant() error checks
- Fix namespace of Errors in libtracker-sparql API
  * Misc:
- Fixes on functional tests

Translations: ca, eu, fur, it


ChangeLog
=
https://download.gnome.org/sources/tracker/1.10/tracker-1.10.1.changes  (2.27K)

Download

https://download.gnome.org/sources/tracker/1.10/tracker-1.10.1.tar.xz (4.79M)
  sha256sum: 67ea78cca8ebbd6633dddcdd40b5205683cc886b872cde987e2a8bae171f4191
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

[Tracker] Fwd: tracker 1.10.0

[Carlos Garnacho]

-- Forwarded message --
From: Carlos Garnacho <install-mod...@master.gnome.org>
Date: Mon, Sep 19, 2016 at 5:10 PM
Subject: tracker 1.10.0
To: FTP Releases <ftp-release-l...@gnome.org>


About Tracker
=

Tracker is a semantic data storage for desktop and mobile devices.
Tracker uses W3C standards for RDF ontologies using Nepomuk with
SPARQL to query and update the data.

Tracker is a central repository of user information, that provides two
big benefits for the user; shared data between applications and
information which is relational to other information (for example:
mixing contacts with files, locations, activities and etc.).

News


Translations: da, el, en_GB


ChangeLog
=
https://download.gnome.org/sources/tracker/1.10/tracker-1.10.0.changes  (315)

Download

https://download.gnome.org/sources/tracker/1.10/tracker-1.10.0.tar.xz (4.76M)
  sha256sum: df95b4a1e7de442f66d1097b725dd3cdd739862f491453fc7d7b1f88181a12fb
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

Re: [Tracker] Fwd: tracker 1.9.2

[Carlos Garnacho]

Hi Michael :)

On Sat, Sep 17, 2016 at 11:31 AM, Michael Biebl <mbi...@gmail.com> wrote:
> Hi
>
> 2016-09-14 22:31 GMT+02:00 Carlos Garnacho <carl...@gnome.org>:
>> -- Forwarded message --
>> From: Carlos Garnacho <install-mod...@master.gnome.org>
>> Date: Wed, Sep 14, 2016 at 10:30 PM
>> Subject: tracker 1.9.2
>> To: FTP Releases <ftp-release-l...@gnome.org>
>>
>>
>> About Tracker
>> =
>>
>> Tracker is a semantic data storage for desktop and mobile devices.
>> Tracker uses W3C standards for RDF ontologies using Nepomuk with
>> SPARQL to query and update the data.
>>
>> Tracker is a central repository of user information, that provides two
>> big benefits for the user; shared data between applications and
>> information which is relational to other information (for example:
>> mixing contacts with files, locations, activities and etc.).
>>
>> News
>> 
>>
>>   * Restore trailing colon in nfo:Equipment URIs
>>   * Add new mime-types for comic books
>>
>
> I do not have functional tests enabled, yet "make install" installs
>
> /usr/share/tracker-tests/01-writeback.py
>
> Is that deliberate or a bug (and should I file a bug report)?

Sounds like the latter, shouldn't be ever needed in real life. Thanks
for noticing!

Cheers,
  Carlos
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

[Tracker] Fwd: tracker 1.9.2

[Carlos Garnacho]

-- Forwarded message --
From: Carlos Garnacho <install-mod...@master.gnome.org>
Date: Wed, Sep 14, 2016 at 10:30 PM
Subject: tracker 1.9.2
To: FTP Releases <ftp-release-l...@gnome.org>


About Tracker
=

Tracker is a semantic data storage for desktop and mobile devices.
Tracker uses W3C standards for RDF ontologies using Nepomuk with
SPARQL to query and update the data.

Tracker is a central repository of user information, that provides two
big benefits for the user; shared data between applications and
information which is relational to other information (for example:
mixing contacts with files, locations, activities and etc.).

News


  * Restore trailing colon in nfo:Equipment URIs
  * Add new mime-types for comic books

Translations: da, fur, fr, gl, kk, ko, lv, pl, pt_BR, sr, sr@latin, sv, vi


ChangeLog
=
https://download.gnome.org/sources/tracker/1.9/tracker-1.9.2.changes  (3.01K)

Download

https://download.gnome.org/sources/tracker/1.9/tracker-1.9.2.tar.xz (4.76M)
  sha256sum: 360214a495aef3a76ce1c2189c44fe164864d313082278c28e0ecc2246667a5a
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

[Tracker] Fwd: tracker 1.9.1

[Carlos Garnacho]

Hi,

A new release! despite the low version number, I consider tracker
feature frozen for 1.10. Apologies for the low rate of releases in
this cycle.

Cheers,
  Carlos

-- Forwarded message --
From: Carlos Garnacho <install-mod...@master.gnome.org>
Date: Tue, Aug 23, 2016 at 2:02 PM
Subject: tracker 1.9.1
To: FTP Releases <ftp-release-l...@gnome.org>


About Tracker
=

Tracker is a semantic data storage for desktop and mobile devices.
Tracker uses W3C standards for RDF ontologies using Nepomuk with
SPARQL to query and update the data.

Tracker is a central repository of user information, that provides two
big benefits for the user; shared data between applications and
information which is relational to other information (for example:
mixing contacts with files, locations, activities and etc.).

News


  * Tracker-resource: New API to describe RDF resources that can be serialized
  into SPARQL updates and various data formats. TrackerSparqlBuilder will
  be eventually phased out by this API.
  * Tracker-extract:
* Use tracker resource integrally.
* Fixed blacklisting of crashy files.
* Fixes in gstreamer module for 32-bit platforms
  * Libtracker-control:
* Expose "index for process" miner API
  * Command line tools:
* Add "tracker extract" subcommand
  * SPARQL:
* Accept INSERT DATA, DELETE DATA and DELETE WHERE syntax again.
  * Libtracker miner: Fix accounting in TrackerPriorityQueue when removing
   elements.

Translations: cs, de, es, fr, hu, id, lt, pl, pt, sk


ChangeLog
=
https://download.gnome.org/sources/tracker/1.9/tracker-1.9.1.changes  (11.0K)

Download

https://download.gnome.org/sources/tracker/1.9/tracker-1.9.1.tar.xz (4.74M)
  sha256sum: 70f38d2f71d1d14b68d60d4f0dc8fa1296687a3b9bc7ac951ba02605cb02883e
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

Re: [Tracker] Crawling-Interval 1

[Carlos Garnacho]

Hi!,

On Fri, Jun 24, 2016 at 2:04 PM, Sam Thursfield  wrote:
> Hi Edward
> The tracker-miner-fs process only does a re-crawl on startup. So
> although 'crawling-interval = 1' should cause it to recrawl every day,
> actually that'll only happen if you restart the process once a day
> too.
>
> That behaviour seems a bit confusing to me -- fixing tracker-miner-fs
> to honour crawling-interval even if it's not restarted would make
> sense; or at least improving the documentation of the
> crawling-interval option so that its limitations are clear).

You're right, I think too this is counterintuitive. Moreover with
servers that don't ever shutdown and laptops that most usually
suspend. It also sounds easy enough to implement, a
g_timeout_add_seconds() idle that calls
tracker_indexing_tree_notify_update() for each root.

Cheers,
  Carlos
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

[Tracker] Fwd: tracker 1.9.0

[Carlos Garnacho]

-- Forwarded message --
From: Carlos Garnacho <install-mod...@master.gnome.org>
Date: Tue, Jun 21, 2016 at 4:45 PM
Subject: tracker 1.9.0
To: FTP Releases <ftp-release-l...@gnome.org>


About Tracker
=

Tracker is a semantic data storage for desktop and mobile devices.
Tracker uses W3C standards for RDF ontologies using Nepomuk with
SPARQL to query and update the data.

Tracker is a central repository of user information, that provides two
big benefits for the user; shared data between applications and
information which is relational to other information (for example:
mixing contacts with files, locations, activities and etc.).

News


  * Adapt to new GKqueue monitor mapping.
  * Remove build time dependency on gnome-common
  * Fix error handling of tracker-extract-persistence
  * Fix tracker-miner-fs to honor all configuration options at runtime.
  * Stop recommending hard resets all through
* tracker reset -r will now warn and request the user to explicitly allow
  the operation.
* tracker-preferences won't show anymore a big "reset and restart" button.
  * Added "tracker reset -f $filename" subcommand. This will recursively reset
all indexed content for the given filename/uri, and trigger reindexing if
appropriate, so contents are just like freshly indexed.
  * Fixed possible crash in MP3 extractor
  * Favor embedded/external cue sheets before flac files' TOC.
  * store albumArtist from TPE2 tag in MP3 extractor
  * Avoid possible integer overflow in GIF extractor
  * Support regular expressions for fn:replace
  * Mark most internal functions as SQLITE_DETERMINISTIC
  * Logging changes in tracker-miner-fs, sparql errors no longer end up with
full insert queries being logged, but a loud warning with instructions to
get more info will be printed instead.
  * Fix FS size calculations on OpenBSD
  * Add MS Office "owner files" to ignored-files
  * Add systemd user services corresponding to D-Bus session services
  * Handle DjVu files
  * Fixes in handling of BIND()
  * miner-fs: Fix handling files moved soon after creating
  * Improved console output of tracker subcommands

Translations: de, es, oc, pt, pt_BR, sk


ChangeLog
=
https://download.gnome.org/sources/tracker/1.9/tracker-1.9.0.changes  (29.3K)

Download

https://download.gnome.org/sources/tracker/1.9/tracker-1.9.0.tar.xz (4.72M)
  sha256sum: cf2b239b4b9a21ac976ca492ee1182ebac4e1f5d8688a5da1f7d97a638d673be
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

Re: [Tracker] Extracting the extractors

[Carlos Garnacho]

Hey Sam :),

On Mon, May 9, 2016 at 1:40 AM, Sam Thursfield  wrote:
> I've made quite a bit of progress on this. The code is mostly there,
> but i've only done a light amount of testing so far so there will be a
> butt load of hidden regressions no doubt.
>
> I have a couple of open questions:
>
> - is it OK to remove API from libtracker-extract ? I seem to remember
> that we've already done that for the TrackerDecorator changes, and the
> developer docs at [1] haven't been updated at all since 0.16, but
> maybe I'm missing something.

Yes, by all means. libtracker-extract was made private before the
switch to 1.x. That was a barely/ever used extension point, now either
patches to tracker-extract or external TrackerDecorators are the way
to go.

>
> - is it ever necessary to have a DELETE statement for a corresponding
> INSERT statement in the output of an extract module?  I ask because I
> noticed the miner-fs removes *everything* about a file in the
> TRACKER_OWN_GRAPH_URN graph whenever a file changes. Which is the
> right thing to do, I think, but it does make me wonder why some
> extract modules then do their own DELETEs...

IIRC those are mostly audio extractors which are deleting+adding
artist info per-file, and perhaps other document formats for authors
(that's also mainly why we create deterministic urns for those). Or
are there more places doing that?

Anyhow, I have a kinda mixed opinion here. I'm sometimes of the
opinion that tracker-extract should use its own graph, so we could
tear down metadata without deleting filesystem structure information,
although then tracker-miner-fs would have to gain knowledge about this
separate graph. I still agree in that tracker-extract shouldn't be
dealing with cleaning up the metadata it produces.

>
> The code is in branch wip/sam/resource, and is mostly ready for review
> now, although i want to do quite a bit more testing to be sure that
> there are no performance or memory leakage regressions.

Thanks, I will try to have a look in the coming days.

Cheers,
  Carlos
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

Re: [Tracker] tracker-1.8 branched

[Carlos Garnacho]

Hey Philip :),

On Sat, Apr 2, 2016 at 9:22 PM, Philip Van Hoof  wrote:
> Hey Carlos,
>
> I noticed we are sometimes doing parallel releases.
>
> Have you ever considered using gitflow as branching technique?

Cheers for the tip, will check how it pans out. TBH I usually make do
on other branches than master/last stable, cherry-picking and
releasing mainly when poked by others, there's barely time for more
:).

Cheers,
  Carlos
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

Re: [Tracker] Extracting the extractors

[Carlos Garnacho]

Hey Sam :),

On Sat, Apr 9, 2016 at 1:39 AM, Sam Thursfield  wrote:
> Hi all
>
> I've always felt like Tracker's extractors should be reusable outside
> Tracker. The design makes that possible but right now they output their
> results as a series of slightly non-standard SPARQL update commands,
> which I don't think is useful for many folk. Lots of people aren't using
> SPARQL databases at all, believe it or not :-)

Just to provide some context, extractors used to generate pieces of
sparql that was passed to tracker-miner-fs, which composed the entire
sparql update(s) out of tracker-extract's and its own extracted
information. Nowadays, tracker-extract is a miner itself, and those
pieces of sparql are executed more or less as-is there.

Today, we can indeed do without this split, tracker extract modules
don't need to logically split the updates performed, those may be
accumulated/executed at once.

>
> The whole point of RDF is to make data interchange easy so I think we
> can do better than that. I've been looking at making the extractors
> optionally output their results in JSON-LD[1] format instead. The cool
> thing about JSON-LD is that if you squint, it's just good old JSON that
> everyone's familiar with. If you look closely it's also Linked Data,
> but in a more human-friendly serialization format than any of the more
> traditional RDF formats.

Cool :), will read up on the JSON-LD format, I see thought that it's
yet another json format compared to application/sparql-results+json as
described in https://www.w3.org/TR/2013/REC-sparql11-results-json-20130321/
. I guess it just caters for a different usecase, but these w3 guys
could sit and breathe before churning out a new format :P

>
> The catch here is that Tracker's extractor modules are all hardwired to
> generate SPARQL using TrackerSparqlBuilder. To be honest I've never
> liked this approach, it's pretty incomprehensible to newcomers and
> overly verbose, especially where we explicitly generate DELETE queries
> to go along with the INSERT queries.

Wholeheartedly agree. It only caters for a very specific subset of
insertion scenarios, not even sufficient for some real cases in
tracker extract modules, and barely useful at all outside
tracker(-extract). eg. things like
https://git.gnome.org/browse/tracker/tree/src/tracker-extract/tracker-extract-gstreamer.c#n1066

>
> so, inspired by something in the Python RDFLib library, I came up with a
> TrackerResource class that the extractors can use instead. This is a
> work in process, but I have a branch in git.gnome.org that adds
> TrackerResource, and converts some of the extractors to use it. The
> TrackerResource class can serialize either to SPARQL update commands or
> to JSON-LD. The branch also adds the `tracker extract` command from
>  so you can try out
> the extractors easily and specify `-o json` or `-o sparql` as you prefer.

Nice! Should it have a turtle serializer too? Do you think this can be
possibly used in the tracker store side to serialize contents?

I'm also wondering these days about exposing better backups/restores.
Tracker has been traditionally used to store data that could be easily
re-extracted, if a database is reset, worse that would happen usually
is that a few nao:Tags are lost. But stored data might be sensibly
harder, or impossible to restore [1].

I know we have org.freedesktop.Tracker1.Backup and the journal, but
some fields (most notably nie:plainTextContent) are taken away when
saving the journal files, so when restored it gives you a randomly
trimmed down database we can't easily recover further from. IIRC it
was done for backup size concerns, but kinda defeats its purpose :(.

So I was wondering whether it should be possible to serialize certain
contents (per class? per graph?) into some format we could restore
from, I first thought good old turtle, but could this be a nice
alternative? It could also be just enough if we make the backup dbus
call include the usually ignored properties, after all it's just doing
what the user asked for...

Sorry to drift off a bit :), I thought it's mildly related.


[1] See eg. https://git.gnome.org/browse/tracker-miner-chatlog/

>
> The results for extractors I've converted so far is promising in terms
> of reducing
> code size:
>
>  src/tracker-extract/tracker-extract-abw.c   |  51 ++--
>  src/tracker-extract/tracker-extract-bmp.c   |  18 +-
>  src/tracker-extract/tracker-extract-dvi.c   |  17 +-
>  src/tracker-extract/tracker-extract-epub.c  | 131 +++-
>  src/tracker-extract/tracker-extract-gstreamer.c | 910
> ++-
>  src/tracker-extract/tracker-extract-mp3.c   | 378
> ---
>  6 files changed, 511 insertions(+), 994 deletions(-)

Looks like a nice reduction :)

>
> Here's an example of auto-generated SPARQL for an MP3 extraction:


>
>
> 

[Tracker] tracker-1.8 branched

[Carlos Garnacho]

Hey all,

FYI, tracker-1.8 has just been branched. Master is open for new features.

Cheers,
  Carlos
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

[Tracker] Fwd: tracker 1.8.0

[Carlos Garnacho]

-- Forwarded message --
From: Carlos Garnacho <install-mod...@master.gnome.org>
Date: Mon, Mar 21, 2016 at 8:52 PM
Subject: tracker 1.8.0
To: FTP Releases <ftp-release-l...@gnome.org>


About Tracker
=

Tracker is a semantic data storage for desktop and mobile devices.
Tracker uses W3C standards for RDF ontologies using Nepomuk with
SPARQL to query and update the data.

Tracker is a central repository of user information, that provides two
big benefits for the user; shared data between applications and
information which is relational to other information (for example:
mixing contacts with files, locations, activities and etc.).

News


Changes since 1.7.5:
  * libtracker-miner: Adapt to libcue 2.0
  * translations: da

Overview of changes since 1.6.x:
  * Better support of the Sparql 1.1 syntax:
- Support for many builtin functions:
  * String: CONCAT, CONTAINS, LCASE, UCASE, STRLEN, SUBSTR,
STRSTARTS, STRENDS,
STRBEFORE, STRAFTER, ENCODE_FOR_URI.
  * Numeric: ABS, ROUND, CEIL, FLOOR, RAND.
  * Date/Time: YEAR, MONTH, DAY, HOUR, MINUTES, SECONDS, NOW.
  * Checksum: MD5, SHA1, SHA256, SHA512.
  * Support for Sparql 1.1's DELETE {} INSERT {} WHERE {} syntax. This
allows for
performing complex updates in a single operation, without the
shortcomings of
the tracker specific INSERT OR REPLACE {} syntax.
  * Support for Sparql 1.1's BIND(). This allows mapping complex
evaluations to a
variable within the triple pattern, so usable within it.
  * The latter two applied allowed tracker-miner-fs to halve the time spent in
handling move operations.
  * Many fixes to tracker-miner-fs and libtracker-miner, it is now a
lot more robust
esp. around cancellation situations (eg. deleting/unmounting the
folder currently
being indexed).
  * Many fixes to tracker-extract modules, many consistency fixes have
been fixed where
modules would produce invalid Sparql (eg. breaking cardinality
limits of certain
variables) if given files with unexpected metadata.
  * As a result of the last two points, 1.8.0 fixes most of the
warnings/criticals that
have been reported as "spamming journald".
  * Tracker now uses GTask, except for vala code.


ChangeLog
=
https://download.gnome.org/sources/tracker/1.8/tracker-1.8.0.changes  (405)

Download

https://download.gnome.org/sources/tracker/1.8/tracker-1.8.0.tar.xz (4.73M)
  sha256sum: a11f31a373bfec3abae38ae719d0a59f666f1f067d8789ade2ed7032a152907d
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

Re: [Tracker] How does tracker-miner-fs work?

[Carlos Garnacho]

Hi Lado,

On Sat, Mar 19, 2016 at 1:12 PM, Lado Kumsiashvili  wrote:
> Hello,
>
> I'm not sure if this is a bug or a feature or it depends on my setup.
>
> I have a folder (/home/user/development) with about 61G development
> files, several projects, libraries, eclipse, intelij etc.

Fist question, do you intend Tracker to index that folder? are its
search features useful to you in code? It seems that you actively
added this folder to be indexed by Tracker, so I'm mostly assuming
yes.

>
>
> The problem is, after each boot the tracker-miners-fs is at 100% and is
> mining and mining

I don't think it's mining. Tracker has a bootstrap process where it
has to recursively descend through folders, in order to:

1) Set up directory monitors
2) Check whether previously indexed contents in the folder where
updated (checking file mtimes)
3) Check the mtime of the folder itself, in order to detect content
deleted, and schedule no longer existing files for removal

Out of those operations, #1 is more expensive than you'd expect,
although obviously altogether this means the bootstrap process depends
on the amount of contents to be indexed. That's just the status quo in
userspace monitoring...

And #2/#3 are not as superfluous as you might think. Usually contents
don't change between boots, but directory monitors are a finite
resource, so it's the only safety net to have contents reindexed
properly in directories that could not be monitored in previous boots.
Given the size of your development folder, I'd say it's safe to assume
that directory monitor exhaustion is feasible to be happening in your
case.

I think the explanation in
https://bugzilla.gnome.org/show_bug.cgi?id=762194#c1 also applies here
somehow...

Basically, in order to cut down on io/cpu usage during startup, you
can only do the following:
- Reducing the amount of contents to be indexed.
- Reducing the frequency of this operation. There is a
/org/freedesktop/tracker/miner/fs/crawling-interval setting that is
intended to help here (not exposed in tracker-preferences, can be
changed with eg. dconf-editor), although I notice it's not being
currently honored, something that'll have to be fixed after 1.8.0...
When fixed, and combined with disabling the enable-monitors setting,
should lead to 0% io/cpu usage in startup until the crawling interval
(in days) is met, of course at the expense of outdated indexed
content.

>
> To test what goes on, I have killed tricker-miners-fs and started it
> with -v 3 option. I left the notebook over night on. In the morning the
> tracker-miner-fs was idle, as it managed to work through the complete
> folder. I have killed the process ans started it again with -v 3 And it
> begun again to index the entire folder from the beginning. But why,
> nothing has been changed there? Another point is the settings option
> "Index content in the background: Only when computer is not being used"
> But it does not work as expected, at least I do not expect to see any
> tracker process (and not at all at 100%), when I work with my IDE or do
> some other things. So the question, what does this option mean? Why it
> does not work as expected?

This boils down to the sched_setscheduler(2) syscall, with SCHED_IDLE
policy. I'd be surprised if the kernel didn't give up those cycles if
that's really necessary.

>
>
> Right now
>
> $ tracker daemon
> Store:
> 19 Mär 2016, 13:07:49:  ✓ Store   - Idle
>
> Miners:
> 19 Mär 2016, 13:07:49:1%  File System(PAUSED) - Crawling
> recursively directory 'file:///home/lado/development'
> 19 Mär 2016, 13:07:49:  ✓ Applications- Idle
> 19 Mär 2016, 13:07:49:  ✗ RSS/ATOM Feeds  - Not running or
> is a disabled plugin
> 19 Mär 2016, 13:07:49:  ✓ Userguides  - Idle
> 19 Mär 2016, 13:07:49:  ✓ Extractor   - Idle
> 19 Mär 2016, 13:07:49:  ✗ Media   - Not running or
> is a disabled plugin
>
> but the top shows 100% of CPU for tracker-miner-fs
>
>
>
>
> I have killed right now all the tracker processes and started them with
>
> tracked daemon -s
>
>
> tracker daemon -f shows
>
> 19 Mär 2016, 13:09:55:1%  File System - Crawling single
> directory 'file:///home/lado'
> 19 Mär 2016, 13:09:57:1%  File System - Crawling single
> directory 'file:///home/lado/Desktop'
> 19 Mär 2016, 13:09:57:1%  File System - Crawling
> recursively directory 'file:///home/lado/development'
>
>
> and tracker-miner-fs is at 100%m crawling in the huge amount if files in
> the development folder.
>
>
>
>  My Home folder is mounted with
> /dev/sda5 on /home type ext4 (rw,noatime,nodiratime,data=ordered)
>
>
> My Version is 1.7.4 with gnome 3.18 under gentoo

Did you modify the ignore-directories-with-content setting previously?
In 1.7.4 I added ".git" there so it'd ignore git repos by default, but
of course dconf will respect any previous user change. It should

Re: [Tracker] Tracker presentation at NLGG

[Carlos Garnacho]

Hey Philip,

Darn, I noticed this too late :), Utrecht is at a feasible distance
from here, the weekend turned out busy enough anyway... Hope the talk
went well :), thanks for doing it.

On Sat, Mar 19, 2016 at 9:14 PM, Philip Van Hoof  wrote:
> Hello,
>
> The main feedback from the people at The Netherlands user group was that
> tracker-needle looks more like a demo-ui than a fully usable user
> interface.

That is a more than fair opinion, IMHO if we want to show compelling
frontends to Tracker, we should be looking into
gnome-music/documents/news/...

We actually are still halfway into doing the split of Tracker
subcomponents, I'd say tracker-needle looks like a candidate for early
splitting, I'd eventually love to remove the gtk dep from at least the
core of tracker (store, libtracker-sparql, libtracker-miner). It's
tracker-preferences which fits oddly here, having it standalone is
nonsense, I guess it could go together with tracker-miner-fs, at least
the settings presented affect there the most, not that I'm fond of it
as an UI, either...

Probably doing some of this splitting would be a nice goal for 1.10 :)

Cheers,
  Carlos
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

[Tracker] Fwd: tracker 1.7.5

[Carlos Garnacho]

-- Forwarded message --
From: Carlos Garnacho <install-mod...@master.gnome.org>
Date: Tue, Mar 15, 2016 at 12:05 AM
Subject: tracker 1.7.5
To: FTP Releases <ftp-release-l...@gnome.org>


About Tracker
=

Tracker is a semantic data storage for desktop and mobile devices.
Tracker uses W3C standards for RDF ontologies using Nepomuk with
SPARQL to query and update the data.

Tracker is a central repository of user information, that provides two
big benefits for the user; shared data between applications and
information which is relational to other information (for example:
mixing contacts with files, locations, activities and etc.).

News


  * Add back .trackerignore match to ignored-directories-with-content
  * tracker-extract: Fix gstreamer module cuesheet handling
  * libtracker-miner: Avoid querying file type in crawling queries
  * libtracker-data: Handle inserts where the subproperty cardinality
is larger than the parents'
  * tracker-extract: Protect all single-valued properties in abiword extractor
  * tracker-extract: Protect all single-valued properties in EPUB extractor
  * tracker-extract: Protect all single-valued properties in ooxml extractor
  * tracker-extract: Protect all single-valued properties in oasis extractor
  * tracker-extract: Protect all single-valued properties in HTML extractor
  * tracker-extract: Check string length before parsing XMP in PDF extractor
  * tracker-extract: Add missing application/msword mimetype
  * libtracker-miner: Cut some slack on the reentry counter
  * libtracker-miner: Avoid changing order of elements in processing queue
  * libtracker-extract: Delete TrackerExtractClient
  * tracker-extract: Remove old dbus interface xml
  * tracker-extract: propagate urn to the TrackerExtractInfo
  * tracker-extract: Use safer method to insert tags in PDF module
  * tracker-extract: Use safer method to insert tags in GIF module
  * tracker-extract: Use safer method to insert tags in JPEG module
  * tracker-extract: Use safer method to insert tags in TIFF module
  * tracker-extract: Use safer method to insert tags in PNG module
  * libtracker-miner: Initialize all NodeData memory
  * libtracker-miner: Ensure the directory root is removed when its
indexing root is


ChangeLog
=
https://download.gnome.org/sources/tracker/1.7/tracker-1.7.5.changes  (7.55K)

Download

https://download.gnome.org/sources/tracker/1.7/tracker-1.7.5.tar.xz (4.70M)
  sha256sum: f1072a0eab98a6da503f6c595f528c6aacf343e43c1938510a2d99febd0a5a44
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

Re: [Tracker] no "nfo:belongsToContainer" property for some files

[Carlos Garnacho]

Hi,

On Fri, Mar 4, 2016 at 3:06 PM, e. Ift  wrote:
> hi all,
> If all images (nfo:Image) have a parent folder why in the result of the
> sparql query,
>
> SELECT nie:url(?u) nie:url(?f) WHERE {
>   ?u a nfo:Image .
>   ?u nfo:belongsToContainer ?f .
> }
>
> nothing is returned for nie:url(?f) for some files (images)?

What version do you see this happening with?
Do these files actually (still) exist?
Is the parent folder actually indexed? is the nfo:belongsToContainer
relationship missing or does it point to an empty urn?
Does this happen on a clean reindex? ("tracker reset -r && tracker
daemon -s" in recent Tracker)

The "tracker info " info for one of these files (and their
parent folder) would be appreciated too.

Cheers,
  Carlos
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

Re: [Tracker] embedded code copy in tracker is problematic

[Carlos Garnacho]

Hey :),

On Tue, Mar 1, 2016 at 11:02 PM, Michael Biebl <mbi...@gmail.com> wrote:
> Hi Carlos
>
> 2016-03-01 19:27 GMT+01:00 Carlos Garnacho <carl...@gnome.org>:
>> I talked with them (Richard Hipp and Dan Kennedy) through private
>> email. The solutions basically seemed to be:
>> - Including a static sqlite copy wherever fts3_tokenizer() is needed
>> - Using FTS5, which offers a way to customize FTS tokenizing that are
>> not affected by this vulnerability
>> - Adding such a similar way to FTS3
>>
>> Basically, the vulnerability is completely intrinsic to the
>> fts3_tokenizer() call with 2 arguments, they can't both fix the cve
>> and keep offering it unchanged. Of those three options, all three
>> require changes in the users of this call, plus for the third we'd
>> have to wait for an hypothetical change, and wouldn't erase 3.11 from
>> earth either...
>>
>> So I took solutions 1 and 2 wherever they apply. I also considered
>> backporting the FTS5 changes to stable branches, but it's too many
>> changes and too bleeding edge for me to be comfortable with it...
>
> Thanks for the explanation. I'm glad to hear that this embedded cope
> copy is only a workaround for the stable 1.6 branch.
> How far away is 1.7/1.8 from being declared stable?

I'm following the gnome schedule, so roughly 3 weeks :). The version
numbers are totally misguiding, but we're supposedly RC1 now.

Cheers,
  Carlos
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

Re: [Tracker] embedded code copy in tracker is problematic

[Carlos Garnacho]

Hey Michael,

On Tue, Mar 1, 2016 at 6:48 PM, Michael Biebl <mbi...@gmail.com> wrote:
> 2016-03-01 18:00 GMT+01:00 Carlos Garnacho <carl...@gnome.org>:
>> Hi Michael,
>>
>> On Tue, Mar 1, 2016 at 4:52 PM, Michael Biebl <mbi...@gmail.com> wrote:
>>> Hi everyone,
>>>
>>> I just noticed that the new tracker 1.6.2 contains a code copy of
>>> sqlite and no longer allows one to use the system sqlite library.
>>> This is problematic for various reasons and distros like Debian [1]
>>> and Fedora strongly discourage such code copies.
>>>
>>> Would it be possible to re-add the ability to link against the system
>>> sqlite and only fall back to the embedded copy if the system library
>>> doesn't meet the requirements of tracker (and output a big fat warning
>>> in this case)?
>>
>> Not sure if you missed the action caused by sqlite 3.11. From that
>> version on, they've hidden by default a sql function that's
>> indispensable for us.
>>
>> https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2015-7036
>
> Seems I missed that, indeed. Most likely because of:
>
> sqlite3 (3.11.0-2) unstable; urgency=low
>
>   * Compile with SQLITE_ENABLE_FTS3_TOKENIZER for backwards compatibility
> (closes: #815499).
>   * Update Standards-Version to 3.9.7 .
>
>  -- Laszlo Boszormenyi (GCS) <g...@debian.org>  Tue, 23 Feb 2016 21:31:39 
> +0100
>
> So this particular change was reverted in Debian.
>
>
>> Tracker itself is not hit by this cve, but we've evidently become
>> colateral damage since this is removed by default.
>>
>> The embedded copy solution has only been done on current stable
>> releases (1.4 and 1.6). It's not one I'm too happy with. But it's
>> surely better than requiring -DSQLITE_ENABLE_FTS3_TOKENIZER
>> system-wide (partly why I just went for always using the embedded
>> copy, this is something distros don't want enabled). For master (and
>> upcoming 1.8), I've opted for using FTS5 (which doesn't have this
>> problem), and still rely on the system sqlite library.
>>
>> I understand and share your concerns, but this is kind of a rough spot
>> we're on :).
>
> Has there been any discussion with sqlite upstream to solve that
> differently? I mean breaking consumers of the sqlite APIs can't be the
> proper fix for that.

I talked with them (Richard Hipp and Dan Kennedy) through private
email. The solutions basically seemed to be:
- Including a static sqlite copy wherever fts3_tokenizer() is needed
- Using FTS5, which offers a way to customize FTS tokenizing that are
not affected by this vulnerability
- Adding such a similar way to FTS3

Basically, the vulnerability is completely intrinsic to the
fts3_tokenizer() call with 2 arguments, they can't both fix the cve
and keep offering it unchanged. Of those three options, all three
require changes in the users of this call, plus for the third we'd
have to wait for an hypothetical change, and wouldn't erase 3.11 from
earth either...

So I took solutions 1 and 2 wherever they apply. I also considered
backporting the FTS5 changes to stable branches, but it's too many
changes and too bleeding edge for me to be comfortable with it...

Sorry about this situation, I know it will also be a pain to you.
There's a lot to grumble about it, we can do that in unison :)

Cheers,
  Carlos
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

Re: [Tracker] embedded code copy in tracker is problematic

[Carlos Garnacho]

Hi Michael,

On Tue, Mar 1, 2016 at 4:52 PM, Michael Biebl  wrote:
> Hi everyone,
>
> I just noticed that the new tracker 1.6.2 contains a code copy of
> sqlite and no longer allows one to use the system sqlite library.
> This is problematic for various reasons and distros like Debian [1]
> and Fedora strongly discourage such code copies.
>
> Would it be possible to re-add the ability to link against the system
> sqlite and only fall back to the embedded copy if the system library
> doesn't meet the requirements of tracker (and output a big fat warning
> in this case)?

Not sure if you missed the action caused by sqlite 3.11. From that
version on, they've hidden by default a sql function that's
indispensable for us.

https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2015-7036

Tracker itself is not hit by this cve, but we've evidently become
colateral damage since this is removed by default.

The embedded copy solution has only been done on current stable
releases (1.4 and 1.6). It's not one I'm too happy with. But it's
surely better than requiring -DSQLITE_ENABLE_FTS3_TOKENIZER
system-wide (partly why I just went for always using the embedded
copy, this is something distros don't want enabled). For master (and
upcoming 1.8), I've opted for using FTS5 (which doesn't have this
problem), and still rely on the system sqlite library.

I understand and share your concerns, but this is kind of a rough spot
we're on :).

Cheers,
  Carlos
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

Re: [Tracker] Tracker error unkown tokenizer

[Carlos Garnacho]

Hi Rick,

On Tue, Mar 1, 2016 at 12:33 PM, Rick van der Zwet
<i...@rickvanderzwet.nl> wrote:
> On 29/02/16 18:06, Carlos Garnacho wrote:
>> On Mon, Feb 29, 2016 at 5:41 PM, Rick van der Zwet
>> <i...@rickvanderzwet.nl> wrote:
>>> Hi Folks,
>>>
>>> $ tracker search foo
>>> Could not get search results, unknown tokenizer: TrackerTokenizer
>>>
>>> $ tracker --version
>>> Tracker 1.7.3
>>>
>>> I also tried to reset:
>>> $ tracker reset -r
>>>
>>> Yet the same result applies. Any hints on what I might be doing wrong?
>>
>> You did nothing wrong, this problem is due to compatibility issues
>> with sqlite >=3.11. Any Tracker version will only find this problem at
>> runtime unless sqlite is compiled with some special flag to preserve
>> compatibility. The issue is being currently addressed at
>> https://bugzilla.gnome.org/show_bug.cgi?id=762226 .
>>
> Thanks for your response and your time in making the software, allowing
> me to search for files et. al, which I use a lot in my daily workflow.

Thanks! Glad to know it's useful to you.

>
> Been testing the wip/fts5 branch of git://git.gnome.org/tracker yet this
> does not seems to-do the trick yet. The nautilus search plug-in for
> example wants to look at a table named ``fts''.

Hmm, I expect that to happen if applications are still linking to the
libtracker-sparql.so as provided by packages. Both the running
tracker-store process and the clients would need to use the same
libtracker-sparql as you compiled it.

AFAICT, the fixes in that branch will just work when those are
installed system-wide through packages.

>
> Will keep a eye on it, let me know if you need help testing.
>
> For the time being on Fedora 23 downgrading seems to be the only (quick)
> fix:
>
>   $ sudo dnf repository-packages fedora install sqlite --allowerasing

Yeah... the fedora folks were too eager to include sqlite 3.11 in
updates repos. Besides having been inconvenient to Tracker, the fix is
worthwhile otherwise.

I've just rolled new 1.4/1.6 releases that include an embedded copy of
sqlite, that will fix the issue in f23/22 (hopefully soon enough!).
The wip/fts5 branch has only been merged to master, and is already
released in 1.7.4, so this approach is reserved for f24/rawhide.

Cheers,
  Carlos
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

[Tracker] Fwd: tracker 1.4.3

[Carlos Garnacho]

-- Forwarded message --
From: Carlos Garnacho <install-mod...@master.gnome.org>
Date: Tue, Mar 1, 2016 at 5:26 PM
Subject: tracker 1.4.3
To: FTP Releases <ftp-release-l...@gnome.org>


About Tracker
=

Tracker is a semantic data storage for desktop and mobile devices.
Tracker uses W3C standards for RDF ontologies using Nepomuk with
SPARQL to query and update the data.

Tracker is a central repository of user information, that provides two
big benefits for the user; shared data between applications and
information which is relational to other information (for example:
mixing contacts with files, locations, activities and etc.).

News


  * Include embedded copy of sqlite

Translations: pt_BR


ChangeLog
=
https://download.gnome.org/sources/tracker/1.4/tracker-1.4.3.changes  (270)

Download

https://download.gnome.org/sources/tracker/1.4/tracker-1.4.3.tar.xz (6.82M)
  sha256sum: aa3728bd503e4849a1996e3721c16267fd45e0b3746b8b25fc4d6942d07f7b07
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

[Tracker] Fwd: tracker 1.6.2

[Carlos Garnacho]

-- Forwarded message --
From: Carlos Garnacho <install-mod...@master.gnome.org>
Date: Tue, Mar 1, 2016 at 5:27 PM
Subject: tracker 1.6.2
To: FTP Releases <ftp-release-l...@gnome.org>


About Tracker
=

Tracker is a semantic data storage for desktop and mobile devices.
Tracker uses W3C standards for RDF ontologies using Nepomuk with
SPARQL to query and update the data.

Tracker is a central repository of user information, that provides two
big benefits for the user; shared data between applications and
information which is relational to other information (for example:
mixing contacts with files, locations, activities and etc.).

News


  * Include embedded copy of sqlite.
  * tracker-extract: Fix small memory leak
  * libtracker-data: Silence a CRITICAL

Translations: lv


ChangeLog
=
https://download.gnome.org/sources/tracker/1.6/tracker-1.6.2.changes  (536)

Download

https://download.gnome.org/sources/tracker/1.6/tracker-1.6.2.tar.xz (5.84M)
  sha256sum: d3583f32e6a06ccb1146ca31939710edb630d7ffe3da37b01f893b45f4480045
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

[Tracker] Fwd: tracker 1.7.4

[Carlos Garnacho]

-- Forwarded message --
From: Carlos Garnacho <install-mod...@master.gnome.org>
Date: Tue, Mar 1, 2016 at 5:27 PM
Subject: tracker 1.7.4
To: FTP Releases <ftp-release-l...@gnome.org>


About Tracker
=

Tracker is a semantic data storage for desktop and mobile devices.
Tracker uses W3C standards for RDF ontologies using Nepomuk with
SPARQL to query and update the data.

Tracker is a central repository of user information, that provides two
big benefits for the user; shared data between applications and
information which is relational to other information (for example:
mixing contacts with files, locations, activities and etc.).

News


  * Update to FTS5.
  * libtracker-miner: Many fixes to TrackerFileNotifier cancellation
  * libtracker-direct: Handle cancellable argument in queries
  * libtracker-miner: Plug fd leak on TrackerCrawler cancellation
  * libtracker-extract: Fix year-only date extraction in gstreamer module
  * libtracker-extract: Use tracker-guarantee to ensure a title in playlists
  * tracker-miner-fs: Ignore git repositories. Modify the
ignored-directories-with-content
  setting if you found this convenient.
  * tracker-miner-fs: Ignore #*# vim backups

Translations: oc


ChangeLog
=
https://download.gnome.org/sources/tracker/1.7/tracker-1.7.4.changes  (6.53K)

Download

https://download.gnome.org/sources/tracker/1.7/tracker-1.7.4.tar.xz (4.70M)
  sha256sum: 297436550b60b12d18b34185390f46a14961d7f1a7fe4120a02de343ffde1256
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

Re: [Tracker] Tracker error unkown tokenizer

[Carlos Garnacho]

Hi Rick,

On Mon, Feb 29, 2016 at 5:41 PM, Rick van der Zwet
 wrote:
> Hi Folks,
>
> $ tracker search foo
> Could not get search results, unknown tokenizer: TrackerTokenizer
>
> $ tracker --version
> Tracker 1.7.3
>
> I also tried to reset:
> $ tracker reset -r
>
> Yet the same result applies. Any hints on what I might be doing wrong?

You did nothing wrong, this problem is due to compatibility issues
with sqlite >=3.11. Any Tracker version will only find this problem at
runtime unless sqlite is compiled with some special flag to preserve
compatibility. The issue is being currently addressed at
https://bugzilla.gnome.org/show_bug.cgi?id=762226 .

Cheers,
  Carlos
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

Re: [Tracker] 3.20 target bugs

[Carlos Garnacho]

Hey,

(CCing the Tracker ML for this one)

On Thu, Feb 18, 2016 at 1:35 PM, Matthias Clasen
 wrote:
> While we are still waiting for 3.19.90 to appear, here is an initial
> review of the bugs that have been marked as "GNOME target: 3.20"
> during this cycle. Since this is the first review, the list is
> somewhat long, and a bit of a mixed bag, I expect us to narrow it down
> for .91. In any case, all these bugs are well worth fixing, and if you
> can get one of the off the list, you will make 3.20 a better release.
>
> Please help out if you can!
>
> Matthias, for the release team
>
>
>
> Fallout from GTK+ changes (CSS and others)
> --
>
> 761765 bijiben Notes have a grey background rather than a custom color
> 762137 nautilus GtkPlacesSidebar: row selection jumps around
> 760525 gtk+ Labels in dialog buttons misaligned
> 760560 gtk+ Icon buttons wider in GTK+ 3.19.6
> 757503 gtk+ Selected text is white on white (invisible) -
> WebKit1 / GTK+ 3.19.7 & Adwaita
> 761686 gtk+ GtkTreeView theming problems
> 758893 gnome-shell Journal spam: Gdk-WARNING **:
> gdk-frame-clock: layout continuously requested, giving up after 4
> tries
>
> Power / Battery life problems
> -
>
> 752070 polari polari uses a lot of cpu
> 762194 tracker Indexes on battery
>
> Deprecation cleanup & build issues
> ---
>
> 760887 NetworkManager Do not depend on deprecated libnm-glib
> or dbus-glib when we only want to build the new libnma library
> 760946 NetworkManager nm-connection-editor still uses dbus-glib
> 757934 gobject-intros g-ir-scanner should not use system-provided 
> CFLAGS
> 751588 evolution Port to WebKit2
> 751185 empathy Use clutter-gst-3.0
> 749001 empathy Port to webkit2
> 728293 bijiben Port to WebKit2 or GtkTextView
> 705069 gnome-music Port from dbus-python to Gio GDBus API
> 686373 general [META] Switch to WebKit2
>
> Accessibility regressions
> 
>
> 762136 nautilus Progress of file and folder operations is no
> longer accessible to screen readers
>
> Wayland issues
> ---
>
> 749913 mutter wayland: Send frame callbacks when native
> hardware cursors get set
> 760745 mutter 100% CPU : Error transferring wayland clipboard to 
> X11
> 762104 mutter handle dnd drops on the root window
> 760567 gtk+ GDK screen size does not count for HiDPI on Wayland
> 756579 gtk+ GTK should let GDK position menus
> 748098 gdm monitors.xml not working in GDM when running
> under Wayland
> 695806 general [TRACKER] Wayland support
>
> Crashes & serious misbehavior
> -
>
> 761613 mutter crash with xwayland glamor
> 761157 libsecret libsecret-0.18.4 seems to crash gnome-shell
> 755721 glib g_inotify_file_monitor_start called with
> nullpointer for dirname causes a segfault
> 761175 librsvg Svg rendering regression from commit 3ae509 onwards
> 750508 gnome-session Logout is broken (a) when session
> inhibitor is active and (b) after logout is canceled once
> 761317 gnome-settings-daemon housekeeping: /tmp/.X11-unix/X0
> socket gets removed during housecleaning

I would like to drop https://bugzilla.gnome.org/show_bug.cgi?id=762226 here.

In short, sqlite has hidden stuff that Tracker needs behind a compile
time option (disabled by default) because of security concerns, and
Tracker only stumbles on this at runtime.

AFAICT the security concerns come from arbitrary execution of SQL
being able to alter later queries by overriding the full-text search
tokenizer, so they don't apply to Tracker (our SQL comes from a
bizarre state machine, but doesn't qualify as "arbitrary"). Most
immediately, I'll add a configure time check for this feature,
although I don't think it's ok to advice distros to enable
fts3_tokenizer system-wide. Solutions I can think of are:

- Including a static copy of sqlite in Tracker, with fts3_tokenizer() enabled
- Updating to the newer but code-wise incompatible fts5 (we use fts4),
which provides other similar hooks we can use. It's probably too
bleeding edge though, it was considered "experimental" not long ago
[1], and I haven't seen a word saying otherwise in later release
notes.

In this situation, the second option is a matter of time, I'd just
wish there were a longer board to walk.

Cheers,
  Carlos

[1] http://sqlite.org/releaselog/3_9_0.html
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

[Tracker] Fwd: tracker 1.7.3

[Carlos Garnacho]

-- Forwarded message --
From: Carlos Garnacho <install-mod...@master.gnome.org>
Date: Tue, Feb 16, 2016 at 12:40 AM
Subject: tracker 1.7.3
To: FTP Releases <ftp-release-l...@gnome.org>


About Tracker
=

Tracker is a semantic data storage for desktop and mobile devices.
Tracker uses W3C standards for RDF ontologies using Nepomuk with
SPARQL to query and update the data.

Tracker is a central repository of user information, that provides two
big benefits for the user; shared data between applications and
information which is relational to other information (for example:
mixing contacts with files, locations, activities and etc.).

News


  * libtracker-miner: Many indexing fixes leading to stale elements in
the database.
  If "UNIQUE constraint failed: nie:DataObject.nie:url" errors are seen in
  journald, running tracker-miner-fs once with the
TRACKER_MINER_FORCE_CHECK_UPDATED
  envvar is recommended. you will need to terminate miners before that with
  tracker daemon -t
  * libtracker-miner: Do not insert partial/empty sparql on error
  * libtracker-miner: Pass a builder in UPDATE state to
TrackerMinerFS::remove-file
  * libtracker-miner: Remove children recursively from queues on
directory deleted
  * libtracker-miner: Fix generated Sparql query in
sparql_contents_compose_query()
  * libtracker-miner: Fix some memory leaks of TrackerTask
  * libtracker-miner: Invalidate files iri recursively in case of file removal
  * libtracker-miner: Reset of reentry counter is not needed anymore
  * libtracker-fts: Fix invalid blob length calculation
  * libtracker-common: Use guint64 for free space calculations
  * libtracker-data, docs, libtracker-miner: Fix compile warnings
  * libtracker-data: misc code fixes
  * libtracker-data: Fix g_warning() missing argument
  * Update AppData to spec version 0.7+

Translations: lv


ChangeLog
=
https://download.gnome.org/sources/tracker/1.7/tracker-1.7.3.changes  (6.95K)

Download

https://download.gnome.org/sources/tracker/1.7/tracker-1.7.3.tar.xz (4.69M)
  sha256sum: b1f61e792db47035fda8524a65a661274fe75b9f8dd3b70ca4c00ae561a4c911
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

Re: [Tracker] Memory usage for tracker-extract and tracker-miner-fs

[Carlos Garnacho]

Hey Joe,

On Thu, Jan 21, 2016 at 3:38 AM, Joe Rhodes  wrote:
> Indeed.  I was mistaken about the memory usage.   Tracker-miner-fs has now 
> stabilized at about 265 MB of RAM.  I force a re-crawl once every two hours 
> to update the index.

Thanks for the update! would be great if you could provide the output of:

  tracker status -a | grep -e "nfo:Folder" -e "nfo:FileDataObject"

So I have some solid numbers instead of ballpark estimations for cases
like yours :).

>
> I've disabled the content searching for now by removing all the rules.  
> Nothing to do with Tracker.  It just took too long for the search results to 
> populate on the Mac clients over Netatalk.  In this case, I *know* the 
> bottleneck is in Netatalk.

Hmm, I guess netatalk queries fall into some performance pitfall, if
that's the case.

>
> So for my purposes, 1.7.2 is looking great.   Thanks!

You're welcome :). Thanks for helping us catch this soon.

Cheers,
  Carlos
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

Re: [Tracker] Memory usage for tracker-extract and tracker-miner-fs

[Carlos Garnacho]

Hi Joe,

On Mon, Jan 18, 2016 at 2:02 AM, Joe Rhodes  wrote:
> Carlos:
>
> I just tried the 1.7.2 tarball.  I think we may still have a leak.  The two 
> processes were heading up over 600 MB's.
>
> Here are the valgrind outputs.  I hope this helps:
>
> https://www.dropbox.com/s/gyndn7ithpge6nc/valgrind-tracker-extract-log-1.7.2.gz?dl=0
> https://www.dropbox.com/s/h250z3lh9mvr2vv/valgrind-tracker-miner-fs-log-1.7.2.gz?dl=0

As expected, the valgrind logs are a lot more favorable.
Tracker-extract reports:

==21759==definitely lost: 168 bytes in 1 blocks
==21759==indirectly lost: 6,046 bytes in 2 blocks

And those seem to be a memory pool in libjpeg that's mistakenly marked
as "lost". In tracker-miner-fs logs we see:

==21669==definitely lost: 0 bytes in 0 blocks
==21669==indirectly lost: 0 bytes in 0 blocks

So I think we're now fine there. Besides these, there's some memory
reported as "possibly lost", but that's the usual glib noise (more
specifically, GObject doesn't deinitialize registered classes/types).
I can also see some "conditional jump depends on uninitialized values"
warnings from the HTML extractor, which I can also reproduce, but need
some more investigation in libxml.

As for the memory used, I must confess 600MB in ps/top seem more
reasonable to me (as opposed to the several GBs you reported
initially). In tracker-miner-fs that sounds like a reasonable number
given your workload (and the fact that we need to keep an in-memory
representation of the directory tree). In tracker-extract sounds a bit
on the high side, but could be explained by memory fragmentation,
because tracker-extract may potentially initialize a lot of libraries
(that may also create their own runtime caches), and those don't/can't
get deinitialized after they are no longer used.

Although keep into account that top/ps is not the best tool to measure
memory usage. A more faithful number is given by the valgrind logs
themselves, eg. for your case in tracker-miner-fs it is:

==21669==still reachable: 143,987,764 bytes in 3,356,013 blocks

So at the time of you hitting Ctrl-C, tracker-miner-fs had allocated
~140MB. That sounds indeed more like it, although I'm not sure if you
let tracker-miner-fs run in its entirety under valgrind (that is,
until all folders were indexed).

Cheers,
  Carlos
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

[Tracker] Fwd: tracker 1.2.7

[Carlos Garnacho]

-- Forwarded message --
From: Carlos Garnacho <install-mod...@master.gnome.org>
Date: Tue, Jan 19, 2016 at 2:23 AM
Subject: tracker 1.2.7
To: FTP Releases <ftp-release-l...@gnome.org>


About Tracker
=

Tracker is a semantic data storage for desktop and mobile devices.
Tracker uses W3C standards for RDF ontologies using Nepomuk with
SPARQL to query and update the data.

Tracker is a central repository of user information, that provides two
big benefits for the user; shared data between applications and
information which is relational to other information (for example:
mixing contacts with files, locations, activities and etc.).

News


- libtracker-miner: Abort async operations once the instace is gone
- libtracker-miner: Cancel pending operations during destruction
- libtracker-miner: Handle failure to get a TrackerSparqlConnection
- libtracker-common: Fix buffer overrun in libunistring unaccenting
- libtracker-control: Fix return value
- libtracker-control: Improve the documentation
- libtracker-data: Silence CRITICAL

Translations: pt


ChangeLog
=
https://download.gnome.org/sources/tracker/1.2/tracker-1.2.7.changes  (3.53K)

Download

https://download.gnome.org/sources/tracker/1.2/tracker-1.2.7.tar.xz (5.75M)
  sha256sum: c9470eff2c5b966f68736129fca6ec05b8c94b05881458ab48bedfb7b7e13e6f
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

[Tracker] Fwd: tracker 1.4.2

[Carlos Garnacho]

-- Forwarded message --
From: Carlos Garnacho <install-mod...@master.gnome.org>
Date: Tue, Jan 19, 2016 at 1:39 AM
Subject: tracker 1.4.2
To: FTP Releases <ftp-release-l...@gnome.org>


About Tracker
=

Tracker is a semantic data storage for desktop and mobile devices.
Tracker uses W3C standards for RDF ontologies using Nepomuk with
SPARQL to query and update the data.

Tracker is a central repository of user information, that provides two
big benefits for the user; shared data between applications and
information which is relational to other information (for example:
mixing contacts with files, locations, activities and etc.).

News


  * tracker CLI: fixed search with -f argument
  * libtracker-miner: Cancel pending async operations during destruction
  * libtracker-miner: Handle failure to get a TrackerSparqlConnection
  * libtracker-common: Fix buffer overrun on libunistring unaccenting
  * libtracker-control: Fix return value in function call
  * libtracker-control: Improve docs
  * libtracker-data: Silence criticals on REGEX() with null strings

Translations: cs, de, hu, id, lt, pl, pt, sv


ChangeLog
=
https://download.gnome.org/sources/tracker/1.4/tracker-1.4.2.changes  (4.00K)

Download

https://download.gnome.org/sources/tracker/1.4/tracker-1.4.2.tar.xz (5.65M)
  sha256sum: 0db364c392992210b2e267bc54e0cad00177e0a1aa92f009ef603f6a8fcaeda1
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

[Tracker] Fwd: tracker 1.7.2

[Carlos Garnacho]

-- Forwarded message --
From: Carlos Garnacho <install-mod...@master.gnome.org>
Date: Sun, Jan 17, 2016 at 8:20 PM
Subject: tracker 1.7.2
To: FTP Releases <ftp-release-l...@gnome.org>


About Tracker
=

Tracker is a semantic data storage for desktop and mobile devices.
Tracker uses W3C standards for RDF ontologies using Nepomuk with
SPARQL to query and update the data.

Tracker is a central repository of user information, that provides two
big benefits for the user; shared data between applications and
information which is relational to other information (for example:
mixing contacts with files, locations, activities and etc.).

News


  * Many leak fixes.
  * libtracker-data: Reverted code to clean up stale Resources, can't
just be done yet.
  * tracker tool: Removed tracker-compatibility CLI wrapper for older commands.
  * libtracker-common: Fix possible warnings on libicu unaccent code
  * ontology: Set domain index on nie:contentCreated for nmo:Message
  * libtracker-miner: Add ::remove-file signal vfunc
  * libtracker-common: Return total available space if running as admin.

Translations: lt


ChangeLog
=
https://download.gnome.org/sources/tracker/1.7/tracker-1.7.2.changes  (2.80K)

Download

https://download.gnome.org/sources/tracker/1.7/tracker-1.7.2.tar.xz (4.69M)
  sha256sum: bd77eb80e29b3a53700ef03fcf4e1fc60b4f2c823b85d3b171ca191b8ef6d0d4
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

Re: [Tracker] Memory usage for tracker-extract and tracker-miner-fs

[Carlos Garnacho]

Hi Joe, Philip,

On Wed, Jan 13, 2016 at 8:55 AM, Philip Van Hoof  wrote:
> Hi Joe,
>
> Ok, that's fine. It would have been nice if we could let you do a
> valgrind re-run of the same use-case to see if we catch all the memory
> leaks before Carlos makes a new release.
>
> But this way we'll do a post-release check of your use-case. If we see
> more memory leaks after your retry we can always create a 7.1.3 release.
>
>
> I'd say: Let the release horses go loose Carlos.

A bit later than hoped, but Tracker 1.7.2 is out :), with some further
last-minute leak fixes. It should run a lot better for you now Joe,
please let us know how it gets along.

Cheers,
  Carlos
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

Re: [Tracker] Memory usage for tracker-extract and tracker-miner-fs

[Carlos Garnacho]

Hi Philip :)

On Mon, Jan 11, 2016 at 11:21 AM, Philip Van Hoof  wrote:
> Hi Carlos,
>
> Looks like my git-account has been closed on GNOME, so here is a patch
> for one of the issues in that valgrind.

Thanks! The patch is now in master and previous stable branches.

And I think your account might have been deactivated during the switch
to FreeIPA in gnome infrastructures, have a look at
https://mail.gnome.org/archives/infrastructure-announce/2014-October/msg1.html
, the "where can I get my first login credentials" has the steps that
should work for you to reactivate it.
https://wiki.gnome.org/AccountsTeam/NewAccounts further other
instructions that might be useful.

Cheers,
  Carlos
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

Re: [Tracker] Memory usage for tracker-extract and tracker-miner-fs

[Carlos Garnacho]

Hi Joe,

On Sun, Jan 10, 2016 at 10:05 PM, Joe Rhodes  wrote:
> Carlos:
>
> Yes, there are a LOT of files on this volume.  The makeup of the 5 TB of data 
> is PDFs, Photoshop files, Word docs, InDesign & Illustrator docs.  There are 
> very few large files like MP3's or videos.   If I disable all the extractors 
> and just build an index based on file names, I get an index of about 3 GB.

Oh wow :), Then your usecase is one of those we deemed "extreme".
People usually fill up those huge HDDs with movies and whatnot, as the
impact in tracker is per-file Tracker still does fine (eg. we don't
usually need to read the file entirely, there's less opendir()/open()
syscalls, etc). But obviously, a few thousands >1GB files are not the
same than a few million >1MB text documents, also in terms of DB
storage, so the meta.db size might be legit after all.

>
> I did notice that I was possibly indexing all of my snapshots of my volumes. 
> I'm using ZFS and they're available under "/volume/.zfs".  I've added that 
> folder to my list of excluded directories:
>
> org.freedesktop.Tracker.Miner.Files ignored-directories ['.zfs', 'ZZZ 
> Snapshots', 'po', 'CVS', 'core-dumps', 'lost+found']

Tracker ignores hidden directories unless they're specifically marked
as an indexing directory in config. You can check with "tracker info
" nonetheless, if there is info spew, Tracker indexed the file
somehow.

>
> I'll see if that makes any difference.  If it was digging into those, that 
> would greatly increase the number of files.
>
> I'm not entirely sure how to start tracker with the valgrind command.  
> Tracker is currently started automatically by the Netatalk file server 
> process.  In order to run the tracker processes, I have to execute the 
> following:
>
> PREFIX="/main-storage"
> export XDG_DATA_HOME="$PREFIX/var/netatalk/"
> export XDG_CACHE_HOME="$PREFIX/var/netatalk/"
> export DBUS_SESSION_BUS_ADDRESS="unix:path=$PREFIX/var/netatalk/spotlight.ipc"
> /usr/local/bin/tracker daemon -t
>
> So after stopping the daemon, I just started tried the following:
>
> valgrind --leak-check=full --log-file=valgrind-tracker-extract-log 
> --num-callers=30 /usr/local/libexec/tracker-extract
> valgrind --leak-check=full --log-file=valgrind-tracker-miner-fs-log 
> --num-callers=30 /usr/local/libexec/tracker-miner-fs
>
> Hopefully that will get you want you want?

Definitely! quite some memory reported as definitely lost, this
spotted a few embarrassing leaks... one now kindly fixed by Philip in
the msoffice module, others were introduced by myself in the 1.7.x
series, while porting part of our async machinery to GTask... I went
again through all my related changes, and fixed some leaks affecting
both tracker-extract and tracker-miner-fs. These fixes should account
for everything I can peek in the valgrind logs you provided.

So, master should work a lot better for you now, I think the leaks
found account for such high memory usage (or the majority of it). If
you can fetch the last 4 commits from git (or try with git itself)
that'd be great. I'll get a 1.7.2 release out of the door this week
nonetheless.

It would be definitely appreciated if you could try running again on
valgrind these patches, and send info back if you see further
"definitely lost" memory reported in the summary at the end of the
valgrind logs.

Cheers,
  Carlos
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

Re: [Tracker] Memory usage for tracker-extract and tracker-miner-fs

[Carlos Garnacho]

Hi Joe,

On Sun, Jan 10, 2016 at 6:40 PM, Joe Rhodes  wrote:
> I have just compiled and installed tracker-1.7.1 on a CentOS 7.1 box.  I
> just used the default configuration ("./configure" with no additional
> options).  I'm indexing around  5 TB of data.  I'm noticing that both the
> tracker-extract   and  tracker-miner-fs processes are using a large amount
> of RAM.  The tracker-extract process is currently using 11 GB of RAM (RES
> not VIRT as reported by top), while the tracker-miner-fs is sitting at 4.5
> GB.
>
> Both processes start out modestly, but continue to grow as they do their
> work.  The tracker-miner-fs levels off at 4.5 GB once it appears to have
> finished crawling the entire volume. (Once the CPU usage goes back down to
> near 0.)   The tracker-extract process also continues to grow as it works.
> Once it is done, it levels off.  Last time it stayed at about 9 GB.
>
> If I restart tracker (with: 'tracker daemon -t' followed by 'tracker daemon
> -s') a similar thing will happen with tracker-miner-fs.  It will grow back
> to 4.5 GB as it crawls its way across the entire volume.  The
> tracker-extract process though, because all of the files were just indexed
> and it doesn't need to do much, uses a very modest amount of RAM. I don't
> have that number right now because I'm re-indexing the entire volume, but
> it's well below 100 MB.
>
> Is this expected behaviour?  Or is there a memory leak?  Or perhaps tracker
> just isn't designed to operate on this large of a volume?

It totally sounds like a memory leak, although it sounds strange that
it hits both tracker-miner-fs and tracker-extract.

There is obviously an impact to running Tracker on large directory
trees, such as:

- Possibly exhausted inotify handles, the directories we fail to
create a monitor for would just be checked/updated on next miner
startup
- More (longer, rather) IO/CPU usage during startup, because the miner
has to check mtimes for all directories and files
- The miner also needs to keep an in-memory representation of the
directory tree for accounting purposes (file monitors, etc). Regular
files are represented in this model only as long as they're being
checked/processed, and disappear soon after. This might account for a
memory peak at startup, if there's many items left to process, because
Tracker dumps files into processing queues ASAP, but I think the
memory usage should be nowhere as big.

So I think nothing accounts for such memory usage in tracker-miner-fs,
the only known source of unbound memory growth is the number of
directories (and regular files for the peak at startup) to be indexed,
but you would need millions of those to have tracker-miner-fs grow up
to 4.5GB.

And tracker-extract has a much shorter memory, it just checks the
files that need extraction in small batches, and processes those one
by one before querying the next batch. 9GB shout memory leak, we've
had other memory leak situations in tracker-extract, and the culprit
most often is in the various libraries we're using in our extract
modules, if many files end up triggering that module (and the leaky
code path in the specific library), the effect will accumulate over
time.

The downside of this situation is that most often we Tracker
developers can't reproduce unless we have a file that triggers the
leak so we can fix it or channel to the appropriate maintainers, so it
would be great if you could provide valgrind logs, just run as:

valgrind --leak-check=full --log-file=valgrind-log --num-callers=30
/path/to/built/tracker-extract

Hit ctrl-C when enough time has passed, and send back the valgrind-log
file. Same applies to tracker-miner-fs.

>
> My tracker meta.db file is about 13 GB right now, though still growing.  I
> suspect it's close to indexed though.

This is also suspicious, you again need either a hideous amount of
files to have meta.db grow as large, or an equally hideous amount of
plain text content that gets indexed. Out of curiosity, how many
directories/files does that partition contain? is the content
primarily video/documents/etc?

Cheers,
  Carlos
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

[Tracker] Fwd: tracker 1.7.0

[Carlos Garnacho]

-- Forwarded message --
From: Carlos Garnacho <install-mod...@master.gnome.org>
Date: Wed, Nov 25, 2015 at 11:34 PM
Subject: tracker 1.7.0
To: FTP Releases <ftp-release-l...@gnome.org>


About Tracker
=

Tracker is a semantic data storage for desktop and mobile devices.
Tracker uses W3C standards for RDF ontologies using Nepomuk with
SPARQL to query and update the data.

Tracker is a central repository of user information, that provides two
big benefits for the user; shared data between applications and
information which is relational to other information (for example:
mixing contacts with files, locations, activities and etc.).

News


  * libtracker-data: Support for Sparql 1.1 functions: CONCAT,
CONTAINS, LCASE/UCASE, STRLEN,
SUBSTR, STRSTARTS/STRENDS, ABS, ROUND, ENCODE_FOR_URI,
STRBEFORE/STRAFTER, CEIL/FLOOR,
YEAR/MONTH/DAY/HOUR/MINUTES/SECONDS, MD5/SHA1/SHA256/SHA512
  * libtracker-miner: Move previous data deletion on file updates to
TrackerMinerFS implementations
  * libtracker-miner/libtracker-data/libtracker-extract: Partial port to GTask
  * tracker tool: Fixes to UID detection
  * libtracker-miner: Fix cancellation of tasks during
TrackerFileNotifier destruction
  * libtracker-miner: Handle failure to get a TrackerSparqlConnection
  * libtracker-common: Fix buffer overrun in libunistring-based unaccenting
  * libtracker-control: Documentation fixes
  * tracker-extract: Photo orientation extraction fixes (TIFF, XMP)
  * Many fixes to functional tests

Translations: eu, it, sr, sr@latin, zh_CN


ChangeLog
=
https://download.gnome.org/sources/tracker/1.7/tracker-1.7.0.changes  (10.5K)

Download

https://download.gnome.org/sources/tracker/1.7/tracker-1.7.0.tar.xz (4.68M)
  sha256sum: 4fbfa4b14a97ed02865451eec8f426612b7257503418f6f2c3e7488c28c9f795
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

[Tracker] Fwd: tracker 1.6.1

[Carlos Garnacho]

-- Forwarded message --
From: Carlos Garnacho <install-mod...@master.gnome.org>
Date: Wed, Nov 25, 2015 at 11:34 PM
Subject: tracker 1.6.1
To: FTP Releases <ftp-release-l...@gnome.org>


About Tracker
=

Tracker is a semantic data storage for desktop and mobile devices.
Tracker uses W3C standards for RDF ontologies using Nepomuk with
SPARQL to query and update the data.

Tracker is a central repository of user information, that provides two
big benefits for the user; shared data between applications and
information which is relational to other information (for example:
mixing contacts with files, locations, activities and etc.).

News


  * tracker tool: Fixes to UID detection
  * libtracker-miner: Fix cancellation of tasks during
TrackerFileNotifier destruction
  * libtracker-miner: Handle failure to get a TrackerSparqlConnection
  * libtracker-common: Fix buffer overrun in libunistring-based unaccenting
  * libtracker-control: Documentation fixes
  * tracker-extract: Photo orientation extraction fixes (TIFF, XMP)
  * Many fixes to functional tests

Translations: eu, it, sr, zh_CN


ChangeLog
=
https://download.gnome.org/sources/tracker/1.6/tracker-1.6.1.changes  (4.70K)

Download

https://download.gnome.org/sources/tracker/1.6/tracker-1.6.1.tar.xz (4.68M)
  sha256sum: 653ed73f4f454b836df56bec1f1141c7a8d77cbeba97ea1e38df9f60a5f0c1ed
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

Re: [Tracker] Fwd: tracker 1.6.0

[Carlos Garnacho]

Hey Philip :),

On Tue, Sep 22, 2015 at 5:40 PM, Philip Van Hoof <phi...@codeminded.be> wrote:
> The sparql 1.1 support didn't make it?

Sorry, no... that's something I want to get back to right after
branching (which I guess we can do right now). The patches already
came by quite late in gnome schedules, not that we've followed them a
lot (as the jump from 1.5.2 to 1.6.0 testifies...) but I preferred to
stay on the safe side wrt freezes, specially given people wouldn't
rush into using sparql 1.1 features either at that stage.

Thanks for the branch review btw! I still got pending to reply to you there.

Cheers,
  Carlos

>
> On Tue, 2015-09-22 at 14:55 +0200, Carlos Garnacho wrote:
>> -- Forwarded message --
>> From: Carlos Garnacho <install-mod...@master.gnome.org>
>> Date: Tue, Sep 22, 2015 at 2:54 PM
>> Subject: tracker 1.6.0
>> To: FTP Releases <ftp-release-l...@gnome.org>
>>
>>
>> About Tracker
>> =
>>
>> Tracker is a semantic data storage for desktop and mobile devices.
>> Tracker uses W3C standards for RDF ontologies using Nepomuk with
>> SPARQL to query and update the data.
>>
>> Tracker is a central repository of user information, that provides two
>> big benefits for the user; shared data between applications and
>> information which is relational to other information (for example:
>> mixing contacts with files, locations, activities and etc.).
>>
>> News
>> 
>>
>>   * tracker-extract: Fix synchronization with tracker-miner-fs when
>> wait-for-miner-fs=TRUE
>>   * tracker-miner-fs: Fix crash during startup
>>   * tracker-extract: Fix builtin dummy module struct
>>
>> Translations: da, de, el, fr, gl, id, ko, pl, pt_BR, nb, sl, sv, tr, ru, 
>> zh_TW
>>
>>
>> ChangeLog
>> =
>> https://download.gnome.org/sources/tracker/1.6/tracker-1.6.0.changes  (2.79K)
>>
>> Download
>> 
>> https://download.gnome.org/sources/tracker/1.6/tracker-1.6.0.tar.xz (4.70M)
>>   sha256sum: 7e2729627224f43f8cd99c18d027a3b984e049fe924a265a9b31857566c9e28a
>> ___
>> tracker-list mailing list
>> tracker-list@gnome.org
>> https://mail.gnome.org/mailman/listinfo/tracker-list
>
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

[Tracker] Fwd: tracker 1.5.2

[Carlos Garnacho]

-- Forwarded message --
From: Carlos Garnacho install-mod...@master.gnome.org
Date: Thu, Aug 20, 2015 at 12:28 AM
Subject: tracker 1.5.2
To: FTP Releases ftp-release-l...@gnome.org


About Tracker
=

Tracker is a semantic data storage for desktop and mobile devices.
Tracker uses W3C standards for RDF ontologies using Nepomuk with
SPARQL to query and update the data.

Tracker is a central repository of user information, that provides two
big benefits for the user; shared data between applications and
information which is relational to other information (for example:
mixing contacts with files, locations, activities and etc.).

News


  * libtracker-data: Fix printf string format
  * libtracker-miner: Fallback to basename checks on hidden files
  * rss: Set website url as a nfo:WebSite
  * rss: Simplify GrssFeedChannel list creation
  * libtracker-data: Clean up stale URIs on startup
  * rss: Optimize deletes
  * rss: Perform extraction/insertion of feed items at once
  * ontology: Remove cardinality limits on nmo:communicationChannel
  * libtracker-common: String to date conversion to return with GError
when null string
  * libtracker-extract: Add builtin dummy extractor
  * tracker-extract: Use dummy fastpath for svg extraction
  * libtracker-extract: Plug leaks
  * libtracker-miner: Cancellation on unmount fixes
  * libtracker-miner: Deprecate tracker_miner_fs_add_directory_without_parent
  * tracker-miner-fs: Keep cache of IndexFile requesters on directories

Translations: ca, cz, lt, pl, pt_BR, pt, sk, tr


ChangeLog
=
https://download.gnome.org/sources/tracker/1.5/tracker-1.5.2.changes  (13.0K)

Download

https://download.gnome.org/sources/tracker/1.5/tracker-1.5.2.tar.xz (4.68M)
  sha256sum: bfdddc8fccdb877b5e75d7e6f7f5e0e2a646986b4d14aaca20d48c7015fa79de
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

Re: [Tracker] Fwd: Re: Linux TOPCRASH 38.1.0. Crash Reports in libsqlite3.*

[Carlos Garnacho]

Sending to the list this time...

Hi Wayne,

On Tue, Jul 28, 2015 at 7:05 PM, Wayne Mery (Thunderbird QA)
vseer...@lehigh.edu wrote:
 The addon is causing a topcrash for opensuse users using THunderbird 38.

 We haven't yet filed a mozilla blocklist request against it. But we will
 need to if a fix cannot be developed in the very new future.

Sorry to hear that, hopefully we can work this out.

I'm unsure how to read the data in the links below, is it true that
the crashes in 1.4.0 drop from the previous 1.2.x version? If that's
the case, I can only recommend updating, 1.4.0 is the latest stable
release (we're now actually heading to the next 1.6 stable series).

Tracker 1.2.x development is in any case discontinued, we could try to
identify the fix and roll a last 1.2 tarball, but I'm not sure how
fast would distros catch up.

Also, is there any chance we may have backtraces or similar?

Cheers,
  Carlos


 distro contact
 Wolf Rosenauer (TB linux OpenSuse distro contact wolfir)
 mozi...@rosenauer.org


  Forwarded Message 
 Subject: Re: Linux TOPCRASH 38.1.0. Crash Reports in libsqlite3.*
 Date: Fri, 24 Jul 2015 13:17:37 -0400
 From: Wayne Mery (Thunderbird QA) vseer...@lehigh.edu
 To: Christian Riechers chriech...@netscape.net

 On 7/24/2015 1:12 PM, Christian Riechers wrote:

 On 07/24/2015 07:04 PM, Wayne Mery (Thunderbird QA) wrote:

 On 7/24/2015 12:40 PM, Christian Riechers wrote:

 On 07/24/2015 04:51 PM, Wayne Mery (Thunderbird QA) wrote:

 the following confirms axs' findings about trackerbird mentioned on IRC

 do a 'find' on libsqlite3.so
 -

 https://crash-analysis.mozilla.com/crash_analysis/20150724/20150724_Thunderbird_38.1.0-interesting-addons-with-versions.txt

 -

 https://crash-analysis.mozilla.com/crash_analysis/20150724/20150724_Thunderbird_38.1.0-interesting-addons.txt


 I'll file a blocklist bug request later today, unless someone beats me
 to it.


 cc: Christian


 Doesn't seem to be a problem here, and I think my SUSE is up to date.

 Information for package libsqlite3-0:
 -
 Repository: openSUSE-13.2-Oss
 Name: libsqlite3-0
 Version: 3.8.6-1.2
 Arch: i586
 Vendor: openSUSE
 Installed: Yes
 Status: up-to-date
 Installed Size: 777.8 KiB
 Summary: Shared libraries for the Embeddable SQL Database Engine


 odd.
 what version of trackerbird do you have installed?


 Not sure what 'trackerbird' is, I suppose this is about 'tracker'.

 Information for package tracker:
 
 Repository: openSUSE-13.2-Update
 Name: tracker
 Version: 1.2.6-9.1
 Arch: i586
 Vendor: openSUSE
 Installed: Yes
 Status: up-to-date
 Installed Size: 2.3 MiB
 Summary: Powerful object database, tag/metadata database, search tool
 and indexer
 Description:
Tracker is a powerful desktop-neutral first class object
database, tag/metadata database, search tool and indexer.

It consists of a common object database that allows entities to
have an almost infinite number of properties, metadata (both
embedded/harvested as well as user definable), a comprehensive
database of keywords/tags and links to other entities.

It provides additional features for file-based objects
including context linking and audit trails for a file object.

It has the ability to index, store, harvest metadata, retrieve
and search all types of files and other first class objects.


 Yes, opensuse users with version 1.2.6 trackerbird and thunderbird 38
 experience a startup crash


 ___
 tracker-list mailing list
 tracker-list@gnome.org
 https://mail.gnome.org/mailman/listinfo/tracker-list
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

[Tracker] tracker 1.5.1

[Carlos Garnacho]

About Tracker
=

Tracker is a semantic data storage for desktop and mobile devices.
Tracker uses W3C standards for RDF ontologies using Nepomuk with
SPARQL to query and update the data.

Tracker is a central repository of user information, that provides two
big benefits for the user; shared data between applications and
information which is relational to other information (for example:
mixing contacts with files, locations, activities and etc.).

News


  * Many fixes to RSS miner:
- Dumps more complete data on tracker-store.
- Stability fixes.
- Leak fixes.
- Performs automatic maintenance of feed messages.
  * Bumped libgrss dependency on 0.7
  * Performance improvements on tracker-store delete operations
  * Performance improvements on tracker-miner-fs delete operation handling.
  * Fix main Resource table id/urn leaks
  * Fix unnecessary queries in tracker-extract

Translations: es, hu, pt


ChangeLog
=
2015-07-21  Carlos Garnacho  carl...@gnome.org

Release 1.5.1

libtracker-miner: Avoid full table scans on recursive sparql buffer queries
If MATCH_CHILDREN is specified for a TrackerTask, we use
tracker:uri-is-descendant(), it's however smarter to use fn:starts-with,
as that'll resort to sqlite tricks that will avoid full table scans.

libtracker-miner: Remove operations on children on deleted folders
This is an optimization to reduce the number of queries that we
perform across the deletion of large directory trees.

libtracker-miner: Only set MATCH_CHILDREN on tasks for directory files
It's a query that can be avoided for non-directories, so better do it.

libtracker-miner: Add tracker_file_notifier_get_file_type()
Just plug the hole from the internal TrackerFileSystem, will be
handy for fast file type checks at the TrackerMinerFS level.

libtracker-miner: Be smarter at not triggering TrackerDecorator activity
There's times where tracker-extract GraphUpdated handler will fire due
to its own inserts. Doing so is harmless, but triggers each time a
query for the count of unhandled elements that effectively goes nowhere
as it's already active.

So handle_updates() is now smarter at not triggering activity unless
a resource of the inspected classes is added, and graph_updated_cb()
won't trigger anymore the count query everytime.

libtracker-data: delete elements from the Resource table
On deletion, items with an specific row ID are removed from all
tables but the Resources one, which holds the urn:uuid:... mappings.

The deletion of that table lead to confusions in the fts_view view
and ultimately the FTS table, as both will indirectly depend on the
elements stored there, so the deleted rows still had FTS representation,
just filled with nulls.

This looks like was just forgotten, if it was there to cover
constraint errors, it'll be better to just open the pandora box, and
fix the bugs we receive. Anyhow, from testing most common scenarios
it works alright.

parser: Optimize 0-length string parsing
We were still creating the ICU parser and trying to feed it with
data, which turned out surprisingly expensive on deletes, as
deleting on FTS just replaces the text with nothing, so we're
creating a parser for each of these.

This reduces the timing of the sparql delete in the previous commit
further down to:
real 1m7.029s
user 0m0.023s
sys 0m0.009s

libtracker-data: Don't schedule all deletes only because of FTS
The limitations in FTS why it made sense to perform the scheduled
delete no longer apply since FTS4 and external content tables
(or rather, we don't need the previous values explicitly).

The scheduled delete is a lot more (if not extremely) thorough,
decomposing the properties and items to be deleted into individual
queries. This has quite an effect on deletes involving a large
number of elements, a query like

delete { ?u a rdfs:Resource; }
where { ?u nie:url ?url .
  FILTER (fn:starts-with (?url, .../linux/))}

on a linux git checkout indexed through tracker-miner-fs used
to involve 7M sqlite queries, with this fast path it's down to
1.6M (and infinitely less sqlite3_stmt cache misses). In result
the timing is improved substantially, time(1) from that query
on the tracker sparql command went from:

real 2m33.377s
user 0m0.021s
sys 0m0.008s

Down to:

real 1m23.625s
user 0m0.021s
sys 0m0.009s

libtracker-data: Add function to delete an entire row from the FTS table
This can be used as an optimization, instead of updating each column
individually as we currently do.

2015-07-19  Carlos Garnacho  carl...@gnome.org

tracker-extract-msoffice: Avoid frequent errors when feeding wrong files
There's mimetypes which detection is too weak (i.e. purely based on
filename extension matches), so it makes sense to avoid the frequent
errors we get when the module gets fed a random file.

tracker-extract-gstreamer: Avoid frequent errors when feeding wrong files
There's mimetypes which detection is too weak (i.e. purely based on
filename extension matches), so it makes

[Tracker] tracker 1.5.0

[Carlos Garnacho]

About Tracker
=

Tracker is a semantic data storage for desktop and mobile devices.
Tracker uses W3C standards for RDF ontologies using Nepomuk with
SPARQL to query and update the data.

Tracker is a central repository of user information, that provides two
big benefits for the user; shared data between applications and
information which is relational to other information (for example:
mixing contacts with files, locations, activities and etc.).

ChangeLog
=

2015-07-13  Carlos Garnacho  carl...@gnome.org

Release 1.5.0

distcheck fixes

tracker-extract: Remove ModulePath from comic/ebook rules
These relied on the dummy extractor, removing the ModulePath is a 
similar
silent shortcut since commit 06f7d4a1a.

extract-gstreamer: Rely better on the GstDiscoverer than mimetype 
sniffing
There's mimetypes that easily fool mimetype detection (eg. OGG videos 
with
.ogg extension instead of .ogv will be detected as audio/ogg), the
GstDiscoverer will however find out correctly whether there's audio 
and/or
video information, so we should rely on it as a last resort, rather than
(weaker) mimetype sniffing.

This prevents .ogg suffixed videos from being played by gnome-music 
(oddly,
with success, in a separate window).

2015-07-13  Daniel Mustieles  daniel.mustie...@gmail.com

Updated Spanish translation

2015-07-12  Carlos Garnacho  carl...@gnome.org

libtracker-extract: Accept rules with no ModulePath
This will enable us to make dummy rules for files that must have some 
RDF
type(s), but don't have an extractor module.

https://bugzilla.gnome.org/show_bug.cgi?id=735610

2015-07-12  Gilles Dartiguelongue  e...@gentoo.org

build: Fix AM_CONDITIONAL position HAVE_{GSTREAMER,LIBAV} definition
https://bugzilla.gnome.org/show_bug.cgi?id=750368

2015-07-06  Carlos Garnacho  carl...@gnome.org

tracker-extract-gstreamer: Fallback to preview image for album art
Some files don't provide a cover image, but just a preview image, we can
use that as albumart.

This commit reuses code from a fallback which looks bogus nowadays, 
since
it just does the same than the while loop above. If we reached there, it
would fail again for sure.

https://bugzilla.gnome.org/show_bug.cgi?id=732236

2015-07-06  Pedro Albuquerque  palbuquerqu...@gmail.com

Updated Portuguese translation

2015-07-06  Philip Withnall  philip.withn...@collabora.co.uk

libtracker-miner: Keep a monitor on root index tree files on deletion
If Tracker is running with some index-recursive-directories set (for
example, ~/Music), and one of those directories is moved, then the file
monitor watching for its existence is deleted. This means that if it is
then moved back to the location listed in index-recursive-directories,
its renewed existence is not detected, and its contents are not
re-indexed.

Fix this by keeping a monitor around on a directory if that directory is
listed as an index tree root, even if the directory is deleted.

https://bugzilla.gnome.org/show_bug.cgi?id=750394

2015-07-06  Ville Skyttä  ville.sky...@iki.fi

tracker-sparql.1: Man page syntax fix
https://bugzilla.gnome.org/show_bug.cgi?id=751723

docs: Spelling fixes
https://bugzilla.gnome.org/show_bug.cgi?id=751724

2015-07-05  Carlos Garnacho  carl...@gnome.org

tracker-miner-fs: Reset retry counter when we need to prepend parents
The situation where a parent directory has to be prepended in order to
process the current file is rare, mainly reserved to IndexFile calls
on files out of watched dirs.

This is a corner situation, but it is a legit place where we have to
put the file back again in the queue, and thus we shouldn't increase
the retry counter.

This nominally fixes the indexing of gnome-documents-getting-started.pdf
on a fresh-out tracker DB, as requested by gnome-shell.

https://bugzilla.gnome.org/show_bug.cgi?id=751992

tracker-preferences: Use the new command line tool
We resort to the tracker CLI for restart/reindex, the calls have
changed to use Posix.system() as there's some operations that are
now mutually exclusive in the CLI tool (eg. reset and start daemons)
and still need to be performed in a single step here.

https://bugzilla.gnome.org/show_bug.cgi?id=748677

tracker-preferences: Remove needless prints
These won't tell a lot, since we're killing the whole thing.

2015-07-05  Sam Thursfield  s...@afuera.me.uk

Fix tracker-encoding-test
After commit ede17cc22b0c6245c030 this was failing to compile.

2015-07-05  Carlos Garnacho  carl...@gnome.org

cli: Improve bash

[Tracker] Fwd: tracker 1.3.6

[Carlos Garnacho]

About Tracker
=

Tracker is a semantic data storage for desktop and mobile devices.
Tracker uses W3C standards for RDF ontologies using Nepomuk with
SPARQL to query and update the data.

Tracker is a central repository of user information, that provides two
big benefits for the user; shared data between applications and
information which is relational to other information (for example:
mixing contacts with files, locations, activities and etc.).

News


  * Fix spurious folder deletes/reindexes (#741852)
  * Fix nie:url UNIQUE constraint asserts on downloaded files (RH#1192224)
  * Clear tracker-store watchdog timeout (#745565)
  * Support fn:replace (#745917)
  * Spam stderr less for not-so-uncommon error conditions (#746256)

Translations: ko, bs, sr, sl, da



ChangeLog
=
https://download.gnome.org/sources/tracker/1.3/tracker-1.3.6.changes  (3.65K)

Download

https://download.gnome.org/sources/tracker/1.3/tracker-1.3.6.tar.xz (5.67M)
  sha256sum: 0738debb8f396951301fcbcb2ba587d9714374f69971c2e354c768c1b3ab2d3d
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

[Tracker] Fwd: tracker 1.3.5

[Carlos Garnacho]

About Tracker
=

Tracker is a semantic data storage for desktop and mobile devices.
Tracker uses W3C standards for RDF ontologies using Nepomuk with
SPARQL to query and update the data.

Tracker is a central repository of user information, that provides two
big benefits for the user; shared data between applications and
information which is relational to other information (for example:
mixing contacts with files, locations, activities and etc.).

News


NEW in 1.3.4 - 2015-03-06
=

  * Fix major database migration bug, skip Tracker 1.3.4. (#745737)
  * Build only libiptc test if libjpeg is enabled (#745583)
  * Put absolute path in shell script (#743738)

Translations: ru



ChangeLog
=
https://download.gnome.org/sources/tracker/1.3/tracker-1.3.5.changes  (1.17K)

Download

https://download.gnome.org/sources/tracker/1.3/tracker-1.3.5.tar.xz (5.75M)
  sha256sum: d95446e7ec50e45b33da442b29afbd269eab867340873309d35a079c667c2f57
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

Re: [Tracker] Review request, bug fixed by a downstream integrator (Jolla)

[Carlos Garnacho]

Hey Philip :),

On Fri, Dec 19, 2014 at 3:23 PM, Philip Van Hoof phi...@codeminded.be wrote:
 -BEGIN PGP SIGNED MESSAGE-
 https://git.gnome.org/browse/tracker/log/?h=crawler-max-depth

 Is this ~ what you proposed?

Yup! pretty much :). That should ensure no subdirs go through
traverse_tree_foreach() before they're meant to. Thanks for doing
this.

Cheers,
  Carlos
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

Re: [Tracker] What to do with libtracker-common?

[Carlos Garnacho]

Hey,

On Mon, Sep 29, 2014 at 11:55 AM, Martyn Russell mar...@lanedo.com wrote:

snip


 Breakdown of libtracker-common
 --

 - tracker-config-file.h

Used for all GSettings/configs, so libtracker-data, libtracker-fts,
tracker-store, and the miners.


The only purpose of this nowadays is to ensure the transition from keyfiles
to gsettings, and will never be used after the keyfile is migrated and
deleted.

IMO we should remove this and fully trust on GSettings API. Right now it
can only hurt to people updating all the way from 0.8, and if any embedded
system wants to use Tracker and has a trouble with gsettings/dconf, they
will surely have the same problem with other parts of the stack, and can
just use a different GSettings backend.


snip


 - tracker-file-utils.h

Used to handle operations like getting file size, mtime, etc and
also for locking files, calculating disk space remaining, etc. One
important reason for APIs like tracker_file_open() is to make sure
extractors open with O_NOATIME and to allow posix_fadvise() use
consistently.


File locking should go, it's currently unused, and only plays well if you
respect the locks, so it really isn't as useful.



 - tracker-ioprio.h

Set the I/O priority to be lower than normal to avoid disk
clobbering. Used by tracker-extract and the miners.

Could actually be in libtracker-miner.


I would think this is too specific API, I wouldn't expect any external
TrackerMiner to thrash the filesystem as massively as tracker-miner-fs does.

If tracker-extract and tracker-miner-fs are kept in the same repo, that
could be private API between both though.



 - tracker-keyfile-object.h

Internal and used exclusively by tracker-config-file.h. So wherever
that ends up, so should this module.


Exactly, I think it deserves the same end...


 - tracker-locale-gconfdbus.h
 - tracker-locale.h
 - tracker-meego.h

Used to notify and keep track of local changes which is needed to
re-create database collations because sorting can vary by locale
(among other things).

The -gconfdbus file is an implementation, which we might even be
able to remove by now? *Is anyone using still using GConf?*

The -meego file is use to translate and get the locale on Meego.
*Is anyone still using Meego?* Would like to remove this.


Please let's remove this outdated stuff, I consider on-the-fly locale
change handling something unachievable today, no matter how much we tape
over it

snip


 - tracker-os-dependant.h

The tracker_spawn*() API here is only used by libtracker-data and 2
extractors, mplayer and ps (where we use gunzip, which is quite a
bad way to do this).


I really don't know why the PS extractor isn't using poppler... I would
prefer us to stick to GStreamer and remove the mplayer module, it's
practically unused in any downstream, after that it can be moved to
libtracker-data, or just make it use g_spawn_async_with_pipes itself.



The tracker_memory_setrlimits() is only used by tracker-extract and
could be moved there directly or put in libtracker-extract.


Please let's just remove this, brings in enough grief.

snip


 Possible solutions
 --
 1. We just make libtracker-common public, perhaps give it a better name
 (like libtracker-core?) and be done with it. We can then use it anywhere.
 Advantages: it's quick and easy, Disadvantages: API has to be stable.

 2. We split libtracker-common up and move code to more specific areas,
 e.g. fundamental type functions (A) go into libtracker-data, etc.
 Advantages: code is more logically placed, Disadvantages: Linking to larger
 libraries for small API calls might not be so attractive.

 3. We create newer smaller libraries with logical parts A to D and.
 Advantages: Smaller concise areas to group API, Disadvantages: Slightly
 larger maintenance and distribution burden in the beginning.

 4. We copy the code to modules they're needed in. Advantages: Less
 libraries to link with, Disadvantages: potentially a larger footprint with
 the same API copied around.


5. Move libtracker-common to a separate git module, set it up on the
relevant repos through git submodule, compile it statically in these repos,
and add a configure script to that submodule that fails and points people
to the right place(s).

Cheers,
  Carlos
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

Re: [Tracker] Review request, bug fixed by a downstream integrator (Jolla)

[Carlos Garnacho]

Hey Philip,

Sorry I missed this...


On Tue, Jul 8, 2014 at 9:48 AM, Philip Van Hoof phi...@codeminded.be
wrote:

 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1

 Hi guys,

 As you know I'm tailing the Tracker package for the Jolla device and I
 noticed that there a bugfix was made by Richard Braakman for a change
 that happened recently:

 https://github.com/nemomobile-packages/tracker/pull/29

 Commit here:

 https://github.com/amtep/tracker/commit/419d680619b5e0d4f3ae308a087dd313e8ce252e



I see how spurious ::file-deleted might happen, the TrackerFileSystem
caches data for directories, so if a reindex happens on a directory that
happened to preserve data, tracker_file_system_traverse() would recurse
deep in the old tree while the crawling phase only updated the most direct
children at that time.

I however think a better fix would be to add a max_depth parameter to
tracker_file_system_traverse(), and pass the same depth that's given on
tracker_crawler_start() (and implicitly performed in queries). AFAICS the
approach in the patch would still fail for files deeper than MAX_DEPTH, as
those won't be yet in the pending_files array. TrackerFileSystem is
private, so API changes are just fine there.

Cheers,
  Carlos
___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list

Re: [Tracker] Tracker 0.17.6

[Carlos Garnacho]

Hey Michael,

On mié, 2014-03-19 at 10:56 +0100, Michael Biebl wrote:
 Hi,
 
 2014-03-18 18:58 GMT+01:00 Martyn Russell mar...@lanedo.com:
* tracker-extract: Add desktop file to autostart process
 
 This is something I'm curious about: What were the reasons to start
 tracker-extract (and tracker-miner-fs) via XDG autostart files? In the
 past IIRC we used D-Bus activation to start those services on-demand.
 Only tracker-store itself was started via an XDG autostart file.

As for tracker-extract, now that's an standalone daemon, I just made it
do the same than tracker-miner-fs (and its autostart file was added in
2009 :).

As a more generic why autorun miners? question, IMO the problem is
more that there is no central point to start miners. on demand doesn't
quite work here because apps generally poke the store, not the miners.
For the specific tracker-extract case it might make sense to have
tracker-miner-fs start tracker-extract (or a watchdog, as martyn says),
but then what does start tracker-miner-fs in a regular session, or any
other miner...

Cheers,
  Carlos

___
tracker-list mailing list
tracker-list@gnome.org
https://mail.gnome.org/mailman/listinfo/tracker-list