Hi all I've always felt like Tracker's extractors should be reusable outside Tracker. The design makes that possible but right now they output their results as a series of slightly non-standard SPARQL update commands, which I don't think is useful for many folk. Lots of people aren't using SPARQL databases at all, believe it or not :-)
The whole point of RDF is to make data interchange easy so I think we can do better than that. I've been looking at making the extractors optionally output their results in JSON-LD[1] format instead. The cool thing about JSON-LD is that if you squint, it's just good old JSON that everyone's familiar with. If you look closely it's also Linked Data, but in a more human-friendly serialization format than any of the more traditional RDF formats. The catch here is that Tracker's extractor modules are all hardwired to generate SPARQL using TrackerSparqlBuilder. To be honest I've never liked this approach, it's pretty incomprehensible to newcomers and overly verbose, especially where we explicitly generate DELETE queries to go along with the INSERT queries. so, inspired by something in the Python RDFLib library, I came up with a TrackerResource class that the extractors can use instead. This is a work in process, but I have a branch in git.gnome.org that adds TrackerResource, and converts some of the extractors to use it. The TrackerResource class can serialize either to SPARQL update commands or to JSON-LD. The branch also adds the `tracker extract` command from <https://bugzilla.gnome.org/show_bug.cgi?id=751991> so you can try out the extractors easily and specify `-o json` or `-o sparql` as you prefer. The results for extractors I've converted so far is promising in terms of reducing code size: src/tracker-extract/tracker-extract-abw.c | 51 ++-- src/tracker-extract/tracker-extract-bmp.c | 18 +- src/tracker-extract/tracker-extract-dvi.c | 17 +- src/tracker-extract/tracker-extract-epub.c | 131 +++----- src/tracker-extract/tracker-extract-gstreamer.c | 910 ++++++++++++++++++------------------------------------- src/tracker-extract/tracker-extract-mp3.c | 378 ++++++++--------------- 6 files changed, 511 insertions(+), 994 deletions(-) Here's an example of auto-generated SPARQL for an MP3 extraction: DELETE { } WHERE { <file:///home/sam/Downloads/Best%20Coast%20-%20The%20Only%20Place.mp3> nie:comment ?nie_comment ; nmm:trackNumber ?nmm_trackNumber ; nmm:performer ?nmm_performer ; nfo:averageBitrate ?nfo_averageBitrate ; nmm:musicAlbum ?nmm_musicAlbum ; nfo:channels ?nfo_channels ; nmm:dlnaProfile ?nmm_dlnaProfile ; nmm:musicAlbumDisc ?nmm_musicAlbumDisc ; rdf:type ?rdf_type ; nfo:duration ?nfo_duration ; nfo:codec ?nfo_codec ; nmm:dlnaMime ?nmm_dlnaMime ; nfo:sampleRate ?nfo_sampleRate ; nie:title ?nie_title . } DELETE { } WHERE { <urn:artist:Best%20Coast> nmm:artistName ?nmm_artistName ; rdf:type ?rdf_type . } INSERT { <urn:artist:Best%20Coast> a nmm:Artist ; nmm:artistName "Best Coast" . } DELETE { } WHERE { <urn:album:The%20Only%20Place> nmm:albumTitle ?nmm_albumTitle ; rdf:type ?rdf_type ; nmm:albumArtist ?nmm_albumArtist . } INSERT { <urn:album:The%20Only%20Place> a nmm:MusicAlbum ; nmm:albumTitle "The Only Place" ; nmm:albumArtist <urn:artist:Best%20Coast> . } DELETE { } WHERE { <urn:album-disc:%D0:%06%02:Disc1> nmm:setNumber ?nmm_setNumber ; nmm:albumDiscAlbum ?nmm_albumDiscAlbum ; rdf:type ?rdf_type . } INSERT { <urn:album-disc:%D0:%06%02:Disc1> a nmm:MusicAlbumDisc ; nmm:setNumber 1 ; nmm:albumDiscAlbum <urn:album:The%20Only%20Place> . } INSERT { <file:///home/sam/Downloads/Best%20Coast%20-%20The%20Only%20Place.mp3> a nmm:MusicPiece , nfo:Audio ; nie:comment "Free download from http://www.last.fm/music/Best+Coast and http://MP3.com"; ; nmm:trackNumber 1 ; nmm:performer <urn:artist:Best%20Coast> ; nfo:averageBitrate 128000 ; nmm:musicAlbum <urn:album:The%20Only%20Place> ; nfo:channels 2 ; nmm:dlnaProfile "MP3" ; nmm:musicAlbumDisc <urn:album-disc:%D0:%06%02:Disc1> ; nfo:duration 164 ; nfo:codec "MPEG" ; nmm:dlnaMime "audio/mpeg" ; nfo:sampleRate 44100 ; nie:title "The Only Place" . } Note there are a lot more DELETE statements than before. I figured that anywhere we want to replace the existing data we need a DELETE statement, and the reason we don't normally do it is because previously it had to be done manually. That said, the TrackerResource class does have a way of avoiding this. If you ever call _set_value() for a property then it assumes you want to *overwrite* it, and will generate a DELETE. If you only use _add_value() then it will assume you want to *add* to it, and won't generate a DELETE. The latter case is needed for stuff like nao:hasTag. I may be misunderstanding things here of course, I didn't actually write any of the extractors myself. Here's a example of JSON-LD output: { "nie:comment" : "Free download from http://www.last.fm/music/Best+Coast and http://MP3.com";, "nmm:trackNumber" : 1, "nmm:performer" : { "@id" : "urn:artist:Best%20Coast", "nmm:artistName" : "Best Coast", "@type" : "nmm:Artist" }, "nfo:averageBitrate" : 128000, "nmm:musicAlbum" : { "@id" : "urn:album:The%20Only%20Place", "nmm:albumTitle" : "The Only Place", "@type" : "nmm:MusicAlbum", "nmm:albumArtist" : { "@id" : "urn:artist:Best%20Coast", "nmm:artistName" : "Best Coast", "@type" : "nmm:Artist" } }, "nfo:channels" : 2, "nmm:dlnaProfile" : "MP3", "nmm:musicAlbumDisc" : { "@id" : "urn:album-disc:%C0:L%01:Disc1", "nmm:setNumber" : 1, "nmm:albumDiscAlbum" : { "@id" : "urn:album:The%20Only%20Place", "nmm:albumTitle" : "The Only Place", "@type" : "nmm:MusicAlbum", "nmm:albumArtist" : { "@id" : "urn:artist:Best%20Coast", "nmm:artistName" : "Best Coast", "@type" : "nmm:Artist" } }, "@type" : "nmm:MusicAlbumDisc" }, "nfo:duration" : 164, "nfo:codec" : "MPEG", "nmm:dlnaMime" : "audio/mpeg", "nfo:sampleRate" : 44100, "nie:title" : "The Only Place" } We can actually do much better than this, right now there's no @context so it kind of misses the point of JSON-LD. I need to finish writing a NamespaceManager class that can track all of the prefixes and generate a suitable JSON-LD context, so that instead of stuff like "nie:title", it can just say "title" and then the @context will link that to <http://www.semanticdesktop.org/ontologies/2007/01/19/nie#title> The code is in branch wip/sam/resource: <https://git.gnome.org/browse/tracker/log/?h=wip/sam/resource>. It's still of course a work in progress but I think it's pretty much taken shape, so please have a look and give feedback on whether you think this is a sane approach! Thanks Sam [1]: http://json-ld.org/ _______________________________________________ tracker-list mailing list [email protected] https://mail.gnome.org/mailman/listinfo/tracker-list
