Re: [Tracker] Proposed future for Tracker - Smaller modules
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 8/09/2014 17:11, Martyn Russell wrote: Hi Martyn, My proposal would be to keep tracker-store, ontology and libtracker-sparql together (as one project). The reason I didn't put libtracker-sparql with tracker-store / core things is that logically it's quite a different thing for applications wanting to JUST run SPARQL queries. Well, nope. They depend on tracker-store either way. Since that's the case, then it's better to make tracker-store an impl. detail of libtracker-sparql I suppose it doesn't make all that much difference, you still require the store and other bits for libtracker-sparql to operate anyway. Precisely. Applications wanting to JUST run SPARQL queries, need tracker-store. Even for reading with direct-access, as tracker-store is the one that assembles meta.db for in case it's not there yet and it needs to be created from initial ontology. I guess you could go as far as to say that you might want to JUST use DBus and none of the libtracker-sparql support at all, another reason for it being separate. The DBus-only non-fd-passing Query APIs on tracker-store are purely for development purposes. Nobody should really use them in production. The only meaningful API on the Resources DBus object is GraphUpdated signal. The Steroids or FD-passing DBus API can be used, but preferably applications always use libtracker-sparql and get the benefits of direct-access mode (process prioritization and no-ipc overhead). The reason for that is that the actual goal of libtracker-sparql was to provide what libsqlite is for a RDMBS and SQL, but for RDF and SPARQL. Yea, but libsqlite3 and sqlite are separate packages (actually I am not sure if they're separate projects at all), but having tracker-store does not mean you have to use libtracker-sparql. sqlite is a package providing the sqlite shell. That would be equivalent to having a package called tracker that provides just the tracker-sparql tool. All of sqlite is in libsqlite, and you can perfectly use libsqlite without the sqlite package, like how I would like libtracker-sparql to be too (tracker is an embedded db vs. an oracle or mysql or postgresql). Without tracker-store, libtracker-sparql can't work. It's more the other way round I had in mind, without libtracker-sparql, tracker-store works. Actually, libtracker-data (the code of tracker-store) is also the direct-access implementation of libtracker-sparql. The tracker-store itself is a bunch of in Vala written D-Bus service wrappers around the same libtracker-data (that is used for libtracker-sparql-direct's implementation) plus an eventloop and checkpointing plus some initialization code and DBus daemonization. That's it, that's what tracker-store is :) Everything is in libtracker-data, which is used by libtracker-sparql By splitting the ontologies into a separate project, managed by the Nepomuk organization or not, we could someday refactor libtracker-sparql to support multiple instances of installed ontologies. I wouldn't split the ontologies out, the store is no good without the database and without the schema it's even more useless :) Sure it is. There are already a few companies that are using Tracker with a completely different ontology. The tracker-ivi project comes to mind. Will run on the dashboards of some nice cars. Besides, all the ontology validation and handling is in the libraries the store depends on. Not correct. All ontology validation rules are in the ontology's own introspection. The libtracker-data's ontology validation rules are created out of that ontology introspection, and nothing about it is hardcoded in it. Meaning that data/ontologies is completely separate from libtracker-data You could install a complete different data/ontologies, and libtracker-data would just happily crunch that and assembly you a different meta.db with different ontology validation rules. In fact, several embedded-solution companies do this today w. tracker. Ideal in that scenario would be that tracker-store becomes an impl. detail of libtracker-sparql (it'll manage the instances of stores on a on-demand basis). The only real three reasons of existence of tracker-store are: - GraphUpdated - The fact that SQLite isn't MVCC and we need WAL journaling and checkpointing done by a separate 'writer' process Yea, single point of update. But whether it comes as a implementation detail of libtracker-sparql or is a central process of a Desktop session; shouldn't be of concern to whoever links with and uses libtracker-sparql. - Providing a ontology Not sure I follow you here, the reason for the store is not to provide an ontology - at least libtracker-data does a lot of that stuff - I would have to double check this. libtracker-data doesn't provide the ontology, data/ontologies does. The libtracker-data implementation (which is also used 1 on 1 by libtracker-sparql's direct-access
Re: [Tracker] Proposed future for Tracker - Smaller modules
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 9/09/2014 18:54, Martyn Russell wrote: On 08/09/14 17:46, Philip Van Hoof wrote: [cut] No reason. libtracker-sparql users shouldn't care. Off the top of my head: 1. To provide a DBus interface (may be a legacy reason, but ...) Only GraphUpdated is of concern here. That should indeed stay and is a difficulty in my idea indeed. The Query and Insert D-Bus DBusMessage based API should at all costs be completely avoided. They are cruelly slow and shouldn't at all be called official API (legacy or not, the legacy is btw from before the 0.8.x release, we've had a major version increment since then). 2. To provide concurrent connections a way to serialise updates Is an implementation detail of libtracker-sparql 3. To provide a central and singleton authority to the DB schema That's what I want to change in my utopian idea :) 4. Because of database locking mechanisms employed by SQLite? Implementation detail of libtracker-sparql 5. To avoid possible race conditions? Implementation detail .. The last two could be wrong, its been a while since I had to care about this stuff :) The list mostly correct. tracker-store exists for the purpose of: - - Initiating the ontology - - Applying ontology changes when detected - - Journalling when enabled (which shouldn't be) - - Providing the Backup and Restore API - - GraphUpdated signal (signals on changes) - - Steroids FD passing D-Bus API - - SQLite WAL checkpointing - - Gathering and returning statistics over D-Bus - - Receiving over Steroids FD passing D-Bus API the from libtracker- sparql rerouted write queries Except for GraphUpdated it has no API that isn't an implementation detail or an internal affair of how libtracker-sparql works. I'm sure many of those things can be improved these days, but you're talking about making this decision upon some utopian idea that is not implemented yet. I am talking about doing this for what we have *now*. That is fair enough. But changes we make *now* should not conflict with these utopian ideas. So let's not split the project, packages and build files up in such a way that a independent libtracker-sparql with an independent ontology package and project can exist. A libtracker-sparql can exists without an ontology installed: the application that uses libtracker-sparql could provide it all by itself, and libtracker-sparql would still have purpose. Just like how libsqlite doesn't come with a SQL schema, instead the applications using libsqlite provide it. Consider the application that only wants to query and not update the DB - they don't want to depend on all the crap needed for updating the DB, just the raw libtracker-sparql part. Except that they already and always depend on tracker-store. You can't avoid it (read libtracker-sparql's initialization code: you always and necessarily depend on it). I guess it depends on what level you care about depending on something. Hard dependency. Unavoidable. Kind regards, Philip -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.20 (MingW32) iQEcBAEBAgAGBQJUD3HNAAoJEEP2NSGEz4aDwoQH/15HZMdxABgnpBhd9irSvqSO g8gkp1qkUroQyx5w8gjIo7G6kZfEc9h1EpIqMI5QJjhgL0wJwXhLqzWgfJVyKul7 bTRxYEJQvj2qc/qYt/JijpRYJxm4PWGo1G+FhdKnxMti88+QK2JnwzPNOrO/pX1D 83oInyQwd3t+U6o5zqxQ3VVn0qcLVZWClVPUDHSjVa9BWIEy7tfpzY++xyhYRH3y yhSQE+0hiNgRJLsN1+mjr8ACBOFmWXYKhEh+tCz43uMcggN69QuiBncT2h9nvCF1 0AhyH8Bc1BgjHoZp3XFtX8x7LnNBX3QbTWyEFEjnQdf2/CBgSLDQtRrUm52+vDQ= =ftvw -END PGP SIGNATURE- ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: [Tracker] Proposed future for Tracker - Smaller modules
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 8/09/2014 12:34, Martyn Russell wrote: Hi Martyn, Module = tracker-store - ontology (database schema) - libstemmer (stemming library used by libtracker-fts) - libtracker-common (type and various utilities) - libtracker-data (database access and implementation) - libtracker-fts (SQLite module for tokenising) - tracker-store (daemon providing DBus interface) Module = libtracker-sparql - libtracker-bus (DBus and read/write access to the DB) - libtracker-direct (readonly access to the DB) - libtracker-sparql-backend (internal implementation) - libtracker-sparql (public facing library to the DB) My proposal would be to keep tracker-store, ontology and libtracker-sparql together (as one project). The reason for that is that the actual goal of libtracker-sparql was to provide what libsqlite is for a RDMBS and SQL, but for RDF and SPARQL. Without tracker-store, libtracker-sparql can't work. By splitting the ontologies into a separate project, managed by the Nepomuk organization or not, we could someday refactor libtracker-sparql to support multiple instances of installed ontologies. Ideal in that scenario would be that tracker-store becomes an impl. detail of libtracker-sparql (it'll manage the instances of stores on a on-demand basis). The only real three reasons of existence of tracker-store are: - GraphUpdated - The fact that SQLite isn't MVCC and we need WAL journaling and checkpointing done by a separate 'writer' process - Providing a ontology When your process links with libtracker-sparql and uses direct-access mode, it effectively has everything it needs to deal with meta.db. The eventual dependency tree would look like this: Your program depends on: | +- nepomuk-ontology | +-libtracker-sparql `- Internally uses libexec/tracker-sparql-store This API: https://developer.gnome.org/libtracker-sparql/stable/TrackerSparqlConnection.html Would look like: #include nepomuk/2006-2008/ontology.h #include tracker-sparql.h static void something () { GError *error = NULL; Ontology *ontology = nepomuk_ontology_2006_2008_new (session id); TrackerSparqlConnection *con = tracker_sparql_connection_get_for_ontology (ontology) TrackerSparqlCursor *cursor = tracker_sparql_connection_query (con, select ?a { ?a nie:title 'something in nepomuk' }, NULL, error); while (tracker_sparql_cursor_next (cursor, NULL, NULL)) { g_print (%s\n, tracker_sparql_cursor_get_string()); } g_object_unref (cursor); g_object_unref (con); g_object_unref (ontology); } This would internally deal with starting and stopping a libexec/tracker-sparql-store, and no global tracker-store would be needed anymore (the fact that multiple users share the same session id, creates the central storage for users of that session id). Storage could go to: ~/.tracker/sessions/$session-id/nepomuk-2006-2008/meta.db or something Kind regards, Philip -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.20 (MingW32) iQEcBAEBAgAGBQJUDcOJAAoJEEP2NSGEz4aDAQQIAJ6arFvREfkh3tvieSwvvPLw gDo7hP8XUHo42GaWC0bfSIV5cZalgN66zZN2uWjKDRba9zQ76F166lQQJ6l/LmlK JQA0DvjuEQvLqR52dc8MDaC0H7D0dXFGWT3lhynCUxioyRPipDHVlwE0WlWEkfnR CGQIeFK7ADgKi4Ae85cxsgJvN0GD3TGwigF10mTSjbLXCwXDyc+XdnBCwx3zFlQ+ H4S2avM3c/DlcPHfudJKy8Bm0qS7YmJxqnOdykkJVYJ3d4TIDf7uOarlkEwuCbxm HraxGgq7xpT/7mIbUlTbS150cFhG/Q03ceeKkewLtfEcmBOEGuB9BRQFi+wJ8LU= =4rYt -END PGP SIGNATURE- ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Announcing GNOME's official GitHub mirror
Richard Stallman wrote on 17/08/2013 1:10: I don't think it'll make much of a difference as neither technical nor philosophical arguments are often part of, not what gets discussed at GNOME but, what gets decided. To make sure that it's noted I do agree with Richard on this: But I don't think that applies to most of what GitHub or Savannah does. Those are communication activities. You couldn't do them by calling a library in your own computer. So it is ok to use services for that (but pay attention to the privacy issues). However, it would be nice if we could do it in a peer-to-peer fashion. I also wonder what Github offers that can't be done peer to peer. Note that I lack experience in the magic of quantum-marketing iCloud stuff. Kind regards, Philip ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: [Tracker] Shell + Tracker + Zeitgeist
On Sat, 2010-06-12 at 18:14 +0200, Seif Lotfy wrote: Looking at the mockups by McCann I am looking for a Tracker + Zeitgeist integration in Shell. Except for Frequent, and Conversations I think everything else can be done over Tracker... I would love to know how Frequent will be categorized... I hope in frequency you could cluster into today, last week, etc... I would encourage using Tracker to get the data on the desktop... And then when asking for frequent and Conversations to ask Zeitgeist. But as an alternative to ease integration I think Tracker guys should look into supporting our ontology... This way Zeitgeist could use its extensions to push into Tracker. This way Tracker will know about the events we have and I think it would make it easier for the shell developers to only query Tracker for frequent stuff etc... What do you think? We already have the extensions ready for you guys thanks to Rob Taylor. OK, we will certainly take a look at the ontology next few days. Feel free to ping us for review a few times, many of our team members are quite busy so you need to conqueror our review time sometimes ;) Let me know what you think :) Cheers Seif ___ tracker-list mailing list tracker-l...@gnome.org http://mail.gnome.org/mailman/listinfo/tracker-list -- Philip Van Hoof freelance software developer Codeminded BVBA - http://codeminded.be ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Module Proposal: Rygel
On Fri, 2010-03-19 at 16:56 +0200, Zeeshan Ali (Khattak) wrote: [CUT] Short note on... Actually this issue is partly solved. I made sure that tracker guys have the right ontology caugh..metadata keys Let's just call these things ontologies. That's just what everybody in RDF calls it. And those alternative versions of the word like the metadata keys, the fields, the schema, description of the things ... are just confusing. It's called an ontology. It wont hurt you. It wont rot your brain. It's terminology that everybody in RDF uses. Let's get over it. Cheers, Philip -- Philip Van Hoof freelance software developer Codeminded BVBA - http://codeminded.be ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Module Proposal: Rygel
On Fri, 2010-03-19 at 12:59 +, Bastien Nocera wrote: On Sun, 2010-02-21 at 18:12 +0200, Zeeshan Ali wrote: Hi, 2 things come to mind: - Tracker shouldn't need to use D-Bus to access its database read-only: https://bugzilla.gnome.org/show_bug.cgi?id=613255 This isn't as easy at it sounds, I posted a comment on the bug you reported explaining what the difficulty is. https://bugzilla.gnome.org/show_bug.cgi?id=613255#c1 Cheers, Philip -- Philip Van Hoof freelance software developer Codeminded BVBA - http://codeminded.be ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Module Proposal: Rygel
On Fri, 2010-03-19 at 16:28 +, Bastien Nocera wrote: On Fri, 2010-03-19 at 15:25 +, Martyn Russell wrote: I tried to sent this several times, but it looks like GMANE is not working for me today ... And Evolution has its reply to all feature all wrong too, apparently. [CUT] What I mean is, what difference does it make if the data is readonly or readwrite? I'm mentioning read-only because I believe it would be possible to implement with your current architecture. If you want read/write access, it would probably involve file locking, and all sort of nastiness that I understand might be problematic. Reading the data through a native API instead of through D-Bus would mean being able to shift very large amounts of data without the marshalling/demarshalling (say, loading 100k audio tracks' metadata would probably take a large amount of time over D-Bus, where it would be of negligeable duration through a more direct API). This isn't as easy at it sounds, I posted a comment on the bug you reported explaining what the difficulty is. https://bugzilla.gnome.org/show_bug.cgi?id=613255#c1 Surely applications would want ONE method not two for communicating with Tracker's. Or did I misunderstand somehow? This would all be hidden behind the tracker libraries for most developers, so it shouldn't matter. You'd get the same result, just much faster. Cheers, Philip -- Philip Van Hoof freelance software developer Codeminded BVBA - http://codeminded.be ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: How do I request a patch review?
On Mon, 2010-03-08 at 09:44 +0100, Andre Klapper wrote: Am Sonntag, den 07.03.2010, 19:56 -0800 schrieb MPR: My assumption is that there must be some patch review process that was not followed. No, there's not. And this isn't a good thing andre Cheers, Philip -- Philip Van Hoof freelance software developer Codeminded BVBA - http://codeminded.be ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: New module proposal: tracker
On Tue, 2009-11-10 at 12:51 +, John Carr wrote: On Tue, Nov 10, 2009 at 10:28 AM, Rob Taylor rob.tay...@codethink.co.uk wrote: One option is that some of this code could be merged into tracker itself, if it is deemed useful, along with exposing TrackerSparqlBuilder (which might be internal atm? can't remember). TrackerSparqlBuilder only supports INSERT and DELETE queries*. * http://jena.hpl.hp.com/~afs/SPARQL-Update.html We could easily expose this as client API, but we'd love to first have support for SELECT too. Otherwise it's a bit limited in usefulness for application developers. Maybe is sparql-glib providing this and could it extend TrackerSparql- Builder? We wouldn't be against accepting patches, if they are sane, to improve on tracker-sparql-builder.vala (even if we wouldn't use it yet ourselves). SPARQL should be generic for all kinds of RDF stores (virtuoso, tracker- store, etc) so it makes sense to keep it separate. Both KDE's metadata projects and tracker-store are indeed going for SPARQL and Nepomuk. It's indeed awesome that the metadata-ppl are in a big agreement on this atm. SPARQL for virtuoso and for tracker-store are of course mostly the same. We just have a few functions that other SPARQL services might not have, and vise-versa. For example fts:match, fts:offsets and fts:range. Standardizing them is part of exploring of SPARQL-on-Desktop. We're of course discussing this with other people in the desktop-metadata-world. If somebody wants to work on this perhaps a GSparqlQueryBuilder and then a TrackerSparqlQueryBuilder that extends, adds and overrides GSparqlQueryBuilder's virtuals with things that are specific for our store's specialized use-cases? /j #tracker and come talk with us. But You are already talking (and coding) with us, John ;), so this is for the other ppl. This client stuff doesn't have to be part of Tracker. We can make it part of Tracker. Let's architect design! :) Just an option, but i'm not strictly in favor of that as i had hoped to see it able to access other SPARQL endpoints like dbpedia.org. Exactly, precisely. That said, theres no reason it couldnt support multiple endpoint types, as im sure the miners would love to query such repositories online. Awesome that some people are getting it! Thanks John. The main problem with sparql-glib as it stands is that it stands is that its still quite low level. You are essentially building an AST by hand and then sparql-glib flattens that into sparql. For the common case we need to come up with something a little higher level i think. Like gobject property getters (pass in multiple properties to fetch, it builds and uses the AST to get sparql). However for complicated queries with lots of relations to other resources this approach won't be as useful, i think. Let's architect design! I'd like to see jalchemy merged in to sparql-glib too, as that is really just a wrapper to make using sparql-glib less verbose from js land. Sure, let's architect design! :) Hehe As long as people who'll actually code, not just talk, take part, we'll most likely join their ideas and help with the coding. People like you, John. So if you know where the others are hiding, go get them! It's absolutely not that we don't want this client stuff to happen, it's that we have quite a list of things to do in Tracker itself. A bit of momentum and contributors will help the client's side a lot, I think. And with that has this thread grown way too large. If all the energy that went into this thread would have been put in coding stuff, we'd have five client libraries by now. -- Philip Van Hoof, freelance software developer home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org http://pvanhoof.be/blog http://codeminded.be ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: New module proposal: tracker
On Sun, 2009-11-08 at 20:21 +0200, Stefan Kost wrote: Patryk Zawadzki schrieb: On Tue, Oct 27, 2009 at 4:56 PM, Shaun McCance sha...@gnome.org wrote: I get the impression that the focus is more on data storage than indexing these days. I respect what the Tracker folks are trying to do, but I just need a good indexer to enable full-text search in Yelp. Wouldn't it make more sense to have a system service available for such resources? Wasn't xsesame meant to be such an (dbus) interface? Imho tracker supported it a some point, Correct, I wrote this support back then. But we since dropped it. dunno how its now and how it relates to sparql. Maybe other solutions should rather have a sparql query iface too. The problems with Xesam 1 are: o. No SPARQL, instead a less powerful home brewed query language. o. Home brewed ontology that is flatter than Nepomuk (fewer relationships). o. Complicated DBus API that among other things requires state. o. Very hard to implement life search capability. o. Has no API for inserting and updating data. Together with other teams, we are discussing with the people who defined Xesam 1. We regularly meet at conferences. It's a long term goal to finish Xesam 2. Among the ideas are to let it use SPARQL, Nepomuk and a less complex life search. Xesam is proposed to be used as community layer around Nepomuk. Tracker 0.7 ships with a DBus API, Nepomuk as ontology, SPARQL as query language, SPARQL UPDATE for inserting and updating data and finally signals-per-classes as life search capability. The DBus APIs are simple but very powerful thanks to SPARQL. -- Philip Van Hoof, freelance software developer home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org http://pvanhoof.be/blog http://codeminded.be ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: New module proposal: tracker
On Sun, 2009-11-08 at 20:43 +, Iain wrote: Tracker 0.7 ships with a DBus API, Nepomuk as ontology, SPARQL as query language, SPARQL UPDATE for inserting and updating data and finally signals-per-classes as life search capability. The DBus APIs are simple but very powerful thanks to SPARQL. From looking at tracker again this weekend, and indeed it does work better than the last time I checked in June (modulo some apparent big bugs and some issues with the concept that I couldn't test due to the bugs) but from a application developer point of view, I'm still disappointed. Essentially, I feel the APIs are far too low level and the documentation is very very incomplete, which ended up leaving me wandering around in dark as to what I was meant to do. The Application Development Manual is useless (I know its not finished, but its no help at all). I know you guys love love love ontolgoies and SPARQL and predicates and all that, but honestly, as an application developer I couldn't care less. This is not what we love, this is simply the vocabulary used in RDF. I know a lot of people hate hate hate buzzwords and all that. But in the end it is actually important to agree on a vocabulary. This is that vocabulary. All I want is to get at the data I want in the easiest possible way, and looking at tracker and discovering that I need to learn yet another language and a whole new load of boring concepts to do that is very offputting. SPARQL, tuples, ontologies should be internal details that you shouldn't need to know about. You'll always have to learn how the things that you are working with look, what relationships with other things can be made, etc. It's the ontology that tells you this. You'll have to understand how to query it, especially if you want to formulate more complex questions about your data. That's SPARQL. Looking at tracker/src I see 7 things begining with libtracker- and none of them look overly helpful. The most useful looks like libtracker-client, but again its too low level. libtracker-client is really only for C developers who want to get some DBus infrastructure quickly at this moment. The API is the DBus API. We haven't yet made any higher level APIs, although we could probably do that. I personally think that SPARQL is, once you get it, so straight forward that any such high level API is simply not necessary. SPARQL is much more simple than SQL. What I would really like to see is some simpler high level calls that enable me to get all data of (for example) Music, allow me to specify what fields I want, and it should be a GObject that I can connect to signals on to find out when a result has been added/removed/changed. Sorry but, with DBusGProxy you already have a GObject that you can immediately connect a signal to and get informed when something gets added, removed and changed. http://live.gnome.org/Tracker/Documentation/SignalsOnChanges With DBusGProxy you also already have a GObject to make a SPARQL query. You just have one method called SparqlQuery that takes a string, and that returns an array of an array of strings. It can't get much more simple than that. Except that with libtracker-client you don't even have to think DBus anymore. We should be looking to make our APIs as simple and easy to use as possible to enable developers to make the applications without distracting them with irrelevant implementation details. If this already exists, I apologise and blame the lack of documentation. The libqttracker library provides this for Qt. We haven't yet had the time to make something as high-level for the GLib world. http://maemo.gitorious.org/maemo-af/libqttracker Libqttracker is what is being used to build various applications for the new Maemo platforms. It generates high level classes like MusicPiece and allows you to get a list of musicpieces, given a query. (Just an example). Libqttracker uses the Nepomuk ontology to generate itself. So it exposes quite an amount of types and classes, etc. -- Philip Van Hoof, freelance software developer home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org http://pvanhoof.be/blog http://codeminded.be ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: New module proposal: tracker
On Fri, 2009-11-06 at 20:24 +0100, Alexander Larsson wrote: I understand that this is not nice :(. But indeed, tracker-store isn't system-wide, instead it's per user. That won't work because you won't have access to the session D-Bus, and certainly wouldn't work with, say, 1k users with NFS home dirs. Not to mention LDAP/NIS or things like that. Its not what you want to do anyway. You want to ship the package with a set of indexes for its help, and then when doing the query for help you want to say search for this string in these given installed index files. A system-wide database for a user's service will come down to using ATTACH in sqlite (that we use internally to translate your SPARQL to). This works nice but the problem is that we'd need to do a UNION on the query to ensure that we involve the system-wide db's tables in it. We already cache the sqlite3_stmt objects so that's not the problem with it, but UNION of course does mean that the query gets executed twice. And because we don't know when to UNION or not, we'd always have to do this. Meaning that all of our queries would become two times slower. We could of course start supporting a light form of named graphs[1] and then we could based on the selected graph select data out of a system wide database instead. We currently have no plans for this, though. It would look like this: SELECT ?title FROM NAMED tracker:system-wide WHERE { ?a a gnome:ManPage ; nie:title ?title . } This kind of feature is of course something we'd have to decide as a team. Although I'd personally like it I can't promise that we'll implement something like this. But I'll bring up to issue of system-wide data versus deployment on large scale systems with thousands of users, in one of our meetings. Thanks for the use case. --- [1] http://www.w3.org/TR/rdf-sparql-query/#namedGraphs -- Philip Van Hoof, freelance software developer home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org http://pvanhoof.be/blog http://codeminded.be ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Module semi-proposal: gnome-shell
On Tue, 2009-11-03 at 09:18 +, Martyn Russell wrote: On 02/11/09 22:59, Jamie McCracken wrote: I dont want to comment too much on zeitgeist but AFAIK all querying, tagging and event logging can be done in the tracker DB. A tracker miner could be created to perform such logging although it seems the zeitgeist team want to use a python daemon instead. Hrrm, I need to read this ML more often :-\ I personally would like to merge zeitgeist functionality into tracker but really its up to the zeitgeist team on whether they are willing to do this (it would mean converting the python code to Vala/Genie or C as well) Why is that necessary? It should work with Python AFAIK. Yes, I was also wondering what Jamie is talking about here. It's perfectly fine to write a 'miner' for Tracker in Python. IF performance is any problem, parts can be rewritten in another language, sure. Such 'miners' don't necessarily have to go into Tracker's repository and project. We want to get as much as possible of them into the project and grow a big family of contributors and side-kick projects. Sounds like a lot of fun to me! A good maintainer is what matters, not the language. We're for example reviewing and working on Adrien Bustany's web miner projects (facebook, flickr, etc) at this moment. Adrien is using Vala for this project. You can follow up this progress here: http://git.gnome.org/cgit/tracker/log/?h=miner-web Hopefully we'll someday have a miner-zeitgeist branch like that, too. Maybe some parts will be done in C, maybe in Vala, or Genie? And since a miner just talks DBus and SPARQL UPDATE with tracker-store, it can also be done in Python. That's fine. At least for me. Maybe somebody should write a 'miner' in JavaScript someday? If that person maintains it well, then that should be fine too. As several people on this thread said before: all these language discussions are bikeshedding. That's true for gnome-shell and for Tracker too. As for storage for the events: At least I wouldn't mind storing a compressed set of events (a cleaned up table) coming from Zeitgeist, in Tracker's stores. It would mean that we need to make a ontology for it. We can do that. Then the Zeitgeist project, and other apps too, can query it over SPARQL. And if SPARQL isn't sufficient we can add functions to the language under a 'zeitgeist'-namespace, for example. If the events are stored outside of Tracker's stores then it'll be more difficult to integrate the fields in an existing SPARQL query. What me and Seif discussed IRL as a possibility a few times is to let Zeitgeist have its own massive store full of all the events, but then synchronize some of the events to Tracker's store (for example a storable conclusion, one that has a 'real query'-usecase). -- Philip Van Hoof, freelance software developer home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org http://pvanhoof.be/blog http://codeminded.be ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: New module proposal: tracker
On Wed, 2009-08-19 at 11:25 +0100, Alan Cox wrote: Hey Alan, thanks for your questions. The tracker-store is a desktop service that offers the application developer a query capability against data that it stores. The data that it stores must be strictly defined by a schema (which is what in RDF is called an ontology). The schemas that we ship by default are the Nepomuk ones. The query language is SPARQL. The service provides the opportunity to the application developer to store. The application developer uses the an extension to SPARQL, SPARQL Update, which we support too. The communication between application and tracker-store happens over DBus. Nepomuk's ontologies: http://www.semanticdesktop.org/ontologies/ Broken link btw at: http://www.semanticdesktop.org/ontologies/ for: http://dev.nepomuk.semanticdesktop.org/repos/trunk/ontologies/pimo/latex/pimo.pdf I will ping upstream Nepomuk maintainers about this problem. Thanks. Note that Tracker at this moment doesn't do the PIMO ontology: we think it's too complicated. This has not been finally decided yet, though. Let me know if that was a helpful description for you. I tried hard not to sound like an old German philosopher ;-). One thing I couldn't quickly tell is whether you are always remembering the source of external information, particularly any externally acquired personal information about someone that is stored in the database. To store the source of external information we need to do so called named graphs (which is a RDF buzzword for namespaces). Right now we don't do named graphs. We are in discussion about adding support for it (actually, a few minutes ago we were), but given the direction the discussion I don't think this feature will make it for the first 0.7.x development releases. More information about using named graphs with SPARQL queries can be found here: http://www.w3.org/TR/rdf-sparql-query/#namedGraphs Our current implementation simply ignores any 'FROM NAMED' parts. That may be important for business users who have to meet data protection/personal information rights legislation. Ditto that tracker doesn't start extracting and organising by anything like religious, medical or ethnic data whose processing is controlled in many countries. Tracker will store this if the applications request storage of it. The issue of protecting the user's personal data is left to the applications using it and the underlying operating system's security features. The database's file will of course have the right UNIX permissions set. -- Philip Van Hoof, freelance software developer home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org http://pvanhoof.be/blog http://codeminded.be ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: New module proposal: tracker
On Wed, 2009-08-19 at 13:07 +0100, Alan Cox wrote: One short coming in this approach will be, It will cause a problem where multiple applications can be associated with a file-type, over a period of time. For instance, for .mbox files, the applications could vary like: Evolution, Mutt, Pine, Claws, Thunderbird, etc. And it is common among some people to switch between applications; not for email but other applications like PDF-viewer, etc. once in few months. This requires some commonality about indexing and the meaning of concepts. There isn't anything wrong with several apps indexing the one file (preferably at the same time so we walk the filestore once). A more interesting problem is heirarchical breakdowns (a multipart mime email of a zip holding a pdf and a jpg file) or xml documents with multiple namespaces in use. The libstreamanalyzer library is ideal for this. I opens a file and then starts reading it using what they call a stream. When they reach a point in the file that can be recursed (like a zip file in a zip file) then they open a stream on top of the root stream and recurse into it. Tracker's FS miner is integrating with libstreamanalyzer for extraction. The libstreamanalyzer library is originally developed by and for KDE's Strigi project by Jos Vandenoever. We're of course in discussion with Jos about various things. We realized that their method of extracting metadata is far superior compared to our more simple FILE and fread() based extractors. The MBox example is a good one for this: a Base64 encoded image/png attachment in an E-mail that can be found somewhere deep in a large MBox file ... can have Exif tags that are indexable (when Base64 decoded). Surely you don't want to Base64 decode all the attachments in the MBox file to files in /tmp, and then extract those file's Exif tags? (well, that's what Tracker did for its Evolution support). Instead, you want to lay a Base64 decoder stream over the root stream for the MBox file, and then analyze the image/png image that comes out of the Base64 decoder stream. That's what libstreamanalyzer does. subject is the metadata etc. So every time the user switches applications, the earlier collected meta-data might need some brushup. That assumes that the old meta data is somehow wrong. When an office changes staff the way stuff is indexed may change a bit but the old index doesn't become invalid or useless. Right many sites exist. For desktop the scale of the things is less, individual application-provided-search is enough and will satisfy the needs of most of the users. ctags, mairix etc. can provide specialized and more effective searching. The notion that the internet and personal file store are separate is one I would question. Exactly. I think RDF metadata stores can be instrumental in bringing the web and the desktop closer together. Which is among our goals. Why for example would I not be running a query across my personal email and a company wide accumulated metadata source of all the internal public mailing lists. This would be possible if we'd first develop a protocol for doing remote queries. Again a long term goal that might not even be part of Tracker itself (we can easily proxy DBus over some TCP/IP or even UDP service). You can probably imagine that this would require things like security policies too? ;-) We wouldn't want random people accessing your data. Of course. Specialized searching is also very different to general contexts. It is better at the one job but cannot answer random queries or associations. /home is a place where you keep stuff nobody else needs, or you want fast access to, or you particularly don't want other people to have access to. Indeed if you backup to an internet connected server its not unreasonable to argue that user filestore is simply a cache, nothing more. Thanks for your input, Alan. It's very helpful. -- Philip Van Hoof, freelance software developer home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org http://pvanhoof.be/blog http://codeminded.be ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: New module proposal: tracker
On Tue, 2009-08-18 at 17:21 +0200, Lennart Poettering wrote: On Tue, 18.08.09 11:18, Jamie McCracken (jamie.mccr...@googlemail.com) wrote: The indexer part is optional The main part tracker-store is just a database with querying and is to be used by zeitgeist If the consensus is that indexer is not suitable for inclusion then the separate tracker-store should be considered for inclusion separately the store does not do any indexing or file monitoring nor does it cosume significant resources I have no idea what tracker-store is. Please elaborate. It sounds as it was the database that is normally filled by the indexing data, but what could it be good for if you rip out the indexer? The tracker-store is a desktop service that offers the application developer a query capability against data that it stores. The data that it stores must be strictly defined by a schema (which is what in RDF is called an ontology). The schemas that we ship by default are the Nepomuk ones. The query language is SPARQL. The service provides the opportunity to the application developer to store. The application developer uses the an extension to SPARQL, SPARQL Update, which we support too. The communication between application and tracker-store happens over DBus. Nepomuk's ontologies: http://www.semanticdesktop.org/ontologies/ SPARQL: http://www.w3.org/TR/rdf-sparql-query/ SPARQL Update: http://jena.hpl.hp.com/~afs/SPARQL-Update.html RDF: http://www.w3.org/RDF/ Let me know if that was a helpful description for you. I tried hard not to sound like an old German philosopher ;-). -- Philip Van Hoof, freelance software developer home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org http://pvanhoof.be/blog http://codeminded.be ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: New module proposal: tracker
On Tue, 2009-08-18 at 17:14 +0100, Martyn Russell wrote: On 18/08/09 17:11, Colin Walters wrote: On Tue, Aug 18, 2009 at 3:32 PM, Jamie McCrackenjamie.mccr...@googlemail.com wrote: we could use the Gtk Recent files stuff for this and that would work for ordinary users but not devs fetching source code or other command line stuff Unless it's really REALLY compelling and fast, I don't want my source code in any kind of database, at least by default. We should leave this to IDEs. The important files to index are word processor documents, graphic files, PDFs, web browser downloads. Recent files could be used for this with a bit of work (i know Firefox would need a patch at least). Yea, we share the same sentiments. I think I will just go ahead and do this then, it seems a common thought amongst developers. I guess we could indeed make a tracker-miner-gtk-recent Since last few months we have been working on cleanly separating the tracker-miner-* from the tracker-store. The tracker-miner-fs is just one such miner. The separating of store and miner is fully completed now. Adrien Bustany, a summer of code student, has been working on miners for Facebook and Flickr this summer. We started discussing and integrating his work this week. A miner like tracker-miner-gtk-recent wouldn't use inotify or any other filemonitoring, but only the gtk-recent library. Adrien Bustany's summer of code project: http://git.mymadcat.com/index.php/p/gsoc/source/tree/master/ -- Philip Van Hoof, freelance software developer home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org http://pvanhoof.be/blog http://codeminded.be ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: New module proposal: tracker
On Tue, 2009-08-18 at 17:36 +0200, Maciej Piechotka wrote: [CUT] Hmm. Is it one-in-all database? Then: - How you keep it out of corruption. Hardware and software errors happens and sometimes one lost files. If it is one file - ok I can live with it. If it is one-in-all file - ops (and please note that average user does not make backup). We have something in mind for this where we'll have a automatic periodically made backup of all of the user's data, plus a journal of not yet backupped such data. This indeed means redundantly storing, but the redundant store ensures that data-loss is virtually impossible (only when both databases get corrupted simultaneously). For that last rather unlikely event, we'll also have normal backup features. The format of that backup will be Turtle, which is a humanly readable w3 standardized format for storing RDF triples. We have also plans to separate indexable data from user unique data, and store the two kinds of datas in different locations (.cache and .local). - Some fs makes operations with small files much more efficient then bigger (reiser* for example). It may have performance impact. We use sqlite which has its own internal performance optimizations that work mostly unaffected by such FS specifics. The FS can of course always have an influence. Use a FS that makes sense, if it does (many apps use sqlite nowadays, any FS that disses sqlite is going to perform badly for a lot of desktop applications. Including Firefox and Evolution, for example). - What with concurrent access? Currently we give priority to read queries and we queue write queries. Read queries are always solved immediately. We don't yet support having two concurrently running read queries (we want to avoid usage threads in tracker-store as much as possible, everything happens on the GMainLoop at this moment). It's not decided that this wont ever change. The Turtle format: http://www.w3.org/TeamSubmission/turtle/ -- Philip Van Hoof, freelance software developer home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org http://pvanhoof.be/blog http://codeminded.be ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: New module proposal: tracker
On Tue, 2009-08-18 at 17:50 +0200, Steve Frécinaux wrote: Xan Lopez wrote: It can be used directly by applications that feed it data through an API. Zeitgeist is an example, another could be bookmarks/history storage in Epiphany. Do you mean storing actual bookmarks into the database ? I would be quite reluctant in mixing actual data with rebuildable caches The tracker-store is not only about rebuildable caches. Again, tracker-miner-fs is just one data provider, like any other. The Tracker project is not only about tracker-miner-fs, and Tracker's purpose is not only to cache rebuildable data. (because when caches get in the way people are eager to just rm them). The unique user data in Tracker isn't cache. An indexer mostly produces data that you can build again -- by running the indexer once more. Tracker's purpose is not only to cache rebuildable data. On which field does tracker-store play ? The tracker-store is a desktop service that offers the application developer a query capability against data that it stores. The data that it stores must be strictly defined by a schema (which is what in RDF is called an ontology). The schemas that we ship by default are the Nepomuk ones. The query language is SPARQL. The service provides the opportunity to the application developer to store. The application developer uses the an extension to SPARQL, SPARQL Update, which we support too. The communication between application and tracker-store happens over DBus. Nepomuk's ontologies: http://www.semanticdesktop.org/ontologies/ SPARQL: http://www.w3.org/TR/rdf-sparql-query/ SPARQL Update: http://jena.hpl.hp.com/~afs/SPARQL-Update.html RDF: http://www.w3.org/RDF/ Let me know if that was a helpful description for you. I tried hard not to sound like an old German philosopher ;-). -- Philip Van Hoof, freelance software developer home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org http://pvanhoof.be/blog http://codeminded.be ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: New module proposal: tracker
On Tue, 2009-08-18 at 19:57 +0200, Andre Klapper wrote: Am Dienstag, den 18.08.2009, 18:05 +0100 schrieb Martyn Russell: On 18/08/09 17:48, Dan Winship wrote: The stigma associated with Tracker is because historically the tracker hackers have been unable to describe what the non-filesystem-indexing part of Tracker does without ok, not good. I agree claiming it solves all possible problems involving data. I don't think I have heard any Tracker developers state it solves all problems involving data. Who, when, where was this? When applying for inclusion for GNOME 2.18 lots of followup mail was just about requesting an exact understandable explanation on what tracker does, and what not. ok, let's avoid that this time The answers were partially confusing and sometimes considered vague and/or inconsistent. To quote jdub from that thread: Great technology loses out to poor communication. Hoping that communication works out better this time... If it doesn't this time, please advise us on how we can be more clear. We'll do our best and are committed to formulate our answers in a non-vague way and improve the communication of the project's members, about the project, towards the community. -- Philip Van Hoof, freelance software developer home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org http://pvanhoof.be/blog http://codeminded.be ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: New module proposal: tracker
On Tue, 2009-08-18 at 19:21 +0100, Michael Meeks wrote: [CUT] Unfortunately, as soon as we have this, it is only a small feature-creep step to lets index all .c/.h files to extract comments in the API documentation - which (I suspect) then commits you to the disaster of irritating a lot of developers - so they turn it off, and getting bogged down indexing things no-one is ever going to want indexed by tracker (?). We can also offer a way to configure very precisely what it is that you want, and what it is that you don't want to mine or index. And then we could on the GNOME desktop install a very sane default of that configuration. Technically ignoring certain directories, or detecting a condition that means that we should ignore it or not, or ... ain't very hard. We already have quite a bit of configurability for this, but we can make it a point of action at the Tracker project to ensure that this becomes even more flexible, and we can help develop that sane default for GNOME. Personally, I'd start by ignoring any directory tree with a configure* script in the top-level, or perhaps a .git / .svn directory - that should reduce the inotify pain :-) Yeah, stuff like that ... So - my point is: are the devs fetching source code at the console - that you are concerned about above, really in the target audience for tracker ? and if so why ? inevitably they will use ctags or equivalent - ie. a tight, generic end-user focused scope is prolly a win. Those devs can change the sane default of the GNOME desktop. -- Philip Van Hoof, freelance software developer home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org http://pvanhoof.be/blog http://codeminded.be ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: New module proposal: tracker
On Tue, 2009-08-18 at 20:26 +0200, Vincent Untz wrote: Le mardi 18 août 2009, à 20:19 +0200, Philip Van Hoof a écrit : We'll do our best and are committed to formulate our answers in a non-vague way and improve the communication of the project's members, about the project, towards the community. Maybe just clearly state what tracker (or tracker-store, the thread already lost me :/) will bring to GNOME if integrated. I don't want to hear about ontology, sparql, data store, indexer, or whatever. I want to know what it will bring me as a user, and what opportunity it gives me as a hacker, for my modules. So, yeah. Just list use cases. (Somebody already gave a few examples in a mail, iirc, but it got lost in the noise for me). Ok use cases.. Note that it's quite a lot harder to communicate these in a non-vague way. I'll put the more concrete ones on top. We are talking with Nokia's Telepathy team about a {I'm not allowed to say the word, I'll call it a schema then} for Instant Messaging. This would make tracker-store usable to implement all the use cases Andre Klapper blogged about last week: http://blogs.gnome.org/aklapper/2009/08/14/wishlist-evolution-instant-messaging-integration/ Other use cases would be to let applications share bookmarks, share the Photo library, share the Music library. All applications would also know about tags that other applications have set on those resources. For example, but not only, Nautilus' tags. It would also be possible to accept another person's user metadata and import it. Fetch it from existing webapps like Flickr, Facebook, Youtube, etc (again, but not only, for example the tags set by other people). Those are all very concrete use-cases. The ones that are either already or almost implemented as we speak. Another perhaps more abstract set of use-cases would be the history behind a contact. Not only would that history display the chat log, but also the E-mail log, the tags that you share with the person (and the resources that the both of you tagged with them), Photos that are relevant to that person, Music that is related, etc. I'm just listing possibilities of course .. A more recent use-case is Geo location tagging. We make it possible to let them set and let applications use Geo location information on any kind of resource. For example the GPS coordinates of where a photo was made, but also the last known location of a contact. Surely this is already possible, but the question is how usable is it for applications that don't deeply integrate with the libraries that make this possible? We want this information to be usable, everywhere. For example, say I was at some location and I had a piece of music that I liked very very much. I mark it as super good! on my mobile device. That mobile device could now store the Geo location on top of the Rating that I've just set. Now we can make an application that will tell you the location where you were when you listened to that super-good piece of music. Or however silly-crazy you want to make it. Point is that it should be up to the application developers, you know those young new guys who have the crazy ideas. Technology should help them, not block them, with their craziest ideas. -- Philip Van Hoof, freelance software developer home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org http://pvanhoof.be/blog http://codeminded.be ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: New module proposal: tracker
On Tue, 2009-08-18 at 21:06 +0200, Lennart Poettering wrote: On Tue, 18.08.09 18:43, Martyn Russell (mar...@lanedo.com) wrote: So you are telling me file system notification does not matter much for Tracker's usefulness? That's news to me indeed. But of course leads to the question: what is it then what it offers that desn't need file system notification? Please explain. Well, for a start, we don't index . prefixed directories, so for applications that want to store their bookmarks, contacts, etc they can use tracker for that. The Nepomuk ontology describes all of that and how the data relates. Those updates do not come from file system updates, they come from applications sending us data. I guess a problem here is that nobody except the Tracker people themselves even know what an ontology or Nepomuk actually is. Just throwing around buzzwords that nobody understands doesn't really help dumb people like me actually get a grip on what tracker actually is, and help them understand that it isn't what they thought it was. You could say the same about pulseaudio ;) Not sure this really compares that well... I didn't really play buzzword bingo that much, I guess. I often use words like 'glitch-free' and stuff, but at least I wrote a couple of blog articles about it, explaing what is supposed to be. Still waiting for a really good explanation on planet gnome what ontology or nepomuk was. Sorry to disappoint you in this reply of mine, but here you are either being intellectual dishonest or you are saying something that you don't know a lot about because you decided to ignore the posts that you just claimed don't exist. We have been writing quite a lot on the subject. Ivan has been writing on the subject, Martyn has written a blog post about it, Jürg recently did, ... I also wrote two tutorials (and these got blogged to pgo too): http://live.gnome.org/Tracker/Documentation/AppDevelopersManual/IntroductionToRDFandSPARQL http://live.gnome.org/Tracker/Documentation/AppDevelopersManual/AdvancedRDFandSPARQL Let's at least keep the discussion fair and professional here. It doesn't matter that you explain to everyone and his dog what tracker is about, but it would be good if at least the desktop-devel folks would understand what it was about. And as I read the comments here I am not the only one who doesn't. Two parts: tracker-store and tracker-miner-fs It appears that you know what tracker-miner-fs is about. Let me know if this isn't accurate then I will repeat a description one more time. I am going to repeat this one: The tracker-store is a desktop service that offers the application developer a query capability against data that it stores. The data that it stores must be strictly defined by a schema (which is what in RDF is called an ontology). The schemas that we ship by default are the Nepomuk ones. The query language is SPARQL. The service provides the opportunity to the application developer to store. The application developer uses an extension to SPARQL, SPARQL Update, which we support too. The communication between application and tracker-store happens over DBus. Nepomuk's ontologies: http://www.semanticdesktop.org/ontologies/ SPARQL: http://www.w3.org/TR/rdf-sparql-query/ SPARQL Update: http://jena.hpl.hp.com/~afs/SPARQL-Update.html RDF: http://www.w3.org/RDF/ Let me know if that was a helpful description for you. I tried hard not to sound like an old German philosopher ;-). (but then again, this is of course a matter of perspective. Maybe I never did a better job explaining what PA is about then you did with explaining what Tracker is -- but then again I never tried to make PA a module of GNOME...) This is not about PulseAudio. Let's stop that thread before it goes ridiculously off topic. -- Philip Van Hoof, freelance software developer home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org http://pvanhoof.be/blog http://codeminded.be ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: New module proposal: tracker
On Tue, 2009-08-18 at 21:43 +0200, Steve Frécinaux wrote: Philip Van Hoof wrote: SPARQL: http://www.w3.org/TR/rdf-sparql-query/ Are people actually intended to use that to get informations from the tracker gui ? No, did somebody tell you that? If you are an application developer then yes, you should read that. Speaking of which, is there a gui or something ? Tracker-explorer is a visualisation tool to browse the relationships http://git.gnome.org/cgit/tracker/tree/src/tracker-explorer Tracker Search Tool is being re-developed by Jamie at this moment, to cope better with the more recent changes. But the Tracker project's purpose is not really to provide a key user interface for what it is storing and how. -- Philip Van Hoof, freelance software developer home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org http://pvanhoof.be/blog http://codeminded.be ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: New module proposal: tracker
On Tue, 2009-08-18 at 21:43 +0200, Maciej Piechotka wrote: On Tue, 2009-08-18 at 15:12 -0400, Jamie McCracken wrote: On Tue, 2009-08-18 at 20:57 +0200, Lennart Poettering wrote: Epiphany does not share its metadata with anyone else nor can you cross query it I believe that gnome do and/or deskbar do it quite well w/out tracker. Let's ask the GNOME Do and Deskbar people about this? Would a system like Tracker help the projects? 2) Zeitgeist integration of events with said search and query capabilities Hmm, I cannot really parse this either. 3) Centralisation of all tags (nautilus, zeitgeist, fspot. facebook/flickr etc). No need to duplicate tagging in different apps Hmm, why should this be in tracker? not in gvfs? flickr/facebook Sorry. But I don't have English parser good enough to parse it. I'm afraid that just like Lennart you aren't trying to parse it. And therefor is Jamie right that it's not productive to go on about this. To end it the way the person you are copycatting ends his sentences: seriously. As far as understood tracker is designed to be something like WinFS. So I operate in terms of peoples, tags and objects instead of folders and files. So user do not think Wait, where I get the information when I meet Joe's to drink a beer or two but I search message from Joe containing meet which gives back all messages from Joe containing meet. Free text search is just one possible way to get such a list. The relationships between resources is going to be the most common way to access related resources. You wont necessarily search, you will select Joe in a list and then you'll provide what it is about Joe that you want (E-mails, Friends, Photos, Locations, Documents, Chat logs). That's not the same as typing in a search term. Although free text search can be utilized to further fine tune a result-set. But not usually as the primary or only criteria entered by the user. Note that it's rare that you have zero context about what the user expects to see next. I guess the only use-case for that is google-com. Which is a nice use-case, but most definitely not the only one. And even for google-com is the context: I want web resources that aren't private but will interest me, if I'm logged in. That's not zero-context. Or - I've seen a picture of myself and my schoolmate/workmate/soulmate etc. in some situation (school trip, conference etc.) but where it was. I enter picture of (me and Alice) tagged London - and here it is - it was posted on Facebook/Flickr (is in epiphany cache?). Sure Cool - for sure. Useful - I don't think so - however I may be wrong. I guess that we should wait until 0.7 release and have it as optional dependency or set of plugins. If it is good enough - we should include it. ok Thanks for your opinion. -- Philip Van Hoof, freelance software developer home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org http://pvanhoof.be/blog http://codeminded.be ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: New module proposal: tracker
On Tue, 2009-08-18 at 23:33 +0200, Sebastian Pölsterl wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Philip Van Hoof schrieb: On Tue, 2009-08-18 at 21:43 +0200, Maciej Piechotka wrote: On Tue, 2009-08-18 at 15:12 -0400, Jamie McCracken wrote: On Tue, 2009-08-18 at 20:57 +0200, Lennart Poettering wrote: Epiphany does not share its metadata with anyone else nor can you cross query it I believe that gnome do and/or deskbar do it quite well w/out tracker. Let's ask the GNOME Do and Deskbar people about this? Would a system like Tracker help the projects? Sure, parsing the epiphany bookmarks file over and over again is not a good solution, but it does the job. Users usually don't care *how* something appears on their screen as long as it actually does. So you're saying that a system like Tracker would indeed help a project like GNOME DO and Deskbar? ok. Depending on tracker only to retrieve epiphany bookmarks in a more sane way is not going to happen. That's of course not going to be Tracker's only purpose. The tracker store may be more useful to deskbar if more applications would actually use it (e.g. evolution, f-spot, rhythmbox, banshee). But as I see it, the ideas of tracker may be great, but far to less applications use it so it could be worthwhile implementing tracker-store support. ok, The fact that applications don't yet use it is a chicken and egg problem on all counts: The Tracker team members have been too busy working on Tracker's own problems to focus on other applications. Besides, Tracker was not ready API and feature-wise either. Although we did squeeze into our personal TODO lists writing a EPlugin for Evolution that pushes E-mail data into us, and the same for KMail (because yeah, we try to take into account the K desktop too), for example. Application maintainers ain't going to just-like-that start accepting patches that will drastically change their data models to use Tracker as storage backend. Not unless they are sure that the way we'd change it is also the way forward for, for example, GNOME. So sure there are a lot of ramblings about us having to adapt all of GNOME's desktop to start using it, but I bet that even if we'd start making patches we'd still get as reply from many of the maintainers of the individual applications: sorry, but we can't depend on Tracker. And they can't depend on Tracker, because Tracker can't go into GNOME. And it can't go into GNOME, because applications ain't using it yet. That's a recursive chicken and egg ;-) Nonetheless, you'll start seeing such patches happening anyhow. Another problem non of the desktop search engines could solve for me is, to provide a way to find out how one result/item can be opened. We have a schema (an ontology, but I'm truly and extremely shockingly deeply frightened to use this word, as apparently people claim it to be a buzzword) for software. You can already in Tracker say which mimeTypes are connected to which softwares. So you can in fact already make the query with which software do I open this resource. I'm not going to type it here because that would just inspire more buzzwordfobists. But join #tracker and I'll show you. You can of course also just keep a table of these, yourself. For deskbar that's a crucial point. It doesn't help at all if I can ask tracker to search for xyz, but don't know how to handle the results. Sure, ok. This is a trivial problem to solve and can already be done ^. To solve this dilemma we currently hard coded the commands to open specific types of results in the beagle module (don't know about the tracker module, it isn't officially part of deskbar anyway). I'd love to get rid of such hacks. Okay, that's fair. Let us know how you need this, what you need and how we can help your application with this. So unless you provide patches/plugins for more applications to fill tracker-store and even more importantly patches for applications that consume this data, it's -1 from me. That's again the chicken-and-egg problem. And again will you nonetheless see these patches happening. But this is a process that takes long. Even if we'd have the patches for ALL applications ready by *next minute*, then still it'll take months for the maintainers of those applications to make up their minds about it. And after that months to rework the patch, as upstream will have massively changed and the patch wont apply anymore. Etc. -- Philip Van Hoof, freelance software developer home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org http://pvanhoof.be/blog http://codeminded.be ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: New module proposal: tracker
On Tue, 2009-08-18 at 18:16 -0400, Colin Walters wrote: On Tue, Aug 18, 2009 at 6:01 PM, Philip Van Hoofs...@pvanhoof.be wrote: So sure there are a lot of ramblings about us having to adapt all of GNOME's desktop to start using it, but I bet that even if we'd start making patches we'd still get as reply from many of the maintainers of the individual applications: sorry, but we can't depend on Tracker. Well, as a maintainer of a different not yet fully baked technology piece, I think it'd be reasonable to get patches in some applications as optional dependencies. As long as you can explain to the application authors why it's valuable and why they should maintain the code in the future. Since you're a DBus service just saying if (org.freedesktop.Tracker isn't on the bus); return isn't that hard. You're right about that. And that's probably also how we'll try to make these patches. I must warn, though, that most of the members of the team have a tight schedule (there's indeed a customer for this, which I don't think is a big secret nor do I think there's anything wrong with that). I hope that this discussion made contributors interested. Ideally they'd start a patch and then we pick it up, improve it, make it ready, support it at the application's mailinglist, etc. Equally ideally a maintainer of an application gets in touch with us and discusses the possibilities. It's going to have to be something like this. But we are openminded, and if you ask on the #tracker mailing list, I'm sure you'll hear that we are helping interested people a lot. Sometimes too much, as we can't always focus on our actual daily tasks if we have to answer a lot. -- Philip Van Hoof, freelance software developer home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org http://pvanhoof.be/blog http://codeminded.be ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Tracker, Zeitgeist, Couchdb...where is the problem ?
On Wed, 2009-08-19 at 00:27 +0200, Rodrigo Moya wrote: On Tue, 2009-08-18 at 16:48 -0400, Matthias Clasen wrote: I think this recent discussion about tracker as a gnome module is somewhat backwards. I don't think it is leading us anywhere to talk about ontologies and rdf and events and timelines and metadata stores and kernel apis before we answer the first question: What is the user problem that we are solving here ? Can that be described in a paragraph ? And if it can, is it something that a 'regular' user would recognize as a problem he has on his computer ? Once we have the problem scoped out, we need to look at the user experience we want to aim for in solving it. Will it be a single search-for-everything dialog ? A query language ? Tagging everywhere ? After that, it might be possible to evaluate whether tracker, zeitgeist, couchdb or something else can be part of the implementation... couchdb provides just the storage of any kind of data, no indexing, searching, etc, so I think they solve different problems. Agree In fact, tracker could just use local files as storage or a couchdb database. If using couchdb, it would get replication and synchronization for free, but it would still provide the indexing We were recently discussing using CouchDB as a possible backup endpoint for Tracker, and/or as a redundant storage location for user-unique metadata. The stores to CouchDB would in latter case be synchronous (not as a result of a backup command, but immediate). With user-unique metadata I mean the non-reproducable / not-cache data: the data that is set by the user and therefor uniquely stored. For example the tags on resources that have no physical representation on your filesystem. (Contacts and tags, for example, don't necessarily have a physical representation on your FS). We evaluated CouchDB as a primary store over sqlite, but CouchDB lacked *very* important features. This makes it undoable. Feel free to get in touch with us to discuss which precise features I mean. -- Philip Van Hoof, freelance software developer home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org http://pvanhoof.be/blog http://codeminded.be ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Call for talks - GNOME Devroom at FOSDEM 2009
I was asked by some of the other Tracker developers to do a talk about Tracker. Not sure about the format yet, but feel free to keep a slot open if you still have one, for Tracker :) On Tue, 2009-01-06 at 22:20 +0100, Christophe Fergeau wrote: Hello :) 2008/12/30 Christophe Fergeau cferg...@gmail.com: As for the last few years, we'll have a GNOME devroom at FOSDEM (7/8 feb in Brussels), and as always, we want *YOU* to give a talk about the cool project you are hacking on in this devroom [...] Please send your talk proposals before Friday 9 January. As always, the sooner is the better wrt organisation. This is a reminder that the deadline is in just a few days, FOSDEM is 3 weeks earlier than usual this year which is why the deadline is so close. So if you want to give a talk in the gnome devroom or the freedesktop devroom (on sunday), please get in touch with me as quickly as possible. Otherwise, your talk won't appear on the printed schedule and people might not be aware of it :-/ Thanks a lot, hope to see you there :) Christophe ___ gtk-devel-list mailing list gtk-devel-l...@gnome.org http://mail.gnome.org/mailman/listinfo/gtk-devel-list -- Philip Van Hoof, freelance software developer home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org http://pvanhoof.be/blog http://codeminded.be ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Thumbnailer specification: Move, Delete, Copy hints (Was: Re: GSD should not housekeep the thumbnails)
(resend, as I was not subscribed on this mailing list) On Tue, 2008-09-16 at 13:30 +0100, Rob Taylor wrote: Emmanuel Pacaud wrote: Hi, On mar, 2008-09-16 at 11:34 +0200, Stephane Delcroix wrote: Hey guys, The plugin does something like this: - clean the thumbnails older than MAX_AGE (default to 60 days) - clean the oldest thumbnails until the cache size is under MAX_SIZE (default to 64MB) I don't get why it work like this. Why remove a thumbnail because it's old ? Why would we want to limit the size of the thumbnail cache to an arbitrary value ? It depends on the disk size, and it will limit itself by the fact the size of the thumbnailed files is largely bigger than the thumbnail size. Wouldn't it be better to just remove thumbnail of files that don't exist anymore ? I guess when Philip's thumbnailer daemon is starting to get widely used, It could do the job of cleaning the cache. This would mean that you would no longer have thumbnails lying around for deleted files and it could also clean out thumbnails that haven't been *accessed* after a certain time period, which should give better performance. Related to this discussion ... adding Andrew den Exter in CC, as I had an interesting conversation with Andrew about this. So although the current Draft specification mentions a Move, Delete and Copy, according to Andrew (and I agree) this ain't necessary at all. http://live.gnome.org/ThumbnailerSpec A properly written thumbnailer, in modern days, would have a file monitor on each thumbnailed original and/or would initially scan all the thumbnails, read the exif info to get the original's location, and would do housekeeping fully automatically this way. I don't know whether or not I should keep the Move, Copy and Delete hints in the specification. I'm in indecision-mode about it. Important note that you want to make thumbnailer daemons activatable by DBus (with a so-called .service file) and that some implementers will want to shut their daemon down after n minutes of inactivity. Meaning that simple file monitoring wont be sufficient. The current infrastructure, that is not guaranteed to be running on every desktop (nor every tablet), are desktop search engines like but not only Tracker, Beagle, Strigi, etc ... these are doing file monitoring at large scales already. It would make sense for them to perform the house-keeping by giving hints to the thumbnailer like Move, Delete, Copy, etc. Which is why I think I should keep the methods in the specification, and explain that there's no requirement for the thumbnailer himself to do housekeeping (although it is allowed to do this automatically). But that there's a requirement to accept the hints. Not a requirement to process the hints, just to accept them, and an advise to process them. No desktop search engine and a wrong thumbnailer could indeed mean that neither of the two will make cached thumbnails get removed. Chicken egg, I don't like it, but I also don't know if I should require anything about this, by specification. Again, I'm in indecision mode about it. -- Philip Van Hoof, freelance software developer home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org http://pvanhoof.be/blog http://codeminded.be ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Thumbnailer specification: Move, Delete, Copy hints (Was: Re: GSD should not housekeep the thumbnails)
On Tue, 2008-09-16 at 13:30 +0100, Rob Taylor wrote: Emmanuel Pacaud wrote: Hi, On mar, 2008-09-16 at 11:34 +0200, Stephane Delcroix wrote: Hey guys, The plugin does something like this: - clean the thumbnails older than MAX_AGE (default to 60 days) - clean the oldest thumbnails until the cache size is under MAX_SIZE (default to 64MB) I don't get why it work like this. Why remove a thumbnail because it's old ? Why would we want to limit the size of the thumbnail cache to an arbitrary value ? It depends on the disk size, and it will limit itself by the fact the size of the thumbnailed files is largely bigger than the thumbnail size. Wouldn't it be better to just remove thumbnail of files that don't exist anymore ? I guess when Philip's thumbnailer daemon is starting to get widely used, It could do the job of cleaning the cache. This would mean that you would no longer have thumbnails lying around for deleted files and it could also clean out thumbnails that haven't been *accessed* after a certain time period, which should give better performance. Related to this discussion ... adding Andrew den Exter in CC, as I had an interesting conversation with Andrew about this. So although the current Draft specification mentions a Move, Delete and Copy, according to Andrew (and I agree) this ain't necessary at all. http://live.gnome.org/ThumbnailerSpec A properly written thumbnailer, in modern days, would have a file monitor on each thumbnailed original and/or would initially scan all the thumbnails, read the exif info to get the original's location, and would do housekeeping fully automatically this way. I don't know whether or not I should keep the Move, Copy and Delete hints in the specification. I'm in indecision-mode about it. Important note that you want to make thumbnailer daemons activatable by DBus (with a so-called .service file) and that some implementers will want to shut their daemon down after n minutes of inactivity. Meaning that simple file monitoring wont be sufficient. The current infrastructure, that is not guaranteed to be running on every desktop (nor every tablet), are desktop search engines like but not only Tracker, Beagle, Strigi, etc ... these are doing file monitoring at large scales already. It would make sense for them to perform the house-keeping by giving hints to the thumbnailer like Move, Delete, Copy, etc. Which is why I think I should keep the methods in the specification, and explain that there's no requirement for the thumbnailer himself to do housekeeping (although it is allowed to do this automatically). But that there's a requirement to accept the hints. Not a requirement to process the hints, just to accept them, and an advise to process them. No desktop search engine and a wrong thumbnailer could indeed mean that neither of the two will make cached thumbnails get removed. Chicken egg, I don't like it, but I also don't know if I should require anything about this, by specification. Again, I'm in indecision mode about it. -- Philip Van Hoof, freelance software developer home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org http://pvanhoof.be/blog http://codeminded.be ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: GSD should not housekeep the thumbnails
On Tue, 2008-09-16 at 13:36 +0100, Rob Taylor wrote: Patryk Zawadzki wrote: On Tue, Sep 16, 2008 at 2:30 PM, Rob Taylor [EMAIL PROTECTED] wrote: could also clean out thumbnails that haven't been *accessed* after a certain time period, which should give better performance. How do you do that when 99% of desktop PCs use noatime precisely for that - perfomance? ;) Don't use atime? The daemon knows when the thumbnails have been accessed because it served them. You are not required to ask the daemon each time. You can just first lookup the thumbnail path for an original, and if it's not there you could ask the thumbnailer to make it for you. And then it'll tell you when it's finished with that. That's also the recommended way. Calculating an MD5 is less expensive than doing a DBus method. Of course is this a task for a toolkit specific library that does thumbnailing for your application developers. -- Philip Van Hoof freelance software developer Codeminded BVBA - http://codeminded.be ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Thumbnailer specification: Move, Delete, Copy hints (Was: Re: GSD should not housekeep the thumbnails)
For the people new to the discussion about the thumbnailer DBus spec, here's[2] a working prototype that will compile on a typical desktop. If you have Gdk and GStreamer, it'll already make thumbnails for all supported formats in GdkPixbuf and for mpeg videos. More formats, if supported by GStreamer, will follow. Of course can specialized plugins be made. Those plugins can be ran either in-process with the prototype's daemon implementation, and as a separate process (if it implements being a specialized thumbnailer as defined by the Draft specification[1]) [1] http://live.gnome.org/ThumbnailerSpec [2] https://stage.maemo.org/svn/maemo/projects/haf/branches/hildon-thumbnail/daemonize With the prototype you can run plugins designed to be ran in-process with the prototype's daemon implementation as a standalone specialized thumbnailer. You use the plugin-runner[3] tool for that. [3] https://stage.maemo.org/svn/maemo/projects/haf/branches/hildon-thumbnail/daemonize/daemon/plugin-runner.c This prototype is designed for desktop neutrality. Although it depends on glib, gobject and dbus-glib, you can easily make plugins that avoid the dependency on GdkPixbuf or GStreamer. You can easily adapt the Makefile.am to either build or not build them. I will improve the code and will accept patches that make it even more desktop neutral (if it makes sense). If even that is not okay, the specification allows for competing implementations. It even allows for a desktop-search engine or a file manager to become an implementer of it. The spec is designed for this. I also want (and can[4]) help competitors with this as I believe that competition is a innovation-force. It's necessary and good to have a competitor, to drive innovation. I'll welcome it, strongly. I strongly believe that thumbnailing is something that will have different characteristics on a mobile compared to a desktop. It indeed make sense to have competing implementations. [4] During daytime I can help with coding tasks related to this. But people need to ask me, of course. The prototype also shows how you could implement a toolkit specific thumbnailer library that could be consumed by your application developers. [5] is a reimplementation of libhildonthumbnail.so that now uses the prototype daemon instead. [5] https://stage.maemo.org/svn/maemo/projects/haf/branches/hildon-thumbnail/daemonize/thumbs/hildon-thumbnail-factory.c Thanks On Tue, 2008-09-16 at 14:42 +0200, Philip Van Hoof wrote: On Tue, 2008-09-16 at 13:30 +0100, Rob Taylor wrote: Emmanuel Pacaud wrote: Hi, On mar, 2008-09-16 at 11:34 +0200, Stephane Delcroix wrote: Hey guys, The plugin does something like this: - clean the thumbnails older than MAX_AGE (default to 60 days) - clean the oldest thumbnails until the cache size is under MAX_SIZE (default to 64MB) I don't get why it work like this. Why remove a thumbnail because it's old ? Why would we want to limit the size of the thumbnail cache to an arbitrary value ? It depends on the disk size, and it will limit itself by the fact the size of the thumbnailed files is largely bigger than the thumbnail size. Wouldn't it be better to just remove thumbnail of files that don't exist anymore ? I guess when Philip's thumbnailer daemon is starting to get widely used, It could do the job of cleaning the cache. This would mean that you would no longer have thumbnails lying around for deleted files and it could also clean out thumbnails that haven't been *accessed* after a certain time period, which should give better performance. Related to this discussion ... adding Andrew den Exter in CC, as I had an interesting conversation with Andrew about this. So although the current Draft specification mentions a Move, Delete and Copy, according to Andrew (and I agree) this ain't necessary at all. http://live.gnome.org/ThumbnailerSpec A properly written thumbnailer, in modern days, would have a file monitor on each thumbnailed original and/or would initially scan all the thumbnails, read the exif info to get the original's location, and would do housekeeping fully automatically this way. I don't know whether or not I should keep the Move, Copy and Delete hints in the specification. I'm in indecision-mode about it. Important note that you want to make thumbnailer daemons activatable by DBus (with a so-called .service file) and that some implementers will want to shut their daemon down after n minutes of inactivity. Meaning that simple file monitoring wont be sufficient. The current infrastructure, that is not guaranteed to be running on every desktop (nor every tablet), are desktop search engines like but not only Tracker, Beagle, Strigi, etc ... these are doing file monitoring at large scales already. It would make sense for them to perform the house-keeping by giving hints to the thumbnailer like Move, Delete, Copy, etc. Which is why I
FWD: Specifying thumbnailers as a service
I started a discussion on a specification for thumbnailers here: http://lists.freedesktop.org/archives/xdg/2008-August/009834.html -- Philip Van Hoof, freelance software developer home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org http://pvanhoof.be/blog http://codeminded.be ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: [Evolution-hackers] Memory consumption and virtual machines
On Wed, 2006-07-19 at 10:58 +0530, Parthasarathi Susarla wrote: Hi all, I second fejjs thoughts. Note that fejj recently improved the patch by removing some ugly pieces of it. Fejj and me discussed (thinks like) the patch on IRC. Note that the patch also fixes a lot non-mmap related memory problems. Also i have been testing the patch for sometime now. Heres the inference: * The patch works in reducing the memory consumed during the initial startup of evolution. And it does a wonderful job of that. * The patch intends to fix the overall consumption of memory during the usage of evolution, and it does *not* do that. I kept evo running with this patch in for more than 8 hours and using Evo as i would use it regularly. The memory that gets saved by the mmap technique is never reclaimed by another (existing nor new) technique. Data that is made available using mmap, is effectively never again made available using another technique. This basically means that it's technically not possible that somehow the patch stops saving the memory that it saves. What it saves from the initial startup, it will always save. It puzzles me a bit how you blame the increase of memory Evolution shows you after eight hours, on the patch. How does it do that then? Evolution consumes a lot real memory when it receives new messages. What you are seeing after eight hours is the real memory being consumed for this new information. The exact same real memory is also being consumed without the patch, there's no difference here. There's no blame on the patch for that. The patch still (for existing information) saves real memory. However. When you'll restart, that new information will be converted to something that can be mmap()ed. Evolution will, after that restart, not consume as much real memory for the same information (that is now on disk and mmap()ed by Evolution). Me, the current Camel maintainer Varadhan and old Camel warrior Fejj are very aware of this. We have a few solutions in mind and will be working on implementing these ideas. The solutions will not require a restart of Evolution. The current patch requires this. This is not a technical impossibility. It's just something that isn't technically implemented by the patch yet. Mostly because tinymail doesn't need this restart (and the patch was initially engineered for the purpose of tinymail). (Reading of new mails, changing foldersblah! blah!) and more than a couple of hundred new mails later and modifying the summary file as many times we lose the memory gain we got initially. I *dont* have fancy graphs to display this information though - but i surely have the data and the necessary information i need. * mmap is *not* the solution to this problem - it helps to a certain extent. * Generating message list each we get a new message is bad enough - we dont want to reload the summary file each time. There's solutions for this. I still have not tested philips latest patches. I gather he has improvised on the older patches. Would report on the patches after i test them enough. -- Philip Van Hoof, software developer at x-tend home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org work: vanhoof at x-tend dot be http://www.pvanhoof.be - http://www.x-tend.be ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: [Evolution-hackers] Memory consumption and virtual machines
On Tue, 2006-07-25 at 21:03 +0200, Philip Van Hoof wrote: Me, the current Camel maintainer Varadhan and old Camel warrior Fejj are very aware of this. We have a few solutions in mind and will be working on implementing these ideas. The solutions will not require a restart of Evolution. The current patch requires this. This is not a technical impossibility. It's just something that isn't technically implemented by the patch yet. Mostly because tinymail doesn't need this restart (and the patch was initially engineered for the purpose of tinymail). Also note that cause of a few reasons involving stress on the side of the Evolution team (feature freeze and stuff like that), and me relaxing a little bit now, I've put the work on the patch temporarily on hold. The current agreement with the Evolution team and idea is to make a branch for evolution-data-server where the experiment will continue. The camel-lite-builder tool* will, however, build you a camel with the summary patch. You can even use the resulting camel with Evolution if you overwrite the so's with the so's that will be installed if you compile a normale evolution-data-server. You can of course also simply apply the patch on evolution-data-server (which is what camel-lite- builder will do for you) https://svn.tinymail.org/svn/camel-lite-builder/trunk -- Philip Van Hoof, software developer at x-tend home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org work: vanhoof at x-tend dot be http://www.pvanhoof.be - http://www.x-tend.be ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Some Vala ideas (Was: What about Embedded?)
On Fri, 2006-07-21 at 11:23 +0200, Philip Van Hoof wrote: On Fri, 2006-07-21 at 07:07 +0200, Jürg Billeter wrote: You might be interested in looking at Vala http://www.paldo.org/vala/ . It's not ready for production use yet but it's available for testing now and with feedback [hint ;) ] from interested developers I believe that we can get a very nice development environment for GNOME ready in relatively short time. Yes Jürg, I have been looking at your Vala stuff very recently indeed. And I indeed have to say it looks very interesting. I even think language binding code generators are not the best solution. Imo it should be handled by the languages themselves and fully automatically. Other than political, I don't see the *real* technical difficulties for something like that. Probably because Vincent asked us to keep on topic, Jürg replied me in private about his idea to have compiler plugins and a standardized interface description (like GIDL). I hope some more people with ideas will contact Jürg and share them. -- Philip Van Hoof, software developer at x-tend home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org work: vanhoof at x-tend dot be http://www.pvanhoof.be - http://www.x-tend.be ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Mac shipments up 12% [Was: focus!]
On Thu, 2006-07-20 at 13:53 +0100, Alex Jones wrote: We need to change the way people use their computers This is why I'm a fan of MacSlow and his lowfat experiments. I hope he's inspiring not only me to start doing equally cool exper- iments. And not just nice-looking OpenGL stuff. Also experiments that do other things in a completely non-conventional way. For me, his stuff is just not about candy or cute looks. His experimenting is extremely important: I know most experiments end up not being used. But a very tiny fraction of the experiments will define the future. Each experiment will contribute to the lessons we will learn. They will ignite new ideas. Let us, the community of free software people, play a role in defining that future. It's both a challenge and my invitation for the young guys that are now, today, reading this mailing list and planning to some day do cool things with our free softwares. -- Philip Van Hoof, software developer at x-tend home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org work: vanhoof at x-tend dot be http://www.pvanhoof.be - http://www.x-tend.be ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Mac shipments up 12% [Was: focus!]
On Thu, 2006-07-20 at 10:58 -0400, Havoc Pennington wrote: Calum Benson wrote: True up to a point, although MacOS in its various (and sometimes more usable than current) guises has been around since before any of those, and although you can't necessarily say it's outlived OS/2, BeOS, Workbench or GEM[1], it's certainly left them eating considerable dust, in terms of widespread adoption. I mentioned this briefly before, but I wonder what percentage of Mac's audience has never been a question of 'switching' but is the publishing/graphics market they have had from the beginning - that they created, that Microsoft never had - if it's a large percentage, the historical frequency of 'switching' starts to look even lower. So lets create our own market?! What about embedded? There's a huge audience waiting for us. To give you a number. A lot people know the Dreambox and its uses in te satellite world. It comes with a Linux kernel and has framebuffer support. If I would say .. a government (which has an insane current economic growth) is interested in bringing Digital Television over two-way satellite to a huge population. And if I would say that the Dreambox, being used in the setup, for could run an embedded version of GNOME, using the framebuffer support GNOME has. If just the right sales people at the companies involved (SES-Astra, Newtec Cy, SES-Global,ST Microelectronics [F], Thomson, NDS) would *KNOW* that GNOME is also about embedded, they certainly would consider it. Yes. Today, these people simply don't know that. I (a developer, a consultant, in the field) has to tell them that. What if I'm not there (for example at another company at another situation)? Do you want to know how many millions of users the software that we at Newtec are building, has to support (this is a per satellite count): ~30,000,000 per satellite for this specific government. That's 10x10 in a few days Jeff. Of course is our SATMODE implementation a demo at this moment. But go figure. All governments are looking at this Digital Television thing. Where's the GNOME marketing? When are going to SCREAM that our platform is totally into embedded? OLPC, PalmSource and Nokia are VERY nice references guys. Please use it! -- Philip Van Hoof, software developer at x-tend home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org work: vanhoof at x-tend dot be http://www.pvanhoof.be - http://www.x-tend.be ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Mac shipments up 12% [Was: focus!]
On Thu, 2006-07-20 at 17:27 +0200, Philip Van Hoof wrote: On Thu, 2006-07-20 at 10:58 -0400, Havoc Pennington wrote: So lets create our own market?! What about embedded? There's a huge audience waiting for us. To give you a number. A lot people know the Dreambox and its uses in te satellite world. It comes with a Linux kernel and has framebuffer support. Oh. And of course I only mentioned satellite business because that is what I'm dealing with at this moment. What about car navigation, music players, instant messaging and VoIP devices and telephones. That's just the obvious ones that exist TODAY already. Tomorrow fridges, household machines like vacuum cleaners. DVD players (or however it will be called). Recorders (DVD, same thing)?! They all need user interfaces. They all need menus. They all need human interaction. Where are we? Why aren't we helping businesses creating that using our framework? That market share is bigger than the desktop world Microsoft is dominating. And it's going to grow the next few years. Embedded is not just handhelds and PDA's. Yet even that market is going to grow. We're very lucky Nokia picked us. That is extremely good for GNOME, indeed. -- Philip Van Hoof, software developer at x-tend home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org work: vanhoof at x-tend dot be http://www.pvanhoof.be - http://www.x-tend.be ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: What about Embedded?
On Thu, 2006-07-20 at 14:46 -0400, Miguel de Icaza wrote: Hello, Indeed, I find it ironic that in light of recent moves to expand the Gnome tent to include Mobile and Embedded devices as at GUADEC this year, that there is at the same time an effort to push MONO into the stack. At what price are these moves being made or considered? Like Havoc said, innovation at the cost of performance and memory usage is not innovation in my book. Mono works just fine on embedded devices, and considering that it consumes less memory than Python when running Gtk applications and people do not have a problem using Python on embedded devices I do not see the problem. In fact. Considering that most of ALSO the application developers in the GNOME context are very good at *not* freeing up their memory, I actually but truly believe that running most GNOME programs under a garbage collector would actually save (a lot) more memory then the virtual machine would add. But that is of course just an assumption. Perhaps I shouldn't make such an assumption if I want to be a nice guy :-)? Nice guys tell you bed-time stories. They don't show you reality. So I run valgrind a lot. And suddenly it's not just an assumption anymore, but more or less a proven fact. I do agree that running it with a garbage collector doesn't fix the problem itself. Leaks must be fixed, not garbage collected. Of course. I would, however, want to point out that I'm not trying to sneak in a look how bad Evolution is here. In terms of truly leaking memory, Evolution is not bad at all. Evolution does consume a lot memory in known memory. Mostly in its summaries. I'm not talking about Evolution here (I know I do a lot, so don't get confused). -- Philip Van Hoof, software developer at x-tend home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org work: vanhoof at x-tend dot be http://www.pvanhoof.be - http://www.x-tend.be ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: What about Embedded?
On Thu, 2006-07-20 at 23:24 +0100, Jamie McCracken wrote: And this is why GNOME should accept and go for higher programming languages and modern development techniques. I totally agree but wouldn't it be better to use native languages that offer all this like the D language (http://www.digitalmars.com/d/). I think most of the people that ever talked with me on IRC already know that I'm very much a fan of the D programming language. No one has ever justified why we need a VM given all its disadvantages (speed - especially when mixing with native code, startup time, resource usage, bloat etc) A program that runs in a virtual machine can benefit from run-time optimization. If you want to do the same with a native application, you would need to feed the compiler with statistics which you need to gather from your users. Which isn't always an easy task. Or you must use alignment tricks and give the compiler a lot hints. But those hints don't hint the compiler about when the user uses it in an unexpected way. Unexpected as in: the user doesn't use the application the way the developer intended. -- I'm sure the heros that implemented our current virtual machines can give you much more information on this than I can. -- But all that doesn't matter for desktop NOR for mobile usage a lot. There's plenty of CPU to waste on a little bit of virtual machine overhead when we are talking about desktop applications. I mean, have you actually seen what some people dare to implement? THAT is the ONLY reason why their stuff runs slow and consumes a lot memory! It mostly has nothing to do with native or virtual machine. Even for mobile devices. But you make a good point with D. I would love to develop using D for the Nokia 770. In fact, I have plans to build myself a cross GCC compiler with an ARM backend and a D frontend soon. And then ... fuck the convention that everything must be done in a popular language to be accepted by some community. Then they don't accept it. That's their problem. I really hope young fresh developers think like this when they will pick their programming language for implementing their cool ideas. Hey young guys, if that GNOME community doesn't like your .NET: so what! DO use what you want to use. Your cool application will get popular even without this official recognition of that GNOME platform. Meanwhile, I do ask the non-Mono (and non-Python, non-Java, non- whatever) GNOME community to start accepting the modern programming languages. Stop fooling yourself and DO look at reality. Also managed languages can still leak and crash and misbehave if the p/invoke is not done properly so it aint a silver bullet. True. Of course. The D language offers the best of all worlds IMO *without* compromising on speed, resource usage or bloat. It would be madness to use a VM instead! There's still some pro VM arguments that D doesn't have. Like a good reflection framework (Hibernate uses this a lot). Oh .. And now that I mention Hibernate. The current popular languages (Java and .NET) have A LOT frameworks (Spring, Hibernate, etc) that don't (yet) exist in the D world. (there's NSpring and NHibernate Java guys, you don't have to tell me I'm selling Java-only tech because you would be lying). Most of them don't even have an equivalent in any of those native languages. I assure you such frameworks DO speed up software devel- opment. I would very much want GNOME application developers to start leveraging such frameworks. (of course its not as integrated into Gnome yet and lacks an IDE but if someone puts the work in you will have a killer platform than no VM based platform can match) -- Philip Van Hoof, software developer at x-tend home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org work: vanhoof at x-tend dot be http://www.pvanhoof.be - http://www.x-tend.be ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: What about Embedded?
On Fri, 2006-07-21 at 00:03 +0100, Jamie McCracken wrote: Philip Van Hoof wrote: On Thu, 2006-07-20 at 23:24 +0100, Jamie McCracken wrote: D *easily* beats mono and java in every benchmark so runtime optimisations count for very little here. http://shootout.alioth.debian.org/debian/benchmark.php?test=all〈=dlanglang2=csharp Most of the tests I see on that page are specific algorithms that utilise already manually optimized techniques. The reality, however, is that most desktop application softwares don't look *at all* like these tests. Most desktop application softwares are very unoptimized pieces of code that do *a lot* disk I/O, inter process communication, write a lot things to the X server (also IPC) and perform an unholy amount of string (and other types of memory) copying (and an almost equal amount of not-freeing that). Real code is very often not at all like these school algorithms that have been developed by keen students. They are used, yes. But those parts of the application-code indeed runs often optimized already. I believe Mono was rejected mostly by Sun Developers with concerns over bloat and memory usage being high on the list (and maybe a few political reasons). I wouldn't say Sun has something to do with this. Let's not accuse people here, shall we? D does not have these issues and if a few of us Gnome developers start using it we may find we can get in Gnome one day :) Just use it. Why do you want it *in* GNOME so fast? Why do we need reflection? I'm not saying your *need* it. I'm saying a lot developers are *using* it. I usually avoid using it myself. If and when Gobject gets full introspection it will become irrelevant. Ugh .. GObject introspection will not at all have all the features reflection in Java and .NET have. Not even close. Reflection does affect performance too which is why D avoided it and stuck to introspectable properties only. D has made sensible design choices by following Delphi's design instead of java/c#. I agree with that. We already have most of what we need in Gnome already for building cool desktop apps - we just need D bindings for the Gnome specific stuff. So help the guys doing just that? -- Philip Van Hoof, software developer at x-tend home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org work: vanhoof at x-tend dot be http://www.pvanhoof.be - http://www.x-tend.be ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: What about Embedded?
On Fri, 2006-07-21 at 00:35 +0100, Emmanuele Bassi wrote: the rationale for some of their choices is risible (uttons are there to pass on user events not executing actions? key bindings and composite widgets anyone?). I haven't at all investigated DUI. I did look at the D specification. The event-handling, however, looks to me a lot like how to do it in C#: From the samples on the site: button.onClick += doCancel; What is wrong with operator overloading + and using delegates for this? In my opinion, that works great in C#. Note that normally I discourage operator overloading. But for Events ... yeah, fantastic idea of Microsoft! My opinion is that it's much cleaner in C#, than these anonymous classes in Java. Which is in my opinion caused by wanting to stick to the observer design pattern TO purely. Microsoft learned from Suns mistakes, and improved it in their own thing. Some call that stealing, I call it innovation. If you can't cope with that, you'll simply lose the game. Totally fair and correct. I guess the DUI guys are still in experimenting phase a little bit. Why don't we (or whoever the experienced GNOME/Gtk+ binding guys are) go out and help them? Lets just go over there and make their stuff the best humans ever created? Why? Because we can. -- Philip Van Hoof, software developer at x-tend home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org work: vanhoof at x-tend dot be http://www.pvanhoof.be - http://www.x-tend.be ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Memory consumption and virtual machines
I'm going to attempt to conclude this mini-thread that got extended to other mailing lists. On Wed, 2006-07-19 at 02:45 -0600, Veerapuram Varadhan wrote: I have created a branch exclusively for the camel mmap summary work, viz., mmapped-camel-summary-branch which will help Phillip to continue his research further. I'm going to let the patch rest for a few days so that you can take a good look at it and decide what needs improvement before it goes into that branch. Phillip: Make sure you don't change the format-of-summary-file too drastically, the \0 is acceptable though. We have the discuss this :). But lets do that in focused mailing lists like evolution-hackers. Federico, I would like it to be maintained and well tested in the branch rather then rushing it up in HEAD. IMO, a branch gives more freedom to do such research work than HEAD. Correct me if I am wrong. :-) I agree here. Bringing it to HEAD might be to early. Nevertheless would I be indeed very pleased if a lot people test this on a lot config- urations. Thanks for considering the patch. -- Philip Van Hoof, software developer at x-tend home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org work: vanhoof at x-tend dot be http://www.pvanhoof.be - http://www.x-tend.be ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: [Evolution-hackers] Memory consumption and virtual machines
On Wed, 2006-07-19 at 09:53 -0500, Federico Mena Quintero wrote: On Wed, 2006-07-19 at 01:10 -0600, Veerapuram Varadhan wrote: On Tue, 2006-07-18 at 23:05 +, Philip Van Hoof wrote: If Novell wants me to implement unit tests (or other tests) for this, I will ask for payment. I am afraid that you won't get paid as Camel already has a neat-test-suite and can be used/extended, IMO. ;-) Aaaah, damn :) ... On the other hand, Philip, next time we meet in person I'll happily buy you dinner :) oh ... what about Boston? :) I'll check with my daytime employer whether it's okay if I can visit the Summit. -- Philip Van Hoof, software developer at x-tend home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org work: vanhoof at x-tend dot be http://www.pvanhoof.be - http://www.x-tend.be ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: focus!
On Wed, 2006-07-19 at 11:26 -0500, Mike Kestner wrote: On Wed, 2006-07-19 at 11:06 -0500, Federico Mena Quintero wrote: Semi-informed reply (Mike Kestner, the gtk-sharp maintainer, will be able to inform you better): - Last I heard inside Novell, gtk-sharp was in the process of adding support for GObject interfaces, which are required to implement the a11y interfaces from C# itself --- you could always implement your accessible interfaces in straight C, but that's not fun :) GInterface registration for managed subclasses is our #1 feature request, other than perhaps Data Binding. GInterface reg will probably be the next substantial addition to Gtk#. Much of the work is already completed, but it won't be included in 2.10.x. 2.10.x looks to be an API tracking release only. This is very good news. Thank you so much for wanting to implement this. When are the C# demos, that implement GTypeInterface's, going to be thrown at us? I can't wait for this to happen! Next to a very good IDE (like MonoDevelop), this is going to help us build a GNOME 3.0 platform that cares about higher level programming languages, that integrates with it, that will leverage the techniques, framework interfaces and components that are typically used with the programming environment. I will definitely be one of the contributors that will be working on bringing GNOME things closer to the .NET world, on helping Mono to be that next generation development platform. Lets innovate. Lets go for it. Lets make things better. http://www.funnypics.cc/en/philips_lets_make_things_better_764.php I thank you. -- Philip Van Hoof, software developer at x-tend home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org work: vanhoof at x-tend dot be http://www.pvanhoof.be - http://www.x-tend.be ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Memory consumption and virtual machines
On Tue, 2006-07-18 at 09:30 -0600, Veerapuram Varadhan wrote: Take for example Evolution. Using ONE WEEK of hacking, I managed to reduce its memory footprint with at least 40 MB of ram. I don't know how many times I need to repeat, because, this keeps coming in lot different threads and I see no progress to make the patch complete. I'm waiting for the decision (yours) of making this optional using a compilation flag or at run-time. Yes, I agree, the patch does reduce *STARTUP-MEMORY-FOOT PRINT* of Evolution as mentioned by Federico in his blog, however, it is *as of now* just-a-hack that the Evolution team cannot take it *as is*. Phillip, as you keep saying the patch needs rework before considering upstream, when are we going to get it? Will the final-patch addresses all the concerns raised by me and Fejj? Guess, you are aware of the GNOME release cycle and API freeze dates. It wouldn't be a good idea to glue this patch to an API freeze or GNOME release cycle as it needs extensive testing that shouldn't be bound by such milestones, but should be bound by test results. The test results have to be done on installations that I don't own. I cannot perform these tests with just my own equipment. I don't have customers that run Evolution on a NFS shared $HOME or run Evolution 4300 times on an application server. I guess this is why you wanted it to be optional. I'm still waiting for your decision of that optionality being at compile time or at run time. Note that at run time means that more errors and bugs are possible (especially when switching the implementation happens). Also note that after a full week of night-hacking on this patch, you have to remember that I'm not getting paid to do this. That I have a daytime job and a girlfriend. -- Philip Van Hoof, software developer at x-tend home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org work: vanhoof at x-tend dot be http://www.pvanhoof.be - http://www.x-tend.be ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: [Evolution-hackers] Memory consumption and virtual machines
On Tue, 2006-07-18 at 13:26 -0500, Federico Mena Quintero wrote: On Tue, 2006-07-18 at 18:29 +0200, Philip Van Hoof wrote: I agree with 1,2,..3 and 4. I will make sure 1 will be finished soon. Probably this evening with a compile-time option (--enable-mmap) I'm waiting for the decision (yours) of making this optional using a compilation flag or at run-time. Let's do this in the usual manner: 0. Polish the patch in the usual way: make sure it follows the indentation and naming conventions of the surrounding code, etc. 1. Branch evolution-data-server into HEAD (development, with Philip's patch), and the stable branch (without the patch). 2. Make the patch *mandatory* in HEAD, so that it gets a good amount of testing. 3. ??? 4. Profit!!! I'd suggest that (3) become write a good stress-test suite for Camel, independent of Evolution. We need that anyway. Novell already has a bunch of LDTP stuff to test the Evo mailer from the user's viewpooint - run those tests on the patched version to see how well they work. [Varadhan, those tests are already part of our QA process, aren't they?] -- Philip Van Hoof, software developer at x-tend home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org work: vanhoof at x-tend dot be http://www.pvanhoof.be - http://www.x-tend.be ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: [Evolution-hackers] Memory consumption and virtual machines
On Tue, 2006-07-18 at 14:46 -0400, Jeffrey Stedfast wrote: I have to wonder if it's even worth ever merging the mmap hack into Evolution at all. If the plan is to finish Zucchi's disk-summary branch, which also solves the memory problems (afaik) as well as: 1. introducing an API for using cursors to get at message infos 2. better designed on-disk format that uses B-Trees When? I'm also very interested in this for tinymail. the problem with philip's mmap file format is that the strings that will be hit for sorting/viewing/etc are all spread out over a huge number of pages. My own tests indicated that it's as-fast as the old implementation. Some test (like sorting) where even faster. I'm guessing mostly because qsort doesn't make large jumps afaik. I mostly fear NFS shared $HOME folders. I just see this being re-examined later to try and design the format to better optimise it by compacting all the strings into a strtab type thing. That would be an excellent idea. I also don't like how it has to reload the summary anytime new messages arrive. This is exactly the same as the current implementation. The current implementation does exactly the same and isn't changed at all. Look at the patch. It doesn't change anything to reloading the summary. I just don't get the feeling this is really all that well thought out and it scares me. So lets test it then?! I'd just hate to see a rush job come out of this Jeff On Tue, 2006-07-18 at 13:26 -0500, Federico Mena Quintero wrote: On Tue, 2006-07-18 at 18:29 +0200, Philip Van Hoof wrote: I'm waiting for the decision (yours) of making this optional using a compilation flag or at run-time. Let's do this in the usual manner: 0. Polish the patch in the usual way: make sure it follows the indentation and naming conventions of the surrounding code, etc. 1. Branch evolution-data-server into HEAD (development, with Philip's patch), and the stable branch (without the patch). 2. Make the patch *mandatory* in HEAD, so that it gets a good amount of testing. 3. ??? 4. Profit!!! I'd suggest that (3) become write a good stress-test suite for Camel, independent of Evolution. We need that anyway. Novell already has a bunch of LDTP stuff to test the Evo mailer from the user's viewpooint - run those tests on the patched version to see how well they work. [Varadhan, those tests are already part of our QA process, aren't they?] Federico ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers -- Philip Van Hoof, software developer at x-tend home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org work: vanhoof at x-tend dot be http://www.pvanhoof.be - http://www.x-tend.be ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: [Evolution-hackers] Memory consumption and virtual machines
On Tue, 2006-07-18 at 16:05 -0500, Federico Mena Quintero wrote: On Tue, 2006-07-18 at 14:46 -0400, Jeffrey Stedfast wrote: I have to wonder if it's even worth ever merging the mmap hack into Evolution at all. If the plan is to finish Zucchi's disk-summary branch, which also solves the memory problems (afaik) as well as: 1. introducing an API for using cursors to get at message infos 2. better designed on-disk format that uses B-Trees Let's put that discussion on Pause until someone actually starts resurrecting the disk-summary branch. If Philip's patch turns out to work well for daily use, it may be a good stopgap measure. The disk-summary branch will need way more QA time than the mmap() stuff, and a lot more performance tuning work as well :) the problem with philip's mmap file format is that the strings that will be hit for sorting/viewing/etc are all spread out over a huge number of pages. I just see this being re-examined later to try and design the format to better optimise it by compacting all the strings into a strtab type thing. I've been talking to Philip on IRC, and gave him these requirements for his patch: 1. Don't change the external ABI of Camel, so that Evo needs no changes, *OR* also submit a patch to update Evo for the changed API. The external API nor ABI has changed by the patches. 2. Make sure the summary format on disk works with older Evos without making *them* rewrite the summaries. This is for deployments which have machines with old and new versions of GNOME, but NFS homedirs accessible from any machine. What about making an internal little tool that converts any summary file it detects to a summary.mmap file, and after that never touches the original summary file. This will allow the format of the mmaped summary file to change, and only an evolution with the mmap patches will only the first time have to perform things. We can also simply change the file-name and let the older evolutions continue using the summary file, and the mmaped evolutions the new file. The older evolutions will not accept any changes to the summary files (for example version changes). They will reload the summary file if it has defects it can't handle, however. 3. Keep the coding style, variable naming convention, indentation, etc. Of course. It may be possible to change the summary format by *just* adding a nul-terminator to strings; that may work with older Evos if we are lucky enough that they'll just ignore the nul byte at the end. This needs testing. I'd say that (1) and (2) are hard requirements. (3) is the usual stuff. I just don't get the feeling this is really all that well thought out and it scares me. I'd just hate to see a rush job come out of this Yeah, it needs good testing. Philip says he'll cook a patch so that I can use it with my system's e-d-s RPM for daily use. Then I can test it with my normal mailbox. It might take me a few days as in fact I was planning to give it a rest for a few days. I've been caring to much about it. I don't know, it might also be finished in a few hours. Oh, no .. it's to late now. -- Philip Van Hoof, software developer at x-tend home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org work: vanhoof at x-tend dot be http://www.pvanhoof.be - http://www.x-tend.be ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Memory consumption and virtual machines
The lovely smell of programming environment flame wars! Part one As the developer of an application that has an extremely high focus on reduced memory consumption and as the author of a patch for Camel that reduced Evolutions memory footprint with ~40 MB (maybe more, but that number I'm certain of) . . . I hereby officially write: If tinymail would have been written in a high level language like Python OR .NET, its memory consumption wouldn't increase to a level of not being usable for its target audience. I could have done tinymail in .NET (or Python) and let it run under the Mono virtual machine (or the Python one) on the Nokia 770 device. I officially regret the fact that I didn't. I have no reason not to do tinymail in for example .NET (I don't like Python, but that says something about me and my development methods, not about Python). Don't force me to use Python because Python was so-called the first higher language for the GNOME platform. Definitely not all developers like the Python style of software development. Definitely not! Part two The few megabytes of memory added by a virtual machine, is insignificant to the current total amount of memory our GNOME desktop consumes. Take for example Evolution. Using ONE WEEK of hacking, I managed to reduce its memory footprint with at least 40 MB of ram. I have a question for you guys. How many virtual machines fit in that 40MB? Everybody who's interested in getting the memory usage of GNOME down: please focus on quality code in stead of trying to tell us virtual machines are the reason. They aren't. WE, the developers, are THE reason. Don't blame technology. Blame yourself. Please do not reply to this message on the mailing list. It's not intended to be your flame war offset. You can reply me in private. -- Philip Van Hoof, software developer at x-tend home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org work: vanhoof at x-tend dot be http://www.pvanhoof.be - http://www.x-tend.be ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Focusing on innovation re: mono, python et al
On Sun, 2006-07-16 at 18:57 +0200, Lluis Sanchez wrote: Hey LLuis, I very much agree with your point of view. And I thank you again and again and again for MonoDevelop. You should be extremely proud of your work. As a developer who is *very* worried about memory consumption of mobile applications like tinymail and desktop applications like Evolution (and a few others), I will most certainly use your MonoDevelop and the Mono virtual machine for many of my upcoming software developments. If GNOME wants to lose software developers, it should definitely keep rejecting innovations like .NET and Mono. By the subtile way GNOME is currently already rejecting Mono and other higher programming environ- ments, it will definitely lose for example me. Most people know I'm very opinionated and working hard to get GNOME to be much more friendly for higher programming languages and environments. I also agree the situation *is* actually improving. All GNOME people should understand this: Software developers, want to work with the finest technologies available on this earth. If GNOME glues itself to GObject and C, it will not be used by the generation of young software developers that will come after us. Yet THEY will be the ones that will build compelling applications. WE will burn out and WE will eventually stop development. FACE THIS. I will even strongly advise them to use another development platform. GObject and C, will not get them anywhere compared to modern other software development infrastructures. You don't even have to face this anymore: it's already the hard reality. GNOME, its community, needs to understand this. If it doesn't, it will not survive. Which would, in that case and in my opinion, be totally fair and totally correct. Innovate. Like MonoDevelop. Thanks, LLuis. You are one of my heros for making MonoDevelop reality. GNOME is useless if there are no applications running on it. That's the key success factor. Applications. It's not so important which applications do gnome include, since distros can take this decision, depending on the specific target of the distro. What's important for gnome is to provide a good infrastructure for developing and running applications. Some of the posts in this thread have been defending the purity of GNOME, and they have been against the inclusion of Mono because it could lead to a bloated and memory hungry GNOME. But purity is an abstract concept, we should look at the reality. The reality shows that people are using everyday Mono based applications, and they are happy with them. The reality shows that new applications are being built with high level languages and frameworks like Mono and Python, not in C. If there are memory and performance problems with Mono or Python, excluding them from GNOME is not a solution, because like it or not users will still use them to run applications. GNOME should adapt to this reality if it wants to survive. And adapting here means giving the best support it can provide to high level languages, like it did in the past for C. The Mono framework overlaps in some areas the existing gnome framework, yes, but this problem is not specific to Mono. That's a problem you will have with any other high level language/framework, because the gnome framework is based on a C API, and some of this API do not fit well in higher level languages. Or maybe it could fit, but at the cost of losing ease of use and a better integration with the rest of the framework. If the goal of a high level language/framework is to make it easier to develop applications, it can't be constrained by the underlying C api. If it means duplicating libraries and having for example two xml parsers in memory, so be it. It's still a good deal. -- Philip Van Hoof, software developer at x-tend home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org work: vanhoof at x-tend dot be http://www.pvanhoof.be - http://www.x-tend.be ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Certification for GNOME apps
On Thu, 2005-07-14 at 12:31 -0400, Behdad Esfahbod wrote: Or one day we may want to refuse certification from companies that support the other operating system :D. I know you're probably being sarcastic. However, for that reason I think it's important to do a very good job of specifying what an application needs to be to become certified. I would for sure stop being a member of any GNOME organisation once the organisation would start picking companies who'd never get such a certification based only on other products they support or their history with the free software communities. If Microsoft would create a GNOME application, we as an organisation should be honest in investigating whether or not their application should be granted the certification label of a GNOME application by looking only at the application. Not at the company. I don't see how supporting a product, like for example Microsoft Office, would have per definition anything to do with the certification of such a application. I also don't see how, for example, supporting the KDE platform would make an entire company not eligible for getting a GNOME certification. I can make a lot such examples. The examples are not my point. If we start being like that, the organisation would end up being far worse than the companies some people hate. Is the foundation going to certify apps or are companies allowed to evaluate and certify their own apps? If the former, should there be any fee for getting GNOME certified? I'd say the fee should be something like the amount of time the Foundation members need to investigate on the application multiplied by the current average cost of a consultant/contract worker. But I'm not a sales or marketing specialist :-) -- Philip Van Hoof, Software Developer @ Cronos home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org work: philip dot vanhoof at cronos dot be junk: philip dot vanhoof at gmail dot com http://www.pvanhoof.be/ ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list