Re: SoC idea: desktop file cache
Le Fri, 15 Apr 2011 15:18:47 -0400, Colin Walters walt...@verbum.org a écrit : On Mon, Mar 28, 2011 at 4:07 AM, Martyn Russell mar...@lanedo.com wrote: Sounds like a god idea. I try to be modest, but it's hard to turn down deification... At this point, I would like to just mention that Tracker already mines applications and it is REALLY fast at this. My first reaction is fear here honestly - part of that is that SPARQL and RDF ontologies are just not something I live and breathe... Understandable, and we'd need in any case a fallback mechanism for if Tracker is not installed. But when you say mines applications, you mean it parses the .desktop files and it's generating some index on the Name fields or something? The application miner parses desktop files, and inserts them in the Tracker database, which has an ontology (data model) for applications. Information extracted includes name, description, category, icon path and command line. Can you point me to some code? Sure, see http://git.gnome.org/browse/tracker/tree/src/miners/fs/tracker-miner-applications.c Cheers Adrien ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: SoC idea: desktop file cache
On Sun, Mar 27, 2011 at 4:20 AM, Vincent Untz vu...@gnome.org wrote: But we still want the merge logic to build menus. So it sounds to me that what you want is to make gnome-menus use a mmaped file. Well, what I think a simple first step is for gnome-menus to just categorize the results of g_desktop_app_info_get_all() effectively. That would at least get gnome-shell and gnome-panel out of the business of reading all .desktop files *twice*. And we should also fix the other gnome-menus issues; I'm very much aware of them and would also love them to disappear). So if the above step was done, then whatever interface we choose ends up living inside gio/gdesktopappinfo.c, which we need to fix anyways since we support apps launching other apps. ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: SoC idea: desktop file cache
Hi Martin, On Fri, Mar 25, 2011 at 2:55 PM, Martin Pitt martin.p...@ubuntu.com wrote: Just for the record I would like to explain the approach that we have had in Ubuntu since December 09. It might not be too useful any more, but can perhaps at least give some pointers. Vincent didn't like it too much back then, and rightly said that it should be implemented using GVariant, but we never got around to this as the current system actually works quite well (and it also dropped out of focus). Thanks, this is useful to hear of. * Being able to ignore NoDisplay/Hide/OnlyShowIn desktop files, which eliminated about half of the .desktop files. Unfortunately I don't think we can use this approach for GNOME; we need to do application tracking even for NoDisplay apps at least. * Not having to do gettext lookups for translations (we use gettext at runtime, so that we can update translations post-release without having to rebuild app packages, see http://bugzilla.gnome.org/show_bug.cgi?id=569829). But even with inline translations you'd save the lookups and reading a lot of extra data, so I recommend to consider per-locale caches. Yes, that makes sense. Also, I'm well aware of the corner case that this would simply ignore desktop files that the admin puts into /usr/share/applications/. We currently treat this case as don't do that then, that's what /usr/local/ is for, but it's of course a small design wart of this. We could probably just get away with declaring that broken, but this kind of thing is already a huge mess for ISVs to figure out. I think you have to at least be correct and check whether the mtime on the directory is newer than the cache, and ignore it if so - this is what gtk+ does for the icon cache. I'll look at your patches, though as I replied to Vincent I think we should be fixing Gio nowadays first, and rebase gnome-menus on top. ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: SoC idea: desktop file cache
On Mon, Mar 28, 2011 at 4:07 AM, Martyn Russell mar...@lanedo.com wrote: Sounds like a god idea. I try to be modest, but it's hard to turn down deification... At this point, I would like to just mention that Tracker already mines applications and it is REALLY fast at this. My first reaction is fear here honestly - part of that is that SPARQL and RDF ontologies are just not something I live and breathe... But when you say mines applications, you mean it parses the .desktop files and it's generating some index on the Name fields or something? Can you point me to some code? ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: SoC idea: desktop file cache
Hi guys, I had to give DBUS a try, so I went and modified the gdbus test
to get some numbers.
Running a thousand transactions takes about 1.5 seconds and there is no
big difference if the we transfer 50 bytes or 4 kbytes.
Here is the output of the test on my machine (Athlon 64 x2 5600+ with
DDR2 ram)
Took 3.020150 seconds to run 2000 iterations of 51 bytes each =
33773.156962 bytes/s
Took 3.193050 seconds to run 2000 iterations of 4079 bytes each =
2554923.975509 bytes/s
Took 0.081563 seconds to run 2 iterations of 1007029 bytes each =
24693280.041195 bytes/s
Took 0.752203 seconds to run 20 iterations of 1007029 bytes each =
26775458.220720 bytes/s
I assume here the bottle neck is the context switch between both server
and client.
So I think DBus speed wont be an issue transferring desktop entries to a
client.
my 2 cents
Juan Pablo
#include gio/gio.h
#include stdlib.h
/* */
static GDBusNodeInfo *introspection_data = NULL;
/* Introspection data for the service we are exporting */
static const gchar introspection_xml[] =
node
interface name='org.gtk.GDBus.TestPeerInterface'
method name='HelloWorld'
arg type='s' name='greeting' direction='in'/
arg type='s' name='response' direction='out'/
/method
/interface
/node;
gint n_entries = 1;
/* */
#define DESKTOP_ENTRY [Desktop Entry] \
Exec=abiword\
Icon=abiword_48\
Terminal=false\
Type=Application\
Categories=Office;WordProcessor;GNOME;GTK;X-Red-Hat-Base;\
StartupNotify=true\
X-Desktop-File-Install-Version=0.9\
MimeType=application/x-abiword;text/x-abiword;text/x-xml-abiword;text/plain;application/msword;application/rtf;\
application/vnd.plain;application/xhtml+xml;text/html;application/x-crossmark;application/docbook+xml;application/x-t602;application/vnd.oasis.opendocument.text;\
application/vnd.oasis.opendocument.text-template;application/vnd.oasis.opendocument.text-web;application/vnd.sun.xml.writer;application/vnd.stardivision.writer;\
text/vnd.wap.wml;application/wordperfect6;application/wordperfect5.1;application/vnd.wordperfect;application/x-abicollab;application/vnd.palm;application/x-applix-word;\
application/x-kword;application/x-mif;application/x-mswrite;application/x-palm-database;text/abiword;text/richtext;text/rtf;application/x-abicollab\
Name=AbiWord\
GenericName=Word Processor\
Comment=Compose, edit, and view documents
static void
handle_method_call (GDBusConnection *connection,
const gchar *sender,
const gchar *object_path,
const gchar *interface_name,
const gchar *method_name,
GVariant *parameters,
GDBusMethodInvocation *invocation,
gpointer user_data)
{
if (g_strcmp0 (method_name, HelloWorld) == 0)
{
const gchar *greeting;
GString *str;
gchar *response;
gint i;
g_variant_get (parameters, (s), greeting);
response = g_strdup_printf (You said '%s'. KTHXBYE!\n, greeting);
str = g_string_new (greeting);
for (i = 0; i n_entries; i++)
g_string_append (str, DESKTOP_ENTRY);
g_dbus_method_invocation_return_value (invocation,
g_variant_new ((s), str-str));
g_free (response);
g_string_free (str, TRUE);
// g_print (Client said: %s\n, greeting);
}
}
static const GDBusInterfaceVTable interface_vtable =
{
handle_method_call,
NULL,
NULL,
};
/* */
static gboolean
on_new_connection (GDBusServer *server,
GDBusConnection *connection,
gpointer user_data)
{
guint registration_id;
GCredentials *credentials;
gchar *s;
credentials = g_dbus_connection_get_peer_credentials (connection);
if (credentials == NULL)
s = g_strdup ((no credentials received));
else
s = g_credentials_to_string (credentials);
g_print (Client connected.\n
Peer credentials: %s\n
Negotiated capabilities: unix-fd-passing=%d\n,
s,
g_dbus_connection_get_capabilities (connection) G_DBUS_CAPABILITY_FLAGS_UNIX_FD_PASSING);
g_object_ref (connection);
registration_id = g_dbus_connection_register_object (connection,
/org/gtk/GDBus/TestObject,
introspection_data-interfaces[0],
interface_vtable,
NULL, /* user_data */
Re: SoC idea: desktop file cache
Le Wed, 30 Mar 2011 15:46:03 -0300, Juan Pablo Ugarte juanpablouga...@gmail.com a écrit : Hi guys, I had to give DBUS a try, so I went and modified the gdbus test to get some numbers. Running a thousand transactions takes about 1.5 seconds and there is no big difference if the we transfer 50 bytes or 4 kbytes. This is probably due to the fact that the daemon writes data in chunks of 4kb... So you probably have to batch in 4kb chunks on app side (when it's possible). Note that for big data transfers, using FD passing and a socket is quite faster, but probably overkill here. Here is the output of the test on my machine (Athlon 64 x2 5600+ with DDR2 ram) Took 3.020150 seconds to run 2000 iterations of 51 bytes each = 33773.156962 bytes/s Took 3.193050 seconds to run 2000 iterations of 4079 bytes each = 2554923.975509 bytes/s Took 0.081563 seconds to run 2 iterations of 1007029 bytes each = 24693280.041195 bytes/s Took 0.752203 seconds to run 20 iterations of 1007029 bytes each = 26775458.220720 bytes/s I assume here the bottle neck is the context switch between both server and client. So I think DBus speed wont be an issue transferring desktop entries to a client. my 2 cents Juan Pablo ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: SoC idea: desktop file cache
Le mercredi 30 mars 2011 à 22:01 +0300, Adrien Bustany a écrit : Note that for big data transfers, using FD passing and a socket is quite faster, but probably overkill here. This is what we use for file transfer in Telepathy. regards, Nicolas signature.asc Description: This is a digitally signed message part ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: SoC idea: desktop file cache
My point is, using dbus to transfer desktop entries is not a bad idea because of speed issues. Besides this would be desktop agnostic and not a gnome custom implementation. Cheers Juan Pablo Enviado desde mi iPhone El 30/03/2011, a las 17:25, Nicolas Dufresne nicolas.dufre...@collabora.co.uk escribió: Le mercredi 30 mars 2011 à 22:01 +0300, Adrien Bustany a écrit : Note that for big data transfers, using FD passing and a socket is quite faster, but probably overkill here. This is what we use for file transfer in Telepathy. regards, Nicolas ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: SoC idea: desktop file cache
On 25/03/11 03:22, Colin Walters wrote:
Bugzilla is down or I'd write this there, and it's a bit lengthy to
fit in the SoC wiki page, so I'll just post here and link from the
wiki.
Hi,
== Kill gnome-menus ==
gnome-menus is the library which reads all this stuff presently, and
it has one serious drawback:
* It is entirely synchronous
This especially hurts since we have to read applications from
gnome-shell, which is the compositor - a blocked compositor means no
redrawing.
Sounds like a god idea.
At this point, I would like to just mention that Tracker already mines
applications and it is REALLY fast at this. On my desktop here it takes
a few seconds to store all information about all applications installed.
There are other, more minor flaws:
* It is quite inefficient; there is a ton of string allocation going on.
* It's not threadsafe (see e.g. the static GSList *pending_events
in menu-monitor.c)
* It wasn't designed for introspection (
https://live.gnome.org/GObjectIntrospection )
Tracker has introspection support in a branch right now and should be
merged to master in the coming week or so.
== Implementing a saner tool ==
There are multiple ways to approach this, but I think by far the
simplest is just an mmap'able file containing all .desktop files found
in the desktop paths. Don't try to implement all the merge logic,
etc.
Tracker uses tracker-miner-fs currently for this. We've been meaning to
split it into a separate miner for applications and there is a bug about
it. Also, this works for Qt/MeeGo applications too.
How do you implement a mmap'd file? Take a look at how dconf does it:
http://git.gnome.org/browse/dconf/tree/gvdb
We use gvdb internally in Tracker already for fast db ontology look ups
and SQLite of course as the db backend ;)
The next question is how does the cache get rebuilt? I think we
basically need a system service with an inotify (GFileMonitor) watch
on all those directories, and it takes care of rebuilding it.
Tracker uses GIO which uses inotify or the next best thing (there are
others supported) depending on the target platform. So this is already
covered in Tracker.
The library api could simply be:
void on_desktop_cache_changed ()
{
DesktopCacheEntryIter iter;
desktop_cache_iter_init (cache,iter);
while (desktop_cache_iter_next (cache,iter)) {
g_print (desktop file path: %s\n,
desktop_cache_iter_get_desktop_file_path (iter));
g_print (desktop name: %s\n, desktop_cache_iter_get_desktop_name
(iter));
}
}
DesktopCache *cache = desktop_cache_get();
g_signal_connect (cache, changed, on_desktop_cache_changed, self);
desktop_cache_read_async (cache, on_desktop_cache_read_complete, self);
The library would simply have an inotify watch on the cache file.
Which should probably simply be defined to live in
/usr/share/applications/applications.cache, say.
libtracker-sparql is a thread safe, async and sync library which has
notifications for updates already available. You could just use that in
your application to listen to specific changes.
Filtering the menus by category would live in a higher layer, in my
case, gnome-shell probably.
Adrien Bustany wrote a patch that integrates Tracker with GNOME shell
last week and with any luck it should be merged into master soon.
So, my suggestion is to use Tracker here, otherwise we're just
duplicating work.
--
Regards,
Martyn
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: SoC idea: desktop file cache
On Mon, 28 Mar 2011 09:07:36 +0100, Martyn Russell wrote:
On 25/03/11 03:22, Colin Walters wrote:
Bugzilla is down or I'd write this there, and it's a bit lengthy to
fit in the SoC wiki page, so I'll just post here and link from the
wiki.
Hi,
== Kill gnome-menus ==
gnome-menus is the library which reads all this stuff presently, and
it has one serious drawback:
* It is entirely synchronous
This especially hurts since we have to read applications from
gnome-shell, which is the compositor - a blocked compositor means no
redrawing.
Sounds like a god idea.
At this point, I would like to just mention that Tracker already
mines applications and it is REALLY fast at this. On my desktop here
it takes a few seconds to store all information about all
applications
installed.
There are other, more minor flaws:
* It is quite inefficient; there is a ton of string allocation going
on.
* It's not threadsafe (see e.g. the static GSList
*pending_events
in menu-monitor.c)
* It wasn't designed for introspection (
https://live.gnome.org/GObjectIntrospection )
Tracker has introspection support in a branch right now and should be
merged to master in the coming week or so.
== Implementing a saner tool ==
There are multiple ways to approach this, but I think by far the
simplest is just an mmap'able file containing all .desktop files
found
in the desktop paths. Don't try to implement all the merge logic,
etc.
Tracker uses tracker-miner-fs currently for this. We've been meaning
to split it into a separate miner for applications and there is a bug
about it. Also, this works for Qt/MeeGo applications too.
How do you implement a mmap'd file? Take a look at how dconf does
it:
http://git.gnome.org/browse/dconf/tree/gvdb
We use gvdb internally in Tracker already for fast db ontology look
ups and SQLite of course as the db backend ;)
The next question is how does the cache get rebuilt? I think we
basically need a system service with an inotify (GFileMonitor) watch
on all those directories, and it takes care of rebuilding it.
Tracker uses GIO which uses inotify or the next best thing (there are
others supported) depending on the target platform. So this is
already
covered in Tracker.
The library api could simply be:
void on_desktop_cache_changed ()
{
DesktopCacheEntryIter iter;
desktop_cache_iter_init (cache,iter);
while (desktop_cache_iter_next (cache,iter)) {
g_print (desktop file path: %s\n,
desktop_cache_iter_get_desktop_file_path (iter));
g_print (desktop name: %s\n,
desktop_cache_iter_get_desktop_name (iter));
}
}
DesktopCache *cache = desktop_cache_get();
g_signal_connect (cache, changed, on_desktop_cache_changed, self);
desktop_cache_read_async (cache, on_desktop_cache_read_complete,
self);
The library would simply have an inotify watch on the cache file.
Which should probably simply be defined to live in
/usr/share/applications/applications.cache, say.
libtracker-sparql is a thread safe, async and sync library which has
notifications for updates already available. You could just use that
in your application to listen to specific changes.
Filtering the menus by category would live in a higher layer, in my
case, gnome-shell probably.
Adrien Bustany wrote a patch that integrates Tracker with GNOME shell
last week and with any luck it should be merged into master soon.
So, my suggestion is to use Tracker here, otherwise we're just
duplicating work.
That said, if we use Tracker we don't solve the problem in a
cross-desktop way... Still worth considering though.
Cheers
Adrien
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: SoC idea: desktop file cache
Le vendredi 25 mars 2011, à 21:52 +, Bastien Nocera a écrit : On Sat, 2011-03-26 at 03:11 +0530, Nirbheek Chauhan wrote: snip I was thinking that this is the kind of problem that has cross-desktop implications, and so I wondered how KDE solves this problem. We should probably take their feedback on this? On the other hand, I think there is a way to do this in a cross-desktop fashion with existing infrastructure as well. We already have `update-desktop-database` for updating the MimeType cache from desktop files, and that's supposed to be run after any new .desktop files are added to $XDG_DATA_DIRS/applications, so I was wondering if caches for the other properties could also be generated by the same tool at the same time? I wouldn't want the code to live in update-desktop-database (which is in shared-mime-info), but we can certainly add support for launching scripts in a specific directory whenever it's called. It's actually in desktop-file-utils, not shared-mime-info :-) Any reason you wouldn't want to add that to the utility? Or was it just that you thought it lived in shared-mime-info which shouldn't know about that kind of stuff? Vincent -- Les gens heureux ne sont pas pressés. ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: SoC idea: desktop file cache
Le jeudi 24 mars 2011, à 23:22 -0400, Colin Walters a écrit : == Kill gnome-menus == gnome-menus is the library which reads all this stuff presently, and it has one serious drawback: * It is entirely synchronous This especially hurts since we have to read applications from gnome-shell, which is the compositor - a blocked compositor means no redrawing. [...] == Implementing a saner tool == There are multiple ways to approach this, but I think by far the simplest is just an mmap'able file containing all .desktop files found in the desktop paths. Don't try to implement all the merge logic, etc. But we still want the merge logic to build menus. So it sounds to me that what you want is to make gnome-menus use a mmaped file. And we should also fix the other gnome-menus issues; I'm very much aware of them and would also love them to disappear). Vincent -- Les gens heureux ne sont pas pressés. ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: SoC idea: desktop file cache
On Sun, 2011-03-27 at 10:17 +0200, Vincent Untz wrote: Le vendredi 25 mars 2011, à 21:52 +, Bastien Nocera a écrit : On Sat, 2011-03-26 at 03:11 +0530, Nirbheek Chauhan wrote: snip I was thinking that this is the kind of problem that has cross-desktop implications, and so I wondered how KDE solves this problem. We should probably take their feedback on this? On the other hand, I think there is a way to do this in a cross-desktop fashion with existing infrastructure as well. We already have `update-desktop-database` for updating the MimeType cache from desktop files, and that's supposed to be run after any new .desktop files are added to $XDG_DATA_DIRS/applications, so I was wondering if caches for the other properties could also be generated by the same tool at the same time? I wouldn't want the code to live in update-desktop-database (which is in shared-mime-info), but we can certainly add support for launching scripts in a specific directory whenever it's called. It's actually in desktop-file-utils, not shared-mime-info :-) Man, I get confused ;) Any reason you wouldn't want to add that to the utility? Or was it just that you thought it lived in shared-mime-info which shouldn't know about that kind of stuff? I wouldn't want to have had to add support for it to shared-mime-info, because that caching mechanism wasn't agreed upon xdg-wide, and there's enough problems with that code as it is (the existing code isn't exactly very forgiving). Having it within update-desktop-database is probably a better idea (and it's not my problem anymore :P) ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: SoC idea: desktop file cache
Bastien Nocera [2011-03-25 21:52 +]: I wouldn't want the code to live in update-desktop-database (which is in shared-mime-info), but we can certainly add support for launching scripts in a specific directory whenever it's called. Or just call it update-desktop-caches and stay a separate script, since (as you already pointed out) it needs to run at different occasions anyway? Note that update-desktop-database is probably not called by packaging managers or application's Makefiles when they don't handle mime-types, so this would be something that would need to be announced, and followed by distributions. At least it wouldn't cause any regression if it isn't called by package managers; it just wouldn't use caching, so this might be acceptable? Thanks, Martin -- Martin Pitt| http://www.piware.de Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org) ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: SoC idea: desktop file cache
Hey Colin, On Thu, Mar 24, 2011 at 11:22 PM, Colin Walters walt...@verbum.org wrote: The next question is how does the cache get rebuilt? I think we basically need a system service with an inotify (GFileMonitor) watch on all those directories, and it takes care of rebuilding it. Sounds good, but we really want a session-service [0] instead of a system-service - otherwise it won't work with apps installed in $HOME/.local or whatever $XDG_CONFIG_DIRS points to. With this, could also implement this as a gnome-settings-daemon plug-in if you don't want the overhead of another process. And you completely bypass the nasty (and often overlooked) implications of dealing with two security contexts instead of just one. And you also bypass other nasty problems like $HOME on NFS or automount issues if you decide to fix this flaw later by having the system daemon look in the users home... Btw, if this service runs in the session, it can also be smart about e.g. synthesizing desktop files for glick-style [1] applications previously seen or downloaded or living in ~/Applications or ~/Downloads or something. Of course someone needs to come up with a design for how runtime-less bundles should work but, at least, let's not make it impossible from the get-go by implementing caching at the system-level. (On a machine with a one or two sessions at a time, the session-instead-of-system approach has approximately the same performance characteristics as the system-service approach. On machines with many many sessions (Largo, FL terminal servers for example) you might want to also have a system-wide cache that each session can use. Could be it doesn't matter doing such a (premature) optimization though.) David [0] : you actually really want a service that is per (user, machine) instead of per-session... this is easiest implemented by using the D-Bus name org.service.machine-id on the session bus and using storage in $HOME/.local/service/machine-machine-id for the cache [1] : http://blogs.gnome.org/alexl/2007/08/07/experiments-with-runtime-less-app-bundles/ ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: SoC idea: desktop file cache
On Fri, Mar 25, 2011 at 6:11 AM, Alan Cox a...@lxorguk.ukuu.org.uk wrote: There are multiple ways to approach this, but I think by far the simplest is just an mmap'able file containing all .desktop files found in the desktop paths. This depends. First comment on the shell is man pthreads, Yeah, it probably wouldn't be hard to make gnome-menus threadsafe, there's no good excuse, I just never did it and kept dreaming of a non-band-aid fix. In both cases a single file cures most of the latency not mmap. In fact read() is often faster unless lots of apps will be sharing a cache. Any application that wants to launch another app needs to read all of them, so yeah, I think there is an argument for mmap. Just from the core, Firefox lets you right click and pick an app to launch a file. Firefox lets you choose an app in the same way, etc. In the read case the OS gets a single unambiguous indication of what you want. In the mmap case it has to guess and you can end up with a lot of fault latency - which can be even worse as it can sometimes clog up all the threads of an app, and which is more SMP load. Your suggestion of telling the OS we are going to read the whole thing would fix that, and it makes sense. mmap MAP_SHARED also makes dealing with external updating really interesting because if the file shrinks you need to handle signal exceptions from touching pages that no longer exist. Not if the cache is replaced with rename(), no? Thanks for the other hints. ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: SoC idea: desktop file cache
On Fri, Mar 25, 2011 at 5:03 AM, David Zeuthen zeut...@gmail.com wrote: Sounds good, but we really want a session-service [0] instead of a system-service I would say system *and* session service. But honestly I'm not too concerned right now about overhead of apps installed in ~/.local, since it's not where huge swaths of crud typically lives. ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: SoC idea: desktop file cache
mmap MAP_SHARED also makes dealing with external updating really interesting because if the file shrinks you need to handle signal exceptions from touching pages that no longer exist. Not if the cache is replaced with rename(), no? If the sequence is build new cache rename unlink old cache then the existing users will continue to see the old file which will only vanish completely on the last close. You can use fstat and the like to check the size for the mmap to avoid racing the rename on guessing sizes etc. It's only if the file is truncated the fun with SIGBUS occurs, or on an unrecoverable read error (eg NFS timeout on soft mount, physical bad block). In those cases unlike read() there is no other reporting mechanism. sigsetjmp/longjmp are normally evil but this may be one case they are needed to build a sort of mock exception handler given C's lack of a proper mechanism. Alan ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: SoC idea: desktop file cache
On Fri, Mar 25, 2011 at 7:43 AM, Alan Cox a...@lxorguk.ukuu.org.uk wrote: then the existing users will continue to see the old file which will only vanish completely on the last close. Right, hence the inotify watch. Though actually, this makes me notice we aren't monitoring the GTK+ icon cache? http://git.gnome.org/browse/gtk+/tree/gtk/gtkiconcache.c Hmm...not good, but I guess no one cared much that you didn't see changed icons until process restart (you will still see new ones, since they won't be found in the cache and we'll still try scanning the filesystem I think). ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: SoC idea: desktop file cache
On Fri, 2011-03-25 at 11:43 +, Alan Cox wrote: mmap MAP_SHARED also makes dealing with external updating really interesting because if the file shrinks you need to handle signal exceptions from touching pages that no longer exist. Not if the cache is replaced with rename(), no? If the sequence is build new cache rename unlink old cache then the existing users will continue to see the old file which will only vanish completely on the last close. You can use fstat and the like to check the size for the mmap to avoid racing the rename on guessing sizes etc. The way I handle this in the gvfs metadata case is to have a rotated int32 in the file header. Then rebuilding does: fd1 = open(cache): create_new_cache(tmp-cache); rename(tmp-cache, cache); // Replaces old cache file atomically seek (fd1, offset_to_rorated_flag); write (fd1, 4, buffer_with_ones); Then, in the public API entry points for reading to the database I always first check the rotated flag, and if set, re-mmap the file. This is generally pretty fast, and all clients will eventually move over to the new file without having to use inotify or some other mainloop-requiring code. -- =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Alexander LarssonRed Hat, Inc al...@redhat.comalexander.lars...@gmail.com He's an unconventional albino shaman on the wrong side of the law. She's a mentally unstable impetuous college professor on the trail of a serial killer. They fight crime! ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: SoC idea: desktop file cache
Hi! After reading Alan's and David's comment it occurs to me that it could be way simpler to have a dbus-service handling the desktop files and anybody wanting to access the desktop files in a fast way can use that service (through a library wrapper). This would make all the cache problems much easier as there would be only one process touching the cache at all. Sync/Async support comes from dbus basically for free. Just my 2 cents, Johannes Am Freitag, den 25.03.2011, 07:16 -0400 schrieb Colin Walters: On Fri, Mar 25, 2011 at 5:03 AM, David Zeuthen zeut...@gmail.com wrote: Sounds good, but we really want a session-service [0] instead of a system-service I would say system *and* session service. But honestly I'm not too concerned right now about overhead of apps installed in ~/.local, since it's not where huge swaths of crud typically lives. ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list signature.asc Description: This is a digitally signed message part ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: SoC idea: desktop file cache
On Fri, Mar 25, 2011 at 11:12 AM, Johannes Schmid j...@jsschmid.de wrote: Hi! After reading Alan's and David's comment it occurs to me that it could be way simpler to have a dbus-service handling the desktop files and anybody wanting to access the desktop files in a fast way can use that service (through a library wrapper). I don't think that would make things faster; in fact it'd be more likely to make them slower. DBus is just not designed for bulk data transfer. Think about it - the bus daemon sits in the middle, copying strings around etc., have to be read in and out... One thing that we should probably promote more about GNOME 3 is how far we are long in kicking GConf out, which had this single-threaded-centralized-daemon model. When I was doing some profiling a while back, I was surprised by the extent to which gconfd acted as a serialization point for the entire desktop, especially if you didn't use PRELOAD_RECURSIVE for reading keys. dconf is just way better. ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: SoC idea: desktop file cache
On Thu, 2011-03-24 at 23:22 -0400, Colin Walters wrote: Bugzilla is down or I'd write this there, and it's a bit lengthy to fit in the SoC wiki page, so I'll just post here and link from the wiki. If it's important for 3.2, it shouldn't be a Google Summer Of Code project. The SoC isn't about free labour, and your student might not manage to have the project reach completion either. Given the discussions, I would advise doing this yourself, or calling out to the community to work on it. ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: SoC idea: desktop file cache
Hello all, Colin Walters [2011-03-25 12:18 +0100]: == Desktop File Cache == There are multiple ways to approach this, but I think by far the simplest is just an mmap'able file containing all .desktop files found in the desktop paths. Don't try to implement all the merge logic, etc. I'm glad that this gets discussed again. Vincent Untz and I were discussing this some two years ago, as we found that reading desktop files was a serious chunk of CPU churn during session startup. Just for the record I would like to explain the approach that we have had in Ubuntu since December 09. It might not be too useful any more, but can perhaps at least give some pointers. Vincent didn't like it too much back then, and rightly said that it should be implemented using GVariant, but we never got around to this as the current system actually works quite well (and it also dropped out of focus). On a Dell Mini 10 (with an Abacus, erm, Atom processor) this saved a little less than one second startup time, and gnome-panel's CPU usage was cut to less than half. (It helped other applications which read the menus as well). This was the pre-GVariant age, and there was no appropriate serialization API available which was both fast and also not a huge pain to use. I played aorund with various file formats, and actually found that with gnome-menus a simple GKeyFile actually worked best, since in the end we want to read all the relevant desktop files anyway, and reading them linearly out of the file cache into the internal DesktopEntry tree representation was fine. The time savings came from: * Being able to ignore NoDisplay/Hide/OnlyShowIn desktop files, which eliminated about half of the .desktop files. * Not having to do gettext lookups for translations (we use gettext at runtime, so that we can update translations post-release without having to rebuild app packages, see http://bugzilla.gnome.org/show_bug.cgi?id=569829). But even with inline translations you'd save the lookups and reading a lot of extra data, so I recommend to consider per-locale caches. * Not having to readdir() /usr/share/applications/ and stat() all the files there. * On rotary disks, reading one file is much faster than reading ~ 150 files which are splattered throughout the disk. This is not that relevant with the various *readahead solutions, of course. So we build /usr/share/applications/dirname.locale.cache, which look very simple [1]. The gnome-menus patch [2] looks for these and uses that instead of parsing all the *.desktop files if present. The next question is how does the cache get rebuilt? In the discussions back in the day we actually considered using a daemon, but we quickly discarded the idea, as this would impose quite a lot of runtime overhead, and we don't actually want to see yet another daemon in the session (which would probably eat up a lot of the CPU savings of the cache by having to inotify and stat everywhere). Instead we found: * /usr/share/applications/ has the vast majority of desktop files; /usr/local, ~/.local, etc. typically only have very few, and trying to cache them isn't worth it. * This is also the only directory which is exclusively in the packages domain, which we can control very well. Other dirs can be modified at any time by the user or admins. So we went for the pragmatic solution and only do caches for /usr/share/applications. (The patch above works with any desktop directory, but we only build cache files for that one). We have a script [3] to build the cache, which gets run by the package manager whenever a package adds/changes/removes/modifies files in /usr/share/applications/. (In dpkg terms this is called a trigger). So we only have some extra overhead at package install/upgrade time (which is already slow enough, so that the extra second doesn't matter), but no extra overhead at runtime. This concept is very hard to generalize as an upstream solution, of course. The gnome-menus patch [2] and the script [3] aren't distro specific at all, but the concept of package triggers simply doesn't exist with upstream tarballs, and thus there will always be some distro specific integration here. Also, I'm well aware of the corner case that this would simply ignore desktop files that the admin puts into /usr/share/applications/. We currently treat this case as don't do that then, that's what /usr/local/ is for, but it's of course a small design wart of this. Thanks, Martin [1] http://people.canonical.com/~pitti/tmp/desktop.de_DE.utf8.cache [2] http://bazaar.launchpad.net/~ubuntu-desktop/gnome-menus/ubuntu/view/head:/debian/patches/22_desktop-cache.patch [3] http://bazaar.launchpad.net/~ubuntu-desktop/gnome-menus/ubuntu/view/head:/debian/update-gnome-menus-cache -- Martin Pitt| http://www.piware.de Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org) signature.asc Description: Digital signature
Re: SoC idea: desktop file cache
Hey everyone, On Sat, Mar 26, 2011 at 12:25 AM, Martin Pitt martin.p...@ubuntu.com wrote: Colin Walters [2011-03-25 12:18 +0100]: == Desktop File Cache == There are multiple ways to approach this, but I think by far the simplest is just an mmap'able file containing all .desktop files found in the desktop paths. Don't try to implement all the merge logic, etc. [snip] I played aorund with various file formats, and actually found that with gnome-menus a simple GKeyFile actually worked best, since in the end we want to read all the relevant desktop files anyway, and reading them linearly out of the file cache into the internal DesktopEntry tree representation was fine. [snip] The next question is how does the cache get rebuilt? [snip] So we went for the pragmatic solution and only do caches for /usr/share/applications. (The patch above works with any desktop directory, but we only build cache files for that one). We have a script [3] to build the cache, which gets run by the package manager whenever a package adds/changes/removes/modifies files in /usr/share/applications/. (In dpkg terms this is called a trigger). So we only have some extra overhead at package install/upgrade time (which is already slow enough, so that the extra second doesn't matter), but no extra overhead at runtime. This concept is very hard to generalize as an upstream solution, of course. The gnome-menus patch [2] and the script [3] aren't distro specific at all, but the concept of package triggers simply doesn't exist with upstream tarballs, and thus there will always be some distro specific integration here. Also, I'm well aware of the corner case that this would simply ignore desktop files that the admin puts into /usr/share/applications/. We currently treat this case as don't do that then, that's what /usr/local/ is for, but it's of course a small design wart of this. I was thinking that this is the kind of problem that has cross-desktop implications, and so I wondered how KDE solves this problem. We should probably take their feedback on this? On the other hand, I think there is a way to do this in a cross-desktop fashion with existing infrastructure as well. We already have `update-desktop-database` for updating the MimeType cache from desktop files, and that's supposed to be run after any new .desktop files are added to $XDG_DATA_DIRS/applications, so I was wondering if caches for the other properties could also be generated by the same tool at the same time? Of course, we'd have to come up with a format for that, but I think the GKeyFile format that the Ubuntu folks have come up with could be reused since the mimeinfo cache is a GKeyFile as well. As I understand it, this would be equivalent to the Ubuntu solution, would be applicable as an upstream solution, and would be useful for other DEs as well. Since we have the time, this should be very doable in time for 3.2. Cheers, -- ~Nirbheek Chauhan Gentoo GNOME+Mozilla Team ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: SoC idea: desktop file cache
On Sat, 2011-03-26 at 03:11 +0530, Nirbheek Chauhan wrote: snip I was thinking that this is the kind of problem that has cross-desktop implications, and so I wondered how KDE solves this problem. We should probably take their feedback on this? On the other hand, I think there is a way to do this in a cross-desktop fashion with existing infrastructure as well. We already have `update-desktop-database` for updating the MimeType cache from desktop files, and that's supposed to be run after any new .desktop files are added to $XDG_DATA_DIRS/applications, so I was wondering if caches for the other properties could also be generated by the same tool at the same time? I wouldn't want the code to live in update-desktop-database (which is in shared-mime-info), but we can certainly add support for launching scripts in a specific directory whenever it's called. Note that update-desktop-database is probably not called by packaging managers or application's Makefiles when they don't handle mime-types, so this would be something that would need to be announced, and followed by distributions. Cheers ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
