I had the same problem on my Xubuntu 12.04 too, but only for new created accounts, not for my existing one. After some hours of trial and error I've got the one file, which made the difference: It was ~/.cache/sessions/xfce4-session-MYCOMPUTERNAME:0 . So simply saving the desktop session once solved the issue for me.
The hole problem and the solution/workaround is easily to reproduce, at least on my machine with Xubuntu 12.04: 1. Create a new user account with an completly empty home-directory. 2. Copy the .Xmodmap file to that home-directory. 3. Log in. There will be a lot of files automatically created, the .Xmodmap file will be read in and Xorg will run at 100%. 4. Ignore what Xorg is doing and go (under XFCE) to Main menu -> Settings -> Settings -> Session and Startup -> Session -> Save session. 5. Wait until the session is saved, log out, log in and everything is fine. The .Xmodmap is read in, but no Xorg running at 100%. Notes: - Maybe you have on an existing account also to disable the xfce4-settings-helper in Main menu -> Settings -> Settings -> Session and Startup -> Application Autostart, but I'm not sure about that. - When I manually read in the .Xmodmap with the command "xmodmap ~/.Xmodmap", there will still be 100% CPU for about one minute, both in the existing and in the new account. But because I do not need to read it in manually, this is, at least for me, no problem. - Debian seems to have the same bug, see http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=687112 . ** Bug watch added: Debian Bug tracker #687112 http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=687112 -- You received this bug notification because you are a member of Desktop Packages, which is subscribed to xorg in Ubuntu. https://bugs.launchpad.net/bugs/998310 Title: .Xmodmap file makes xorg temporarily reach high cpu usage (90%-100%) after resume or when coming back from tty to X session Status in “xorg” package in Ubuntu: Confirmed Bug description: Each time when I resume my laptop, my computer nearly freezes for about 5 minutes and Xorg is using 90-100% cpu. This issue disappears and then everything is back to normal. ProblemType: Bug DistroRelease: Ubuntu 12.04 Package: xorg 1:7.6+12ubuntu1 ProcVersionSignature: Ubuntu 3.2.0-24.38-generic 3.2.16 Uname: Linux 3.2.0-24-generic x86_64 ApportVersion: 2.0.1-0ubuntu7 Architecture: amd64 Date: Fri May 11 22:57:03 2012 EcryptfsInUse: Yes InstallationMedia: Kubuntu 11.04 "Natty Narwhal" - Release amd64 (20110427) ProcEnviron: LANGUAGE=ca_ES TERM=xterm LANG=ca_ES.UTF-8 SHELL=/bin/bash SourcePackage: xorg UpgradeStatus: Upgraded to precise on 2012-03-25 (47 days ago) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/xorg/+bug/998310/+subscriptions -- Mailing list: https://launchpad.net/~desktop-packages Post to : [email protected] Unsubscribe : https://launchpad.net/~desktop-packages More help : https://help.launchpad.net/ListHelp

