D9840: Tentative patch to reduce I/O overhead of plasmashell when copying files

[Milian Wolff]

mwolff added a comment.


  I guess this can be abandoned now that the real culprit was found, no?

REPOSITORY
  R309 KService

REVISION DETAIL
  https://phabricator.kde.org/D9840

To: jtamate, #frameworks, dfaure
Cc: mwolff, ngraham, mmustac, michaelh

D9840: Tentative patch to reduce I/O overhead of plasmashell when copying files

[Jaime Torres Amate]

jtamate added a comment.


  In D9840#204587 , @dfaure wrote:
  
  > Can you strace vivaldi to see what it's doing? It must be doing something 
wrong if it's triggering so much I/O.
  
  
  This is related to https://phabricator.kde.org/D10342
  
  I've straced vivaldi and main process is continuously doing futex and gettid, 
and the active child the same + write + sendto. Every tab is a process, just 
like chromium.

REPOSITORY
  R309 KService

REVISION DETAIL
  https://phabricator.kde.org/D9840

To: jtamate, #frameworks, dfaure
Cc: ngraham, mmustac, michaelh

D9840: Tentative patch to reduce I/O overhead of plasmashell when copying files

[David Faure]

dfaure added a comment.


  Can you strace vivaldi to see what it's doing? It must be doing something 
wrong if it's triggering so much I/O.

REPOSITORY
  R309 KService

REVISION DETAIL
  https://phabricator.kde.org/D9840

To: jtamate, #frameworks, dfaure
Cc: ngraham, mmustac, michaelh

D9840: Tentative patch to reduce I/O overhead of plasmashell when copying files

[Jaime Torres Amate]

jtamate added a comment.


  > ! In D9840#203803 , @ngraham 
wrote:
  >  Do any of the active or recently-landed patches address this?
  
  No, but I have one in https://phabricator.kde.org/D10342

REPOSITORY
  R309 KService

REVISION DETAIL
  https://phabricator.kde.org/D9840

To: jtamate, #frameworks, dfaure
Cc: ngraham, mmustac, michaelh

D9840: Tentative patch to reduce I/O overhead of plasmashell when copying files

[Nathaniel Graham]

ngraham added a comment.


  In https://phabricator.kde.org/D9840#203563, @jtamate wrote:
  
  > Just to let you know that I know how to reproduce the high I/O of 
plasmashell and continuous calls to Kservice.
  >  You can reproduce it just having opened the vivaldi web browser. (I never 
though it could be a problem).
  >  If it goes to a webpage that still uses flash, the effect is even bigger.
  >  F5700774: plasmashell_disk.png .
  
  
  Do any of the active or recently-landed patches address this?

REPOSITORY
  R309 KService

REVISION DETAIL
  https://phabricator.kde.org/D9840

To: jtamate, #frameworks, dfaure
Cc: ngraham, mmustac, michaelh

D9840: Tentative patch to reduce I/O overhead of plasmashell when copying files

[Jaime Torres Amate]

jtamate added a comment.


  Just to let you know that I know how to reproduce the high I/O of plasmashell 
and continuous calls to Kservice.
  You can reproduce it just having opened the vivaldi web browser. (I never 
though it could be a problem).
  If it goes to a webpage that still uses flash, the effect is even bigger.
  F5700774: plasmashell_disk.png .

REPOSITORY
  R309 KService

REVISION DETAIL
  https://phabricator.kde.org/D9840

To: jtamate, #frameworks, dfaure
Cc: ngraham, mmustac, michaelh

D9840: Tentative patch to reduce I/O overhead of plasmashell when copying files

[Jaime Torres Amate]

jtamate abandoned this revision.

REPOSITORY
  R309 KService

REVISION DETAIL
  https://phabricator.kde.org/D9840

To: jtamate, #frameworks, dfaure
Cc: ngraham, mmustac

D9840: Tentative patch to reduce I/O overhead of plasmashell when copying files

[Jaime Torres Amate]

jtamate added a comment.


  Yes, you are right, it doesn't pass the unittests with this tentative patch.
  
  Some local changes should have triggered the heavy load, because now I'm not 
able to reproduce them.
  
  But there is a problem somewhere (I'll investigate further) because today, 
everytime I change directory in dolphin (even in /d/), or change tabs in 
konsole, or change file to edit in kate or save a file in kate, a messagebox is 
shown or a notification is shown, there is a check for the timestamps, and this 
check has to check all the directories in the QMap:
  
  checking file timestamps for  QMap(("/etc/xdg/menus", 
1515667084956)("/home/jtorres/.config/menus", 
1515507849227)("/home/jtorres/.local/share/applications", 
1515507849219)("/home/jtorres/.local/share/kservices5", 
0)("/home/jtorres/.local/share/kservicetypes5", 
0)("/home/jtorres/.local/share/mime", 
1512993366020)("/usr/local/share/applications", 
1515667254872)("/usr/local/share/kservices5", 
0)("/usr/local/share/kservicetypes5", 0)("/usr/local/share/mime", 
0)("/usr/share/applications", 1515667274588)("/usr/share/kservices5", 
1515667109372)("/usr/share/kservicetypes5", 1515667086640)("/usr/share/mime", 
1515667195828))
  
  And the directories watched by KDirWatch m_fileWatcher are:
  kf5.kcoreaddons.kdirwatch: Entries watched:
  kf5.kcoreaddons.kdirwatch:[ Entry for "/", dir, using INotify 
inotify_wd=6, has 0 clients , nonexistent subentries: 0x20e45f0 "/.directory" ]
  kf5.kcoreaddons.kdirwatch: dependent entries:
  kf5.kcoreaddons.kdirwatch:0x20e45f0 "/.directory" NonExistent
  kf5.kcoreaddons.kdirwatch:[ Entry for "/.directory", file, non-existent, 
using INotify inotify_wd=-1, has 2 clients ]
  kf5.kcoreaddons.kdirwatch: by  "KDirWatch-3"  ( 1  times) ""
  kf5.kcoreaddons.kdirwatch: by  "KDirWatch-5"  ( 1  times) ""
  kf5.kcoreaddons.kdirwatch:[ Entry for "/etc/xdg/taskmanagerrulesrc", 
file, using INotify inotify_wd=9, has 1 clients ]
  kf5.kcoreaddons.kdirwatch: by  "KDirWatch-8"  ( 1  times) ""
  kf5.kcoreaddons.kdirwatch:[ Entry for 
"/home/jtorres/.cache/ksycoca5_es_EE55KafscqW408BVSKZuHAbUvuI=", file, using 
INotify inotify_wd=1, has 1 clients ]
  kf5.kcoreaddons.kdirwatch: by  "KDirWatch-1"  ( 2  times) ""
  kf5.kcoreaddons.kdirwatch:[ Entry for "/home/jtorres/.config", dir, using 
INotify inotify_wd=8, has 0 clients , nonexistent subentries: 0x43c53f0 
"/home/jtorres/.config/taskmanagerrulesrc" 0x446ba20 
"/home/jtorres/.config/klaunchrc" ]
  kf5.kcoreaddons.kdirwatch: dependent entries:
  kf5.kcoreaddons.kdirwatch:0x43c53f0 
"/home/jtorres/.config/taskmanagerrulesrc" NonExistent
  kf5.kcoreaddons.kdirwatch:0x446ba20 "/home/jtorres/.config/klaunchrc" 
NonExistent
  kf5.kcoreaddons.kdirwatch:[ Entry for "/home/jtorres/.config/klaunchrc", 
file, non-existent, using INotify inotify_wd=-1, has 1 clients ]
  kf5.kcoreaddons.kdirwatch: by  "KDirWatch-9"  ( 1  times) ""
  kf5.kcoreaddons.kdirwatch:[ Entry for 
"/home/jtorres/.config/ksmserverrc", file, using INotify inotify_wd=7, has 1 
clients ]
  kf5.kcoreaddons.kdirwatch: by  "KDirWatch-7"  ( 1  times) ""
  kf5.kcoreaddons.kdirwatch:[ Entry for 
"/home/jtorres/.config/plasma-org.kde.plasma.desktop-appletsrc", file, using 
INotify inotify_wd=5, has 1 clients ]
  kf5.kcoreaddons.kdirwatch: by  "KDirWatch-2"  ( 1  times) ""
  kf5.kcoreaddons.kdirwatch:[ Entry for "/home/jtorres/.config/plasmarc", 
file, using INotify inotify_wd=2, has 1 clients ]
  kf5.kcoreaddons.kdirwatch: by  "KDirWatch-2"  ( 60  times) ""
  kf5.kcoreaddons.kdirwatch:[ Entry for 
"/home/jtorres/.config/taskmanagerrulesrc", file, non-existent, using INotify 
inotify_wd=-1, has 1 clients ]
  kf5.kcoreaddons.kdirwatch: by  "KDirWatch-8"  ( 1  times) ""
  kf5.kcoreaddons.kdirwatch:[ Entry for 
"/usr/share/icons/breeze/index.theme", file, using INotify inotify_wd=4, has 1 
clients ]
  kf5.kcoreaddons.kdirwatch: by  "KDirWatch-2"  ( 1  times) ""
  kf5.kcoreaddons.kdirwatch:[ Entry for 
"/usr/share/plasma/desktoptheme/default/metadata.desktop", file, using INotify 
inotify_wd=3, has 1 clients ]
  kf5.kcoreaddons.kdirwatch: by  "KDirWatch-2"  ( 1  times) ""
  kf5.kcoreaddons.kdirwatch:[ Entry for "/usr/share/solid/actions", dir, 
using INotify inotify_wd=10, has 1 clients ]
  kf5.kcoreaddons.kdirwatch: by  "KDirWatch-10"  ( 1  times) ""

REPOSITORY
  R309 KService

REVISION DETAIL
  https://phabricator.kde.org/D9840

To: jtamate, #frameworks, dfaure
Cc: ngraham, mmustac

D9840: Tentative patch to reduce I/O overhead of plasmashell when copying files

[David Faure]

dfaure requested changes to this revision.
dfaure added a comment.
This revision now requires changes to proceed.


  This goes against the whole redesign of ksycoca that I did some time ago. 
It's supposed to check timestamps on dirs, to rebuild the cache on demand, much 
like many other caches out there. Did you run the unittests in kservice after 
this change? I strongly doubt they pass.
  
  I don't understand why a copy triggered by the user (so, I assume, not 
related to ~/.local) would trigger an unusual effort on the part of ksycoca? 
How can this be related?
  It's just the I/O of the copying happening at the same time as the I/O from 
ksycoca when using the K menu or something?

REPOSITORY
  R309 KService

REVISION DETAIL
  https://phabricator.kde.org/D9840

To: jtamate, #frameworks, dfaure
Cc: ngraham, mmustac

D9840: Tentative patch to reduce I/O overhead of plasmashell when copying files

[Elvis Angelaccio]

elvisangelaccio added a reviewer: dfaure.

REPOSITORY
  R309 KService

REVISION DETAIL
  https://phabricator.kde.org/D9840

To: jtamate, #frameworks, dfaure
Cc: ngraham, mmustac

D9840: Tentative patch to reduce I/O overhead of plasmashell when copying files

[Jaime Torres Amate]

jtamate added a comment.


  In https://phabricator.kde.org/D9840#189862, @mmustac wrote:
  
  > Does this patch then also affects the speed of the copy operation?
  
  
  Not that I've noticed. Only the cpu in io/wait.

REPOSITORY
  R309 KService

REVISION DETAIL
  https://phabricator.kde.org/D9840

To: jtamate, #frameworks
Cc: mmustac

D9840: Tentative patch to reduce I/O overhead of plasmashell when copying files

[Marijo Mustac]

mmustac added a comment.


  Does this patch then also affects the speed of the copy operation?

REPOSITORY
  R309 KService

REVISION DETAIL
  https://phabricator.kde.org/D9840

To: jtamate, #frameworks
Cc: mmustac

D9840: Tentative patch to reduce I/O overhead of plasmashell when copying files

[Jaime Torres Amate]

jtamate created this revision.
jtamate added a reviewer: Frameworks.
Restricted Application added a project: Frameworks.
jtamate requested review of this revision.

REVISION SUMMARY
  kservice uses kdirwatch to be notified when one of the directories containing 
it's source of information changes.
  But also the method TimestampChecker().checkTimestamps(allResourceDirs) is 
called once every 1500 miliseconds.
  When copying files from any directory to another one, this check introduces a 
big use of io/wait.

TEST PLAN
  Before, when copying 20Gb files, the cpu spent in io/wait, as seen by 
ksysguard, was around 80%.
  After, when copying 20Gb files, the cpu spent in io/wait, as seen by 
ksysguard, is around 50%

REPOSITORY
  R309 KService

REVISION DETAIL
  https://phabricator.kde.org/D9840

AFFECTED FILES
  src/sycoca/ksycoca.cpp

To: jtamate, #frameworks