T11859: Understanding and reducing bad performance impact of running Baloo under certain conditions

[Kai Krakow]

hurikhan77 updated the task description.

TASK DETAIL
  https://phabricator.kde.org/T11859

To: hurikhan77
Cc: #frameworks, bruns, #baloo, hurikhan77, lots0logs, LeGast00n, cblack, 
fbampaloukas, domson, ashaposhnikov, michaelh, spoorun, ngraham, abrahams

T11859: Understanding and reducing bad performance impact of running Baloo under certain conditions

[Kai Krakow]

hurikhan77 updated the task description.

TASK DETAIL
  https://phabricator.kde.org/T11859

To: hurikhan77
Cc: #frameworks, bruns, #baloo, hurikhan77, lots0logs, LeGast00n, cblack, 
fbampaloukas, domson, ashaposhnikov, michaelh, spoorun, ngraham, abrahams

T11859: Understanding and reducing bad performance impact of running Baloo under certain conditions

[Kai Krakow]

hurikhan77 updated the task description.

TASK DETAIL
  https://phabricator.kde.org/T11859

To: hurikhan77
Cc: #frameworks, bruns, #baloo, hurikhan77, lots0logs, LeGast00n, fbampaloukas, 
GB_2, domson, ashaposhnikov, michaelh, astippich, spoorun, ngraham, abrahams

T11859: Understanding and reducing bad performance impact of running Baloo under certain conditions

[Kai Krakow]

hurikhan77 updated the task description.

TASK DETAIL
  https://phabricator.kde.org/T11859

To: hurikhan77
Cc: #frameworks, bruns, #baloo, hurikhan77, lots0logs, LeGast00n, fbampaloukas, 
GB_2, domson, ashaposhnikov, michaelh, astippich, spoorun, ngraham, abrahams

D24540: [Database] Disable read-ahead to reduce thrashing

[Kai Krakow]

hurikhan77 added a comment.


  Here's a quote:
  https://github.com/bmatsuo/lmdb-go/issues/118#issuecomment-325449496
  
  I suspect that we can safely say: The Baloo DB is always bigger than the 
available RAM (we shouldn't fight with what the desktop needs, so let's say we 
have maybe 20% of the installed RAM at our disposal which is already a high 
assumptions for typical systems).

REPOSITORY
  R293 Baloo

REVISION DETAIL
  https://phabricator.kde.org/D24540

To: hurikhan77, ngraham, #baloo, bruns
Cc: bruns, kde-frameworks-devel, ngraham, #baloo, hurikhan77, lots0logs, 
LeGast00n, fbampaloukas, GB_2, domson, ashaposhnikov, michaelh, astippich, 
spoorun, abrahams

T11859: Understanding and reducing bad performance impact of running Baloo under certain conditions

[Kai Krakow]

hurikhan77 updated the task description.

TASK DETAIL
  https://phabricator.kde.org/T11859

To: hurikhan77
Cc: #frameworks, bruns, #baloo, hurikhan77, lots0logs, LeGast00n, fbampaloukas, 
GB_2, domson, ashaposhnikov, michaelh, astippich, spoorun, ngraham, abrahams

T11859: Understanding and reducing bad performance impact of running Baloo under certain conditions

[Kai Krakow]

hurikhan77 renamed this task from "Reduce bad performance impact of running 
Baloo under certain conditions" to "Understanding and reducing bad performance 
impact of running Baloo under certain conditions".
hurikhan77 updated the task description.

TASK DETAIL
  https://phabricator.kde.org/T11859

To: hurikhan77
Cc: #frameworks, bruns, #baloo, hurikhan77, lots0logs, LeGast00n, fbampaloukas, 
GB_2, domson, ashaposhnikov, michaelh, astippich, spoorun, ngraham, abrahams

T11859: Reduce bad performance impact of running Baloo under certain conditions

[Kai Krakow]

hurikhan77 updated the task description.

TASK DETAIL
  https://phabricator.kde.org/T11859

To: hurikhan77
Cc: #frameworks, bruns, #baloo, hurikhan77, lots0logs, LeGast00n, fbampaloukas, 
GB_2, domson, ashaposhnikov, michaelh, astippich, spoorun, ngraham, abrahams

T11859: Reduce bad performance impact of running Baloo under certain conditions

[Kai Krakow]

hurikhan77 updated the task description.

TASK DETAIL
  https://phabricator.kde.org/T11859

To: hurikhan77
Cc: #frameworks, bruns, #baloo, hurikhan77, lots0logs, LeGast00n, fbampaloukas, 
GB_2, domson, ashaposhnikov, michaelh, astippich, spoorun, ngraham, abrahams

D24540: [Database] Disable read-ahead to reduce thrashing

[Kai Krakow]

hurikhan77 updated this revision to Diff 67684.
hurikhan77 edited the summary of this revision.
hurikhan77 added a comment.


  Added depend line according to arc docs

REPOSITORY
  R293 Baloo

CHANGES SINCE LAST UPDATE
  https://phabricator.kde.org/D24540?vs=67681=67684

BRANCH
  fixes/readahead

REVISION DETAIL
  https://phabricator.kde.org/D24540

AFFECTED FILES
  src/engine/database.cpp

To: hurikhan77, ngraham, #baloo, bruns
Cc: bruns, kde-frameworks-devel, ngraham, #baloo, hurikhan77, lots0logs, 
LeGast00n, fbampaloukas, GB_2, domson, ashaposhnikov, michaelh, astippich, 
spoorun, abrahams

D24540: [Database] Disable read-ahead to reduce thrashing

[Kai Krakow]

hurikhan77 updated this revision to Diff 67681.
hurikhan77 added a comment.


  Resend diff with arc

REPOSITORY
  R293 Baloo

CHANGES SINCE LAST UPDATE
  https://phabricator.kde.org/D24540?vs=67629=67681

BRANCH
  fixes/readahead

REVISION DETAIL
  https://phabricator.kde.org/D24540

AFFECTED FILES
  src/engine/database.cpp

To: hurikhan77, ngraham, #baloo, bruns
Cc: bruns, kde-frameworks-devel, ngraham, #baloo, hurikhan77, lots0logs, 
LeGast00n, fbampaloukas, GB_2, domson, ashaposhnikov, michaelh, astippich, 
spoorun, abrahams

D24539: [Database] Rework handling environment flags

[Kai Krakow]

hurikhan77 updated this revision to Diff 67680.
hurikhan77 added a comment.


  Using single-patch-per-branch approach
  
  This feels unnatural... :-\

REPOSITORY
  R293 Baloo

CHANGES SINCE LAST UPDATE
  https://phabricator.kde.org/D24539?vs=67677=67680

BRANCH
  fixes/prepare-readahead

REVISION DETAIL
  https://phabricator.kde.org/D24539

AFFECTED FILES
  src/engine/database.cpp

To: hurikhan77, ngraham, #baloo
Cc: kde-frameworks-devel, ngraham, #baloo, hurikhan77, lots0logs, LeGast00n, 
fbampaloukas, GB_2, domson, ashaposhnikov, michaelh, astippich, spoorun, bruns, 
abrahams

D24539: [Database] Rework handling environment flags

[Kai Krakow]

hurikhan77 added a comment.


  Now arc squashed all my commits into one single diff... I'm not sure if I 
like that.

REPOSITORY
  R293 Baloo

REVISION DETAIL
  https://phabricator.kde.org/D24539

To: hurikhan77, ngraham, #baloo
Cc: kde-frameworks-devel, ngraham, #baloo, hurikhan77, lots0logs, LeGast00n, 
fbampaloukas, GB_2, domson, ashaposhnikov, michaelh, astippich, spoorun, bruns, 
abrahams

D24539: [Database] Rework handling environment flags

[Kai Krakow]

hurikhan77 updated this revision to Diff 67677.
hurikhan77 added a comment.


  Rebase and submit with arc
  
  - [Database] Rework handling environment flags
  - [Database] Disable read-ahead to prevent thrashing

REPOSITORY
  R293 Baloo

CHANGES SINCE LAST UPDATE
  https://phabricator.kde.org/D24539?vs=67628=67677

BRANCH
  fixes/readahead

REVISION DETAIL
  https://phabricator.kde.org/D24539

AFFECTED FILES
  src/engine/database.cpp

To: hurikhan77, ngraham, #baloo
Cc: kde-frameworks-devel, ngraham, #baloo, hurikhan77, lots0logs, LeGast00n, 
fbampaloukas, GB_2, domson, ashaposhnikov, michaelh, astippich, spoorun, bruns, 
abrahams

D24539: [Database] Rework handling environment flags

[Kai Krakow]

hurikhan77 added a comment.


  In D24539#545019 , @ngraham wrote:
  
  > This doesn't apply cleanly on master. Can you rebase it?
  
  
  This is strange because I rebased it to latest master just before submitting 
it here. Maybe I used an outdated mirror. Let me check.
  
  > Also it would be nice to use `arc` to submit patches: 
https://community.kde.org/Infrastructure/Phabricator#Using_Arcanist
  
  Ah, there it is. It isn't in the default Gentoo repository.
  
  Will check back soon.

REPOSITORY
  R293 Baloo

REVISION DETAIL
  https://phabricator.kde.org/D24539

To: hurikhan77, ngraham, #baloo
Cc: kde-frameworks-devel, ngraham, #baloo, hurikhan77, lots0logs, LeGast00n, 
fbampaloukas, GB_2, domson, ashaposhnikov, michaelh, astippich, spoorun, bruns, 
abrahams

D24540: [Database] Disable read-ahead to reduce thrashing

[Kai Krakow]

hurikhan77 added a reviewer: Baloo.

REPOSITORY
  R293 Baloo

REVISION DETAIL
  https://phabricator.kde.org/D24540

To: hurikhan77, ngraham, #baloo
Cc: kde-frameworks-devel, ngraham, #baloo, hurikhan77, lots0logs, LeGast00n, 
fbampaloukas, GB_2, domson, ashaposhnikov, michaelh, astippich, spoorun, bruns, 
abrahams

D24539: [Database] Rework handling environment flags

[Kai Krakow]

hurikhan77 added a reviewer: Baloo.

REPOSITORY
  R293 Baloo

REVISION DETAIL
  https://phabricator.kde.org/D24539

To: hurikhan77, ngraham, #baloo
Cc: kde-frameworks-devel, ngraham, #baloo, hurikhan77, lots0logs, LeGast00n, 
fbampaloukas, GB_2, domson, ashaposhnikov, michaelh, astippich, spoorun, bruns, 
abrahams

D24540: [Database] Disable read-ahead to reduce thrashing

[Kai Krakow]

hurikhan77 added a dependency: D24539: [Database] Rework handling environment 
flags.

REPOSITORY
  R293 Baloo

REVISION DETAIL
  https://phabricator.kde.org/D24540

To: hurikhan77, ngraham
Cc: kde-frameworks-devel, ngraham, #baloo, lots0logs, LeGast00n, fbampaloukas, 
GB_2, domson, ashaposhnikov, michaelh, astippich, spoorun, bruns, abrahams

D24539: [Database] Rework handling environment flags

[Kai Krakow]

hurikhan77 added a dependent revision: D24540: [Database] Disable read-ahead to 
reduce thrashing.

REPOSITORY
  R293 Baloo

REVISION DETAIL
  https://phabricator.kde.org/D24539

To: hurikhan77, ngraham
Cc: kde-frameworks-devel, ngraham, #baloo, lots0logs, LeGast00n, fbampaloukas, 
GB_2, domson, ashaposhnikov, michaelh, astippich, spoorun, bruns, abrahams

D24540: [Database] Disable read-ahead to reduce thrashing

[Kai Krakow]

hurikhan77 created this revision.
hurikhan77 added a reviewer: ngraham.
hurikhan77 added a project: Baloo.
Herald added a project: Frameworks.
Herald added subscribers: Baloo, kde-frameworks-devel.
hurikhan77 requested review of this revision.

REVISION SUMMARY
  Prevent filesystem thrashing in low-mem situations by turning off
  read-ahead. Access patterns of baloo are unlikely to benefit from
  read-ahead and preserving cache is more valuable than reading ahead
  potentially unused data.
  
  In low-memory conditions, the random access patterns of Baloo combined
  with read-ahead tend to dominate the cache. Let Baloo be the slow player
  here, and leave system cache and memory for interactive performance.

REPOSITORY
  R293 Baloo

REVISION DETAIL
  https://phabricator.kde.org/D24540

AFFECTED FILES
  src/engine/database.cpp

To: hurikhan77, ngraham
Cc: kde-frameworks-devel, ngraham, #baloo, lots0logs, LeGast00n, fbampaloukas, 
GB_2, domson, ashaposhnikov, michaelh, astippich, spoorun, bruns, abrahams

D24539: [Database] Rework handling environment flags

[Kai Krakow]

hurikhan77 created this revision.
hurikhan77 added a reviewer: ngraham.
hurikhan77 added a project: Baloo.
Herald added a project: Frameworks.
Herald added subscribers: Baloo, kde-frameworks-devel.
hurikhan77 requested review of this revision.

REVISION SUMMARY
  This will keep the code of the following changes simpler to read. It 
introduces no functional change.

REPOSITORY
  R293 Baloo

REVISION DETAIL
  https://phabricator.kde.org/D24539

AFFECTED FILES
  src/engine/database.cpp

To: hurikhan77, ngraham
Cc: kde-frameworks-devel, ngraham, #baloo, lots0logs, LeGast00n, fbampaloukas, 
GB_2, domson, ashaposhnikov, michaelh, astippich, spoorun, bruns, abrahams

D24538: [Database] Disable read-ahead to prevent thrashing

[Kai Krakow]

hurikhan77 abandoned this revision.
hurikhan77 added a comment.


  My submission didn't work as expected...

REPOSITORY
  R293 Baloo

REVISION DETAIL
  https://phabricator.kde.org/D24538

To: hurikhan77, ngraham
Cc: kde-frameworks-devel, ngraham, #baloo, lots0logs, LeGast00n, fbampaloukas, 
GB_2, domson, ashaposhnikov, michaelh, astippich, spoorun, bruns, abrahams

D24538: [Database] Disable read-ahead to prevent thrashing

[Kai Krakow]

hurikhan77 updated this revision to Diff 67627.
hurikhan77 retitled this revision from "[Database] Rework handling environment 
flags" to "[Database] Disable read-ahead to prevent thrashing".
hurikhan77 edited the summary of this revision.
hurikhan77 added a comment.


  Prevent filesystem thrashing in low-mem situations by turning off
  read-ahead. Access patterns of baloo are unlikely to benefit from
  read-ahead and preserving cache is more valuable than reading ahead
  potentially unused data.
  
  In low-memory conditions, the random access patterns of Baloo combined
  with read-ahead tend to dominate the cache. Let Baloo be the slow player
  here, and leave system cache and memory for interactive performance.

REPOSITORY
  R293 Baloo

CHANGES SINCE LAST UPDATE
  https://phabricator.kde.org/D24538?vs=67626=67627

REVISION DETAIL
  https://phabricator.kde.org/D24538

AFFECTED FILES
  src/engine/database.cpp

To: hurikhan77, ngraham
Cc: kde-frameworks-devel, ngraham, #baloo, lots0logs, LeGast00n, fbampaloukas, 
GB_2, domson, ashaposhnikov, michaelh, astippich, spoorun, bruns, abrahams

D24538: [Database] Rework handling environment flags

[Kai Krakow]

hurikhan77 created this revision.
hurikhan77 added a reviewer: ngraham.
hurikhan77 added a project: Baloo.
Herald added a project: Frameworks.
Herald added subscribers: Baloo, kde-frameworks-devel.
hurikhan77 requested review of this revision.

REVISION SUMMARY
  This will keep the code of the following changes simpler to read. It 
introduces no functional change.

REPOSITORY
  R293 Baloo

REVISION DETAIL
  https://phabricator.kde.org/D24538

AFFECTED FILES
  src/engine/database.cpp

To: hurikhan77, ngraham
Cc: kde-frameworks-devel, ngraham, #baloo, lots0logs, LeGast00n, fbampaloukas, 
GB_2, domson, ashaposhnikov, michaelh, astippich, spoorun, bruns, abrahams

D24492: [FilteredDirIterator] Reduce stack pressure

[Kai Krakow]

hurikhan77 abandoned this revision.
hurikhan77 added a comment.


  Let's close this in favor of https://phabricator.kde.org/D24502

REPOSITORY
  R293 Baloo

REVISION DETAIL
  https://phabricator.kde.org/D24492

To: hurikhan77, #baloo
Cc: bruns, cullmann, shubham, kde-frameworks-devel, lots0logs, LeGast00n, 
fbampaloukas, GB_2, domson, ashaposhnikov, michaelh, astippich, spoorun, 
ngraham, abrahams

D24492: [FilteredDirIterator] Reduce stack pressure

[Kai Krakow]

hurikhan77 added a comment.


  I think we can also assume that using possibly unlimited recursion is also 
not good practice.
  
  Let me change that into a proper loop.

REPOSITORY
  R293 Baloo

REVISION DETAIL
  https://phabricator.kde.org/D24492

To: hurikhan77, #baloo
Cc: cullmann, shubham, kde-frameworks-devel, lots0logs, LeGast00n, 
fbampaloukas, GB_2, domson, ashaposhnikov, michaelh, astippich, spoorun, 
ngraham, bruns, abrahams

D24492: [FilteredDirIterator] Reduce stack pressure

[Kai Krakow]

hurikhan77 created this revision.
hurikhan77 added a reviewer: Baloo.
Herald added projects: Frameworks, Baloo.
Herald added a subscriber: kde-frameworks-devel.
hurikhan77 requested review of this revision.

REVISION SUMMARY
  Let's not call ourselves recursively in next() when the currently iterated 
file is going to be skipped.

REPOSITORY
  R293 Baloo

REVISION DETAIL
  https://phabricator.kde.org/D24492

AFFECTED FILES
  src/file/filtereddiriterator.cpp

To: hurikhan77, #baloo
Cc: kde-frameworks-devel, lots0logs, LeGast00n, fbampaloukas, GB_2, domson, 
ashaposhnikov, michaelh, astippich, spoorun, ngraham, bruns, abrahams