[frameworks-baloo] [Bug 404057] Uses an insane amount of memory (RSS/PSS) and writes a *ton* of data while indexing

2019-10-11 Thread Kai Krakow
https://bugs.kde.org/show_bug.cgi?id=404057 --- Comment #29 from Kai Krakow --- During the re-index job after reboot, baloo shows very strange behavior when evaluating the indexSize: $ balooctl indexSize File Size: 6,83 GiB Used: 25,48 MiB PostingDB: 1.018,96 MiB 3998.360 %

[frameworks-baloo] [Bug 404057] Uses an insane amount of memory (RSS/PSS) and writes a *ton* of data while indexing

2019-10-11 Thread Kai Krakow
https://bugs.kde.org/show_bug.cgi?id=404057 --- Comment #28 from Kai Krakow --- (In reply to Kai Krakow from comment #19) > After testing this a few days, with my patches it works flawlessly: No > performance impact, krunner finds result immediately without thrashing the > HDD, etc. That is,

[frameworks-baloo] [Bug 404057] Uses an insane amount of memory (RSS/PSS) and writes a *ton* of data while indexing

2019-10-11 Thread Kai Krakow
https://bugs.kde.org/show_bug.cgi?id=404057 Kai Krakow changed: What|Removed |Added Attachment #122925|0 |1 is obsolete|

[frameworks-baloo] [Bug 404057] Uses an insane amount of memory (RSS/PSS) and writes a *ton* of data while indexing

2019-10-09 Thread Kai Krakow
https://bugs.kde.org/show_bug.cgi?id=404057 Kai Krakow changed: What|Removed |Added Attachment #122918|0 |1 is obsolete|

[frameworks-baloo] [Bug 404057] Uses an insane amount of memory (RSS/PSS) and writes a *ton* of data while indexing

2019-10-09 Thread Kai Krakow
https://bugs.kde.org/show_bug.cgi?id=404057 --- Comment #25 from Kai Krakow --- Yes, that's my plan. But I'd like to refine them a bit first. Especially turning fsync() off seems to involve a big controversy discussion about whether this should be done or not. So I will research the side effects

[frameworks-baloo] [Bug 404057] Uses an insane amount of memory (RSS/PSS) and writes a *ton* of data while indexing

2019-10-09 Thread Nate Graham
https://bugs.kde.org/show_bug.cgi?id=404057 --- Comment #24 from Nate Graham --- Nice work. Would you be interested in submitting the other patches on phabricator too? -- You are receiving this mail because: You are watching all bug changes.

[frameworks-baloo] [Bug 404057] Uses an insane amount of memory (RSS/PSS) and writes a *ton* of data while indexing

2019-10-08 Thread soredake
https://bugs.kde.org/show_bug.cgi?id=404057 soredake changed: What|Removed |Added CC||fds...@krutt.org -- You are receiving this mail

[frameworks-baloo] [Bug 404057] Uses an insane amount of memory (RSS/PSS) and writes a *ton* of data while indexing

2019-10-08 Thread Kai Krakow
https://bugs.kde.org/show_bug.cgi?id=404057 --- Comment #23 from Kai Krakow --- I'll have a look at that soon. First I'd like to get the "Reduce stack pressure" patch upstreamed (also as a learning curve because this is my first KDE contribution). I've uploaded it to Phabricator and already

[frameworks-baloo] [Bug 404057] Uses an insane amount of memory (RSS/PSS) and writes a *ton* of data while indexing

2019-10-08 Thread Martin Steigerwald
https://bugs.kde.org/show_bug.cgi?id=404057 --- Comment #22 from Martin Steigerwald --- @Kai: Fine with me. I don't care all that much which bug report is about which problem. Do you have any idea why it re-indexes all the files already? -- You are receiving this mail because: You are watching

[frameworks-baloo] [Bug 404057] Uses an insane amount of memory (RSS/PSS) and writes a *ton* of data while indexing

2019-10-08 Thread Patrick Silva
https://bugs.kde.org/show_bug.cgi?id=404057 Patrick Silva changed: What|Removed |Added CC||bugsefor...@gmx.com -- You are receiving this

[frameworks-baloo] [Bug 404057] Uses an insane amount of memory (RSS/PSS) and writes a *ton* of data while indexing

2019-10-08 Thread Kai Krakow
https://bugs.kde.org/show_bug.cgi?id=404057 Kai Krakow changed: What|Removed |Added See Also||https://bugs.kde.org/show_b |

[frameworks-baloo] [Bug 404057] Uses an insane amount of memory (RSS/PSS) and writes a *ton* of data while indexing

2019-10-08 Thread Kai Krakow
https://bugs.kde.org/show_bug.cgi?id=404057 Kai Krakow changed: What|Removed |Added See Also||https://bugs.kde.org/show_b |

[frameworks-baloo] [Bug 404057] Uses an insane amount of memory (RSS/PSS) and writes a *ton* of data while indexing

2019-10-08 Thread Kai Krakow
https://bugs.kde.org/show_bug.cgi?id=404057 --- Comment #21 from Kai Krakow --- @Martin I think your bug report is already really about this issue: Re-indexing all files over and over again and consuming a lot of memory and IO that way. The performance aspects of this are already covered by

[frameworks-baloo] [Bug 404057] Uses an insane amount of memory (RSS/PSS) and writes a *ton* of data while indexing

2019-10-07 Thread Martin Steigerwald
https://bugs.kde.org/show_bug.cgi?id=404057 --- Comment #20 from Martin Steigerwald --- Dear Kai, (In reply to Kai Krakow from comment #19) > After testing this a few days, with my patches it works flawlessly: No > performance impact, krunner finds result immediately without thrashing the >

[frameworks-baloo] [Bug 404057] Uses an insane amount of memory (RSS/PSS) and writes a *ton* of data while indexing

2019-10-07 Thread Kai Krakow
https://bugs.kde.org/show_bug.cgi?id=404057 --- Comment #19 from Kai Krakow --- After testing this a few days, with my patches it works flawlessly: No performance impact, krunner finds result immediately without thrashing the HDD, etc. That is, until you reboot: While with the patches it has no

[frameworks-baloo] [Bug 404057] Uses an insane amount of memory (RSS/PSS) and writes a *ton* of data while indexing

2019-09-28 Thread Kai Krakow
https://bugs.kde.org/show_bug.cgi?id=404057 --- Comment #18 from Kai Krakow --- (In reply to Martin Steigerwald from comment #16) > I bet I can just compile baloo via kdesrc. Does it have many dependencies on > KF libraries? I do not like to break my system at this point in time and it > has

[frameworks-baloo] [Bug 404057] Uses an insane amount of memory (RSS/PSS) and writes a *ton* of data while indexing

2019-09-28 Thread Kai Krakow
https://bugs.kde.org/show_bug.cgi?id=404057 --- Comment #17 from Kai Krakow --- (In reply to Martin Steigerwald from comment #15) > Kai, thank you very much for your work on this! About an alternative to > LMDB… I am not sure at the moment. Will think about it. Currently with my patches it

[frameworks-baloo] [Bug 404057] Uses an insane amount of memory (RSS/PSS) and writes a *ton* of data while indexing

2019-09-28 Thread Martin Steigerwald
https://bugs.kde.org/show_bug.cgi?id=404057 --- Comment #16 from Martin Steigerwald --- I bet I can just compile baloo via kdesrc. Does it have many dependencies on KF libraries? I do not like to break my system at this point in time and it has been a long time I last used kdesrc. -- You are

[frameworks-baloo] [Bug 404057] Uses an insane amount of memory (RSS/PSS) and writes a *ton* of data while indexing

2019-09-28 Thread Martin Steigerwald
https://bugs.kde.org/show_bug.cgi?id=404057 --- Comment #15 from Martin Steigerwald --- Kai, thank you very much for your work on this! About an alternative to LMDB… I am not sure at the moment. Will think about it. -- You are receiving this mail because: You are watching all bug changes.

[frameworks-baloo] [Bug 404057] Uses an insane amount of memory (RSS/PSS) and writes a *ton* of data while indexing

2019-09-28 Thread Kai Krakow
https://bugs.kde.org/show_bug.cgi?id=404057 --- Comment #14 from Kai Krakow --- Meanwhile, my patched indexer started content indexing phase. I also added back all the expensive directories I excluded previously. It's currently indexing with a mixed R/W workload of up to 200 MB/s (most time

[frameworks-baloo] [Bug 404057] Uses an insane amount of memory (RSS/PSS) and writes a *ton* of data while indexing

2019-09-28 Thread Kai Krakow
https://bugs.kde.org/show_bug.cgi?id=404057 --- Comment #13 from Kai Krakow --- I've added some patches to my experimental patchset after crunching through some of the documentation and articles available. The system responsiveness has improved a lot. Can anyone confirm that these patches help?

[frameworks-baloo] [Bug 404057] Uses an insane amount of memory (RSS/PSS) and writes a *ton* of data while indexing

2019-09-28 Thread Kai Krakow
https://bugs.kde.org/show_bug.cgi?id=404057 --- Comment #12 from Kai Krakow --- Created attachment 122927 --> https://bugs.kde.org/attachment.cgi?id=122927=edit Don't fsync the file-system Let's not stress the system with fsync() after each DB transaction. This database can be easily rebuild

[frameworks-baloo] [Bug 404057] Uses an insane amount of memory (RSS/PSS) and writes a *ton* of data while indexing

2019-09-28 Thread Kai Krakow
https://bugs.kde.org/show_bug.cgi?id=404057 --- Comment #11 from Kai Krakow --- Created attachment 122926 --> https://bugs.kde.org/attachment.cgi?id=122926=edit Disable read-ahead of mmap access We should not read-ahead when accessing the database because it may introduce thrashing during

[frameworks-baloo] [Bug 404057] Uses an insane amount of memory (RSS/PSS) and writes a *ton* of data while indexing

2019-09-28 Thread Kai Krakow
https://bugs.kde.org/show_bug.cgi?id=404057 --- Comment #10 from Kai Krakow --- Created attachment 122925 --> https://bugs.kde.org/attachment.cgi?id=122925=edit Prepare simpler coding of environment flags This simply prepares the following patches and introduces no functional change. -- You

[frameworks-baloo] [Bug 404057] Uses an insane amount of memory (RSS/PSS) and writes a *ton* of data while indexing

2019-09-28 Thread Kai Krakow
https://bugs.kde.org/show_bug.cgi?id=404057 --- Comment #9 from Kai Krakow --- Here's more evidence of why LMDB may be a particularly bad choice for the workload applied by baloo: It is btree organized, and writing and maintaining btrees will result in a lot of random I/O. At some point in time,

[frameworks-baloo] [Bug 404057] Uses an insane amount of memory (RSS/PSS) and writes a *ton* of data while indexing

2019-09-28 Thread Kai Krakow
https://bugs.kde.org/show_bug.cgi?id=404057 --- Comment #8 from Kai Krakow --- Created attachment 122919 --> https://bugs.kde.org/attachment.cgi?id=122919=edit Experimental: Reduce mmap by one magnitude This patch reduces the memory map size for LMDB by one order of magnitude (16 instead of

[frameworks-baloo] [Bug 404057] Uses an insane amount of memory (RSS/PSS) and writes a *ton* of data while indexing

2019-09-28 Thread Kai Krakow
https://bugs.kde.org/show_bug.cgi?id=404057 --- Comment #7 from Kai Krakow --- Created attachment 122918 --> https://bugs.kde.org/attachment.cgi?id=122918=edit Reduce stack pressure This patch reduces pressure on the used stack size by looping instead of recursively calling itself when

[frameworks-baloo] [Bug 404057] Uses an insane amount of memory (RSS/PSS) and writes a *ton* of data while indexing

2019-04-16 Thread Martin Steigerwald
https://bugs.kde.org/show_bug.cgi?id=404057 --- Comment #6 from Martin Steigerwald --- This now happened also for the second user on this laptop. Totaling to another 16 GiB of database and baloo_file_extractor again at somewhat between 4.4 / 5 GiB RSS. So I disabled Baloo completely for the

[frameworks-baloo] [Bug 404057] Uses an insane amount of memory (RSS/PSS) and writes a *ton* of data while indexing

2019-02-08 Thread Kai Krakow
https://bugs.kde.org/show_bug.cgi?id=404057 --- Comment #5 from Kai Krakow --- @Martin Yes, you're right. Let's keep this nice and tidy. Let's see how a solution works out. I'll then look into maybe opening new issues or looking into the source code to suggest a patch. -- You are receiving

[frameworks-baloo] [Bug 404057] Uses an insane amount of memory (RSS/PSS) and writes a *ton* of data while indexing

2019-02-07 Thread Martin Steigerwald
https://bugs.kde.org/show_bug.cgi?id=404057 Martin Steigerwald changed: What|Removed |Added Ever confirmed|0 |1 Status|REPORTED

[frameworks-baloo] [Bug 404057] Uses an insane amount of memory (RSS/PSS) and writes a *ton* of data while indexing

2019-02-07 Thread Kai Krakow
https://bugs.kde.org/show_bug.cgi?id=404057 Kai Krakow changed: What|Removed |Added CC||k...@kaishome.de --- Comment #3 from Kai Krakow

[frameworks-baloo] [Bug 404057] Uses an insane amount of memory (RSS/PSS) and writes a *ton* of data while indexing

2019-02-07 Thread Martin Steigerwald
https://bugs.kde.org/show_bug.cgi?id=404057 --- Comment #2 from Martin Steigerwald --- I disabled it completely for now, cause an index size of 17 GiB, is more that I am willing to accept: LANG=C balooctl status Baloo File Indexer is running Indexer state: Idle Indexed 542390 / 559358 files

[frameworks-baloo] [Bug 404057] Uses an insane amount of memory (RSS/PSS) and writes a *ton* of data while indexing

2019-02-07 Thread Nate Graham
https://bugs.kde.org/show_bug.cgi?id=404057 Nate Graham changed: What|Removed |Added CC||n...@kde.org -- You are receiving this mail

[frameworks-baloo] [Bug 404057] Uses an insane amount of memory (RSS/PSS) and writes a *ton* of data while indexing

2019-02-07 Thread Martin Steigerwald
https://bugs.kde.org/show_bug.cgi?id=404057 --- Comment #1 from Martin Steigerwald --- I stopped it as laptop became unresponsive due to Baloo's activity. This is not going to fly. I may let it do its work over night or so when I do not use the laptop. -- You are receiving this mail because: