[frameworks-baloo] [Bug 502641] baloo hangs on file, has to be restarted, hangs again
https://bugs.kde.org/show_bug.cgi?id=502641 --- Comment #21 from Niklāvs Koļesņikovs <[email protected]> --- Seems reasonable. Thank you! I expect our distros will receive this within days or even hours of the next release containing the fix. -- You are receiving this mail because: You are watching all bug changes.
[frameworks-baloo] [Bug 502641] baloo hangs on file, has to be restarted, hangs again
https://bugs.kde.org/show_bug.cgi?id=502641 [email protected] changed: What|Removed |Added Resolution|--- |FIXED Status|REOPENED|RESOLVED --- Comment #20 from [email protected] --- (In reply to Niklāvs Koļesņikovs from comment #18) > And just to be clear, this has nothing to do with systemd, this is Baloo > limit they chose themselves. There's now a patch to set the MemoryHigh limit to 25% (instead of the rather tough 512MB) https://invent.kde.org/frameworks/baloo/-/merge_requests/277 It has been merged, not sure how long it takes to get to your distro... I think it's good to have this done. I'll close the bug now, I think the root cause is resolved... -- You are receiving this mail because: You are watching all bug changes.
[frameworks-baloo] [Bug 502641] baloo hangs on file, has to be restarted, hangs again
https://bugs.kde.org/show_bug.cgi?id=502641 --- Comment #19 from [email protected] --- Ah. So here is the right place to discuss exactly this problem. -- You are receiving this mail because: You are watching all bug changes.
[frameworks-baloo] [Bug 502641] baloo hangs on file, has to be restarted, hangs again
https://bugs.kde.org/show_bug.cgi?id=502641 --- Comment #18 from Niklāvs Koļesņikovs <[email protected]> --- And just to be clear, this has nothing to do with systemd, this is Baloo limit they chose themselves. -- You are receiving this mail because: You are watching all bug changes.
[frameworks-baloo] [Bug 502641] baloo hangs on file, has to be restarted, hangs again
https://bugs.kde.org/show_bug.cgi?id=502641 --- Comment #17 from Niklāvs Koļesņikovs <[email protected]> --- Gentoo like Arch has a long standing policy of not changing upstream defaults. It's not something you'll get through easily (don't even ask me how I know, please). -- You are receiving this mail because: You are watching all bug changes.
[frameworks-baloo] [Bug 502641] baloo hangs on file, has to be restarted, hangs again
https://bugs.kde.org/show_bug.cgi?id=502641 --- Comment #16 from [email protected] --- Coming from gentoo either the systemd people have to be convinced for this limit or gentoo has to apply its own rules. -- You are receiving this mail because: You are watching all bug changes.
[frameworks-baloo] [Bug 502641] baloo hangs on file, has to be restarted, hangs again
https://bugs.kde.org/show_bug.cgi?id=502641 --- Comment #15 from Niklāvs Koļesņikovs <[email protected]> --- The solution is as simple as MemoryHigh=10% . Also consider, if it would make sense to maybe just have a hard kill via MemoryMax=10% instead, since I really do not see the point of slowing a daemon down to an utter crawl. Ultimately those systemd limits are safeguards against bugs or other misbehavior and to me they don't seem like a good primary line of resource control, which should be handled by the daemon itself. -- You are receiving this mail because: You are watching all bug changes.
[frameworks-baloo] [Bug 502641] baloo hangs on file, has to be restarted, hangs again
https://bugs.kde.org/show_bug.cgi?id=502641 --- Comment #14 from [email protected] --- > Sorry, you got the problem when Baloo only had 512MB "space" and you still > have it with 2GB? Or did changing to 2GB help? With 2GB it seems to work correctly. > A very, very rough "rule of thumb" is that Baloo will slow down if the index > is bigger than the RAM it has been given. Maybe that will be scarcely > noticable, but it can be that Baloo fights for space when indexing... But who decides what the best size is? Currently systemd offers 512MB for everybody. This doesn't seem to fit. But I'm not familiar how such values are determined. So I leave it to the experts. -- You are receiving this mail because: You are watching all bug changes.
[frameworks-baloo] [Bug 502641] baloo hangs on file, has to be restarted, hangs again
https://bugs.kde.org/show_bug.cgi?id=502641 --- Comment #13 from [email protected] --- (In reply to peer.griebel from comment #12) > > You see Baloo using the 2G (or 4G?) you've allowed it or has it dropped back > > to 512MB? > > The problem reappeared when using the default 512MB. > Currently I'm at 2G. > > ... My index is 3.1G ... Sorry, you got the problem when Baloo only had 512MB "space" and you still have it with 2GB? Or did changing to 2GB help? A very, very rough "rule of thumb" is that Baloo will slow down if the index is bigger than the RAM it has been given. Maybe that will be scarcely noticable, but it can be that Baloo fights for space when indexing... -- You are receiving this mail because: You are watching all bug changes.
[frameworks-baloo] [Bug 502641] baloo hangs on file, has to be restarted, hangs again
https://bugs.kde.org/show_bug.cgi?id=502641 --- Comment #12 from [email protected] --- > You see Baloo using the 2G (or 4G?) you've allowed it or has it dropped back > to 512MB? The problem reappeared when using the default 512MB. Currently I'm at 2G. > Maybe it's worth asking how big the .local/share/baloo/index file is and > whether you are getting "multiple" results when you do a baloosearch6 for a > known file. My index is 3.1G. If I search for my surname I get 19389 hits. -- You are receiving this mail because: You are watching all bug changes.
[frameworks-baloo] [Bug 502641] baloo hangs on file, has to be restarted, hangs again
https://bugs.kde.org/show_bug.cgi?id=502641 --- Comment #11 from [email protected] --- (In reply to peer.griebel from comment #10) > No, it does not work. Same behavior as before :( I assume you've repeated the troubleshooting steps... You see Baloo using the 2G (or 4G?) you've allowed it or has it dropped back to 512MB? Maybe it's worth asking how big the .local/share/baloo/index file is and whether you are getting "multiple" results when you do a baloosearch6 for a known file. -- You are receiving this mail because: You are watching all bug changes.
[frameworks-baloo] [Bug 502641] baloo hangs on file, has to be restarted, hangs again
https://bugs.kde.org/show_bug.cgi?id=502641 --- Comment #10 from [email protected] --- No, it does not work. Same behavior as before :( -- You are receiving this mail because: You are watching all bug changes.
[frameworks-baloo] [Bug 502641] baloo hangs on file, has to be restarted, hangs again
https://bugs.kde.org/show_bug.cgi?id=502641 --- Comment #9 from [email protected] --- I just lightly tested the current version. Everything seems to work. Thank you for your support. -- You are receiving this mail because: You are watching all bug changes.
[frameworks-baloo] [Bug 502641] baloo hangs on file, has to be restarted, hangs again
https://bugs.kde.org/show_bug.cgi?id=502641 Sam James changed: What|Removed |Added See Also||https://bugs.gentoo.org/sho ||w_bug.cgi?id=953972 CC||[email protected] -- You are receiving this mail because: You are watching all bug changes.
[frameworks-baloo] [Bug 502641] baloo hangs on file, has to be restarted, hangs again
https://bugs.kde.org/show_bug.cgi?id=502641 --- Comment #8 from [email protected] --- (In reply to Méven from comment #7) > Here is a stack trace as baloo_file was using 93% cpu hanging... Is it managing to process any files or just spinning its wheels? You might be able to tell with iotop... It's baloo_file rather than baloo_file_extractor so it's not stuck doing content indexing, baloo_file can get overloaded if you have a large index and then do a "big delete". If you've set up and then deleted a large folder tree, Baloo removes the indexed files one by one. That's hard... I think the recommendations above to check the memory use (it Baloo up against the systemd limit?) and grant it more work space (setting MemoryHigh=25% for example). I'm afraid dumpcracking is not one of my skills -- You are receiving this mail because: You are watching all bug changes.
[frameworks-baloo] [Bug 502641] baloo hangs on file, has to be restarted, hangs again
https://bugs.kde.org/show_bug.cgi?id=502641 Méven changed: What|Removed |Added CC||[email protected] --- Comment #7 from Méven --- Here is a stack trace as baloo_file was using 93% cpu hanging: ``` 0x7f212114b972 in ?? () from /usr/lib/liblmdb.so --Type for more, q to quit, c to continue without paging-- Function(s) ^std::(move|forward|as_const|(__)?addressof) will be skipped when stepping. Function(s) ^std::(shared|unique)_ptr<.*>::(get|operator) will be skipped when stepping. Function(s) ^std::(basic_string|vector|array|deque|(forward_)?list|(unordered_|flat_)?(multi)?(map|set)|span)<.*>::(c?r?(begin|end)|front|back|data|size|empty) will be skipped when stepping. Function(s) ^std::(basic_string|vector|array|deque|span)<.*>::operator.] will be skipped when stepping. (gdb) bt #0 0x7f212114b972 in ?? () from /usr/lib/liblmdb.so #1 0x7f212114e495 in ?? () from /usr/lib/liblmdb.so #2 0x7f2121154f31 in ?? () from /usr/lib/liblmdb.so #3 0x7f21211569ef in ?? () from /usr/lib/liblmdb.so #4 0x7f212202120c in Baloo::PositionDB::del(QByteArray const&) () from /usr/lib/libKF6BalooEngine.so.6 #5 0x7f212202f528 in Baloo::WriteTransaction::commit() () from /usr/lib/libKF6BalooEngine.so.6 #6 0x7f212202f612 in Baloo::Transaction::commit() () from /usr/lib/libKF6BalooEngine.so.6 #7 0x564acff89100 in ?? () #8 0x7f2121b27b65 in ?? () from /usr/lib/libQt6Core.so.6 #9 0x7f2121b1ee69 in ?? () from /usr/lib/libQt6Core.so.6 #10 0x7f21212a57eb in ?? () from /usr/lib/libc.so.6 #11 0x7f212132918c in ?? () from /usr/lib/libc.so.6 ``` -- You are receiving this mail because: You are watching all bug changes.
[frameworks-baloo] [Bug 502641] baloo hangs on file, has to be restarted, hangs again
https://bugs.kde.org/show_bug.cgi?id=502641 --- Comment #6 from Niklāvs Koļesņikovs <[email protected]> --- Okay, so the TL;DR is that Baloo should: 1) deal with excessively large indexes, since there's a long history of that happening not due to huge filesystems but rather Baloo bugs 2) set a reasonable MemoryHigh value based on percentage as a backstop for OOM'ing the system 3) be more resource aware and either avoid using too much RAM (or even defaulting to off) when resource constrained or on the contrary going all out with locking its index to memory for best indexing performance, when there's headroom for that. Yeah, but if it the expected behavior is to consume a lot more than 512M, then I'm not sure it does anyone any good to slow Baloo to a crawl, when there's plenty of resources available. Case in point, I have 32GiB of RAM and I honestly and literally *do not care* about 512M here or there, because on a typical day I have about 20G free and Baloo is more than welcome to use a few G, if it needs to. Meanwhile on our circular economy "TV" with 2 GiB RAM and integrated graphics perhaps Baloo should either switch to a memory conserving mode or just straight up default to off, since that system barely plays YouTube without dropping frames and there's no memory, CPU or I/O budget for extras such as Baloo, that I only use on my main system for tagging family photos. Of course, disabling Baloo is the first thing I did with that system but the point of my example is that probably should be the default on resource constrained systems. Regarding that Btrfs issue, it's supposed to be fixed, although I did catch Baloo earlier this month with again an almost 1G index and stuck indexing some random file while consuming __half__ of the RAID array bandwith for a *month* (yikes and mea culpa for ignoring the unusual drive activity for so long). Presumably it was this same issue at heart but I was able to "solve" it by just banning the folders where it got stuck and purging, since I'll never need Baloo to find anything in them. This is outside my area of expertise but perhaps it would make sense to use an established database meant for storing large data sets and is already optimized for good performance on modern hardware? Furthermore, a file being memory mapped does not guarantee it's actually in the memory. The only real guarantee is to lock that index into memory but that requires either CAP_IPC_LOCK or large enough memlock (`ulimit -l`) and Baloo better makes sure that the system has enough resources to spare at all times to avoid a silly OOM situation, because the locked memory will not get paged out. -- You are receiving this mail because: You are watching all bug changes.
[frameworks-baloo] [Bug 502641] baloo hangs on file, has to be restarted, hangs again
https://bugs.kde.org/show_bug.cgi?id=502641 --- Comment #5 from [email protected] --- (In reply to Niklāvs Koļesņikovs from comment #4) > This is insane. Presumably Baloo project set that MemoryHigh=512M value, so, > if > it's known to be a problem, please raise it to a reasonable value. Previously there was no limit and the 512MB was a reaction to the problems that caused. Those problems correlated to trouble with BTRFS, where files were indexed multiple times and therefore the index grew dramatically. The two together caused havoc. > I don't know the nature of the Baloo cache The index is a memory mapped file, so "close to the metal". A record can be just pulled off disk and used so lookups are *fast*, but that means more work when indexing. I don't think Baloo can determine what pages are in memory, how many of them are clean, how many of them dirty. The systemd/cgroups limit applies external pressure on the memory and seems to work pretty well. In a way it is good to have something external to Baloo enforcing limits, it takes any arguments away from Baloo itself. I really agree that the default should be changed. I have been recommending MemoryHigh=25%, which would be no worse on a 2GB Guest VM and give far better headroom elsewhere. -- You are receiving this mail because: You are watching all bug changes.
[frameworks-baloo] [Bug 502641] baloo hangs on file, has to be restarted, hangs again
https://bugs.kde.org/show_bug.cgi?id=502641 [email protected] changed: What|Removed |Added CC||[email protected] --- Comment #1 from [email protected] --- (In reply to peer.griebel from comment #0) > 1. restart baloo (systemctl --user restart kde-baloo.service) > 2. balooctl6 status reports (very fast) that 27 files have still to be > examined > 3. balooctl6 monitor shows that some files get indexed > 4. balooctl6 status takes very long time to answer > 5. repeat whole process, yields same result > > ADDITIONAL INFORMATION > The list of files which are reported by balooctl6 monitor does not help: > Even if I remove the last files in this list, baloo stops working on some > file further to the front of the list. I've not seen exactly that behaviour but something close. When running Baloo under systemd it is limited to using 512 MB RAM. You can see the memory use with $ systemctl --user status kde-baloo Look for a "Memory:" line. If are indexing a lot of files, Baloo might be fighting to work within the constraints. Initially, you see the impact as Baloo reads and drops, rereads and drops pages from the database - it has to drop pages it has read in order to read others. If Baloo is content indexing and building a large transaction, it will not be able to "drop" those pages, it will have to ask for more memory. When reaching the 512MB limit, there'll be a gradually increasing delay in allocating it. Potentially Baloo will start swapping, which is bad news... If this it the problem, a potential "quick fix" is to allow Baloo more RAM, try 25% rather than a fixed 512MB. $ systemctl --user edit kde-baloo and add the lines to the "override" file (this change is just for your logged on user): [Service] MemoryHigh=25% and restart: $ systemctl --user restart kde-baloo > I would like to debug this problem but I do not know how. If you want to check to see whether Baloo is hitting the limits, you can use iotop and see whether Baloo has started to read and reread pages and whether the CPU use is dropping as a result of the delays in allocating more memory. An alternative is to manually index the remaining files individually with "balooctl6 index ", that indexing is done in the foreground and without the systemd limits on RAM usage. Maybe this will give error messages for files... Good luck and lets us know how the tests went... -- You are receiving this mail because: You are watching all bug changes.
[frameworks-baloo] [Bug 502641] baloo hangs on file, has to be restarted, hangs again
https://bugs.kde.org/show_bug.cgi?id=502641 Niklāvs Koļesņikovs <[email protected]> changed: What|Removed |Added CC||[email protected] Status|RESOLVED|REOPENED Resolution|FIXED |--- Ever confirmed|0 |1 --- Comment #4 from Niklāvs Koļesņikovs <[email protected]> --- This is insane. Presumably Baloo project set that MemoryHigh=512M value, so, if it's known to be a problem, please raise it to a reasonable value. With modern systems in mind you could easily do MemoryHigh=10% and it would still result in considerable increase for almost everyone but the few recluses still running on 4 GiB or less. I don't know the nature of the Baloo cache that's consuming the large amount of RAM but modern systems do have low latency and high bandwidth solid state storage as well as, I think, reasonably good OS level caching, so maybe Baloo can dial back its own caching to avoid the high memory usage in the first place? -- You are receiving this mail because: You are watching all bug changes.
[frameworks-baloo] [Bug 502641] baloo hangs on file, has to be restarted, hangs again
https://bugs.kde.org/show_bug.cgi?id=502641 --- Comment #3 from [email protected] --- (In reply to peer.griebel from comment #2) > The initial status was: Memory: 536M (high: 512M available: 0B peak: 536.3M) > But even 2G is not enough: Memory: 1.9G (high: 2G available: 32K peak: 2G) > So I set it to 4G. Baloo will use the memory as cache. It will of course be happier with more but may not need it, what you see with systemctl status on a quiet system is memory Baloo has used and wants to keep around. You might find that 2GB is quite OK... Good that the issue is sorted :-) -- You are receiving this mail because: You are watching all bug changes.
[frameworks-baloo] [Bug 502641] baloo hangs on file, has to be restarted, hangs again
https://bugs.kde.org/show_bug.cgi?id=502641 [email protected] changed: What|Removed |Added Resolution|--- |FIXED Status|REPORTED|RESOLVED --- Comment #2 from [email protected] --- (In reply to tagwerk19 from comment #1) Thank you very much for your detailed suggestions! They did the trick! The initial status was: Memory: 536M (high: 512M available: 0B peak: 536.3M) But even 2G is not enough: Memory: 1.9G (high: 2G available: 32K peak: 2G) So I set it to 4G. I think I should report it to Gentoo's bugzilla. -- You are receiving this mail because: You are watching all bug changes.
