Re: Call fo comments - raising vfs.ufs.dirhash_reclaimage?
On Monday, October 07, 2013 1:34:24 pm Davide Italiano wrote: What would perhaps be better than a hardcoded reclaim age would be to use an LRU-type approach and perhaps set a target percent to reclaim. That is, suppose you were to reclaim the oldest 10% of hashes on each lowmem call (and make the '10%' the tunable value). Then you will always make some amount of progress in a low memory situation (and if the situation remains dire you will eventually empty the entire cache), but the effective maximum age will be more dynamic. Right now if you haven't touched UFS in 5 seconds it throws the entire thing out on the first lowmem event. The LRU-approach would only throw the oldest 10% out on the first call, but eventually throw it all out if the situation remains dire. -- John Baldwin ___ freebsd...@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-fs To unsubscribe, send any mail to freebsd-fs-unsubscr...@freebsd.org I liked your idea more than what's available in HEAD right now and I implemented it. http://people.freebsd.org/~davide/review/ufs_direclaimage.diff I was unsure what kind of heuristic I should choose to select which (10% of) entries should be evicted so I just removed the first 10% ones from the head of the ufs_dirhash list (which should be the oldest). The code keeps rescanning the cache until 10% (or, the percentage set via SYSCTL) of the entry are freed, but probably we can discuss if this limit could be relaxed and just do a single scan over the list. Unfortunately I haven't a testcase to prove the effectiveness (or non-effectiveness) of the approach but I think either Ivan or Peter could be able to give it a spin, maybe. I think this looks good. One cosmetic nit is that I think this: if (!try_lock()) continue; else memfreed += ufsdirhash_destroy(); Looks a bit odd. I would either drop the else (which the old code did in its failsafe case) or just do this: if (try_lock()) memfreed += ufsdirhash_destroy(); -- John Baldwin ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Call fo comments - raising vfs.ufs.dirhash_reclaimage?
On Tue, 8 Oct 2013 16:01:25 -0700 Davide Italiano wrote: This could be probably changed -- from what | see even under high memory pressure this wasn't a problem but all in all I agree with you that we shouldn't loop forever but limit the number of pass on the list to a somewhat constant number, to prevent pathological cases. I don't see any need to loop. I don't believe that's true. Under most circustances the existing scheme free more memory. The only case when yours frees more is when 90% of the entries are locked. Well, no. Here you can set the threshold to a more aggressive value so that you reclaim more memory every time. Note that this was not possible in the previous version, so, if you could have a situation where all your cache entries were not touched for 15 or even 30 seconds they would have kept around, and you can destroy up to 10% of them everytime lowmem event is called. Outside of contrived stress tests I think it's very rare for a significant fraction of the cache to have been accessed in the last minute - particularly on large caches where this matters most. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Call fo comments - raising vfs.ufs.dirhash_reclaimage?
On Mon, Oct 07, 2013 at 07:34:24PM +0200, Davide Italiano wrote: What would perhaps be better than a hardcoded reclaim age would be to use an LRU-type approach and perhaps set a target percent to reclaim. That is, suppose you were to reclaim the oldest 10% of hashes on each lowmem call (and make the '10%' the tunable value). Then you will always make some amount of progress in a low memory situation (and if the situation remains dire you will eventually empty the entire cache), but the effective maximum age will be more dynamic. Right now if you haven't touched UFS in 5 seconds it throws the entire thing out on the first lowmem event. The LRU-approach would only throw the oldest 10% out on the first call, but eventually throw it all out if the situation remains dire. -- John Baldwin ___ freebsd...@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-fs To unsubscribe, send any mail to freebsd-fs-unsubscr...@freebsd.org I liked your idea more than what's available in HEAD right now and I implemented it. http://people.freebsd.org/~davide/review/ufs_direclaimage.diff I was unsure what kind of heuristic I should choose to select which (10% of) entries should be evicted so I just removed the first 10% ones from the head of the ufs_dirhash list (which should be the oldest). The code keeps rescanning the cache until 10% (or, the percentage set via SYSCTL) of the entry are freed, but probably we can discuss if this limit could be relaxed and just do a single scan over the list. Unfortunately I haven't a testcase to prove the effectiveness (or non-effectiveness) of the approach but I think either Ivan or Peter could be able to give it a spin, maybe. I gave this patch a spin for 12 hours without finding any problems. I can do more testing at a later time, if you want to. - Peter ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Call fo comments - raising vfs.ufs.dirhash_reclaimage?
Hi, Please try it out on a -10 VM with something RAM limited - say, 128mb w/ GENERIC. See how it behaves. I've successfully done buildworlds on 10-i386 with 128mb RAM. Let's try not to break that before releng/10 is cut. thanks, -adrian On 7 October 2013 23:34, Peter Holm pe...@holm.cc wrote: On Mon, Oct 07, 2013 at 07:34:24PM +0200, Davide Italiano wrote: What would perhaps be better than a hardcoded reclaim age would be to use an LRU-type approach and perhaps set a target percent to reclaim. That is, suppose you were to reclaim the oldest 10% of hashes on each lowmem call (and make the '10%' the tunable value). Then you will always make some amount of progress in a low memory situation (and if the situation remains dire you will eventually empty the entire cache), but the effective maximum age will be more dynamic. Right now if you haven't touched UFS in 5 seconds it throws the entire thing out on the first lowmem event. The LRU-approach would only throw the oldest 10% out on the first call, but eventually throw it all out if the situation remains dire. -- John Baldwin ___ freebsd...@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-fs To unsubscribe, send any mail to freebsd-fs-unsubscr...@freebsd.org I liked your idea more than what's available in HEAD right now and I implemented it. http://people.freebsd.org/~davide/review/ufs_direclaimage.diff I was unsure what kind of heuristic I should choose to select which (10% of) entries should be evicted so I just removed the first 10% ones from the head of the ufs_dirhash list (which should be the oldest). The code keeps rescanning the cache until 10% (or, the percentage set via SYSCTL) of the entry are freed, but probably we can discuss if this limit could be relaxed and just do a single scan over the list. Unfortunately I haven't a testcase to prove the effectiveness (or non-effectiveness) of the approach but I think either Ivan or Peter could be able to give it a spin, maybe. I gave this patch a spin for 12 hours without finding any problems. I can do more testing at a later time, if you want to. - Peter ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Call fo comments - raising vfs.ufs.dirhash_reclaimage?
On Tue, Oct 8, 2013 at 1:25 PM, Adrian Chadd adr...@freebsd.org wrote: Hi, Hi Adrian, Please try it out on a -10 VM with something RAM limited - say, 128mb w/ GENERIC. See how it behaves. I've successfully done buildworlds on 10-i386 with 128mb RAM. Let's try not to break that before releng/10 is cut. thanks, This is not supposed to hit 10-STABLE, at least not before proper review. This is the reason why it was proposed on mailing lists. Also, if you read the patch you'll end up with realizing this should behave better on low memory environment because it unconditionally cleans 10% of the cache every time. Previous changes in this area just did the opposite keeping a lot more of memory around. I hope this makes sense to you. Thanks, -- Davide There are no solved problems; there are only problems that are more or less solved -- Henri Poincare ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Call fo comments - raising vfs.ufs.dirhash_reclaimage?
On Tue, Oct 8, 2013 at 1:32 PM, Davide Italiano dav...@freebsd.org wrote: On Tue, Oct 8, 2013 at 1:25 PM, Adrian Chadd adr...@freebsd.org wrote: Hi, Hi Adrian, Please try it out on a -10 VM with something RAM limited - say, 128mb w/ GENERIC. See how it behaves. I've successfully done buildworlds on 10-i386 with 128mb RAM. Let's try not to break that before releng/10 is cut. thanks, This is not supposed to hit 10-STABLE, at least not before proper review. This is the reason why it was proposed on mailing lists. Also, if you read the patch you'll end up with realizing this should behave better on low memory environment because it unconditionally cleans 10% of the cache every time. Previous changes in this area just did the opposite keeping a lot more of memory around. I hope this makes sense to you. Thanks, -- Davide There are no solved problems; there are only problems that are more or less solved -- Henri Poincare Also, right now you can set up a value which indicates the percentage of memory you can reclaim. It's 10% by default, but we can discuss if this could be adjusted to a more reasonable default. Feel free to give your opinion. -- Davide There are no solved problems; there are only problems that are more or less solved -- Henri Poincare ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Call fo comments - raising vfs.ufs.dirhash_reclaimage?
On Tue, 8 Oct 2013 13:32:58 +0200 Davide Italiano wrote: On Tue, Oct 8, 2013 at 1:25 PM, Adrian Chadd adr...@freebsd.org wrote: Hi, Hi Adrian, Please try it out on a -10 VM with something RAM limited - say, 128mb w/ GENERIC. See how it behaves. Be aware that any test that doesn't cause vfs.ufs.dirhash_lowmemcount to increment isn't testing the change at all. This is not supposed to hit 10-STABLE, at least not before proper review. This is the reason why it was proposed on mailing lists. Also, if you read the patch you'll end up with realizing this should behave better on low memory environment because it unconditionally cleans 10% of the cache every time. The current version deletes anything older than the time limit and if this doesn't reclaim 10% it makes a second pass through the list until the target is met. Your version tries to delete 10% (or whatever) by looping around the list until this is achieved. And if there aren't enough unlocked entries it will loop indefinitely until there are. I hope you know what you are doing because that looks very wrong. The original version looks better to me given that it already tries to free a minimum of 10%. The low-memory handler is called under very low memory conditions when the system is probably thrashing or even on the verge of killing processes. Holding on to entries that haven't been used for a minute seems like a luxury. Previous changes in this area just did the opposite keeping a lot more of memory around. I don't believe that's true. Under most circustances the existing scheme free more memory. The only case when yours frees more is when 90% of the entries are locked. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Call fo comments - raising vfs.ufs.dirhash_reclaimage?
On Tue, Oct 8, 2013 at 3:38 PM, RW rwmailli...@googlemail.com wrote: On Tue, 8 Oct 2013 13:32:58 +0200 Davide Italiano wrote: On Tue, Oct 8, 2013 at 1:25 PM, Adrian Chadd adr...@freebsd.org wrote: Hi, Hi Adrian, Please try it out on a -10 VM with something RAM limited - say, 128mb w/ GENERIC. See how it behaves. Be aware that any test that doesn't cause vfs.ufs.dirhash_lowmemcount to increment isn't testing the change at all. This is not supposed to hit 10-STABLE, at least not before proper review. This is the reason why it was proposed on mailing lists. Also, if you read the patch you'll end up with realizing this should behave better on low memory environment because it unconditionally cleans 10% of the cache every time. The current version deletes anything older than the time limit and if this doesn't reclaim 10% it makes a second pass through the list until the target is met. Your version tries to delete 10% (or whatever) by looping around the list until this is achieved. And if there aren't enough unlocked entries it will loop indefinitely until there are. I hope you know what you are doing because that looks very wrong. Hi (RW..?), This could be probably changed -- from what | see even under high memory pressure this wasn't a problem but all in all I agree with you that we shouldn't loop forever but limit the number of pass on the list to a somewhat constant number, to prevent pathological cases. The original version looks better to me given that it already tries to free a minimum of 10%. The low-memory handler is called under very low memory conditions when the system is probably thrashing or even on the verge of killing processes. Holding on to entries that haven't been used for a minute seems like a luxury. Previous changes in this area just did the opposite keeping a lot more of memory around. I don't believe that's true. Under most circustances the existing scheme free more memory. The only case when yours frees more is when 90% of the entries are locked. Well, no. Here you can set the threshold to a more aggressive value so that you reclaim more memory every time. Note that this was not possible in the previous version, so, if you could have a situation where all your cache entries were not touched for 15 or even 30 seconds they would have kept around, and you can destroy up to 10% of them everytime lowmem event is called. Thanks, -- Davide There are no solved problems; there are only problems that are more or less solved -- Henri Poincare ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Call fo comments - raising vfs.ufs.dirhash_reclaimage?
On Wed, Aug 28, 2013 at 3:56 PM, Ivan Voras ivo...@freebsd.org wrote: Hi, Prodded by davide@, I'd like to collect opinions about raising the vfs.ufs.dirhash_reclaimage sysctl from 5 to 60, committed at: http://svnweb.freebsd.org/changeset/base/254986 What it does: Used in lowmem handler at http://fxr.watson.org/fxr/source/ufs/ufs/ufs_dirhash.c#L1247 when determining which cache entries to evict; it skips (keeps in the cache) entries which are younger than this number of seconds. This lowmem handler only frees up to 10% of the dirhash cache at a time. I don't think this is correct. The first loop scans over the whole ufsdirhash_list and there's nothing that sets the cap to 10%. It might happen that UFS is unused for some minutes and you'll end up with all the cache free'd. -- Davide There are no solved problems; there are only problems that are more or less solved -- Henri Poincare ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Call fo comments - raising vfs.ufs.dirhash_reclaimage?
What would perhaps be better than a hardcoded reclaim age would be to use an LRU-type approach and perhaps set a target percent to reclaim. That is, suppose you were to reclaim the oldest 10% of hashes on each lowmem call (and make the '10%' the tunable value). Then you will always make some amount of progress in a low memory situation (and if the situation remains dire you will eventually empty the entire cache), but the effective maximum age will be more dynamic. Right now if you haven't touched UFS in 5 seconds it throws the entire thing out on the first lowmem event. The LRU-approach would only throw the oldest 10% out on the first call, but eventually throw it all out if the situation remains dire. -- John Baldwin ___ freebsd...@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-fs To unsubscribe, send any mail to freebsd-fs-unsubscr...@freebsd.org I liked your idea more than what's available in HEAD right now and I implemented it. http://people.freebsd.org/~davide/review/ufs_direclaimage.diff I was unsure what kind of heuristic I should choose to select which (10% of) entries should be evicted so I just removed the first 10% ones from the head of the ufs_dirhash list (which should be the oldest). The code keeps rescanning the cache until 10% (or, the percentage set via SYSCTL) of the entry are freed, but probably we can discuss if this limit could be relaxed and just do a single scan over the list. Unfortunately I haven't a testcase to prove the effectiveness (or non-effectiveness) of the approach but I think either Ivan or Peter could be able to give it a spin, maybe. -- Davide There are no solved problems; there are only problems that are more or less solved -- Henri Poincare ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Call fo comments - raising vfs.ufs.dirhash_reclaimage?
On Wednesday, August 28, 2013 12:40:15 pm Ivan Voras wrote: On 28 August 2013 18:12, Gary Jennejohn gljennj...@googlemail.com wrote: So, if I understand this correctly, a normal desktop user won't notice any real change, except that buildworld might get faster, and big servers will benefit? Basically, yes, but read on... But could this negatively impact small, embedded systems, which usually have only small memory footprints? Although I suppose one could argue that they usually don't have large numbers of files cached in memory at any given time. Unless I'm wrong, the only pathological case coming from this change would be the following sequence of events: 1) Memory is scarce [*] 2) There's a sudden surge of requests for a huge number of different directories 3) There's an urgent lowmem event which is observed by dirhash, which attempts to free memory but is prevented in doing so for the next 60 seconds because all entries are young (the idea behind dirhash being that if a directory is accessed, it will probably soon be accessed again - think ls then fopen, so we won't evict him until reclaimage seconds) 4) the kernel runs out of memory, game over. Just to play devil's advocate, the only way your change can benefit is if: 1) Memory is scarce thus triggering a lowmem event 2) There are requests for a huge number of directories that haven't been accessed in over 5 seconds. That is to say, what your change does is increase the relative importance of dirhash memory relative to other memory in the machine when the machine is under memory pressure. If the machine is not under memory pressure then the lowmem handler will not be triggered and your change will never matter. Keep in mind that if pagedaemon is able to keep up, the lowmem event handler will not be called. This handler only triggers when you are really low on memory and trying to allocate it faster than pagedaemon can reclaim free pages. In that sort of environment you generally want caches to return pages sooner rather than later. What would perhaps be better than a hardcoded reclaim age would be to use an LRU-type approach and perhaps set a target percent to reclaim. That is, suppose you were to reclaim the oldest 10% of hashes on each lowmem call (and make the '10%' the tunable value). Then you will always make some amount of progress in a low memory situation (and if the situation remains dire you will eventually empty the entire cache), but the effective maximum age will be more dynamic. Right now if you haven't touched UFS in 5 seconds it throws the entire thing out on the first lowmem event. The LRU-approach would only throw the oldest 10% out on the first call, but eventually throw it all out if the situation remains dire. -- John Baldwin ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Call fo comments - raising vfs.ufs.dirhash_reclaimage?
On Wed, 28 Aug 2013 15:56:30 +0200 Ivan Voras ivo...@freebsd.org wrote: [jump to the chase] Why not leave it for sysadmins to tune it themselves if they want it: 1) They usually don't know about it until it's too late. 2) Dirhash is typically miniscule compared to todays memory sizes - a few dozen MBs even on very busy systems, and there are no typical situations where a large number of entries are filled in at the same time which block eviction of a large-ish amount of memory, so having reclaimage higher will automatically help in file-system intensive spikes without harming other uses. So, if I understand this correctly, a normal desktop user won't notice any real change, except that buildworld might get faster, and big servers will benefit? But could this negatively impact small, embedded systems, which usually have only small memory footprints? Although I suppose one could argue that they usually don't have large numbers of files cached in memory at any given time. -- Gary Jennejohn ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Call fo comments - raising vfs.ufs.dirhash_reclaimage?
On 28 August 2013 18:12, Gary Jennejohn gljennj...@googlemail.com wrote: So, if I understand this correctly, a normal desktop user won't notice any real change, except that buildworld might get faster, and big servers will benefit? Basically, yes, but read on... But could this negatively impact small, embedded systems, which usually have only small memory footprints? Although I suppose one could argue that they usually don't have large numbers of files cached in memory at any given time. Unless I'm wrong, the only pathological case coming from this change would be the following sequence of events: 1) Memory is scarce [*] 2) There's a sudden surge of requests for a huge number of different directories 3) There's an urgent lowmem event which is observed by dirhash, which attempts to free memory but is prevented in doing so for the next 60 seconds because all entries are young (the idea behind dirhash being that if a directory is accessed, it will probably soon be accessed again - think ls then fopen, so we won't evict him until reclaimage seconds) 4) the kernel runs out of memory, game over. Note that this sequence of events could still happen right now, only over a span of 5 seconds, not 60 seconds. Note also that all of this has nothing to do with regular file cache, dirhash is a very specific corner-case of UFS. [*] Keep in mind that dirhash cache even on large and busy systems is usually ~~15-25 MB; on 16 GB machines the auto-tuning code caps it at 25 MB. As an illustration on how tiny dirhash is: a du -c on /usr/ports increases dirhash_mem on my desktop from 103945 to 501507 bytes. One of the issues raised by davide is that the benefits from this are also miniscule and hard to prove. A simple buildworld is not a big enough load. I've seen on my own skin how increasing reclaimage helped, but that was under such specific circumstances that I'm still trying to figure out how to create a self-sustained benchmark (basically - how to provoke lowmem events?). Basically, this change will have no effect for 99.9% of users, but could save that 0.1% from going crazy. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
