Re: btrfs and numa - needing drop_caches to keep speed up
Hi, Am 14.10.2016 um 15:19 schrieb Stefan Priebe - Profihost AG: > Dear julian, > > Am 14.10.2016 um 14:26 schrieb Julian Taylor: >> On 10/14/2016 08:28 AM, Stefan Priebe - Profihost AG wrote: >>> Hello list, >>> >>> while running the same workload on two machines (single xeon and a dual >>> xeon) both with 64GB RAM. >>> >>> I need to run echo 3 >/proc/sys/vm/drop_caches every 15-30 minutes to >>> keep the speed as good as on the non numa system. I'm not sure whether >>> this is related to numa. >>> >>> Is there any sysctl parameter to tune? >>> >>> Tested with vanilla v4.8.1 >>> >>> Greets, >>> Stefan >> >> hi, >> why do you think this is related to btrfs? > > was just an idea as i couldn't find any other difference between those > systems. > >> This is easy to diagnose but recording some kernel stacks during the > > problem with perf. > > you just mean perf top? Does it also show locking problems? As i see not > much CPU usage in that case. perf top looks like this: 5,46% libc-2.19.so [.] memset 5,26% [kernel] [k] page_fault 3,63% [kernel] [k] clear_page_c_e 1,38% [kernel] [k] _raw_spin_lock 1,06% [kernel] [k] get_page_from_freelist 0,83% [kernel] [k] copy_user_enhanced_fast_string 0,79% [kernel] [k] release_pages 0,68% [kernel] [k] handle_mm_fault 0,57% [kernel] [k] free_hot_cold_page 0,55% [kernel] [k] handle_pte_fault 0,54% [kernel] [k] __pagevec_lru_add_fn 0,45% [kernel] [k] unmap_page_range 0,45% [kernel] [k] __mod_zone_page_state 0,43% [kernel] [k] page_add_new_anon_rmap 0,38% [kernel] [k] free_pcppages_bulk > >> The only known issue that has this type of workaround that I know of are >> transparent huge pages. > > I already disabled thp by: > echo never > /sys/kernel/mm/transparent_hugepage/enabled > > cat /proc/meminfo says: > HugePages_Total: 0 > HugePages_Free:0 > HugePages_Rsvd:0 > HugePages_Surp:0 > > > > Greets, > Stefan > >> >> cheers, >> Julian -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs and numa - needing drop_caches to keep speed up
Dear julian, Am 14.10.2016 um 14:26 schrieb Julian Taylor: > On 10/14/2016 08:28 AM, Stefan Priebe - Profihost AG wrote: >> Hello list, >> >> while running the same workload on two machines (single xeon and a dual >> xeon) both with 64GB RAM. >> >> I need to run echo 3 >/proc/sys/vm/drop_caches every 15-30 minutes to >> keep the speed as good as on the non numa system. I'm not sure whether >> this is related to numa. >> >> Is there any sysctl parameter to tune? >> >> Tested with vanilla v4.8.1 >> >> Greets, >> Stefan > > hi, > why do you think this is related to btrfs? was just an idea as i couldn't find any other difference between those systems. > This is easy to diagnose but recording some kernel stacks during the > problem with perf. you just mean perf top? Does it also show locking problems? As i see not much CPU usage in that case. > The only known issue that has this type of workaround that I know of are > transparent huge pages. I already disabled thp by: echo never > /sys/kernel/mm/transparent_hugepage/enabled cat /proc/meminfo says: HugePages_Total: 0 HugePages_Free:0 HugePages_Rsvd:0 HugePages_Surp:0 Greets, Stefan > > cheers, > Julian -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs and numa - needing drop_caches to keep speed up
On 2016-10-14 02:28, Stefan Priebe - Profihost AG wrote: Hello list, while running the same workload on two machines (single xeon and a dual xeon) both with 64GB RAM. I need to run echo 3 >/proc/sys/vm/drop_caches every 15-30 minutes to keep the speed as good as on the non numa system. I'm not sure whether this is related to numa. Is there any sysctl parameter to tune? Tested with vanilla v4.8.1 This may sound odd, but does echoing 1 or 2 to /proc/sys/vm/drop_caches help at all, or is it just 3? The value itself is actually a bit-field with just two bits defined. 1 just drops the page-cache, while 2 just drops freeable SLAB objects, and 3 drops both. The thing is, both have an impact when dealing with filesystems (page-cache contains cached file contents, while freeable SLAB objects includes cached dentries and inodes), so knowing whether one or the other or only both is what's helping things can help diagnose this further. Regardless of that, you might try adjusting vm.vfs_cache_pressure. Increasing that will make the page-cache reclaim more aggressive, while decreasing it will make it less aggressive. It defaults to 100, and I wouldn't suggest setting it below 50 or above about 150. Keep in mind that increasing that will mean you're likely to put more load on the storage devices. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs and numa - needing drop_caches to keep speed up
On 10/14/2016 08:28 AM, Stefan Priebe - Profihost AG wrote: Hello list, while running the same workload on two machines (single xeon and a dual xeon) both with 64GB RAM. I need to run echo 3 >/proc/sys/vm/drop_caches every 15-30 minutes to keep the speed as good as on the non numa system. I'm not sure whether this is related to numa. Is there any sysctl parameter to tune? Tested with vanilla v4.8.1 Greets, Stefan hi, why do you think this is related to btrfs? The only known issue that has this type of workaround that I know of are transparent huge pages. This is easy to diagnose but recording some kernel stacks during the problem with perf. If there is very high system cpu usage in a spinlock called by compaction functions during page faults it is the synchronous memory defragmentation needed for thp. Should that be the case the better workaround is disabling it in /sys/kernel/mm/transparent_hugepage/defrag cheers, Julian -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
btrfs and numa - needing drop_caches to keep speed up
Hello list, while running the same workload on two machines (single xeon and a dual xeon) both with 64GB RAM. I need to run echo 3 >/proc/sys/vm/drop_caches every 15-30 minutes to keep the speed as good as on the non numa system. I'm not sure whether this is related to numa. Is there any sysctl parameter to tune? Tested with vanilla v4.8.1 Greets, Stefan -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
