Thank you very much for this update. I think this thread will be very useful to guide robinhood users in their tunings.
Regards, Thomas On 06/08/15 09:57, Carmelo Ponti (CSCS) wrote: > Hi Chris and Thomas > > for completeness I can tell you that since my last email (~ mount ago) > robinhood is working well. I cannot tell you if this it's the result of > the tunning you suggested (now vfs_cache_pressure is 125) or if our > users didn't stress the file systems anymore (it's not so usual somebody > create or remove million of files in few hours), but now I'm getting a > very good read speed average (~ 10000 record/sec). > > Thank you again, > Carmelo > > On Mon, 2015-05-11 at 18:36 +0200, Carmelo Ponti (CSCS) wrote: >> Hi Chris >> >> I read something about the dangers to use vm.drop_caches so I was in >> doubt. During the weekend our users didn't stress the system, the number >> of changelog went down to ca. 0 and for this reason I cannot give you >> any useful feedback about last settings. I took advantage of the fact >> thee machine was empty to perform a "rbh-config optimize_db" which gave >> me some benefits too (now the read speed varies between 2500 to 10000 >> record/sec). Anyway I'm waiting users start stressing the machine again >> to see what it will happen. I will let you know. >> >> regards >> Carmelo >> >> On Mon, 2015-05-11 at 11:06 -0400, Chris Hunter wrote: >>> Hi Carmelo, >>> >>> Glad you to hear you are making progress. >>> vm.drop_caches is disruptive, it will likely dump the cache for your >>> local filesystem, mysql buffer, etc. Too aggressive vfs_cache_pressure >>> values can cause similar problems. >>> I found running the lru_size=clear command hourly gave about the same >>> results as setting lru_max_age to one hour, YMMV. >>> >>> regards, >>> chris hunter >>> yale hpc group >>> >>> On 05/08/2015 10:47 AM, Carmelo Ponti (CSCS) wrote: >>>> Chris >>>> >>>> On Thu, 2015-05-07 at 11:56 -0400, Chris Hunter wrote: >>>>> Hi Carmelo, >>>>> >>>>> What I found is once the local vfs cache is full, robinhood GET_INFO_FS >>>>> speed drops. You can monitor (via /proc/meminfo) buffer, cached & slab >>>>> memory usage when starting robinhood, you will see these values reach >>>>> threshold after a time. Once this state is reached, parameters >>>>> lru_max_age and vfs_cache_pressure are important. >>>> I see your point now. I increased vfs_cache_pressure to 125 and I >>>> monitoring buffer, cached & slab as you suggested. I will check on >>>> Monday and let you know. >>>> >>>>> You can also "flush" lustre cached ldlm locks by echoing "clear" to >>>>> lru_size parameter. >>>> I scheduled this every hour in cron. >>>> In addition on >>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.kernel.org_doc_Documentation_sysctl_vm.txt&d=AwICaQ&c=-dg2m7zWuuDZ0MUcV7Sdqw&r=d_G2h_sZYG4xtHMeKo8QgjDmOcMVdQvYgM-5Dri1AOY&m=2e-uiK2hYQaNO4gSQJmS-sbkcCSotxt3gGa7oYx5vds&s=j5zzS6pU4jPUsa4HvMfpFTrBqUXcffJvk08rLyK0sNg&e= >>>> I'm reading about /proc/sys/vm/drop_caches. Do you think it could help >>>> dropping slab objects and pagecache? (echo 3 >>>>> /proc/sys/vm/drop_caches). >>>> Carmelo >>>> ------------------------------------------------------------------------------ _______________________________________________ robinhood-support mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/robinhood-support
