I've been poking around at ZFS with large configurations and of late have been trying out filebench to extend and stretch it in different ways. One thing I wanted to do was create a very large number of files, ranging from 200K to 1M for loads such as varmail and webserver.
Being the impatient sort, the time it takes to do the laydown of these filesets (~50 minutes for 200K, several hours for 1M) was irksome so I decided to add 'paralloc' to the fileset definition. This, however, didn't exactly have the desired effect. To make a long story short(-ish) I'm seeing a problem where the fileset_freemem() routine is causing severe lock contention in lwpchan_delete_mapping() because of the mmap()/msync()/munmap() loop (lwpchan_delete_mapping() takes the p_lcp_lock which is per-process and there's only one filebench process with up to MAX_PARALLOC_THREADS threads). If I'm not mistaken the purpose of this loop is to evict the mapping from the cache so that when the benchmark is run it has to be pulled from disk. Since this is also accomplished by the fs_flush script I chopped out that block of code and now the same 200K fileset is written out in ~200 seconds; adding in the zpool export/import of the fs_flush script brings the total time to just about three minutes. Needless to say that improves my ability to run several iterations of slightly different tests without worrying about a aged fileset causing performance anomalies. Is my understanding of the purpose of fileset_freemem() accurate or am I missing something further? If it's accurate then is the issue I'm seeing of interest to anyone but myself, and are there better ideas of how to solve / work around the issue? In the mean time, I suppose working on my patience would be a good idea. ;) - Pete _______________________________________________ perf-discuss mailing list perf-discuss@opensolaris.org
