On Fri, Jul 11, 2008 at 3:59 PM, Bob Friesenhahn <[EMAIL PROTECTED]> wrote: > There are definitely issues with programs like 'ls' when listing a > directory with a million files since 'ls' sorts its output by default. > My Windows system didn't like it at all when accessing it with CIFS > and the file browser since it wants to obtain all file information > before doing anything else. System backup with hundreds of millions > of files sounds like fun.
Millions of files in a directory has historically been the path to big performance problems. Even if zfs can handle millions, other tools (ls, backup programs, etc.) will choke. Create a hierarchy and you will be much happier. FWIW, I created 10+ million files and the necessary directories to make it so that no directory had more than 10 entries (dirs or files) in it. I found the creation time to be quite steady at about 2500 file/directory creations per second over the entire exercise. I saw the kernel memory usage (kstat -p unix:0:system_pages:pp_kernel) slowly and steadily increase while arc_c slowly decreased. Out of curiosity I crashed the box then ran ::findleaks to find that there was just over 32KB leaked. I've not dug in further to see where the rest of the memory was used. In the past when I was observing file creations on UFS, VxFS, and NFS with millions of files in a single directory, the file operation time was measured in seconds per operation, rather than operations per second. This was with several (<100) processes contending for reading directory contents, file creations, and file deletions. This is where I found the script that though that "touch $dir/test.$$" (followed by rm) was the right way to check to see if a directory is writable. -- Mike Gerdts http://mgerdts.blogspot.com/ _______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
