Yes, I think Samba does exactly that, and netatalk doesn't. The biggest netatalk share I'm looking at has about 130,000,000 files in 13,000,000 directories.
Presumably the real villain here is the OS X Finder. When I look at it with fs_usage (an OS X tool to watch file system operations using Dtrace), it thrashes mindlessly down the folder structure of whatever folder windows happen to be open, over and over. Unfortunately, replacing the Finder (and the OS X open file dialog) is not an option here... The sad part is that performance was much better on a Synology NAS, which runs netatalk on Linux, than on this server that has 10 times the horsepower. I'm seeing hints that getcwd() is much less expensive on Linux. Best, Chris Am 06.08.2014 um 17:23 schrieb Garrett D'Amore via smartos-discuss <[email protected]>: > I was thinking about how you could do this. I think you need to cache the > cwd as you traverse. This could make the hot case fast but would fall down in > the case of symlinks or when the underlying directory changes. > > Otherwise I don't know how to make it fast. Perhaps a reverse cache indexed > by inode? > > Sent from my iPhone > >> On Aug 6, 2014, at 7:48 AM, "Robert Mustacchi via smartos-discuss" >> <[email protected]> wrote: >> >>> On 08/06/2014 07:31 AM, Chris Ferebee via smartos-discuss wrote: >>> Following up on my previous posts, I have now successfully increased the >>> size of the DNLC from 450,000 (default) to 10,000,000, by following the >>> /etc/system override method explained at >>> >>> >>> <http://dtrace.org/blogs/wesolows/2013/12/28/anonymous-tracing-on-smartos/> >>> >>> Unfortunately, this has not solved the problem. I am still seeing very poor >>> performance from the OS X 10.9 Finder when viewering directories served >>> from the SmartOS server using netatalk 3.1.2. Subjectively, performance is >>> somewhat better, but opening folders still takes 10-20 seconds when it >>> should be almost instantaneous. >>> >>> A new Flame Graph shows that now the kernel is still spending most of its >>> time in getcwd(), though now that is divided to a large amount between >>> mutex_enter() and mutex_leave(), with the lesser part actually spent in >>> dnlc_reverse_lookup(). The new Flame Graph is here: >>> >>> <http://fere.be/p/netatalk-flamegraph-2.svg> >>> >>> and a new Dtrace analysis from afp.d is here: >>> >>> <https://gist.github.com/ferebee/0f8169cd0ea19b656c51> >>> >>> Netatalk developer Ralph Böhme suggests that the observed slow performance >>> of getcwd might be specific to Illumos, and this comment from the netatalk >>> mailing list would seem to point in a similar direction: >>> >>> <http://sourceforge.net/p/netatalk/mailman/message/24468503/> >>> >>> It references a comment from the Samba source code >>> <https://ftp.samba.org/pub/unpacked/samba_3_next/source3/smbd/vfs.c> which >>> says: >>> >>> /* >>> * We don't have the information to hand so rely on traditional >>> * methods. The very slow getcwd, which spawns a process on some >>> * systems, or the not quite so bad getwd. >>> */ >>> >>> This raises the question, what is the expected behavior of getcwd() on >>> SmartOS/Illumos/Solaris? If it is in fact very slow, and netatalk relies >>> heavily on it, perhaps Illumos is not a good choice for a netatalk server? >>> >>> Or is there a way to optimize this after all within SmartOS? >> >> Hey Chris, >> >> From looking at the code, I'm not sure that increasing the size of the >> DNLC is going to actually help things. The DNLC appears to be optimized >> for going from a path to a vnode, where as when we do the reverse >> lookup, we end up actually doing an entire scan of the DNLC. >> >> While we appear to cache this data, it clearly isn't enough. Because if >> you're looking at lot of files only once and never coming back to them, >> then we're going to be going down the expensive path. >> >> There is probably some kernel work that could be done in illumos here to >> improve the uncached aspects of the lookup in the DNLC so as to make >> reverse lookups not scan the entire table. >> >> Robert >> >> >> ------------------------------------------- >> smartos-discuss >> Archives: https://www.listbox.com/member/archive/184463/=now >> RSS Feed: https://www.listbox.com/member/archive/rss/184463/22103350-51080293 >> Modify Your Subscription: https://www.listbox.com/member/?&; >> Powered by Listbox: http://www.listbox.com > > > ------------------------------------------- > smartos-discuss > Archives: https://www.listbox.com/member/archive/184463/=now > RSS Feed: https://www.listbox.com/member/archive/rss/184463/24804823-eebbfb1e > Modify Your Subscription: https://www.listbox.com/member/?&; > Powered by Listbox: http://www.listbox.com ------------------------------------------- smartos-discuss Archives: https://www.listbox.com/member/archive/184463/=now RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00 Modify Your Subscription: https://www.listbox.com/member/?member_id=25769125&id_secret=25769125-7688e9fb Powered by Listbox: http://www.listbox.com
