Hi Thomas, 10.6 or 10.7 to 10.11 is a big jump with lots of changes. If you could hone it down to which major release the performance regression occurred (it might have been in 10.8, 10.9, or 10.10), it would help.
Also, an Instruments Time Profiler trace including kernel callstacks should show where the aditional time is being spent. > On Jun 3, 2016, at 3:50 AM, Thomas Tempelmann <tempelm...@gmail.com> wrote: > > As the author of a file search application, I've been contacted by a user who > claims that searching the same SMB-mounted volume on 10.6.8 was noticable > faster than with 10.11. Furthermore, a second search was significantly faster > on 10.6, whereas there was almost no speed-up on 10.11. > > For instance, the old system was able to look at all dir entries in 33 > seconds while the new system takes over 4 minutes. This is bad. > > My app simply recurses thru all folder contents, starting at the volume's > root, using FSGetCatalogInfoBulk, fetching only the file names > > I understand that... > > 1. Using FSRef based APIs requires the FS layer to cache the refs along with > their paths (because, at the lowest level, all is still path-based, right?) FSRefs are not and have never been cached. However, the data used to create an FSRef is cached by either the file system/kernel (for file systems that support VOL_CAP_FMT_PATH_FROM_ID and 32-bit IDs), or by coreservicesd. And yes, the file system is path-based. > 2. Since FSRefs have no reference counter, they are cached in an arbitrary > way and can get kicked out at any time when the OS decides it doesn't need > them any more. This will cause a cache miss next time the same Ref is > accessed, leading to a new network request, slowing everything down > significantly. Again, the caching is done at lower layers. > 3. Since my app needs to access every file only once, there's no need for > explicit caching, so there's little reason for me so far to switch to using > CFURLs - that's only needed when I have to reference the item later again. > And so I use CFURLs only if I have found a match, because then I'll keep > referencing that item for displaying it to the user. But for items that do > not match the user's search criteria, I don't need to keep them and can let > the FS cache drop them as needed. If you don't need the "track a location by file ID" behavior that FSRefs (and File Reference URLs) provides, path-based API is usually faster. Getting attributes adds to the expense. For example, on my system, if I use CFURLEnumerator to recursively enumerate "/Applications" getting file path URLs, it takes: 320616 urls and 0 errors in 1.780873 seconds With file reference URLs, it takes: 320616 urls and 0 errors in 2.551695 seconds And those times don't show the time it would take to convert the file reference URL back to its current POSIX path when it is used. > But why is 10.11 so much slower now than 10.6? > > The 10.6 runs suggest that all the FSRefs stayed in the cache so that the > second search was much faster. But with 10.11 this is not the cache any more, > so what has changed about the Cache, and can that be controlled to make it > work better again? > > Mind you, it makes no sense for me app to keep a CFURL for every item that > the app searched (i.e. for every item on the volume) because they'd all be > freed when the app quits, and thus would not keep them in cache if the user > quits the search app and re-runs it a few minutes later. Yet, with 10.6, my > app could be quit and still the next search would be faster, i.e. the FSRefs > were still cached, apparently. > > Also, why has even the initial search become slower on 10.11 vs 10.6? It's > the very same program of mine on both systems, with the same attributes > requested, so I don't think it's my app that's causing this. Rather something > changed about FSRefs caching or general smb handling that has degraded this > straight-forward performance. > > (Note: The user tried SMB1 on both OS versions, by mounting using cifs: which > I believe is the way to force SMB1. But even with SMB2 it still is slower on > 10.11 than it was on 10.6, he says). > > The server is a Windows Server, BTW. > > Actually, anohter customer just contacted me who noticed the same slowdown at > even greater intensity when he upgraded from 10.7 (I believe) to 10.11. His > disk is near-full, so probably has a highly fragmented directory, and he has > only 8 GB of RAM (while I have 20). Acitivity monitor shows that the "Cached > Files" is about 2 GB in his system, and about 5 on mine. So it seems that > recent OSX versions limit the file/dir cache to a percentage of the RAM, > whereas old systems used it all when available. Is that so? > > Is there a remedy, e.g. a setting to increase the file cache, at least > temporarily? > > -- > Thomas Tempelmann, http://www.tempel.org/ > Follow me on Twitter: https://twitter.com/tempelorg > Read my programming blog: http://blog.tempel.org/ > _______________________________________________ > Do not post admin requests to the list. They will be ignored. > Filesystem-dev mailing list (Filesystem-dev@lists.apple.com) > Help/Unsubscribe/Update your Subscription: > https://lists.apple.com/mailman/options/filesystem-dev/luther.j%40apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Filesystem-dev mailing list (Filesystem-dev@lists.apple.com) Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/filesystem-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
