Re: Network Trash

[Jim Luther]

Apple doesn't support trash directories (or temporary item directories) on 
network volumes and hasn't for quite a while because there is no good mechanism 
to clean up trash on network volumes (i.e., if a user mounts a network volume, 
moves files to the trash, unmounts the network volume, and never returns to 
that network volume, files are left in the trash forever). The only network 
locations that still support trash are network home directories -- in this 
case, the trash folder is within the network home directory so any user quotas 
can be enforced and documents moved to the trash are still within that user's 
control (no privacy issues).

- Jim

> On Apr 12, 2016, at 2:38 AM, Charles Francoise  wrote:
> 
> Hi,
> 
> I’m currently working on a network filesystem that can only be used by one 
> user at any time (similarly to a cloud drive).
> 
> Since only one user manipulates the files at any time, it would be 
> interesting to have a network Trash. Is this something that can be done? 
> Either by exposing a “Trash enabled” capability for the filesystem, or 
> perhaps in the kernel extension for the filesystem?
> 
> Thanks,
> --
> Charles Francoise
> Lead Mac Developer @ Lima
> char...@meetlima.com 
> 
> 
> 
> 
> 
> 
> ___
> Do not post admin requests to the list. They will be ignored.
> Filesystem-dev mailing list  (Filesystem-dev@lists.apple.com)
> Help/Unsubscribe/Update your Subscription:
> https://lists.apple.com/mailman/options/filesystem-dev/luther.j%40apple.com
> 
> This email sent to luthe...@apple.com

 ___
Do not post admin requests to the list. They will be ignored.
Filesystem-dev mailing list  (Filesystem-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/filesystem-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Re: APFS questions

[Jim Luther]

> On Jun 30, 2016, at 6:49 PM, tridiak  wrote:
> 
> Some questions concerning APFS

Have you looked at an APFS file system in the WWDV seed? Have you watched the 
WWDC presentation? Those would answer a lot of your questions.

> - My guess is resource forks are being dropped. What will happen to the API 
> that takes in account resource forks?

Resource forks are still supported.

> - What is going to happen to 'finder info’ data (getattrlist() specifically)?

ATTR_CMN_FNDRINFO is still supported.

> - Besides data type size changes, any other changes concerning the stat() 
> family and get/setattrlist()?

The stat() API work the same.

Both getattrlist() and setattrlist() work the same.

The ATTR_CMN_BKUPTIME is no longer supported (it has been considered deprecated 
functionality since the Carbon File Manager was deprecated).

ATTR_VOL_ATTRIBUTES can be used to determine what attributes are supported on a 
particular file system including APFS. (Note: the ATTR_CMN_BKUPTIME is still 
returned set by ATTR_VOL_ATTRIBUTES in the current seed but that will be fixed).

> - Will there be a way to force the filing to actually create a new copy of a 
> file instead of just a reference?

Use copyfile(3) without either the COPYFILE_CLONE_FORCE or COPYFILE_CLONE flag. 
However, there are many reasons why you wouldn't want a clone. It is faster, 
saves disk space, and causes fewer writes (which will make SSDs last longer).

> - Is file cloning ‘same-volume-only’ or if a file is copied to a different 
> volume, will it still be a clone?

I believe a clone is always on the same filesystem.

> 
> Mark
> ___
> Do not post admin requests to the list. They will be ignored.
> Filesystem-dev mailing list  (Filesystem-dev@lists.apple.com)
> Help/Unsubscribe/Update your Subscription:
> https://lists.apple.com/mailman/options/filesystem-dev/luther.j%40apple.com
> 
> This email sent to luthe...@apple.com


 ___
Do not post admin requests to the list. They will be ignored.
Filesystem-dev mailing list  (Filesystem-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/filesystem-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Re: NSURL getResourceValue: for NSURLVolumeMaximumFileSizeKey returns nil

[Jim Luther]

> On Apr 3, 2017, at 8:56 AM, Scott Talbert <s...@techie.net> wrote:
> 
> It's getting a value from the getattrlist path.
> 
> The fact that the data is cached at mount time is an interesting clue. It 
> suggest that perhaps there may be some window just shortly after mount that 
> the incorrect data is getting reported and cached.
> 
> It's a FUSE filesystem.  However, some of the more commonly-used FUSE 
> filesystems (e.g., sshfs) seem to behave the same way.

That sounds like a bug in FUSE. Once a file system is mounted, the kernel sends 
out a VQ_MOUNT kernel event and that's what causes the kNotifyVFSMount BSD 
notification to be sent. At that time, file systems must be ready to respond to 
requests correctly. If they don't, they have a bug.

- Jim

> On Sat, 1 Apr 2017, Jim Luther wrote:
> 
>> Scott, if you walk through that code, do you see which path it's getting a 
>> value from -- pathconf or getattrlist?
>> 
>> The code in our frameworks that gets that value runs when a file system is 
>> first mounted. We have code listening for kNotifyVFSMount BSD notifications. 
>> When a notification comes in, we look to see what new volume(s) are 
>> available, and then collect and cache the volume's immutable properties 
>> (like this capability).
>> 
>> Knowing more about the file system (what file system type, how it's 
>> connected, etc) where you are seeing the problem might help diagnose things.
>> 
>> - Jim
>> 
>>> On Mar 31, 2017, at 5:10 PM, Scott Talbert <s...@techie.net> wrote:
>>> 
>>> Thanks for the information.  Unfortunately, that leaves me even more
>>> confused as, using that code, I get the correct answer:
>>> 
>>> #include 
>>> #include 
>>> #include 
>>> #import 
>>> 
>>> int getMaxFileSize(const char *path, int64_t *maxFileSize)
>>> {
>>>   int result;
>>>   struct statfs statfsBuf;
>>> 
>>> 
>>>   // get the file system's mount point path for the input path
>>>   result = statfs(path, );
>>>   if ( result == 0 ) {
>>>   long fileSizeBits = pathconf(statfsBuf.f_mntonname,
>>> _PC_FILESIZEBITS);
>>>   if (fileSizeBits == -1) {
>>>   // if _PC_FILESIZEBITS isn't supported, check for
>>> VOL_CAP_FMT_2TB_FILESIZE
>>>   bool fileSystemSupports2TBFileSize = false;
>>> 
>>> 
>>>   // get the supported capabilities
>>>   struct attrlist attrList;
>>>   struct volCapabilitiesBuf {
>>>   u_int32_t length;
>>>   vol_capabilities_attr_t capabilities;
>>>   } __attribute__((aligned(4), packed));
>>>   struct volCapabilitiesBuf volCaps;
>>> 
>>> 
>>>   memset(, 0, sizeof(attrList));
>>>   attrList.bitmapcount = ATTR_BIT_MAP_COUNT;
>>>   attrList.volattr = ATTR_VOL_INFO | ATTR_VOL_CAPABILITIES;
>>>   result = getattrlist(statfsBuf.f_mntonname, ,
>>> , sizeof(volCaps), 0);
>>>   if ( result == 0 ) {
>>>   fileSystemSupports2TBFileSize =
>>> ((volCaps.capabilities.capabilities[VOL_CAPABILITIES_FORMAT]
>>> & VOL_CAP_FMT_2TB_FILESIZE) &&
>>> (volCaps.capabilities.valid[VOL_CAPABILITIES_FORMAT] &
>>> VOL_CAP_FMT_2TB_FILESIZE));
>>>   }
>>> 
>>> 
>>>   if ( fileSystemSupports2TBFileSize ) {
>>>   // use Supports2TBFileSize
>>>   *maxFileSize = LONG_LONG_MAX;
>>>   }
>>>   else {
>>>   // otherwise, we don't know
>>>   *maxFileSize = -1LL;
>>>   }
>>>   }
>>>   else if ( fileSizeBits > 64 ) {
>>>   // off_t is signed long long, so it cannot be more than
>>> LONG_LONG_MAX
>>>   *maxFileSize = LONG_LONG_MAX;
>>>   }
>>>   else if ( fileSizeBits < 32 ) {
>>>   // POSIX spec says 'Minimum Acceptable Value: 32' for
>>> FILESIZEBITS
>>>   *maxFileSize = INT_MAX;
>>>   }
>>>   else {
>>>   // 32...64 bits: shift off the bits we don't need
>>>   *maxFileSize = (int64_t)((u_int64_t)0xLL >>
>>> (u_int64_t)(65 - fileSizeBits));
>>>   }
>>>   }
>>>   return result;
>>> }
>>> 
>>> int main()
>>> {
>>> const char *mount

Re: NSURL getResourceValue: for NSURLVolumeMaximumFileSizeKey returns nil

[Jim Luther]

As -[NSURL getResourceValue:forKey:error:] is documented, "If this method 
returns YES and the value is populated with nil, it means that the resource 
property is not available for the specified resource, and that no errors 
occurred when determining that the resource property was unavailable." So YES 
and nil is a valid response.

The NSURLVolumeMaximumFileSizeKey property value comes from two sources: 
pathconf() with _PC_FILESIZEBITS, or from getattrlist() from the 
ATTR_VOL_CAPABILITIES attribute and the VOL_CAP_FMT_2TB_FILESIZE capability.

Here's code (not the real code but the exact same algorithm) that shows how the 
value is calculated:

int getMaxFileSize(const char *path, int64_t *maxFileSize)
{
int result;
struct statfs statfsBuf;

// get the file system's mount point path for the input path
result = statfs(path, );
if ( result == 0 ) {
long fileSizeBits = pathconf(statfsBuf.f_mntonname, _PC_FILESIZEBITS);
if (fileSizeBits == -1) {
// if _PC_FILESIZEBITS isn't supported, check for 
VOL_CAP_FMT_2TB_FILESIZE
bool fileSystemSupports2TBFileSize = false;

// get the supported capabilities
struct attrlist attrList;
struct volCapabilitiesBuf {
u_int32_t length;
vol_capabilities_attr_t capabilities;
} __attribute__((aligned(4), packed));
struct volCapabilitiesBuf volCaps;

memset(, 0, sizeof(attrList));
attrList.bitmapcount = ATTR_BIT_MAP_COUNT;
attrList.volattr = ATTR_VOL_INFO | ATTR_VOL_CAPABILITIES;
result = getattrlist(statfsBuf.f_mntonname, , , 
sizeof(volCaps), 0);
if ( result == 0 ) {
fileSystemSupports2TBFileSize = 
((volCaps.capabilities.capabilities[VOL_CAPABILITIES_FORMAT] & 
VOL_CAP_FMT_2TB_FILESIZE) && 
(volCaps.capabilities.valid[VOL_CAPABILITIES_FORMAT] & 
VOL_CAP_FMT_2TB_FILESIZE));
}

if ( fileSystemSupports2TBFileSize ) {
// use Supports2TBFileSize
*maxFileSize = LONG_LONG_MAX;
}
else {
// otherwise, we don't know
*maxFileSize = -1LL;
}
}
else if ( fileSizeBits > 64 ) {
// off_t is signed long long, so it cannot be more than 
LONG_LONG_MAX
*maxFileSize = LONG_LONG_MAX;
}
else if ( fileSizeBits < 32 ) {
// POSIX spec says 'Minimum Acceptable Value: 32' for FILESIZEBITS
*maxFileSize = INT_MAX;
}
else {
// 32...64 bits: shift off the bits we don't need
*maxFileSize = (int64_t)((u_int64_t)0xLL >> 
(u_int64_t)(65 - fileSizeBits));
}
}
return result;
}

if NSURLVolumeMaximumFileSizeKey is returning nil with no errors, maxFileSize 
is -1LL, and the only way maxFileSize will be -1LL is if pathconf() returned an 
error, and the file system is saying VOL_CAP_FMT_2TB_FILESIZE is not valid and 
set (capabilities).

Hope that helps™

- Jim

> On Mar 31, 2017, at 2:33 PM, Scott Talbert  wrote:
> 
> Hello,
> 
> Can anyone tell me why calling NSURL -getResourceValue:forKey:error: with key 
> NSURLVolumeMaximumFileSizeKey would return a nil resource value for a given 
> volume?  The function returns YES and error is nil also.
> 
> I can see from running dtruss that it in turn calls getattrlist().  If I call 
> getattrlist() myself on the same volume, I can see the bits I would think it 
> is looking at are set correctly - VOL_CAP_FMT_2TB_FILESIZE.
> 
> Thanks,
> Scott
> ___
> Do not post admin requests to the list. They will be ignored.
> Filesystem-dev mailing list  (Filesystem-dev@lists.apple.com)
> Help/Unsubscribe/Update your Subscription:
> https://lists.apple.com/mailman/options/filesystem-dev/luther.j%40apple.com
> 
> This email sent to luthe...@apple.com

 ___
Do not post admin requests to the list. They will be ignored.
Filesystem-dev mailing list  (Filesystem-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/filesystem-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Re: NSURL getResourceValue: for NSURLVolumeMaximumFileSizeKey returns nil

[Jim Luther]

No. Immutable properties and capabilities like NSURLVolumeMaximumFileSizeKey 
are cached by coreservicesd on macos. The URL methods only remove cached values 
from the URL object.

> On Apr 5, 2017, at 6:29 AM, Scott Talbert <s...@techie.net> wrote:
> 
> Do -[NSURL removeCachedResourceValueForKey:] and -[NSURL 
> removeAllCachedResourceValues] clear the cached information in this case? It 
> doesn't seem that they do.
> 
> Scott
> 
> On Mon, 3 Apr 2017, Scott Talbert wrote:
> 
>> It sure sounds that way.  However, I haven't been able to spot a problem in 
>> the FUSE kext thus far.  It seems to be setting VOL_CAP_FMT_2TB_FILESIZE 
>> when it should be.
>> 
>> On Mon, 3 Apr 2017, Jim Luther wrote:
>> 
>>> 
>>>  On Apr 3, 2017, at 8:56 AM, Scott Talbert <s...@techie.net>
>>>  wrote:
>>> It's getting a value from the getattrlist path.
>>> The fact that the data is cached at mount time is an interesting
>>> clue. It suggest that perhaps there may be some window just shortly
>>> after mount that the incorrect data is getting reported and cached.
>>> It's a FUSE filesystem.  However, some of the more commonly-used FUSE
>>> filesystems (e.g., sshfs) seem to behave the same way.
>>> That sounds like a bug in FUSE. Once a file system is mounted, the kernel
>>> sends out a VQ_MOUNT kernel event and that's what causes the
>>> kNotifyVFSMount BSD notification to be sent. At that time, file systems
>>> must be ready to respond to requests correctly. If they don't, they have a
>>> bug.
>>> - Jim
>>> 
>>>  On Sat, 1 Apr 2017, Jim Luther wrote:
>>> 
>>>Scott, if you walk through that code, do you see
>>>which path it's getting a value from -- pathconf or
>>>getattrlist?
>>> 
>>>The code in our frameworks that gets that value
>>>runs when a file system is first mounted. We have
>>>code listening for kNotifyVFSMount BSD
>>>notifications. When a notification comes in, we
>>>look to see what new volume(s) are available, and
>>>then collect and cache the volume's immutable
>>>properties (like this capability).
>>> 
>>>Knowing more about the file system (what file
>>>system type, how it's connected, etc) where you are
>>>seeing the problem might help diagnose things.
>>> 
>>>- Jim
>>> 
>>>  On Mar 31, 2017, at 5:10 PM, Scott
>>>  Talbert <s...@techie.net> wrote:
>>> 
>>>  Thanks for the information.
>>>   Unfortunately, that leaves me even
>>>  more
>>>  confused as, using that code, I get the
>>>  correct answer:
>>> 
>>>  #include 
>>>  #include 
>>>  #include 
>>>  #import 
>>> 
>>>  int getMaxFileSize(const char *path,
>>>  int64_t *maxFileSize)
>>>  {
>>>int result;
>>>struct statfs statfsBuf;
>>> 
>>>// get the file system's mount point
>>>  path for the input path
>>>result = statfs(path, );
>>>if ( result == 0 ) {
>>>long fileSizeBits =
>>>  pathconf(statfsBuf.f_mntonname,
>>>  _PC_FILESIZEBITS);
>>>if (fileSizeBits == -1) {
>>>// if _PC_FILESIZEBITS isn't
>>>  supported, check for
>>>  VOL_CAP_FMT_2TB_FILESIZE
>>>bool
>>>  fileSystemSupports2TBFileSize = false;
>>> 
>>>// get the supported
>>>  capabilities
>>>struct attrlist attrList;
>>>struct volCapabilitiesBuf {
>>>u_int32_t length;
>>>vol_capabilities_attr_t
>>>  capabilities;
>>>} __attribute__((aligned(4),
>>>  packed));
>>>  

Re: searchfs support on APFS

[Jim Luther]

FSCatalogSearch is part of the deprecated Carbon File Manager API. The Carbon 
File Manager API exposes file IDs and directory IDs as 32-bit values and so 
FSCatalogSearch will never be supported on APFS (which use 64-bit file IDs and 
directory IDs).

In 10.13:
• PBCatSearch on APFS volumes will return paramErr.
• FSCatalogSearch on APFS volumes will return errFSOperationNotSupported.
• PBGetVolParms and FSGetVolumeParms on APFS volumes will return with 
vMAttrib.bHasCatSearch and vMExtendedAttributes.bSupportsFSCatalogSearch clear.

(There was a bug that made it look like PBCatSearch and FSCatalogSearch were 
supported on APFS.)

> On Jul 6, 2017, at 7:49 AM, Thomas Tempelmann  wrote:
> 
> Two things:
> 
> 1.
> A long while ago the APFS team at Apple asked for input on what's needed - I 
> then asked for fast catalog search support, in the ways of the searchfs BSD 
> function. Has anything happened in that direction?
> 
> 2.
> And then there's a possible issue around reporting support for this operation 
> (as of 10.13 beta 2):
> 
> My search tool unsuccessfully attempts to use FSCatalogSearch on APFS volumes 
> because it's being told by the API that the APFS volume supports that 
> operation. Which it doesn't do, though, as my app then gets no results from 
> the search. I haven't been able to look deeper into this yet, I get this from 
> customer reports that all have the same problem now after installing High 
> Sierra and upgrading to APFS.
> 
> So, could it be that the volume flags are incorrectly set, indicating that 
> the file system supports CatalogSearch even though it doesn't? (I'm just 
> throwing this in as early as possible, but I'll make sure to file a bug 
> report should my suspicion be confirmed).
> 
> -- 
> Thomas Tempelmann, http://www.tempel.org/ 
> Follow me on Twitter: https://twitter.com/tempelorg 
> 
> Read my programming blog: http://blog.tempel.org/ 
> ___
> Do not post admin requests to the list. They will be ignored.
> Filesystem-dev mailing list  (Filesystem-dev@lists.apple.com)
> Help/Unsubscribe/Update your Subscription:
> https://lists.apple.com/mailman/options/filesystem-dev/luther.j%40apple.com
> 
> This email sent to luthe...@apple.com

 ___
Do not post admin requests to the list. They will be ignored.
Filesystem-dev mailing list  (Filesystem-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/filesystem-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Re: from sidebar to NSURLDocumentIdentifierKey

[Jim Luther]

As things work today...

When a user drags a file system volume out the Devices sidebar list, the 
Visibility is set to NeverVisible for that device in the list (as you can see 
below). The Bookmark is a URL bookmark object (serialized) used to determine 
which file system volume should be hidden in the future. After reboot, the 
Bookmark is resolved to find the file system volume. If the entry is 
NeverVisible, that file system volume is not shown in the Devices sidebar list. 
So, it sounds like the Bookmark to your device (file system volume) don't 
resolve between boots (or maybe between unmounts and remounts).

What happens if you create a Finder Alias file to your filesystem, reboot, and 
then attempt to resolve the Finder Alias file? i.e.

1 - Select your file system volume in the Finder.
2 - Use File->Make Alias (command-L) to create a Finder Alias file.
3 - Reboot
4 - Open (double-click) the Alias file in the Finder.

It should resolve to your file system volume and open it in a Finder window. If 
it doesn't, let's continue to talk.

- Jim

> On Jul 25, 2017, at 10:08 PM, Jorgen Lundman  wrote:
> 
> 
> Hello list,
> 
> The issue that started it:
> 
> Finder has a sidebar, showing all the mounted devices. You can drag one of
> these devices out to "remove" it, and Finder won't bother to show it again,
> after reboots etc. This does not work for us, each reboot brings back the
> device/mount.
> 
> 
> Best guess is that it is the
> 
> ./Library/Preferences/com.apple.sidebarlists.plist
> 
> file that controls this, and when you remove a device (fs2 in this case) it
> adds an entry for it:
> 
>   CustomItemProperties
>   
> com.apple.LSSharedFileList.TemplateSystemSelector
>   1935821166
>   
>   EntryType
>   261
>   Name
>   fs2
>   Visibility
>   NeverVisible
>   Bookmark
>   
>   $BASE64
>   
> 
> Inside the $BASE64 data, best guess is that this is the line:
> 
> NSURLDocumentIdentifierKey:
> 
> da47360e295885ecd68baeb6197af4a6f5745705;;;0020;com.apple.app-sandbox.read-write;0001;3206;0002;/volumes/boom/fs2
> 
> and this value appears to change between boots:
> 
> d28986a7141072273d5c69996b92d5e909e49a32;;;0020;com.apple.app-sandbox.read-write;0001;3206;0002;/volumes/boom/fs2
> 
> So if all those guesses are good, the next question is then, how does
> NSURLDocumentIdentifierKey produce it. There is practically no information
> on it that I can find.
> 
> I do suspect that it sets UF_TRACKED on the mountpoint (/volumes/boom/fs2)
> and using ATTR_CMN_DOCUMENT_ID to get something persistent.
> 
> But I can confirm that we handle UF_TRACKED, and set va_document_id
> consistently, between boots:
> 
> zfs_setattr_generate_id: 'fs2' -> 3779942736 (0xE14D5950)
> 
> zfs_setattr_generate_id: 'fs2' -> 3779942736 (0xE14D5950)
> 
> 
> I also compiled the hfs/test/test-doc_tombstone.c file, which passes
> without causing asserts. So I feel confident that we handle DOCUMENT_ID as
> expected.
> 
> So perhaps NSURLDocumentIdentifierKey mixes in some other information to
> generate its identifier?
> 
> Are there any hints as to what it could be?
> 
> Am I even barking at the right tree, looking at
> sidebarlists.plist:bookmark:data:unknownbinaryblob ?
> 
> Lund
> 
> 
> -- 
> Jorgen Lundman   | 
> Unix Administrator   | +81 (0)90-5578-8500
> Shibuya-ku, Tokyo| Japan
> 
> ___
> Do not post admin requests to the list. They will be ignored.
> Filesystem-dev mailing list  (Filesystem-dev@lists.apple.com)
> Help/Unsubscribe/Update your Subscription:
> https://lists.apple.com/mailman/options/filesystem-dev/luther.j%40apple.com
> 
> This email sent to luthe...@apple.com

 ___
Do not post admin requests to the list. They will be ignored.
Filesystem-dev mailing list  (Filesystem-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/filesystem-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Re: from sidebar to NSURLDocumentIdentifierKey

[Jim Luther]

What version of the system? I've been told the dev seeds have a bug in this 
area.

> On Jul 26, 2017, at 5:10 PM, Jorgen Lundman <lund...@lundman.net> wrote:
> 
> 
> Jim,
> 
> I have created an alias to "fs2", called "fs2 alias" in Desktop. I have no
> problems following (double clicking) the alias after reboots. Even reboots
> where a removed "fs2" comes back to the sidebar.
> 
> 
> There is one other thing I had noticed is broken, but I hesitate to mention
> it in case it just adds confusion (ie, another unrelated problem we need to
> fix later).
> 
> Feel free to ignore this following section:
> 
> 
> The sidebarlist.plist can also have an "Alias" section, as well as
> "Bookmark". But the alias section is not always present, so I assume the
> magic is in the Bookmark.  But, there is something curious in Alias's data:
> 
> A working HFS entry's alias's data field:
> 
> 0040 00 14 00 09 00 54 00 65 00 73 00 74 00 65 00 72 |.T.e.s.t.e.r|
> 0050 00 20 00 48 00 44 00 0f 00 14 00 09 00 54 00 65 |. .H.D...T.e|
> 0060 00 73 00 74 00 65 00 72 00 20 00 48 00 44 00 12 |.s.t.e.r. .H.D..|
> 0070 00 00 00 13 00 01 2f 00 ff ff 00 00 |../.|
> 
> My filesystem's section:
> 
> 0040 00 66 00 73 00 33 00 0f 00 08 00 03 00 66 00 73 |.f.s.3...f.s|
> 0050 00 33 00 12 00 00 00 13 00 11 2f 56 6f 6c 75 6d |.3/Volum|
> 0060 65 73 2f 42 4f 4f 4d 2f 66 73 33 00 ff ff 00 00 |es/BOOM/fs3.|
> 
> First difference is HFS has just volume name "Tester HD" and the other has
> the mountpoint "/Volumes/BOOM/fs3". Curious.
> 
> But also, the HFS entry is 2-byte chars (utf16?) whereas mine is single
> byte chars, but only on the volume name, the filesystem name is "correct".
> 
> AFAIK, the only places to query VolumeName is vfs_getattr() and calling
> "Filesystem/fs.bundle/fs.util -p" - both which are single byte chars. So it
> is curious that the working HFS somehow ends up with 2 byte chars.
> 
> When trying to parse the Alias data, it will fail as the string length 36,
> being longer than the rest of the data, presumably because the length 18
> (0x12) as been doubled in anticipation of 2 byte chars...
> 
> Lund
> 
> 
> Jim Luther wrote:
>> As things work today...
>> 
>> When a user drags a file system volume out the Devices sidebar list, the 
>> Visibility is set to NeverVisible for that device in the list (as you can 
>> see below). The Bookmark is a URL bookmark object (serialized) used to 
>> determine which file system volume should be hidden in the future. After 
>> reboot, the Bookmark is resolved to find the file system volume. If the 
>> entry is NeverVisible, that file system volume is not shown in the Devices 
>> sidebar list. So, it sounds like the Bookmark to your device (file system 
>> volume) don't resolve between boots (or maybe between unmounts and remounts).
>> 
>> What happens if you create a Finder Alias file to your filesystem, reboot, 
>> and then attempt to resolve the Finder Alias file? i.e.
>> 
>> 1 - Select your file system volume in the Finder.
>> 2 - Use File->Make Alias (command-L) to create a Finder Alias file.
>> 3 - Reboot
>> 4 - Open (double-click) the Alias file in the Finder.
>> 
>> It should resolve to your file system volume and open it in a Finder window. 
>> If it doesn't, let's continue to talk.
>> 
>> - Jim
>> 
>>> On Jul 25, 2017, at 10:08 PM, Jorgen Lundman <lund...@lundman.net> wrote:
>>> 
>>> 
>>> Hello list,
>>> 
>>> The issue that started it:
>>> 
>>> Finder has a sidebar, showing all the mounted devices. You can drag one of
>>> these devices out to "remove" it, and Finder won't bother to show it again,
>>> after reboots etc. This does not work for us, each reboot brings back the
>>> device/mount.
>>> 
>>> 
>>> Best guess is that it is the
>>> 
>>> ./Library/Preferences/com.apple.sidebarlists.plist
>>> 
>>> file that controls this, and when you remove a device (fs2 in this case) it
>>> adds an entry for it:
>>> 
>>> CustomItemProperties
>>> 
>>> com.apple.LSSharedFileList.TemplateSystemSelector
>>> 1935821166
>>> 
>>> EntryType
>>> 

Re: So what does VOL_CAP_INT_USERACCESS actually mean?

[Jim Luther]

Here's something that is kind of related that I ran into recently.

If you have code that is checking the ownership and permissions of a file 
system object and attempting to correct the ownership and permissions if it is 
wrong, don't bother if the MNT_IGNORE_OWNERSHIP mount flag is set -- nothing 
will change . In addition, that attempt to set the ownership and permissions 
WILL cause an FSEvent which causes the Finder (and other things listening to 
FSEvents) do wake up and do some work.

I had some code which was called quite often and was making that useless 
attempt to set the ownership and permissions -- it kept the Finder very busy. 
Oops.

- Jim

> On Feb 23, 2018, at 5:17 PM, Vivek Verma  wrote:
> 
> 
>> On Feb 23, 2018, at 3:56 PM, James Bucanek  wrote:
>> 
>> Hello,
>> 
>> I have a (low priority) question about the VOL_CAP_INT_USERACCESS volume 
>> attribute.
>> 
>> I often get the ATTR_CMN_USERACCESS value for files via getattrlist().
> 
> Surely then you would want to question VOL_CAP_INT_ATTRLIST :-)  ?
> 
>> Today, while doing unrelated research, I stumbled across the description for 
>> VOL_CAP_INT_USERACCESS which says "If this bit is set the volume format 
>> implementation supports the ATTR_CMN_USERACCESS attribute."
>> 
>> This took me quit by surprise, since ATTR_CMN_USERACCESS has never failed 
>> me. In testing I've now identified volumes that report 
>> VOL_CAP_INT_USERACCESS=0, but when I get the attributes for a file and 
>> request ATTR_CMN_USERACCESS it reports completely reasonable values.
>> 
>> So what does it mean, in terms of requesting the ATTR_CMN_USERACCESS 
>> attribute, if a volume reports VOL_CAP_INT_USERACCESS=0.
>> 
> 
> getattrlist(2) used to be part of the Filesystem implementation and so it was 
> possible for you to potentially not get ATTR_CMN_USERACCESS  back from a 
> particular implementation. getattrlist(2) today is implemented in VFS so this 
> has become somewhat non useful for getattrlist(2) itself  but can be useful 
> for getdirentriesattr(2) (which is deprecated in favor of 
> getattrlistbulk(2)). 
> 
> (As for  VOL_CAP_INT_ATTRLIST, we OR it in always so you'll never see it as 0)
> 
> This sounds like something we should fix in our documentation so please file 
> a bug.
> 
> 
>> Thanks,
>> 
>> James
>> ___
>> Do not post admin requests to the list. They will be ignored.
>> Filesystem-dev mailing list  (Filesystem-dev@lists.apple.com)
>> Help/Unsubscribe/Update your Subscription:
>> https://lists.apple.com/mailman/options/filesystem-dev/vivek_verma%40apple.com
>> 
>> This email sent to vivek_ve...@apple.com
> 
> ___
> Do not post admin requests to the list. They will be ignored.
> Filesystem-dev mailing list  (Filesystem-dev@lists.apple.com)
> Help/Unsubscribe/Update your Subscription:
> https://lists.apple.com/mailman/options/filesystem-dev/luther.j%40apple.com
> 
> This email sent to luthe...@apple.com

 ___
Do not post admin requests to the list. They will be ignored.
Filesystem-dev mailing list  (Filesystem-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/filesystem-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Re: readdir vs. getdirentriesattr

[Jim Luther]

In contentsOfDirectoryAtURL, instead of "includingPropertiesForKeys:nil", use 
"includingPropertiesForKeys:@[NSURLVolumeIdentifierKey]" (and add whatever 
other property keys you know you'll need). The whole purpose of the 
includingPropertiesForKeys argument is so the enumerator code can pre-fetch the 
properties you need as efficiently as possible. The enumeration will be a bit 
slower, but the entire operation of enumerating and getting the properties from 
the URLs returned will be faster.

Also, use -[enumeratorAtURL:includingPropertiesForKeys:options:errorHandler:] 
instead of 
-[contentsOfDirectoryAtURL:includingPropertiesForKeys:options:error:] unless 
you really need an NSArray of NSURLs. If your code is just processing all of 
the URLs and has no need to keep them after processing, there's no reason to 
add them to an array (which takes time and adds to peak memory pressure).

-[enumeratorAtURL:includingPropertiesForKeys:options:errorHandler:] also 
supports recursive enumeration (which stops at device boundaries -- you'll see 
mount points but not their contents) so you don't have to do that yourself.

- Jim

> On Apr 29, 2019, at 8:01 AM, Thomas Tempelmann  wrote:
> 
> Doing more performance tests for directory traversal I ran into a performance 
> issue with [NSURL contentsOfDirectoryAtURL:]:
> 
> See this typical code for scanning a directory:
> 
>   NSArray *contentURLs = [fileMgr contentsOfDirectoryAtURL:parentURL 
> includingPropertiesForKeys:nil options:0 error:nil];
>   for (NSURL *url in contentURLs) {
> id value;
> [url getResourceValue: forKey:NSURLVolumeIdentifierKey error:nil];
> 
> I would have expected the call for fetching NSURLVolumeIdentifierKey to be 
> rather fast because the upper file system layer should know which volume this 
> belong to because it has to know which FS driver it has to pass the calls to. 
> I.e., asking for the volume ID should be much faster than fetching actual 
> directory data such as the file size, for instance.
> 
> However, it turns out that this is just as slow as getting actual data from 
> the lower levels.
> 
> Could it be that the call is not optimized for returning this information as 
> earlier as possible but that it passes the call down to the lowest level 
> regardless of need?
> 
> I mention this because it degrades the performance of a recursive directory 
> scan significantly in my tests (on both APFS and HFS) - by more than 30%! The 
> only thing even slower would be to call stat() instead (for getting the 
> st_dev value).
> 
> Is this worth having looked at? If so, should I report this via bugreporter 
> (though, when I'm then asked to provide a system profiler report then, it's 
> not going anywhere)?
> 
> Thomas
> 
> ___
> Do not post admin requests to the list. They will be ignored.
> Filesystem-dev mailing list  (Filesystem-dev@lists.apple.com)
> Help/Unsubscribe/Update your Subscription:
> https://lists.apple.com/mailman/options/filesystem-dev/luther.j%40apple.com
> 
> This email sent to luthe...@apple.com

 ___
Do not post admin requests to the list. They will be ignored.
Filesystem-dev mailing list  (Filesystem-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/filesystem-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Re: readdir vs. getdirentriesattr

[Jim Luther]

> On Apr 29, 2019, at 1:19 PM, Thomas Tempelmann  wrote:
> 
> Jim,
> 
> In contentsOfDirectoryAtURL, instead of "includingPropertiesForKeys:nil", use 
> "includingPropertiesForKeys:@[NSURLVolumeIdentifierKey]" (and add whatever 
> other property keys you know you'll need). The whole purpose of the 
> includingPropertiesForKeys argument is so the enumerator code can pre-fetch 
> the properties you need as efficiently as possible. The enumeration will be a 
> bit slower, but the entire operation of enumerating and getting the 
> properties from the URLs returned will be faster.
> 
> I know. That's the theory, but my benchmarking says it makes no difference in 
> that case. And that's quite logical because the pre-caching is meant for data 
> that has to come from the lowest level, i.e. where the catalog data is 
> fetched - it makes sense to combine multiple property requests into one, just 
> like the getdirentriesattr is meant to used like. However, as I explained the 
> volume ID is not stored in the catalog but at a higher level, and therefore 
> pre-fetching this at the lowest level makes no difference, at requires no 
> catalog access, right?

The volume ID is at a higher layer, but the enumeration code attempts to 
retrieve the value less than once per URL returned. That said, if the directory 
hierarchy has few items per directory, the number of times it is retrieved will 
be higher. You can write a bug report and I'll look to see if there are ways to 
improve the performance.

In the meantime, there's something you could do to improve the performance 
(even if our code changes). You can get the volumeIdentifier for the directory 
you start enumerating from. It will be the same for the entire enumeration 
except when directories are seen on other file systems (today, that's volume 
mount points and mount triggers). Like this:

NSURL *directoryURL = [NSURL 
fileURLWithPath:@"/System/Applications/Utilities/" isDirectory:YES];
// get the volume identifier for most of the enumeration
id mainVolumeIdentifier;
[directoryURL getResourceValue: 
forKey:NSURLVolumeIdentifierKey error:nil];
NSDirectoryEnumerator *directoryEnumerator = 
[NSFileManager.defaultManager enumeratorAtURL:directoryURL 
includingPropertiesForKeys:nil options:0 errorHandler:nil];
for (NSURL *url in directoryEnumerator) {
NSNumber *isVolume;
NSNumber *isMountTrigger;
if ( ([url getResourceValue: forKey:NSURLIsVolumeKey 
error:nil] && isVolume.boolValue)
|| ([url getResourceValue: 
forKey:NSURLIsMountTriggerKey error:nil] && isMountTrigger.boolValue) ) {
// get the volume identifier for the volume or mount 
trigger
id otherVolumeIdentifier ;
[directoryURL getResourceValue: 
forKey:NSURLVolumeIdentifierKey error:nil];
}
}

> My performance tests always runs twice in fast succession, so that in the 
> second run, due to caching, all data's ready and does not incur random delays 
> that would give imprecise measurements. Sure, this does not give me the worst 
> case, but it gives me the best case results at least. And these best case 
> results say: Scanning "/System" on my Mac without getting the Volume ID takes 
> less than 3s, but with (with and without pre-fetching) getting it takes over 
> 6s. That's TWICE as much time. With smaller dir tree the difference is less, 
> possibly because then there's other caches helping.
> 
> I assume that when I re-run the scan, after having released all NSURLs from 
> the previous scan (even by restarting the test app), the framework creates, 
> fresh, NSURL objects, right? It's not that there is only one NSURL instance 
> on the entire system per volume item, shared between all processes, or is 
> there? The only caching, once I release an NSURL, is at the volume block 
> cache level, isn't it?
> 
> Also, use -[enumeratorAtURL:includingPropertiesForKeys:options:errorHandler:] 
> instead of 
> -[contentsOfDirectoryAtURL:includingPropertiesForKeys:options:error:] unless 
> you really need an NSArray of NSURLs. If your code is just processing all of 
> the URLs and has no need to keep them after processing, there's no reason to 
> add them to an array (which takes time and adds to peak memory pressure).
> 
> Thanks, that makes sense.
> 
> -[enumeratorAtURL:includingPropertiesForKeys:options:errorHandler:] also 
> supports recursive enumeration (which stops at device boundaries -- you'll 
> see mount points but not their contents) so you don't have to do that 
> yourself.
> 
> Is that based on fts_read? Because I found that this is much faster on local 
> volumes (not on network vols, though) than all other ways I've tried. And it 
> brings along the st_dev value without time penalty, unlike 
> contentsOfDirectoryAtURL.

It used to be based on heavily modified fts(3). I rewrote it for Mojave 

Re: readdir vs. getdirentriesattr

[Jim Luther]

I don’t really have time to look at the current fts implementation, but… it has 
several options that effect performance (in particular, the FTS_NOCHDIR, 
FTS_NOSTAT, FTS_NOSTAT_TYPE, and FTS_XDEV options). If you are trying to 
compare fts to CFURLEnumerator (for example), use FTS_NOCHDIR and FTS_XDEV, but 
don’t use FTS_NOSTAT and FTS_NOSTAT_TYPE.

> On Apr 22, 2019, at 9:59 AM, Thomas Tempelmann  wrote:
> 
> Jim,
> thanks for your comments.
> 
> If all you need is filenames and no other attributes, readdir is usually 
> faster than getattrlistbulk because it doesn't have to do as much work. 
> However, if you need additional attributes, getattrlistbulk is usually much 
> faster. Some of that extra work done by getattrlistbulk involves checking to 
> see what attributes were requested and packing the results into the result 
> buffer. 
> 
> What's interesting is that on HFS+, readdir is not faster in my tests, but on 
> a recent and fast Mac (i.e. not on my MacPro 2010), it can be twice as fast 
> as the others when scanning an APFS volume. I wonder why. Is the 
> implementation for getattrlistbulk in the APFS driver inefficient compared to 
> the one in HFS+? The source code for the APFS FS driver has still not be 
> published, or has it?
> 
> You'll find that lstat is slightly faster than getattrlist (when getattrlist 
> is returning the same set of attributes) for the same reason. There's no 
> extra code needed in lstat to see what attributes were requested and packing 
> the results into the result buffer.
> 
> It's also significantly faster than using NSURL's getResourceValue, even if 
> the NSURL has already been created regardless. That's probably due to all the 
> objc overhead.
> 
> By the way, I haven't tested this but I would expect 
> enumeratorAtURL:includingPropertiesForKeys:options:errorHandler: (followed by 
> a "for (NSURL *fileURL in directoryEnumerator)" loop) to be slightly faster 
> than contentsOfDirectoryAtURL:includingPropertiesForKeys:options:error: 
> because the URLs aren't retained in a NSArray. Using CFURLEnumerator may also 
> be slightly faster than NSFileManager's directory enumeration.
> 
> Now, that's something I had not considered, yet. Will try.
>  
> Using POSIX/BSD APIs will be the fastest, but that means you have to deal 
> with the different capabilities between file systems yourself (although 
> getattrlistbulk helps with that a lot).
> 
> Most interesting, though:
> 
> Today someone pointed out fts_read. This does, so far always beat all other 
> methods, especially if I also need extra attributes (e.g. file size).
> 
> Can you give some more information about the fts implementation? Is this 
> user-library-level oder kernel code that's doing this? I had expected that 
> this would only be a convenience userland function that uses readdir or 
> similar BSD functions, but it appears to beat them all, suggesting this is 
> optimized at a lower level.
> 
> 
> I have updated my test project accordingly (with the fts code) in case anyone 
> likes to run their own tests:
> 
>   http://files.tempel.org/Various/DirScanner.zip 
> 
> 
> Also, I am wondering if using concurrent threads will speed up scanning a dir 
> tree on an SSD as well, by distributing each directory read to one thread (or 
> dispatch queue). Will eventually try, but probably not soon. Gotta get my 
> program out of the door soon, first.
> 
> Thomas
> 

 ___
Do not post admin requests to the list. They will be ignored.
Filesystem-dev mailing list  (Filesystem-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/filesystem-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Re: readdir vs. getdirentriesattr

[Jim Luther]

If all you need is filenames and no other attributes, readdir is usually faster 
than getattrlistbulk because it doesn't have to do as much work. However, if 
you need additional attributes, getattrlistbulk is usually much faster. Some of 
that extra work done by getattrlistbulk involves checking to see what 
attributes were requested and packing the results into the result buffer. 
You'll find that lstat is slightly faster than getattrlist (when getattrlist is 
returning the same set of attributes) for the same reason. There's no extra 
code needed in lstat to see what attributes were requested and packing the 
results into the result buffer.

The original implementation of CFURLEnumerator (which is the implementation 
under NSFileManager's directory enumeration) was readdir followed by 
getattrlist requests to get the additional attributes on each item. Before we 
even shipped SnowLeopard, the implementation was changed to use 
getdirentriesattr if the file system supported it (getattrlistbulk was not 
available until several releases later) because of performance improvements.

By the way, I haven't tested this but I would expect 
enumeratorAtURL:includingPropertiesForKeys:options:errorHandler: (followed by a 
"for (NSURL *fileURL in directoryEnumerator)" loop) to be slightly faster than 
contentsOfDirectoryAtURL:includingPropertiesForKeys:options:error: because the 
URLs aren't retained in a NSArray. Using CFURLEnumerator may also be slightly 
faster than NSFileManager's directory enumeration. Using POSIX/BSD APIs will be 
the fastest, but that means you have to deal with the different capabilities 
between file systems yourself (although getattrlistbulk helps with that a lot).

- Jim

> On Apr 21, 2019, at 7:35 PM, Thomas Tempelmann  wrote:
> 
> I like to add some info on a thread from 2015:
> 
> I recently worked on my file search tool (FAF) and wanted to make sure that I 
> use the best method to deep-scan directory contents.
> 
> I had expected that getattrlistbulk() would always be the best choice, but it 
> turns out that opendir/readdir perform much better in some cases, oddly (this 
> is about reading just the file names, no other attributes).
> 
> See my blog post: https://blog.tempel.org/2019/04/dir-read-performance.html 
> <https://blog.tempel.org/2019/04/dir-read-performance.html>
> 
> There's also a test project trying out the various methods.
> 
> Any comments, insights, clarifications and bug reports are most welcome.
> 
> Enjoy,
>  Thomas Tempelmann
> 
> 
>> On 12. Jan 2015, at 17:33, Jim Luther > <mailto:luthe...@apple.com>> wrote:
>> 
>> getattrlistbulk() works on all file systems. If the file system supports 
>> bulk enumeration natively, great! If it does not, then the kernel code takes 
>> care of it. In addition, getattrlistbulk() supports all non-volume 
>> attributes (getattrlistbulk only supported a large subset).
>> 
>> The API calling convention for getattrlistbulk() is slightly different than 
>> getattrlistbulk() — read the man page carefully. In particular:
>> 
>> • ATTR_CMN_NAME and ATTR_CMN_RETURNED_ATTRS are required (requiring 
>> ATTR_CMN_NAME allowed us to get rid of the newState argument).
>> • A new attribute, ATTR_CMN_ERROR, can be requested to detect error 
>> conditions for a specific directory entry.
>> • The method for determining when enumeration is complete is different. You 
>> just keep calling getattrlistbulk() until 0 entries are returned.
>> 
>> - Jim
>> 
>>> On Jan 11, 2015, at 9:31 PM, James Bucanek >> <mailto:subscri...@gloaming.com>> wrote:
>>> 
>>> Eric,
>>> 
>>> I would just like to clarify: the new getattrlistbulk() function works on 
>>> all filesystem. We don't have to check the volume's VOL_CAP_INT_READDIRATTR 
>>> capability before calling it, correct?
>>> 
>>> James Bucanek
>>> 
>>>>Eric Tamura December 10, 2014 at 5:57 PM
>>>> It should be much faster.
>>>> 
>>>> Also note that as of Yosemite, we have added a new API: 
>>>> getattrlistbulk(2), which is like getdirentriesattr(), but supported in 
>>>> VFS for all filesystems. getdirentriesattr() is now deprecated. 
>>>> 
>>>> The main advantage of the bulk call is that we can return results in most 
>>>> cases without having to create a vnode in-kernel, which saves on I/O: HFS+ 
>>>> on-disk layout is such that all of the directory entries in a given 
>>>> directory are clustered together and we can get multiple directory entries 
>>>> from the same cached on-disk blocks.
> __