Re: NSURL getResourceValue: for NSURLVolumeMaximumFileSizeKey returns nil

2017-04-10 Thread Scott Talbert 

Still trying to sort this out, but not having any luck.

The real problem I'm trying to solve is this error message from Finder:

The item "" can't be copied because it is too large for the 
volume's format


Do you know under what conditions Finder will produce this error message? 
My hunch has been that it is related to the NSURL result, but strangely, 
we only see this Finder error consistently on one specific machine.


On Wed, 5 Apr 2017, Jim Luther wrote:


No. Immutable properties and capabilities like NSURLVolumeMaximumFileSizeKey 
are cached by coreservicesd on macos. The URL methods only remove cached values 
from the URL object.


On Apr 5, 2017, at 6:29 AM, Scott Talbert  wrote:

Do -[NSURL removeCachedResourceValueForKey:] and -[NSURL 
removeAllCachedResourceValues] clear the cached information in this case? It 
doesn't seem that they do.

Scott

On Mon, 3 Apr 2017, Scott Talbert wrote:


It sure sounds that way.  However, I haven't been able to spot a problem in the 
FUSE kext thus far.  It seems to be setting VOL_CAP_FMT_2TB_FILESIZE when it 
should be.

On Mon, 3 Apr 2017, Jim Luther wrote:



 On Apr 3, 2017, at 8:56 AM, Scott Talbert 
 wrote:
It's getting a value from the getattrlist path.
The fact that the data is cached at mount time is an interesting
clue. It suggest that perhaps there may be some window just shortly
after mount that the incorrect data is getting reported and cached.
It's a FUSE filesystem.  However, some of the more commonly-used FUSE
filesystems (e.g., sshfs) seem to behave the same way.
That sounds like a bug in FUSE. Once a file system is mounted, the kernel
sends out a VQ_MOUNT kernel event and that's what causes the
kNotifyVFSMount BSD notification to be sent. At that time, file systems
must be ready to respond to requests correctly. If they don't, they have a
bug.
- Jim

 On Sat, 1 Apr 2017, Jim Luther wrote:

   Scott, if you walk through that code, do you see
   which path it's getting a value from -- pathconf or
   getattrlist?

   The code in our frameworks that gets that value
   runs when a file system is first mounted. We have
   code listening for kNotifyVFSMount BSD
   notifications. When a notification comes in, we
   look to see what new volume(s) are available, and
   then collect and cache the volume's immutable
   properties (like this capability).

   Knowing more about the file system (what file
   system type, how it's connected, etc) where you are
   seeing the problem might help diagnose things.

   - Jim

 On Mar 31, 2017, at 5:10 PM, Scott
 Talbert  wrote:

 Thanks for the information.
  Unfortunately, that leaves me even
 more
 confused as, using that code, I get the
 correct answer:

 #include 
 #include 
 #include 
 #import 

 int getMaxFileSize(const char *path,
 int64_t *maxFileSize)
 {
   int result;
   struct statfs statfsBuf;

   // get the file system's mount point
 path for the input path
   result = statfs(path, );
   if ( result == 0 ) {
   long fileSizeBits =
 pathconf(statfsBuf.f_mntonname,
 _PC_FILESIZEBITS);
   if (fileSizeBits == -1) {
   // if _PC_FILESIZEBITS isn't
 supported, check for
 VOL_CAP_FMT_2TB_FILESIZE
   bool
 fileSystemSupports2TBFileSize = false;

   // get the supported
 capabilities
   struct attrlist attrList;
   struct volCapabilitiesBuf {
   u_int32_t length;
   vol_capabilities_attr_t
 capabilities;
   } __attribute__((aligned(4),
 packed));
   struct volCapabilitiesBuf
 volCaps;

   memset(, 0,
 sizeof(attrList));
   attrList.bitmapcount =
 ATTR_BIT_MAP_COUNT;
   attrList.volattr =
 ATTR_VOL_INFO | ATTR_VOL_CAPABILITIES;
   result =
 getattrlist(statfsBuf.f_mntonname,
 ,
 , sizeof(volCaps), 0);
   if ( result == 0 ) {
   fileSystemSupports2TBFileSize
 =
 ((volCaps.capabilities.capabilities[VOL_CAPABILITIES_FORMAT]
 &

Re: NSURL getResourceValue: for NSURLVolumeMaximumFileSizeKey returns nil

2017-04-05 Thread Jim Luther 
No. Immutable properties and capabilities like NSURLVolumeMaximumFileSizeKey 
are cached by coreservicesd on macos. The URL methods only remove cached values 
from the URL object.

> On Apr 5, 2017, at 6:29 AM, Scott Talbert  wrote:
> 
> Do -[NSURL removeCachedResourceValueForKey:] and -[NSURL 
> removeAllCachedResourceValues] clear the cached information in this case? It 
> doesn't seem that they do.
> 
> Scott
> 
> On Mon, 3 Apr 2017, Scott Talbert wrote:
> 
>> It sure sounds that way.  However, I haven't been able to spot a problem in 
>> the FUSE kext thus far.  It seems to be setting VOL_CAP_FMT_2TB_FILESIZE 
>> when it should be.
>> 
>> On Mon, 3 Apr 2017, Jim Luther wrote:
>> 
>>> 
>>>  On Apr 3, 2017, at 8:56 AM, Scott Talbert 
>>>  wrote:
>>> It's getting a value from the getattrlist path.
>>> The fact that the data is cached at mount time is an interesting
>>> clue. It suggest that perhaps there may be some window just shortly
>>> after mount that the incorrect data is getting reported and cached.
>>> It's a FUSE filesystem.  However, some of the more commonly-used FUSE
>>> filesystems (e.g., sshfs) seem to behave the same way.
>>> That sounds like a bug in FUSE. Once a file system is mounted, the kernel
>>> sends out a VQ_MOUNT kernel event and that's what causes the
>>> kNotifyVFSMount BSD notification to be sent. At that time, file systems
>>> must be ready to respond to requests correctly. If they don't, they have a
>>> bug.
>>> - Jim
>>> 
>>>  On Sat, 1 Apr 2017, Jim Luther wrote:
>>> 
>>>Scott, if you walk through that code, do you see
>>>which path it's getting a value from -- pathconf or
>>>getattrlist?
>>> 
>>>The code in our frameworks that gets that value
>>>runs when a file system is first mounted. We have
>>>code listening for kNotifyVFSMount BSD
>>>notifications. When a notification comes in, we
>>>look to see what new volume(s) are available, and
>>>then collect and cache the volume's immutable
>>>properties (like this capability).
>>> 
>>>Knowing more about the file system (what file
>>>system type, how it's connected, etc) where you are
>>>seeing the problem might help diagnose things.
>>> 
>>>- Jim
>>> 
>>>  On Mar 31, 2017, at 5:10 PM, Scott
>>>  Talbert  wrote:
>>> 
>>>  Thanks for the information.
>>>   Unfortunately, that leaves me even
>>>  more
>>>  confused as, using that code, I get the
>>>  correct answer:
>>> 
>>>  #include 
>>>  #include 
>>>  #include 
>>>  #import 
>>> 
>>>  int getMaxFileSize(const char *path,
>>>  int64_t *maxFileSize)
>>>  {
>>>int result;
>>>struct statfs statfsBuf;
>>> 
>>>// get the file system's mount point
>>>  path for the input path
>>>result = statfs(path, );
>>>if ( result == 0 ) {
>>>long fileSizeBits =
>>>  pathconf(statfsBuf.f_mntonname,
>>>  _PC_FILESIZEBITS);
>>>if (fileSizeBits == -1) {
>>>// if _PC_FILESIZEBITS isn't
>>>  supported, check for
>>>  VOL_CAP_FMT_2TB_FILESIZE
>>>bool
>>>  fileSystemSupports2TBFileSize = false;
>>> 
>>>// get the supported
>>>  capabilities
>>>struct attrlist attrList;
>>>struct volCapabilitiesBuf {
>>>u_int32_t length;
>>>vol_capabilities_attr_t
>>>  capabilities;
>>>} __attribute__((aligned(4),
>>>  packed));
>>>struct volCapabilitiesBuf
>>>  volCaps;
>>> 
>>>memset(, 0,
>>>  sizeof(attrList));
>>>attrList.bitmapcount =
>>>  ATTR_BIT_MAP_COUNT;
>>>attrList.volattr =
>>>  ATTR_VOL_INFO | ATTR_VOL_CAPABILITIES;
>>>result =
>>>  getattrlist(statfsBuf.f_mntonname,
>>>  ,
>>>  , sizeof(volCaps), 0);
>>>if ( result == 0 ) {
>>>fileSystemSupports2TBFileSize
>>>  =
>>>  
>>> ((volCaps.capabilities.capabilities[VOL_CAPABILITIES_FORMAT]
>>>  & VOL_CAP_FMT_2TB_FILESIZE) &&
>>>

Re: NSURL getResourceValue: for NSURLVolumeMaximumFileSizeKey returns nil

2017-04-03 Thread Jim Luther 

> On Apr 3, 2017, at 8:56 AM, Scott Talbert  wrote:
> 
> It's getting a value from the getattrlist path.
> 
> The fact that the data is cached at mount time is an interesting clue. It 
> suggest that perhaps there may be some window just shortly after mount that 
> the incorrect data is getting reported and cached.
> 
> It's a FUSE filesystem.  However, some of the more commonly-used FUSE 
> filesystems (e.g., sshfs) seem to behave the same way.

That sounds like a bug in FUSE. Once a file system is mounted, the kernel sends 
out a VQ_MOUNT kernel event and that's what causes the kNotifyVFSMount BSD 
notification to be sent. At that time, file systems must be ready to respond to 
requests correctly. If they don't, they have a bug.

- Jim

> On Sat, 1 Apr 2017, Jim Luther wrote:
> 
>> Scott, if you walk through that code, do you see which path it's getting a 
>> value from -- pathconf or getattrlist?
>> 
>> The code in our frameworks that gets that value runs when a file system is 
>> first mounted. We have code listening for kNotifyVFSMount BSD notifications. 
>> When a notification comes in, we look to see what new volume(s) are 
>> available, and then collect and cache the volume's immutable properties 
>> (like this capability).
>> 
>> Knowing more about the file system (what file system type, how it's 
>> connected, etc) where you are seeing the problem might help diagnose things.
>> 
>> - Jim
>> 
>>> On Mar 31, 2017, at 5:10 PM, Scott Talbert  wrote:
>>> 
>>> Thanks for the information.  Unfortunately, that leaves me even more
>>> confused as, using that code, I get the correct answer:
>>> 
>>> #include 
>>> #include 
>>> #include 
>>> #import 
>>> 
>>> int getMaxFileSize(const char *path, int64_t *maxFileSize)
>>> {
>>>   int result;
>>>   struct statfs statfsBuf;
>>> 
>>> 
>>>   // get the file system's mount point path for the input path
>>>   result = statfs(path, );
>>>   if ( result == 0 ) {
>>>   long fileSizeBits = pathconf(statfsBuf.f_mntonname,
>>> _PC_FILESIZEBITS);
>>>   if (fileSizeBits == -1) {
>>>   // if _PC_FILESIZEBITS isn't supported, check for
>>> VOL_CAP_FMT_2TB_FILESIZE
>>>   bool fileSystemSupports2TBFileSize = false;
>>> 
>>> 
>>>   // get the supported capabilities
>>>   struct attrlist attrList;
>>>   struct volCapabilitiesBuf {
>>>   u_int32_t length;
>>>   vol_capabilities_attr_t capabilities;
>>>   } __attribute__((aligned(4), packed));
>>>   struct volCapabilitiesBuf volCaps;
>>> 
>>> 
>>>   memset(, 0, sizeof(attrList));
>>>   attrList.bitmapcount = ATTR_BIT_MAP_COUNT;
>>>   attrList.volattr = ATTR_VOL_INFO | ATTR_VOL_CAPABILITIES;
>>>   result = getattrlist(statfsBuf.f_mntonname, ,
>>> , sizeof(volCaps), 0);
>>>   if ( result == 0 ) {
>>>   fileSystemSupports2TBFileSize =
>>> ((volCaps.capabilities.capabilities[VOL_CAPABILITIES_FORMAT]
>>> & VOL_CAP_FMT_2TB_FILESIZE) &&
>>> (volCaps.capabilities.valid[VOL_CAPABILITIES_FORMAT] &
>>> VOL_CAP_FMT_2TB_FILESIZE));
>>>   }
>>> 
>>> 
>>>   if ( fileSystemSupports2TBFileSize ) {
>>>   // use Supports2TBFileSize
>>>   *maxFileSize = LONG_LONG_MAX;
>>>   }
>>>   else {
>>>   // otherwise, we don't know
>>>   *maxFileSize = -1LL;
>>>   }
>>>   }
>>>   else if ( fileSizeBits > 64 ) {
>>>   // off_t is signed long long, so it cannot be more than
>>> LONG_LONG_MAX
>>>   *maxFileSize = LONG_LONG_MAX;
>>>   }
>>>   else if ( fileSizeBits < 32 ) {
>>>   // POSIX spec says 'Minimum Acceptable Value: 32' for
>>> FILESIZEBITS
>>>   *maxFileSize = INT_MAX;
>>>   }
>>>   else {
>>>   // 32...64 bits: shift off the bits we don't need
>>>   *maxFileSize = (int64_t)((u_int64_t)0xLL >>
>>> (u_int64_t)(65 - fileSizeBits));
>>>   }
>>>   }
>>>   return result;
>>> }
>>> 
>>> int main()
>>> {
>>> const char *mount = "/Volumes/Filesystem";
>>> 
>>> int64_t maxFileSize;
>>> int ret1 = getMaxFileSize(mount, );
>>> NSLog(@"getMaxFileSize %d %lld", ret1, maxFileSize);
>>> 
>>> NSString *path = [NSString stringWithUTF8String:mount];
>>> NSURL *url = [NSURL fileURLWithPath:path];
>>> NSNumber *maxfilesize;
>>> NSError *error;
>>> BOOL ret = [url getResourceValue:
>>> forKey:NSURLVolumeMaximumFileSizeKey error:];
>>> NSLog(@"NSURLVolumeMaximumFileSize: %d %@ %@", ret, maxfilesize, error);
>>> 
>>> }
>>> 
>>> $ ./blahm
>>> 2017-03-31 20:04:34.358 blahm[45735:2028260] getMaxFileSize 0
>>> 9223372036854775807
>>> 2017-03-31 20:04:34.365 blahm[45735:2028260] NSURLVolumeMaximumFileSize: 1
>>> (null) (null)
>>> 
 As -[NSURL getResourceValue:forKey:error:] is documented, "If this method
 returns YES and the value is populated with nil, it means that the
 resource property is not available for the

Re: NSURL getResourceValue: for NSURLVolumeMaximumFileSizeKey returns nil

2017-03-31 Thread Jim Luther 
As -[NSURL getResourceValue:forKey:error:] is documented, "If this method 
returns YES and the value is populated with nil, it means that the resource 
property is not available for the specified resource, and that no errors 
occurred when determining that the resource property was unavailable." So YES 
and nil is a valid response.

The NSURLVolumeMaximumFileSizeKey property value comes from two sources: 
pathconf() with _PC_FILESIZEBITS, or from getattrlist() from the 
ATTR_VOL_CAPABILITIES attribute and the VOL_CAP_FMT_2TB_FILESIZE capability.

Here's code (not the real code but the exact same algorithm) that shows how the 
value is calculated:

int getMaxFileSize(const char *path, int64_t *maxFileSize)
{
int result;
struct statfs statfsBuf;

// get the file system's mount point path for the input path
result = statfs(path, );
if ( result == 0 ) {
long fileSizeBits = pathconf(statfsBuf.f_mntonname, _PC_FILESIZEBITS);
if (fileSizeBits == -1) {
// if _PC_FILESIZEBITS isn't supported, check for 
VOL_CAP_FMT_2TB_FILESIZE
bool fileSystemSupports2TBFileSize = false;

// get the supported capabilities
struct attrlist attrList;
struct volCapabilitiesBuf {
u_int32_t length;
vol_capabilities_attr_t capabilities;
} __attribute__((aligned(4), packed));
struct volCapabilitiesBuf volCaps;

memset(, 0, sizeof(attrList));
attrList.bitmapcount = ATTR_BIT_MAP_COUNT;
attrList.volattr = ATTR_VOL_INFO | ATTR_VOL_CAPABILITIES;
result = getattrlist(statfsBuf.f_mntonname, , , 
sizeof(volCaps), 0);
if ( result == 0 ) {
fileSystemSupports2TBFileSize = 
((volCaps.capabilities.capabilities[VOL_CAPABILITIES_FORMAT] & 
VOL_CAP_FMT_2TB_FILESIZE) && 
(volCaps.capabilities.valid[VOL_CAPABILITIES_FORMAT] & 
VOL_CAP_FMT_2TB_FILESIZE));
}

if ( fileSystemSupports2TBFileSize ) {
// use Supports2TBFileSize
*maxFileSize = LONG_LONG_MAX;
}
else {
// otherwise, we don't know
*maxFileSize = -1LL;
}
}
else if ( fileSizeBits > 64 ) {
// off_t is signed long long, so it cannot be more than 
LONG_LONG_MAX
*maxFileSize = LONG_LONG_MAX;
}
else if ( fileSizeBits < 32 ) {
// POSIX spec says 'Minimum Acceptable Value: 32' for FILESIZEBITS
*maxFileSize = INT_MAX;
}
else {
// 32...64 bits: shift off the bits we don't need
*maxFileSize = (int64_t)((u_int64_t)0xLL >> 
(u_int64_t)(65 - fileSizeBits));
}
}
return result;
}

if NSURLVolumeMaximumFileSizeKey is returning nil with no errors, maxFileSize 
is -1LL, and the only way maxFileSize will be -1LL is if pathconf() returned an 
error, and the file system is saying VOL_CAP_FMT_2TB_FILESIZE is not valid and 
set (capabilities).

Hope that helps™

- Jim

> On Mar 31, 2017, at 2:33 PM, Scott Talbert  wrote:
> 
> Hello,
> 
> Can anyone tell me why calling NSURL -getResourceValue:forKey:error: with key 
> NSURLVolumeMaximumFileSizeKey would return a nil resource value for a given 
> volume?  The function returns YES and error is nil also.
> 
> I can see from running dtruss that it in turn calls getattrlist().  If I call 
> getattrlist() myself on the same volume, I can see the bits I would think it 
> is looking at are set correctly - VOL_CAP_FMT_2TB_FILESIZE.
> 
> Thanks,
> Scott
> ___
> Do not post admin requests to the list. They will be ignored.
> Filesystem-dev mailing list  (Filesystem-dev@lists.apple.com)
> Help/Unsubscribe/Update your Subscription:
> https://lists.apple.com/mailman/options/filesystem-dev/luther.j%40apple.com
> 
> This email sent to luthe...@apple.com

 ___
Do not post admin requests to the list. They will be ignored.
Filesystem-dev mailing list  (Filesystem-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/filesystem-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com