Re: readdir vs. getdirentriesattr

[James Bucanek]

Eric,

I would just like to clarify: the new getattrlistbulk() function works 
on all filesystem. We don't have to check the volume's 
VOL_CAP_INT_READDIRATTR capability before calling it, correct?


James Bucanek


Eric Tamura <mailto:etam...@apple.com>
December 10, 2014 at 5:57 PM
It should be much faster.

Also note that as of Yosemite, we have added a new API: 
getattrlistbulk(2), which is like getdirentriesattr(), but supported 
in VFS for all filesystems. getdirentriesattr() is now deprecated.


The main advantage of the bulk call is that we can return results in 
most cases without having to create a vnode in-kernel, which saves on 
I/O: HFS+ on-disk layout is such that all of the directory entries in 
a given directory are clustered together and we can get multiple 
directory entries from the same cached on-disk blocks.


 ___
Do not post admin requests to the list. They will be ignored.
Filesystem-dev mailing list  (Filesystem-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/filesystem-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Re: readdir vs. getdirentriesattr

[James Bucanek]

Jim, Eric, list,

I have to say, the new getattrlistbulk() function is working very well 
here. Requiring ATTR_CMN_RETURNED_ATTRS made me reorganize my code a 
bit, but that was minor. Specifically, it's nice to have a single 
routine to call (instead of getdirentriesattr() and a readdir() fallback 
when it's not available). It's nice to have a less complicated use 
pattern (i.e. no count-estimate/returned-count or state 
management/reread code to write). And, I can confirm that's it's fast. :)


James


Jim Luther <mailto:luthe...@apple.com>
January 12, 2015 at 9:33 AM
getattrlistbulk() works on all file systems. If the file system 
supports bulk enumeration natively, great! If it does not, then the 
kernel code takes care of it. In addition, getattrlistbulk() supports 
all non-volume attributes (getattrlistbulk only supported a large subset).


The API calling convention for getattrlistbulk() is slightly different 
than getattrlistbulk() — read the man page carefully. In particular:


• ATTR_CMN_NAME and ATTR_CMN_RETURNED_ATTRS are required (requiring 
ATTR_CMN_NAME allowed us to get rid of the newState argument).
• A new attribute, ATTR_CMN_ERROR, can be requested to detect error 
conditions for a specific directory entry.
• The method for determining when enumeration is complete is 
different. You just keep calling getattrlistbulk() until 0 entries are 
returned.


- Jim


James Bucanek <mailto:subscri...@gloaming.com>
January 11, 2015 at 10:31 PM
Eric,

I would just like to clarify: the new getattrlistbulk() function works 
on all filesystem. We don't have to check the volume's 
VOL_CAP_INT_READDIRATTR capability before calling it, correct?


James Bucanek

Eric Tamura <mailto:etam...@apple.com>
December 10, 2014 at 5:57 PM
It should be much faster.

Also note that as of Yosemite, we have added a new API: 
getattrlistbulk(2), which is like getdirentriesattr(), but supported 
in VFS for all filesystems. getdirentriesattr() is now deprecated.


The main advantage of the bulk call is that we can return results in 
most cases without having to create a vnode in-kernel, which saves on 
I/O: HFS+ on-disk layout is such that all of the directory entries in 
a given directory are clustered together and we can get multiple 
directory entries from the same cached on-disk blocks.


How big are the directories in question? How many times are you 
calling this?


Eric





___
Do not post admin requests to the list. They will be ignored.
Filesystem-dev mailing list (Filesystem-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/filesystem-dev/subscriber%40gloaming.com

This email sent to subscri...@gloaming.com
 ___
Do not post admin requests to the list. They will be ignored.
Filesystem-dev mailing list  (Filesystem-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/filesystem-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Testing the 2TB file size limit

[James Bucanek]

Quick question,

I know I can check the VOL_CAP_FMT_2TB_FILESIZE volume capabilities bit 
to determine if files larger than 4GB are supported on a volume.


Is there a reasonable test to determine if a filesystem supports files 
larger than 2TB?


James


___
Do not post admin requests to the list. They will be ignored.
Filesystem-dev mailing list  (Filesystem-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/filesystem-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

API to determine status of System Integrity Protection?

[James Bucanek]

Greetings,

Is there an API that can be queried to determine if System Integrity 
Protection is currently enabled on the system? Or, more specifically, is 
there a way to determine if the filesystem protections are being 
enforced? For example, is this status reported by sysctl() or statfs()?


I know there's the csrutil command line tool, which I could launch and 
parse the results, but ... really?


I develop backup and recovery software and I'd like to make decisions 
based on whether SIP is enabled or not.

___
Do not post admin requests to the list. They will be ignored.
Filesystem-dev mailing list  (Filesystem-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/filesystem-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Re: API to determine status of System Integrity Protection?

[James Bucanek]

I'll assume by the raging silence that there is no such API.

I've filed an enhancement request: rdar #22614669

James


James Bucanek <mailto:subscri...@gloaming.com>
September 4, 2015 at 1:47 PM
Greetings,

Is there an API that can be queried to determine if System Integrity 
Protection is currently enabled on the system? Or, more specifically, 
is there a way to determine if the filesystem protections are being 
enforced? For example, is this status reported by sysctl() or statfs()?


I know there's the csrutil command line tool, which I could launch and 
parse the results, but ... really?


I develop backup and recovery software and I'd like to make decisions 
based on whether SIP is enabled or not.

___
Do not post admin requests to the list. They will be ignored.
Filesystem-dev mailing list  (Filesystem-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/filesystem-dev/subscriber%40gloaming.com 



This email sent to subscri...@gloaming.com


 ___
Do not post admin requests to the list. They will be ignored.
Filesystem-dev mailing list  (Filesystem-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/filesystem-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

FSEventsCopyUUIDForDevice() occasionally changes value

[James Bucanek]

Greetings,

I was doing some debugging today and I noticed that 
FSEventsCopyUUIDForDevice() occasionally returns a different value for a 
given device. The frequency varies, but the UUID seems to change about 
every week or so. This apparently has been going on for years.


Is this expected behavior?

I would have expected the UUID to be more persistent. I certainly 
haven't reformatted my drives or anything, and I notice that the event 
ID haven't rolled over either. Are there other reasons why the UUID 
isn't stable (on my system)?


Thanks,

James

___
Do not post admin requests to the list. They will be ignored.
Filesystem-dev mailing list  (Filesystem-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/filesystem-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Re: FSEventsCopyUUIDForDevice() occasionally changes value

[James Bucanek]

Dominic Giampaolo 
November 20, 2015 at 5:09 AM
It's expected if something causes fseventsd to lose track of events 
and thus reset its history. The main reasons fseventsd can lose track 
of events is if the system crashes or reboots unexpectedly. An event 
overrun or fseventsd itself crashing can also cause it but that's 
pretty rare. The other thing that can cause it is if a volume is 
modified by someone other than an fseventsd capable system. In that 
case we don't know what changed and thus have to toss the history and 
generate a new volume uuid.


I knew some of that. The volume wouldn't be touched by any non-OS X 
systema, but I do regularly boot into newer/older versions of OS X for 
testing. Could that cause it?


Generally fseventsd will print a log message when this happens. Check 
your recent system logs to see if there are any messages.


I will look for those.


Note that the uuid returned by FSEventsCopoyUUIDForDevice() is not at 
all related to the per-volume uuid vended by through the DiskArb api's.


I do understand that this UUID is exclusively for determining if two 
FSEvent streams are equivalent, which is what I'm using it for. My issue 
was that I was keeping a dictionary with UUID/last-event-id pairs in a 
persistent store. What I discovered is, with the UUID occasionally 
changing, is that I had one system with almost 200 entries in this 
dictionary. Not a big deal, and easily dealt with (I now just toss 
entries if they're more than a month or so old), but it was surprising.


Thanks for the info,

James
 ___
Do not post admin requests to the list. They will be ignored.
Filesystem-dev mailing list  (Filesystem-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/filesystem-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Re: FSEventStream and rename notifications

[James Bucanek]

tridiak 
March 5, 2016 at 9:14 PM


I want a way to know a file what a file's new name is.
Having event ID X (old name) & X+1 (new name) for file rename (& move) 
would be really useful.


Is there another way to find out what a file's new name is? I am 
currently iterating through the parent directory comparing file sys 
IDs to get the new name.

A file's file ID will never changed under rename (or move*)?


A simpler approach would probably be to create a bookmark of the 
original file and make sure the bookmark data prefers the file ID 
(kCFURLBookmarkCreationPreferFileIDResolutionMask). Then, all you have 
to do is resolve the bookmark and get the file's new name and location 
(from the returned file URL). Very fast, very neat.


Optionally, if you use NSURL to create the bookmark, you can include the 
URL's original filename (as an URL resource) in the bookmark data. 
Later, the bookmark will provide both the original name (saved during 
creation) and its new name (obtained after resolution).


 ___
Do not post admin requests to the list. They will be ignored.
Filesystem-dev mailing list  (Filesystem-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/filesystem-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Re: FSEventStream and rename notifications

[James Bucanek]

Jim Luther <mailto:luthe...@apple.com>
March 7, 2016 at 9:01 AM

On Mar 6, 2016, at 5:04 PM, James Bucanek <mailto:subscri...@gloaming.com>> wrote:


A simpler approach would probably be to create a bookmark of the 
original file and make sure the bookmark data prefers the file ID 
(kCFURLBookmarkCreationPreferFileIDResolutionMask). Then, all you 
have to do is resolve the bookmark and get the file's new name and 
location (from the returned file URL). Very fast, very neat.


That won’t work. kCFURLBookmarkCreationPreferFileIDResolutionMask was 
never implemented and the option didn’t make sense anyway (if 
anything, it should have been a resolution option, not a creation option).

Well, that's very sad. :(

Nevertheless, bookmarks must still do something equivalent, correct?

That is, I can create a bookmark for a file, rename/move that file, and 
the bookmark will still find it.


James
 ___
Do not post admin requests to the list. They will be ignored.
Filesystem-dev mailing list  (Filesystem-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/filesystem-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Unravelling the mysteries of F_PREALLOCATE (on APFS and HFS+)

[James Bucanek]

Greetings,

I'm trying to determine exactly what F_PREALLOCATE does and how to 
effectively use it.


I recently had an F_PREALLOCATE bug report thrown back at me with "this 
is a bug in your code". I'd like to disagree, and point out that my 
F_PREALLOCATE call is perfectly valid and it's fcntl() that is not doing 
what it should. But after drilling down on this, I came to a disturbing 
realization:


I don't know, exactly, how F_PREALLOCATE works.

Here's the problem:

I have an app that writes to a very large (GB to TB) file. It's 
essentially a database, consisting of relatively small data and control 
records. While performing a transaction, it will append a bunch of new 
data records. To complete the transaction, it must write a modest number 
of control records to tie together the data records and record the 
transaction.


If, for any reason, the transaction is interrupted, I want to guarantee 
that the necessary control records can be written in order to complete 
the transaction and leave the file in a valid state. One of these 
interruptions would be, obviously, running out of fee disk space.


Here was my solution:

My "solution" to this was to begin by requesting a small F_PREALLOCATE 
(say, 2MB) before starting.


After writing some new data records (appending, say, an additional 1MB), 
I would assume that there is now only 1MB of pre-allocated file space 
remaining as a safety net. At this point, the code performs another 2MB 
F_PREALLOCATE so the file again has 2MB of preallocated space to finish 
its transaction.


This logic repeats, indefinately, until the F_PREALLOCATE returns an 
out-of-disk-space error, which cancels the transactions and uses the 
previously preallocated disk space to wrap up.


And here's the can of worms:

This seemed to work just fine on HFS+ (as far as I can tell). Then my 
APFS customers starting getting weird errors (error 22, invalid 
parameter) from the F_PREALLOCATE request. So I filed a bug.


Now in trying to defend this bug, I realize I have a lot more questions 
about F_PREALLOCATE than the documentation (what little there is) addresses.


The F_PREALLOCATE command passes an fstore_t structure with the 
following fields:


fst_flags: a combination of F_ALLOCATECONTIG (request a "contiguous 
allocation") and F_ALLOCATEALL ("allocate all of the requested space or 
fail and allocation nothing"). These seem pretty clear, and I don't use 
either.


fst_posmode: this must be either F_PEOFPOSMODE ("allocate from the 
physical eof") or F_VOLPOSMODE ("allocate from volume offset"). I have 
no idea what the latter means, but since I want additional space past 
the file's eof to get preallocated I've always used F_PEOFPOSMODE.


Which brings me to my first (and biggest) question: F_PEOFPOSMODE 
allocates from the "physical" end of file. What is the physical end of 
file? Let's say I have a 1MB file and request a 2MB preallocation. 
Afterwards, is the "physical" eof 1MB or 3MB? If I perform another 2MB 
preallocation will the preallocated space remain at 2MB or will it grow 
to 4MB? If the latter, how does one determine the "physical" end of file?


fst_offset and fst_length: The offset to the start of the "region" and 
the length of the preallocation request. I've always assumed (see 
fst_posmode) that the offset was relative to the file's logical EOF, but 
now I'm not sure.


fst_bytesalloc: This is the return field that reports the amount of 
space actually allocated. The documentation says "the space that is 
allocated can be the same size or larger than the space requested". That 
always made sense to me. If I requested a paltry 3 bytes, I'm sure the 
filesystem would round that up at least to the nearest block size.


Problem/question number two: For HFS+ volumes, the fst_bytesalloc 
returned was always the size I requested (unless the drive was out of 
space). In APFS, however, I get numbers much smaller than what was 
requested, even when the return value indicates success, in direct 
contradiction of the documentation. For example, after performing a few 
preallocations, the next request might be to preallocate 3MB, but the 
value returned in fst_bytesalloc will be 20K.


Finally, I've got some new questions as I explore using "holes" and 
sparse files. Specifically, if I punch a "hole" in a files with 
F_PUNCHHOLE, can I later use F_PREALLOCATE to re-allocate those blocks 
before I write into them? If so, how would one determine the offset of a 
hole when setting up fst_posmode, fst_offset, and fst_length?


I'm just hoping there are fcntl() and/or APFS gurus out there that know 
the answers to these questions.


James Bucanek

___
Do not post admin requests to the list. They will be ignored.
Filesyst

Re: Unravelling the mysteries of F_PREALLOCATE (on APFS and HFS+)

[James Bucanek]

Dominic Giampaolo 
December 14, 2017 at 9:17 AM


Which brings me to my first (and biggest) question: F_PEOFPOSMODE allocates from the 
"physical" end of file. What is the physical end of file?


Is is the current size of the file as reported by stat() in the st_size field.  
This is different than the amount of data allocated to the file which is 
returned in st_blocks.  The difference between st_blocks * fs-block-size and 
st_size is how much the file has pre-allocated.  (well you should round up 
st_size to a multiple of the fs block size before doing the subtraction).


Thanks a million. That piece of information has helped me immensely. I 
have now (a) figured out what's going wrong with F_PREALLOCATE, (b) 
discovered that APFS and HFS+ treat F_PREALLOCATE differently, and (c) 
believe I've figured out a workaround.



Let's say I have a 1MB file and request a 2MB preallocation. Afterwards, is the 
"physical" eof 1MB or 3MB? If I perform another 2MB preallocation will the preallocated 
space remain at 2MB or will it grow to 4MB? If the latter, how does one determine the 
"physical" end of file?


As noted above, the physical end of file is simply the size reported for the 
file.

But this is the crux of your issue: you're asking to grow the file but the amount 
you want to grow it by doesn't exceed the current amount already pre-allocated for 
the file and so apfs returns EINVAL (error 22).  Put another way, if (st_size + 
amount_you_asked_to_allocate)<  (st_blocks * fs_block_size) then apfs returns 
EINVAL.

Agreed, and I think this is the root of my problem.

#1 The APFS F_PREALLOCATE bug

Back to my bug report. I can now reproduce the bug in several different 
ways, and I also believe I understand (at least indirectly) why it's 
failing.


I modified my test code to perform an fstat() on the file before, and 
again after, each preallocation request. This let me compare the 
preallocation request (fst_length -> fst_bytesalloc) against the actual 
change in the file's allocated size (st_blocks).


My first success at failure was by requesting a 1MB pre-allocation on an 
empty file, writing a small (20K) block of data, then requesting a 1MB 
pre-allocation again, repeating until it failed. Here's what happened


<- before F_PREALLOC request > <- request / result -> 
<--- after request --->
#: leof st_size st_blocks(*512) fst_length
alloc st_size st_blocks(*512)  blocks-delta
1:0   0 0(   0)1048576  
1048576   0  2048( 1048576) 1048576
2:20480   20480  2048( 1048576)1048576
20480   20480  2088( 1069056)   20480
3:40960   40960  2088( 1069056)1048576
20480   40960  2128( 1089536)   20480
4:61440   61440  2128( 1089536)1048576
20480   61440  2168( 1110016)   20480


Before the first preallocation, the file is empty, st_blocks is 0. After 
requesting a 1MB preallocation, it changes to 2048 (1MB). This is as 
expected and agrees with the amount returned in fst_bytesalloc (the 
alloc column).


After writing 20K, another 1MB preallocation is requested. This time 
there is 1MB-20K of preallocated space still left, so F_PREALLOCATE 
allocates only the difference between what's already preallocated and 
the request, which is exactly 20K. The request reports that 20K was 
preallocated (fst_bytesalloc) which agrees with the change in st_blocks 
(from 2048 to 2088, or 20K).


This goes on for some time without any problems. Then, the surprise...

 4845: 9920512099205120195768(   100233216)1048576
2048099205120195808(   100253696)   20480
 4846: 9922560099225600195808(   100253696)1048576
2048099225600195848(   100274176)   20480
 4847: 9924608099246080195848(   100274176)1048576
2048099246080228616(   117051392)16777216

preallocation failed: errno=22

Request #4,846 requests 1MB, allocates 20K, the file allocation grows by 
20K, and all is good. Then request #4,847 requests exactly the same, but 
this time the preallocation reports that it allocated 20K but actually 
allocated 16,777,216 (16MB), based on the change in the st_blocks.


The next request for 1MB fails with errno 22 (EINVAL). As explained 
earlier, this is likely because the amount requested (1MB) is less than 
the amount already allocated.


Here's my theory: I think APFS is applying it's own preemptive 
preallocation logic based on how the file is getting written, and at 
some point it internally decides to preallocate more space so all of 
those little write() calls won't create a horribly fragmented file. But 
because the F_PREALLOCATE now requires that you don't request less that 
wha't already been preallocated, the next call fails.

Re: Unravelling the mysteries of F_PREALLOCATE (about holes)

[James Bucanek]

Changing the subject, because this is really a separate question...

Dominic Giampaolo 
December 14, 2017 at 9:17 AM

Finally, I've got some new questions as I explore using "holes" and sparse files. 
Specifically, if I punch a "hole" in a files with F_PUNCHHOLE, can I later use 
F_PREALLOCATE to re-allocate those blocks before I write into them?


No.  Just write real data to the hole and apfs will handle allocating blocks 
for it.  You don't have to worry about that.
I understand that the filesystem will handle allocating the blocks, just 
as I understand writing past the EOF will allocate more blocks.


What I'm asking is that, before I start filling a hole, can I 
preallocate the space for that hole so I won't be surprised by a disk 
full error in the middle of filling it (and to optimize the allocation 
of that space)?


Or, said another way, and probably an easier question to answer: Do the 
blocks used to satisfy writing into holes come from the same pool of 
preallocated blocks?



 ___
Do not post admin requests to the list. They will be ignored.
Filesystem-dev mailing list  (Filesystem-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/filesystem-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

So what does VOL_CAP_INT_USERACCESS actually mean?

[James Bucanek]

Hello,

I have a (low priority) question about the VOL_CAP_INT_USERACCESS volume 
attribute.


I often get the ATTR_CMN_USERACCESS value for files via getattrlist().

Today, while doing unrelated research, I stumbled across the description 
for VOL_CAP_INT_USERACCESS which says "If this bit is set the volume 
format implementation supports the ATTR_CMN_USERACCESS attribute."


This took me quit by surprise, since ATTR_CMN_USERACCESS has never 
failed me. In testing I've now identified volumes that report 
VOL_CAP_INT_USERACCESS=0, but when I get the attributes for a file and 
request ATTR_CMN_USERACCESS it reports completely reasonable values.


So what does it mean, in terms of requesting the ATTR_CMN_USERACCESS 
attribute, if a volume reports VOL_CAP_INT_USERACCESS=0.


Thanks,

James
___
Do not post admin requests to the list. They will be ignored.
Filesystem-dev mailing list  (Filesystem-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/filesystem-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

How to use mandatory file locking on afpfs volumes

[James Bucanek]

More annoying questions...

How does one use "AFP-style mandatory file locking" in macOS 
(10.9...current)?


I'm trying to implement file locking/coordination across various 
filesystems and servers.


The man page for getattrlist() includes the following filesystem 
capability test:


VOL_CAP_INT_MANLOCK  If this bit is set, the volume format implementation
 supports AFP-style mandatory byte range locks
 via ioctl(2).

AFPFS (a.k.a. AppleShare) volumes, particularly those on AirPort Extreme 
/ Time Capsule devices, return 1 for this capability. These volumes do 
not support flock() or fcntl(F_SETLK) style advisory locking.


I looked at the man page for ioctl(2) and it contains no real 
information. It says the commands for ioctl(2) are documented in 
.


So I had a look at  and found very little there; it's 
basically an umbrella header for several other headers. So I went 
through them one at a time: , , 
 (the one I would have assumed it would be in), and 
. I could found no mention of any kind of file or byte 
range locking commands defined.


So, is it possible to lock files/ranges on an AFPFS volume using 
ioctl(2), and if so where are these commands defined?



___
Do not post admin requests to the list. They will be ignored.
Filesystem-dev mailing list  (Filesystem-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/filesystem-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Re: How to use mandatory file locking on afpfs volumes

[James Bucanek]

Follow up on my own post...

My intention is to perform file locking on remote volumes, including 
AFPFS volumes. So far, I've found no documentation on how to implement 
"AFP-style" mandatory locking using fcntl().


Then I remember that, while deprecated, the now-ancient FS API is still 
around. Plan B was born: use FSOpenFork's deny-read & deny-write 
permissions to implement file locking, then turn around and use open(2) 
to access the data.


That didn't work. As soon as I opened the file via FSOpenFork, open(2) 
would fail with "resource not available" error.


Hmm, that sounds exactly like what happens when open(2) tries to open a 
file that has been opened by another process with O_SHLOCK or O_EXLOCK.


So it would appear that, rather than being completely different file 
locking mechanisms, FSOpenFork() bridges to the advisory locking 
provided by open(2). And when I rewrite my test code to use open(2) 
O_SHLOCK and O_EXLOCK to access the same AFPFS volume from two different 
clients, it works perfectly!


Now I'm wondering why the AFPFS volume (this on a Time Capsule, by the 
way) reports VOL_CAP_INT_FLOCK=NO and VOL_CAP_INT_ADVLOCK=NO. The 
functionality of open(2) on that volume would appear to directly 
contradict that capability test.


Can anyone explain this?

Thanks!

James



James Bucanek <mailto:subscri...@gloaming.com>
February 26, 2018 at 1:11 PM
More annoying questions...

How does one use "AFP-style mandatory file locking" in macOS 
(10.9...current)?


I'm trying to implement file locking/coordination across various 
filesystems and servers.


The man page for getattrlist() includes the following filesystem 
capability test:


VOL_CAP_INT_MANLOCK  If this bit is set, the volume format implementation
 supports AFP-style mandatory byte range locks
 via ioctl(2).

AFPFS (a.k.a. AppleShare) volumes, particularly those on AirPort 
Extreme / Time Capsule devices, return 1 for this capability. These 
volumes do not support flock() or fcntl(F_SETLK) style advisory locking.


I looked at the man page for ioctl(2) and it contains no real 
information. It says the commands for ioctl(2) are documented in 
.


So I had a look at  and found very little there; it's 
basically an umbrella header for several other headers. So I went 
through them one at a time: , , 
 (the one I would have assumed it would be in), and 
. I could found no mention of any kind of file or byte 
range locking commands defined.


So, is it possible to lock files/ranges on an AFPFS volume using 
ioctl(2), and if so where are these commands defined?



___
Do not post admin requests to the list. They will be ignored.
Filesystem-dev mailing list  (Filesystem-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/filesystem-dev/subscriber%40gloaming.com 



This email sent to subscri...@gloaming.com



 ___
Do not post admin requests to the list. They will be ignored.
Filesystem-dev mailing list  (Filesystem-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/filesystem-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Re: How to use mandatory file locking on afpfs volumes

[James Bucanek]

Kevin,

I tested this in a variety of configurations. The tests consisted of 
setting up an AFP shared volume and then testing open(2) using O_SHLOCK 
and O_EXLOCK options when opening a file on that volume from two 
different client processes.


AFP server = Airport Extreme, Client #1 on machine A, Client #2 on machine B
AFP server = Airport Extreme, Client #1 on machine A, Client #2 on machine A
AFP server = machine C with macOS AFP Sharing, Client #1 on machine A, 
Client #2 on machine B
AFP server = machine C with macOS AFP Sharing, Client #1 on machine A, 
Client #2 on machine A
AFP server = machine C with macOS AFP Sharing, Client #1 on machine A, 
Client #2 on machine C (same computer running the AFP server)


In all cases, the open(2) O_SHLOCK and O_EXLOCK worked exactly as they 
would with a volume that reports it supports POSIX file locking 
(VOL_CAP_INT_FLOCK=YES). That is, if one client obtains a shared lock no 
other clients can obtain an exclusive lock, and if one client obtains an 
exclusive lock no other clients can obtain any lock. Most importantly, 
the open() call does NOT return an EOPNOTSUPP error, which is what the 
man page says it will do if "the underlying filesystem does not support 
locking".


This is consistent in every configuration, with one notable exception: 
if you open a file on a remote APF volume with O_SHLOCK or O_EXLOCK from 
a client machine and you then attempt to perform a regular open() 
(without either O_SHLOCK or O_EXLOCK) from a second process, the open 
will fail with a "resource unavailable" error, just as if you had 
requested a file lock. This is consistent with AFP-style mandatory 
locking, but inconsistent with POSIX-style advisory locking.


James


Kevin Elliott <mailto:kelli...@mac.com>
March 28, 2018 at 5:15 PM


Were you testing this on two different machines simultaneously? The 
point of AFP locking is to protect the file contents from simultaneous 
access on different machines, not within the same machine.


-Kevin


James Bucanek <mailto:subscri...@gloaming.com>
March 13, 2018 at 12:18 PM
Follow up on my own post...

My intention is to perform file locking on remote volumes, including 
AFPFS volumes. So far, I've found no documentation on how to implement 
"AFP-style" mandatory locking using fcntl().


Then I remember that, while deprecated, the now-ancient FS API is 
still around. Plan B was born: use FSOpenFork's deny-read & deny-write 
permissions to implement file locking, then turn around and use 
open(2) to access the data.


That didn't work. As soon as I opened the file via FSOpenFork, open(2) 
would fail with "resource not available" error.


Hmm, that sounds exactly like what happens when open(2) tries to open 
a file that has been opened by another process with O_SHLOCK or O_EXLOCK.


So it would appear that, rather than being completely different file 
locking mechanisms, FSOpenFork() bridges to the advisory locking 
provided by open(2). And when I rewrite my test code to use open(2) 
O_SHLOCK and O_EXLOCK to access the same AFPFS volume from two 
different clients, it works perfectly!


Now I'm wondering why the AFPFS volume (this on a Time Capsule, by the 
way) reports VOL_CAP_INT_FLOCK=NO and VOL_CAP_INT_ADVLOCK=NO. The 
functionality of open(2) on that volume would appear to directly 
contradict that capability test.


Can anyone explain this?



 ___
Do not post admin requests to the list. They will be ignored.
Filesystem-dev mailing list  (Filesystem-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/filesystem-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Re: How to use mandatory file locking on afpfs volumes

[James Bucanek]

Vivek Verma 
April 3, 2018 at 11:46 AM

Advisory locking as defined by POSIX is bunch of sentences thrown 
together which make no sense when you actually try to use it in a real 
world application.


AFP's  locking protocol 
(https://developer.apple.com/library/content/documentation/Networking/Conceptual/AFP/UsingForkCommands/UsingForkCommands.html#//apple_ref/doc/uid/TP4854-CH225-SW1) 
 borrows some API elements (on macOS) from the advisory locking 
interfaces , so some calls may on the surface appear to work the same 
but it does _not_  completely support POSIX style advisory locking and 
that's why the macOS AFP client  does not claim to support it.  It has 
its own locking implementation which interface-wise may overlap a bit 
with the interfaces for advisory locking.


I appreciate this position, and I'm aware that POSIX locking has some 
holes. I also appreciate that AFP mandatory locking and POSIX file 
locking are not interchangeable and therefor AFP can't claim to support 
them.


However, at the end of the day, I really just want to find out what 
works and get some rudimentary understanding as to why and what the 
pitfalls are / might be.


I have a real issue with file corruption from multiple clients writing 
to the same set of database files/package. All of my code has been 
re-written to use the POSIX API (i.e. open(2) et al). If the POSIX API 
bridges to the AFP file locking mechanism on afpfs volumes in any 
predictable way—and I can deal with some inconsistencies—that would be 
really good news for my application.


I've already got POSIX file locking and advisory locking, via fcntl(2), 
to work. Getting any kind of locking on AFP volumes is the last big hole 
I want to fill.


James
 ___
Do not post admin requests to the list. They will be ignored.
Filesystem-dev mailing list  (Filesystem-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/filesystem-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Some questions about firm links

[James Bucanek via Filesystem-dev]

Greetings,

I have so many questions... ;)

But let's start with a few basic ones. I'm upgrading my backup and 
recovery tools for Catalina and I find myself making assumptions about 
firm links that I'd like confirm. Here are my burning questions:



(1) Are firm links and their target always the same relative path?

I've written a few "firm link exploration" tools and I find that 
(thankfully!) the relative path of a firm link and its target are always 
the same, with respect to their volumes. For example:


/Users -> /System/Volumes/Data/Users
/usr/share/snmp -> /System/Volumes/Data/usr/share/snmp
/System/Library/CoreServices/CoreTypes.bundle/Contents/Library -> 
/System/Volumes/Data/System/Library/CoreServices/CoreTypes.bundle/Contents/Library


Said another way, is this a requirement of firm links or is this just a 
Catalina convention? Is Apple reserving the right to create something 
like this in a future version:


/usr/share/snmp -> /System/Volumes/Data/Library/Application 
Support/Network/SNMP



(2) Are firm links always on the R/O volume?

In other words, are firm links only created on the R/O system volume and 
always target something on its data volume? (I wrote some code to scan 
for firm links and I can only find them on the R/O system volume, but I 
wanted to ask.)



(3) Are firm links readable?

In the WWDC 2019 video "What's New in the Apple File Systems", firm 
links are described as "a special kind of sym link". Are these readable? 
I haven't found a new enum vtype defined (i.e. no new VFRMLNK), nor have 
I discovered a firm link equivalent for O_NOFOLLOW or O_SYMLINK that 
would allow one to open the contents of a firm link. Without these, it 
would appear that firm links are opaque filesystem structures that 
user-land apps cannot plumb.


This question is just idle curiosity; I don't currently have any real 
need to read firm links.




Cheers,

James Bucanek
___
Do not post admin requests to the list. They will be ignored.
Filesystem-dev mailing list  (Filesystem-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/filesystem-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Exploring APFS volume groups in Catalina

[James Bucanek via Filesystem-dev]

Hello again,

Is there an API to discover/detect/explore an APFS volume group in Catalina?

I'm currently inferring the relationship between volumes in a group by 
getting the F_GETPATH_NOFIRMLINK path of a file and comparing that with 
its F_GETPATH path. Is this reliable or is there a better, more 
definitive, method for determining the relationship of a system/data 
volume pair?


Thanks,

James Bucanek
___
Do not post admin requests to the list. They will be ignored.
Filesystem-dev mailing list  (Filesystem-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/filesystem-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com