I've received off-list help on this problem, which has been invaluable!
Turns out it isn't really a problem of "safe save" - we already handle
DOCUMENT_ID, and the related IOCTLs including transfer, as well as a
version of "tombstones" that suits us.
It appears to be the HFS+ file compression "decmpfs".
For a given source file, it appears that decmpfs will create a named stream
(vnop_makenamedstream) named "com.apple.ResourceFork" where it saves the
compressed version. It also saves a "compression header" (vnop_setxattr) as
"com.apple.decmpfs". The next step is truncating the (real? data? source?)
file to 0 byte length, and finally sets (chflags / vnop_setattr) UF_COMPRESSED.
So for us, we would have an empty 0 byte length file and everything would
fail, like Mail trying to send mail.
In HFS, it has additional logic in vnop_read to detect that the file is
compressed, and decompress it into UPL first. There is additional logic in
vnop_getattr to handle returning the filesize, as well as in vnop_write to
decompress the file completely should it be written to. This logic was/is
missing for us.
As we did not advertise VOL_CAP_FMT_DECMPFS_COMPRESSION, we ignored the
compression paths in HFS (we totally should not have ignored that code)
Indeed when using "ditto --hfsCompression" as a tester tool, ditto will
skip compression if VOL_CAP_FMT_DECMPFS_COMPRESSION is not set.
Unfortunately, this is not true of "Mail". It will compress no matter what.
We tried returning errors in vnop_makenamedstream, vnop_setxattr, and
vnop_setattr, but all errors are ignored and it will carrying on with
compression and things will fail.
For now, we have managed to give temporary support to the situation. As the
last task it performs is to set UF_COMPRESSED, we now add these lines:
cp = spl_decmpfs_cnode_alloc();
spl_decmpfs_cnode_init(cp);
iscompressed = spl_decmpfs_file_is_compressed(ap->a_vp, cp);
if (iscompressed == 1)
error = spl_decmpfs_decompress_file(ap->a_vp, cp,
/*tosize*/ -1, /*truncateok*/ 1, /*skiplock*/ 1 );
spl_decmpfs_cnode_destroy(cp);
spl_decmpfs_cnode_free(cp);
It is a shame to decompress it immediately, but it was the quickest
solution "for now" (watch it become permanent!), while we ponder a proper
implementation. Ultimately, we do not really want decmpfs, as the user can
pick better compression, like that of gzip-9 (I mean, if they really wanted
to..)
None of the decmpfs_*() functions are exported, but that is a separate problem.
I would never been able to figure this out on my own, so big thanks to the
hints I received.
We still have some incompatibilities to solve, why "bird" repeatedly
crashes, and fonts are considered invalid, but we have to leave something
to tinker with on a rainy day.
Lund
Jorgen Lundman wrote:
>
> Hello list,
>
> So with 10.13 we have now noticed that Mail also uses what I would call
> "safe save" feature when sending email, I'm not sure what it is really
> called? Versioning? AtomicFile? It is something we have not been able to
> unravel for ZFS of yet, but so far we have lived without it. Mail makes
> that a bit awkward. We just get a completely-empty email when you click
> "send" and sending fails. Clicking Edit-Message then just shows empty message.
>
> So I'm wondering if I am allowed to know what underlying feature that
> mechanism uses. I have for a while tried to pin it down with dtrace,
> perhaps VNOP_EXCHANGE, or VNOP_CLONEFILE, or VNOP_COPYFILE, or renamex
> (RENAME_SWAP), or ATTR_VOL_DOCUMENT_ID with UF_TRACKED,
> HFSIOC_TRANSFER_DOCUMENT_ID, or ... I have checked quite a few other things...
>
> Dtrace has not given me any clues as to what we are missing. It could even
> be my xmas present!
>
> Sincerely,
>
> Lund
>
>
> _______________________________________________
> Do not post admin requests to the list. They will be ignored.
> Filesystem-dev mailing list (Filesystem-dev@lists.apple.com)
> Help/Unsubscribe/Update your Subscription:
> https://lists.apple.com/mailman/options/filesystem-dev/lundman%40lundman.net
>
> This email sent to lund...@lundman.net
>
--
Jorgen Lundman | <lund...@lundman.net>
Unix Administrator | +81 (0)90-5578-8500
Shibuya-ku, Tokyo | Japan
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Filesystem-dev mailing list (Filesystem-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/filesystem-dev/archive%40mail-archive.com
This email sent to arch...@mail-archive.com
