[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

One encryption related fix to add to the this PR, zfsonlinux/zfs@d7e4b30a672c1d7eb77dbbe2cdcf14cac102839a. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/openzfs/openzfs/pull/489#issuecomment-428354045

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@GernotS Unfortunately no, that wasn't intended to fix your issue. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/openzfs/openzfs/pull/489#issuecomment-427883495 --

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

Was recounted DSL meant to fix my above issue with unencrypted send/receive? If so it was not successful, I still get: BAD TRAP: type=e (#pf Page fault) rp=fe003dd7dee0 addr=60 occurred in module "zfs" due to a NULL pointer dereference zfs: #pf Page fault Bad kernel fault at addr=0x60

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

Refcounted DSL added - one missing `zvol_create_minors()` call in illumos, but I believe that is as expected. Added a couple of `(void)` before `refcount_add()` due to lint. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@lundman pushed 1 commit. bc64070 Refcounted DSL Crypto Key Mappings -- You are receiving this because you are subscribed to this thread. View it on GitHub: https://github.com/openzfs/openzfs/pull/489/files/e2df9b70741ea3c091a30c03c08847c7bacec46e..bc640707c92028b91bfe86437929b63fe484ea9f

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@lundman FYI, I added 3 commits to fix a few more issues. There were some changes on linux that were not directly related to encryption but which the encryption code depends on. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@ahrens pushed 3 commits. 3de57cc fix error handling in arc_read 5548355 spill blocks are metadata e2df9b7 zfs_receive_one needs to restore keylocation prop -- You are receiving this because you are subscribed to this thread. View it on GitHub:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@GernotS Thanks for your report. I was not able to reproduce this with a simple test. Could you upload/send me the crash dump? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

yeah its me again, with a new bug. Trying to do an ls -l /dev/zvol/rdsk/pool/disk0 on an encrypted zvol (key loaded) will produce a core: panic[cpu3]/thread=fe003e350c20: assertion failed: spa_do_crypt_objset_mac_abd(B_TRUE, spa, dsobj, zio->io_abd, psize, (bp)->blk_prop) >> (63)) &

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@lundman pushed 1 commit. 3972768 Do not call dmu_objset_disown twice -- You are receiving this because you are subscribed to this thread. View it on GitHub: https://github.com/openzfs/openzfs/pull/489/files/6f85552ff7a285fef0af14f31d322cc82bebd07c..3972768dc18f2f6b4b2c3c90ac9ce516f8e1cf99

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@Ahrens glad to help a bit getting this forward, but the credit belongs to you, Tom, Jorgen and all the others who keep on pushing on with it. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

mav pointed out something to me: dmu_objset_disown() is called in both zfsvfs_create() and zfsvfs_create_impl() in case of error. At the end of zfsvfs_create(), remove the if after `error = zfsvfs_create_impl(zfvp, zfsvfs, os);` -- You are receiving this because you are subscribed to this

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@GernotS That's great to hear! It makes sense that the "Refactor arc_hdr_realloc_crypt()" commit would address the panics that you were seeing when using L2ARC + encryption. I think that the other bug (causing an incorrect MAC on the target when doing encrypted send, causing some panics due

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

Woo, I contributed! \o/ -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/openzfs/openzfs/pull/489#issuecomment-407324973 -- openzfs: openzfs-developer Permalink:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

so far no further crashes on my side :) -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/openzfs/openzfs/pull/489#issuecomment-407324688 -- openzfs:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

In the interest in making `dsl_crypto.c` and `zio_crypto.c` to be the same as ZOL - I added a few commits that messed around in there. [1] The 3a5bf20 commit will replace the gist I pasted in https://github.com/openzfs/openzfs/pull/489#issuecomment-404715592 [1] minus the QAT changes. --

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@GernotS Working on a theory to a couple of bugs over in OsX port, which might just help here as well. If you have the time/energy, could you see if this patch fixes things with "cache + crypto" and "raw receive" ? https://gist.github.com/lundman/5c269ae5582bac1adf2a58e7b9513c1b -- You are

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@lundman Yes, we'd need to get `zfs recv -o/-x` in illumos in general before the encryption-specific `-o/-x` support. I think we would be best to leave both for a separate PR. Thanks for staying on top of the linux encryption work, though! -- You are receiving this because you are

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

I wouldn't worry about the encryption related `zfs recv -o / -x` changes until the PR has been merged into ZoL. It's passing tests right now, but I still need to add some documentation / man pages to it. -- You are receiving this because you are subscribed to this thread. Reply to this email

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

Hmm alas, commit https://github.com/zfsonlinux/zfs/pull/7650/commits/5248401f8bf3d8149ba6e559fa94a075bfdff9d4 depends on https://github.com/zfsonlinux/zfs/commit/a3eeab2de68670a4481eab3d086982aff23b6906 and we have a fair bit of divergence in the property parsing between the two. I'm just

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

another pull request: https://github.com/zfsonlinux/zfs/pull/7650 thanks -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/openzfs/openzfs/pull/489#issuecomment-403027939

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@lundman pushed 1 commit. 58e8434 Odd git merge decisions -- You are receiving this because you are subscribed to this thread. View it on GitHub: https://github.com/openzfs/openzfs/pull/489/files/d123482538a05e9ed8dcea8f2872ee72a52b0a3f..58e8434463cc1ffd99b4fedd8995e11ea3c4af7c

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@lundman pushed 1 commit. d123482 Correct typo in zfs.1m -- You are receiving this because you are subscribed to this thread. View it on GitHub: https://github.com/openzfs/openzfs/pull/489/files/ab2d8ac94cf227207f80ca9b221ef7bd3bcd6e99..d123482538a05e9ed8dcea8f2872ee72a52b0a3f

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

Dropped and re-added 7632, added 7667 and 7637. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/openzfs/openzfs/pull/489#issuecomment-402581297 -- openzfs:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@lundman pushed 3 commits. 23eb923 Raw receive fix and encrypted objset security fix 58dd6e4 Add ASSERT to debug encryption key mapping issues ab2d8ac Fix coverity defects: CID 176037 -- You are receiving this because you are subscribed to this thread. View it on GitHub:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

gdamore commented on this pull request. > .It Sy filesystem_count The total number of filesystems and volumes that exist under this location in the dataset tree. This value is only available when a .Sy filesystem_limit has been set somewhere in the tree under which the dataset resides.

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@lundman would you mind integrating https://github.com/zfsonlinux/zfs/pull/7667 and https://github.com/zfsonlinux/zfs/pull/7637 ? I belive https://github.com/zfsonlinux/zfs/pull/7632 had some changes after integrating here as well. thanks! -- You are receiving this because you are subscribed

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

Hi, is there a way to try the work done on Smartos ? Thank you Paolo -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/openzfs/openzfs/pull/489#issuecomment-399044477

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@lundman pushed 1 commit. d9fa7a3 [PATCH] Raw receive fix and encrypted objset security fix -- You are receiving this because you are subscribed to this thread. View it on GitHub:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@lundman pushed 1 commit. 02dceca [PATCH] Make zvol update volsize operation synchronous. -- You are receiving this because you are subscribed to this thread. View it on GitHub:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@lundman @GernotS I added 3 commits which should gather more debugging info. @GernotS if you can run your workload with the latest bits, hopefully we will get more info to diagnose the issues you've been seeing. -- You are receiving this because you are subscribed to this thread. Reply to

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@lundman pushed 1 commit. d95dfde Remove test not yet ported to illumos -- You are receiving this because you are subscribed to this thread. View it on GitHub: https://github.com/openzfs/openzfs/pull/489/files/473f80e35fb25e0778ab1678fa0f4fa7ed3a48c0..d95dfdee9d978010290cfde087b25f4e0da4f56d

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@lundman pushed 2 commits. 5da0a65 Raw receive functions must not decrypt data 473f80e Update the correct abd in l2arc_read_done() -- You are receiving this because you are subscribed to this thread. View it on GitHub:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@ahrens pushed 1 commit. 9c0a780 mismerge in ztest_freeze() -- You are receiving this because you are subscribed to this thread. View it on GitHub: https://github.com/openzfs/openzfs/pull/489/files/7d80e85eb7d645666127053da17be18e17f4..9c0a780d61739bd28e15c5d564df0dbaaff284d6

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

Ah, so that is the last outstanding issue? Do you want anything squashed? Squashed-to-some, or Squashed-to-one ? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

The above problem (*.37) was determined to be a mismerge between @GernotS's version of illumos and the code in this repo. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@GernotS I got the dump from your latest panic (*.37) and I'm still not sure what's going on there. I am running some stress tests with encryption and l2arc and haven't hit a panic yet. Is there any chance you can reproduce with debug bits? Even just debug zfs kernel module would be helpful

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

I added a commit to fix a mismerge in ztest_freeze(). -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/openzfs/openzfs/pull/489#issuecomment-391141740 -- openzfs:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@ahrens pushed 1 commit. 56bbddd mismerge in ztest_freeze() -- You are receiving this because you are subscribed to this thread. View it on GitHub: https://github.com/openzfs/openzfs/pull/489/files/d1dab40ab051d5f52296e064fd3c4b990f65b37c..56bbddd350db21002218dab709bdfea65a53a388

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@GernotS Can you share a dump from the page fault you're hitting? I looked through the dumps I have from you and I don't think I have one with that stack. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

I still get the PF from above after some time. The issue is gone as soon as I remove the l2arc device. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/openzfs/openzfs/pull/489#issuecomment-390955163

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@lundman i hope it can be integrated before next year anniversary :) -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/openzfs/openzfs/pull/489#issuecomment-389523909

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

latest rebase was non-trivial due to the parallel mount commit. I had to guess a bit. But good news, we are coming up to 2 year anniversary! \o/ -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@lundman pushed 1 commit. 1fa5148 Add support for decryption faults in zinject -- You are receiving this because you are subscribed to this thread. View it on GitHub:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

and I think https://github.com/openzfs/openzfs/pull/489/commits/31a00d3962866cf5c0f0c9755b4620a978fc312a is not what you intended to do? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

Probably unrelated, but there seems to be some additional error handling in https://github.com/illumos/illumos-gate/commit/fa98e487a9619b7902f218663be219e787a57dad Can you please rebase against its once more? -- You are receiving this because you are subscribed to this thread. Reply to this

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

Not sure about that one, could be the new commits in master triggering something new, is probably beyond my abilities. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

Issue remains: panic[cpu1]/thread=fe2ce8fbd180: BAD TRAP: type=e (#pf Page fault) rp=fe003e0dd580 addr=2e occurred in module "zfs" due to a NULL pointer dereference qemu-system-x86_: #pf Page fault Bad kernel fault at addr=0x2e pid=51237, pc=0xf7e3ef45, sp=0xfe003e0dd670,

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@lundman pushed 1 commit. dad661b minor dbuf_read_done work -- You are receiving this because you are subscribed to this thread. View it on GitHub: https://github.com/openzfs/openzfs/pull/489/files/7743d9912c456bce8cc6085031ef7080c8c848a3..dad661b040d6e0d69089efc0513a450d004d61ca

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

Not sure about that, but in that place, ZOL has the following ``` ASSERT(db->db.db_data == NULL); if (db->db_level == 0 && db->db_freed_in_flight) { /* we were freed in flight; disregard any error */ +if (buf == NULL) { +buf =

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

OK, this one get me another dump after a few hours: panic[cpu2]/thread=fe44aab2c020: BAD TRAP: type=e (#pf Page fault) rp=fe003d7615e0 addr=0 occurred in module "zfs" due to a NULL pointer dereference #pf Page fault Bad kernel fault at addr=0x0 pid=114332, pc=0xf7d90366,

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@lundman pushed 1 commit. 7743d99 DMU objset should not be encrypted -- You are receiving this because you are subscribed to this thread. View it on GitHub: https://github.com/openzfs/openzfs/pull/489/files/099741e7ff87c29c3876debac50acbc99ce0ab26..7743d9912c456bce8cc6085031ef7080c8c848a3

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

Hmm odd it wasnt too big a conflict in `arc.c` but I'll go over it again -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/openzfs/openzfs/pull/489#issuecomment-385909026

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

this one gets me an assert on booting, screenshot is at https://imgur.com/aXvuamR -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/openzfs/openzfs/pull/489#issuecomment-385907991

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@lundman pushed 1 commit. 099741e Correct ztest build -- You are receiving this because you are subscribed to this thread. View it on GitHub: https://github.com/openzfs/openzfs/pull/489/files/77c699a32931847d5557f0ebadcec07a680504b1..099741e7ff87c29c3876debac50acbc99ce0ab26

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

Nah it's my fault, my VM is super slow, so its faster to rely on this build setup :) -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/openzfs/openzfs/pull/489#issuecomment-385876826

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

sorry, another one: Makefile:294: recipe for target '0-illumos-stamp' failed gmake: *** [0-illumos-stamp] Error 1 [root@526d5932-f213-cc59-9414-c8d239762b3a ~/smartos-live]# vi /root/smartos-live/projects/illumos/log/latest/nightly.log + /root/smartos-live/proto.strap/usr/bin/gcc -fident

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@lundman pushed 1 commit. 77c699a Correct merge collision -- You are receiving this because you are subscribed to this thread. View it on GitHub: https://github.com/openzfs/openzfs/pull/489/files/cd8b6763fa2413d2f3950c719a6f4f7024b5a768..77c699a32931847d5557f0ebadcec07a680504b1

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

latest commit does not compile (at least for me): Makefile:294: recipe for target '0-illumos-stamp' failed gmake: *** [0-illumos-stamp] Error 1 + /root/smartos-live/proto.strap/usr/bin/gcc -fident -finline -fno-inline-functions -fno-builtin -fno-asm -fdiagnostics-show-option -nodefaultlibs

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

i've been using this in production (in the form of ZoL) for a few months and it's worked very well since the project quota fix was merged. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

rebased and up to date -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/openzfs/openzfs/pull/489#issuecomment-385841807 -- openzfs: openzfs-developer Permalink:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@lundman pushed 1 commit. a381944 Fix issues found with zfs diff -- You are receiving this because you are subscribed to this thread. View it on GitHub: https://github.com/openzfs/openzfs/pull/489/files/03c0787a23cdd0fc44a47690d69ffcfae64eb72b..a3819442d8617598d8e1463f8b52ea0589fe96cf

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

yea, sorry, that is probably the root cause :) -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/openzfs/openzfs/pull/489#issuecomment-382299920 -- openzfs:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

Are you hinting that you want it rebased against master maybe? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/openzfs/openzfs/pull/489#issuecomment-382294674 --

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

this is the commit: https://github.com/illumos/illumos-gate/commit/3a4b1be953ee5601bab748afa07c26ed4996cde6#diff-c6fefc71095424603bca3dd5d115010b -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

Hmm I don't even have `vdev_indirect.c:1356` ? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/openzfs/openzfs/pull/489#issuecomment-382291271 -- openzfs:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

Oppsie, my bad, one sec -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/openzfs/openzfs/pull/489#issuecomment-382289822 -- openzfs: openzfs-developer Permalink:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

I have a build failure, probably related to latest Illumos commits: ../../../uts/common/fs/zfs/vdev_indirect.c: In function 'vdev_indirect_checksum_error': ../../../uts/common/fs/zfs/vdev_indirect.c:1356: error: passing argument 3 of 'zfs_ereport_post_checksum' from incompatible pointer type

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@lundman pushed 1 commit. dc23634 Fix issues with raw sends of spill blocks -- You are receiving this because you are subscribed to this thread. View it on GitHub:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@lundman pushed 2 commits. bc5a90b Correct swapped keylocation error messages 5d6cd7d Fix race in dnode_check_slots_free() -- You are receiving this because you are subscribed to this thread. View it on GitHub:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@ahrens pushed 1 commit. f46f092 add codes to truss -- You are receiving this because you are subscribed to this thread. View it on GitHub: https://github.com/openzfs/openzfs/pull/489/files/893c519c9dcb7eae00acb466c9d19c2c37b26ed3..f46f0922baeffdc1b7b8750a4fe9ed4ef202ade6

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@lundman pushed 1 commit. 893c519 receive_spill does not byte swap spill contents -- You are receiving this because you are subscribed to this thread. View it on GitHub:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@ahrens pushed 3 commits. 363b893 assertion failure in arc_release() during encrypted receive 48f5098 tom feedback 2b622bf more feedback -- You are receiving this because you are subscribed to this thread. View it on GitHub:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@lundman pushed 1 commit. ef73ffd Move enum zio_encrypt into sys/fs/zfs.h -- You are receiving this because you are subscribed to this thread. View it on GitHub:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

> Well, there's an unfortunate precedent of userland (at least libzfs) > including sys/zfs headers directly For what it's worth, this is something we're slowly working to clean up so we should be able to pull back these kind of changes. -- You are receiving this because you are subscribed to

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@lundman let's move `enum zio_encrypt` to `sys/fs/zfs.h`. Thanks! -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/openzfs/openzfs/pull/489#issuecomment-380618411 --

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@lundman pushed 1 commit. bf72974 Correct manpage for --raw -- You are receiving this because you are subscribed to this thread. View it on GitHub: https://github.com/openzfs/openzfs/pull/489/files/097d997308675d4d2387d5bb8a786fb09a189dfc..bf72974e371a6b1d097024f7cb567e9ea78fa65b

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

Not sure I'm seeing it. The difference to ZOL is that they have `zfs_main.c` add ``` #include "libzfs_impl.h" ``` But I can not add that to IllumOS, as none of the -I paths include `libzfs_impl.h` nor `sys/zio.h` - are you suggesting I change Makefiles to add the include path? As for existing

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

Well, there's an unfortunate precedent of userland (at least libzfs) including sys/zfs headers directly, so we could probably just try to include them here. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

Hmm that's defined in `zio.h` but the `cmd`'s tend not to include the sys/zfs* headers directly? Should they be in `zfs_context`? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

Problem with 097d997: zfs_main.c: In function 'share_mount_one': zfs_main.c:6072: error: 'ZIO_CRYPT_OFF' undeclared (first use in this function) zfs_main.c:6072: error: (Each undeclared identifier is reported only once zfs_main.c:6072: error: for each function it appears in.) -- You are

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@lundman pushed 1 commit. 097d997 Make encrypted "zfs mount -a" failures consistent -- You are receiving this because you are subscribed to this thread. View it on GitHub:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@ahrens pushed 1 commit. c3623f8 lint -- You are receiving this because you are subscribed to this thread. View it on GitHub: https://github.com/openzfs/openzfs/pull/489/files/b7734bfecda4baac05c3416bf20211086fea7275..c3623f89d3a48ffce3e6b9f0a825e1125af7f414

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

I pushed 2 commits. I'm going to try to open ZoL PR's for each of them as well. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/openzfs/openzfs/pull/489#issuecomment-378776850

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@ahrens pushed 2 commits. c8b50d0 fail fraction b7734bf fixes for SNPRINTF_BLKPTR with encrypted BP's -- You are receiving this because you are subscribed to this thread. View it on GitHub:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

ahrens commented on this pull request. > @@ -409,6 +489,26 @@ _NOTE(CONSTCOND) } while (0) #defineBP_GET_LEVEL(bp)BF64_GET((bp)->blk_prop, 56, 5) #defineBP_SET_LEVEL(bp, x) BF64_SET((bp)->blk_prop, 56, 5, x) +/* encrypted, authenticated, and MAC

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@GernotS That last panic is quite bizarre, and doesn't appear to be related to encryption. We have an indirect block (level 1) of a space map, and it's filled with a pattern like 0xdadada... but every 16th byte is incremented by 0x10. The block is in the L2ARC. Of course, checksums should

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@lundman pushed 3 commits. bdcf9d7 Encrypted dnode blocks should be prefetched raw f016fc4 Decryption error handling improvements 9b1e59f Remove ASSERT() in l2arc_apply_transforms() -- You are receiving this because you are subscribed to this thread. View it on GitHub:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

While I cannot reproduce l2arc panic anymore, I had another (new) one reagding zvols: panic[cpu3]/thread=ff003fe9ec40: blkptr at ff0dbeb82200 has invalid TYPE 218 ff003fe9e330 genunix:vcmn_err+42 () ff003fe9e3a0 zfs:zfs_panic_recover+51 () ff003fe9e400

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@GernotS The problem causing the panic in sa.c is that zio_decrypt() was not properly propagating a decryption / MAC verification error. I added a commit to this PR which fixes that. You should now get an error message when doing the `ls -l`, rather than a panic. The next issue is why it's

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@ahrens pushed 1 commit. 1a503d4 dont swallow error! -- You are receiving this because you are subscribed to this thread. View it on GitHub: https://github.com/openzfs/openzfs/pull/489/files/a048a01fa13fac430a9f9c838f70569e54122101..1a503d4121bb673a3f99a9a16da28b6f9bf3349d

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

I looks like I need to increase my dump device: savecore -f vmdump.118 savecore: incomplete dump on dump device savecore: System dump time: Thu Mar 29 09:50:51 2018 savecore: saving system crash dump in /var/crash/volatile/{unix,vmcore}.118 Constructing namelist

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@GernotS Thanks. I downloaded vmdump.118 but it doesn't seem to have come through correctly. Any chance you uploaded it while it was still being written? The file length I have is `3347091456` bytes, and I get: ``` $ savecore -f vmdump.118 . savecore: bad magic number 78201ff0 savecore:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@Ahrens ls -l /zones/data/fritz gives me /zones/data/fritz: Bad exchange descriptor trying other directories gives me the known crash -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

Thanks! Can you try ls -l /zones/data/fritz too? Does that crash? On Thu, Mar 29, 2018 at 5:28 AM GernotS wrote: > @ahrens > > steps to reproduce > zfs send -w zones|datatest@1 |zfs recv zones/data (in my case about 700GB) > zfs set

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@GernotS Thanks. I'm still not sure what's going on with the sa.c dump, and have a few more questions for you: Can you describe how you reproduce the sa.c dump? I could gather some of the sequence below but if you can confirm that and fill in any other details that would be helpful. It

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

another dump for the assert in sa.c, hopefully with debug bits now https://www.magentacloud.de/lnk/I0rB1RA1 -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/openzfs/openzfs/pull/489#issuecomment-376915757

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

I have now moved around a few TBs using raw send, compressed send, etc with L2ARC devices, but I cannot reproduce the issue any more. Is it possible kmem_flags hides this? Will try to get a debug build done. -- You are receiving this because you are subscribed to this thread. Reply to this

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@GernotS Thanks. I'm also very interested in getting this merged :-) kmem_flags is great, but in addition it would be good to try a debug build, i.e. using the `nightly` rather than `nightly-nd` packages. This might be able to hit an assertion sooner - closer to where the error occurred.

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

Dump for the assert in sa.c https://www.magentacloud.de/lnk/4xLBV469 -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/openzfs/openzfs/pull/489#issuecomment-376474714

  1   2   >