[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

@lundman pushed 1 commit. e418fcb Small bug fixes -- You are receiving this because you are subscribed to this thread. View it on GitHub: https://github.com/openzfs/openzfs/pull/124/files/c26140e60da6f8a885be38fd339e9c5e2781007c..e418fcbb7204fafc5f0b77881739fa0289175352

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

@lundman pushed 1 commit. ed02f22 bug fixes -- You are receiving this because you are subscribed to this thread. View it on GitHub: https://github.com/openzfs/openzfs/pull/124/files/2992551a9e789513fdd9a5c7868771d78c12b8bf..ed02f22933e7c5c9ae31cbd3cbaf1322c302f299

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

A few of the tests fail that might not be related, which tests do I need to look at wrt to this PR ? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/openzfs/openzfs/pull/124#issuecomment-316238036

[developer] Re: [openzfs/openzfs] backout: 8021 ARC buf data scatter-ization (causes 8100) (4ee0199)

If you are curious: ``` src == dst and c 2 x 2 size 0 src == dst and c 2 x 2 size 0 src == dst and c 2 x 2 size 0

[developer] Re: [openzfs/openzfs] backout: 8021 ARC buf data scatter-ization (causes 8100) (4ee0199)

The code we added to show it was happening was: ``` @@ -976,7 +976,8 @@ vdev_raidz_reconstruct_q(raidz_map_t *rm, int *tgts, int ntg ts) dst = rm->rm_col[x].rc_abd; if (c == rm->rm_firstdatacol) { - abd_copy(dst, src, size); +

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

Ah volumes, so something like ``` # zpool create -f BOOM c2t3d0 # zfs create BOOM/plain # zfs create -o encryption=on -o keyformat=passphrase BOOM/ccm Enter passphrase: Re-enter passphrase: # zfs create -V 1G BOOM/vol # zfs snapshot BOOM/vol@send # zfs send BOOM/vol@send | zfs recv BOOM/ccm/evol

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

@GernotS are these the commands used? I've tried it with latest commit: ``` # zpool create -f BOOM c2t3d0 # zfs create -o encryption=aes-256-ccm -o keyformat=passphrase BOOM/ccm # zfs create BOOM/plain # zfs snapshot BOOM/plain@send # zfs send BOOM/plain@send | zfs recv BOOM/ccm/new # zfs get

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

@zettabot go -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/openzfs/openzfs/pull/124#issuecomment-323239573 -- openzfs-developer Archives:

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

I can confirm that a `send --raw plain` into encrypted panics, which shouldn't be allowed. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/openzfs/openzfs/pull/124#issuecomment-323559993

[developer] Re: [openzfs/openzfs] 8100 8021 seems to cause random BAD TRAP: type=d (#gp General protection) (#375)

lundman approved this pull request. Runs through the tester for us, which is an improvement. `abd_copy()` still gets called with `src==dst`, but this is a noop generally. As we added "ABD move" to our repo to decrease defragmentation, we will add code to skip the null copy, but since that is

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

The PR is just 3 weeks away from its first birthday, yaaay \o/ -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/openzfs/openzfs/pull/124#issuecomment-302028603 --

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

@lundman pushed 1 commit. 9170e97 Correct code in raidz_checksum_error() -- You are receiving this because you are subscribed to this thread. View it on GitHub: https://github.com/openzfs/openzfs/pull/124/files/84fbf78cb5c75fe5027fb3edcaeacb496e25400c..9170e977ffe1f483173d85dece954b08e2ff

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

`rngd` is a Linux helper for random, it can just be `echo` or similar on Illumos. I'll have a look at the tests in the morning. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

@lundman pushed 1 commit. 1045b9c Remove rngd from tests -- You are receiving this because you are subscribed to this thread. View it on GitHub: https://github.com/openzfs/openzfs/pull/124/files/7e91e1af19bceb66dcddfc8776e281e7a72d685e..1045b9c5762c2f0bad1110fe1303f461a1b96aff

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

Ok I went through the FAILs I got locally ``` functional/cli_root/zpool_get/zpool_get_002_pos (run as root) [00:00] [FAIL] ``` It seems `zpool_get.cfg` is missing the `bootsize` property, which is outside of the scope of this PR, so I will ignore it. ``` 14:17:01.36 Assertion failed:

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

@lundman pushed 1 commit. a19f544 zfs-tests corrections -- You are receiving this because you are subscribed to this thread. View it on GitHub: https://github.com/openzfs/openzfs/pull/124/files/1045b9c5762c2f0bad1110fe1303f461a1b96aff..a19f54495262141f5a551d525c9d6179276c066e

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

The test is the one immediate after ``` /opt/zfs-tests/tests/functional/cli_root/zpool_clear/setup ``` ie ``` zpool_clear_001_pos.ksh ``` -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

Pushed the latest commits, I did not do the manpage changes, specifically those in https://github.com/zfsonlinux/zfs/pull/5769/commits/705a2bfc5eabb45facf1e0f7a38a3ee5a69a54ea#diff-42027d93bce823d185f48bd265de4ac0

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

@lundman pushed 2 commits. 24aa9ff Various fixes and improvements 0f52c02 Fixes and improvements after 4th round of review -- You are receiving this because you are subscribed to this thread. View it on GitHub:

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

@lundman pushed 1 commit. ea53ac7 Compile fixes -- You are receiving this because you are subscribed to this thread. View it on GitHub: https://github.com/openzfs/openzfs/pull/124/files/0f52c029de5953d706bec7b3d288b5cc6d6f0072..ea53ac7930549f31a8d6480ab697c404ffbc05f4

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

@lundman pushed 1 commit. 6c89a04 Delinting -- You are receiving this because you are subscribed to this thread. View it on GitHub: https://github.com/openzfs/openzfs/pull/124/files/ea53ac7930549f31a8d6480ab697c404ffbc05f4..6c89a04a59661b4bedc7e4b892c0a100204695da

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

@lundman pushed 1 commit. f68ee3c Attempt to update the manpages -- You are receiving this because you are subscribed to this thread. View it on GitHub: https://github.com/openzfs/openzfs/pull/124/files/79798fac7fde50d17bd90222d13ee0ac3d806be3..f68ee3ce27b2a9758ae41e0addf24b511fcd1cb4

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

@lundman pushed 1 commit. dc4e629 Fixes -- You are receiving this because you are subscribed to this thread. View it on GitHub: https://github.com/openzfs/openzfs/pull/124/files/6c89a04a59661b4bedc7e4b892c0a100204695da..dc4e629f2d708e7241963ddaa8b94996863075d9

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

It was my guess that all the zdb cores (there are many of them) made the tests run slow, and ended up in a timeout.. I will take a look at the zdb assert and see if it is something obvious. Cheers for your feedback -- You are receiving this because you are subscribed to this thread. Reply to

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

@lundman pushed 2 commits. bbf9bd3 Fixes and improvements after 5th round of review 439cd9c Fixes after rebase and more review -- You are receiving this because you are subscribed to this thread. View it on GitHub:

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

@lundman pushed 2 commits. 27bc76c ABD released too early in abd_hmac check 785086e Corrections in zio_checksum for crypto -- You are receiving this because you are subscribed to this thread. View it on GitHub:

[developer] Re: [openzfs/openzfs] 8648 Fix range locking in ZIL commit codepath (#462)

It is probably not related, and perhaps not even worth mentioning. But due to deadlocks in OsX version from zget and rangelock in `zfs_get_data()` we now call both zget and rangelock in `zil_lwb_commit()` before passing them into `zilog->zl_get_data()`. But I don't think you are dealing with

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

This PR has been updated with latest commits up and including Oct 2nd "060e4cf33002c4". -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/openzfs/openzfs/pull/124#issuecomment-334077937

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

@lundman pushed 1 commit. 0b4ee95 prevent panic when non-raidz checksum error detected -- You are receiving this because you are subscribed to this thread. View it on GitHub:

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

@lundman pushed 1 commit. 57291b0 de-linting -- You are receiving this because you are subscribed to this thread. View it on GitHub: https://github.com/openzfs/openzfs/pull/124/files/8e2c3b855a7eb155744586ec610d642c2c46a5c9..57291b074f93cd588106df1a5c0147778c600c6e

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

Now contains the send/recv fixes from ZOL. Please test. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/openzfs/openzfs/pull/124#issuecomment-324526413 --

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

@lundman pushed 1 commit. 807e3db Minor code style changes -- You are receiving this because you are subscribed to this thread. View it on GitHub: https://github.com/openzfs/openzfs/pull/124/files/0b4ee95274efe6e3103402a8d390e9206cca2669..807e3dbdbda5090243d8d534c010ee6ccb34187a

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

@lundman pushed 2 commits. 53406a2 More corrections to testers bf0d5b0 Disable dump on encrypted zvol -- You are receiving this because you are subscribed to this thread. View it on GitHub:

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

@lundman pushed 1 commit. 599316e OpenZFS repo runs delphix.run in tests, so update it to match -- You are receiving this because you are subscribed to this thread. View it on GitHub:

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

Looks like we managed to pass all tests finally. I plan to squash everything into one single commit unless there is some other number of commits people would prefer..? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

[developer] Re: [openzfs/openzfs] fixed arc_cksum_is_equal() that doesn't take into account ABD-logic (#498)

Or maybe I am overselling, perhaps the hours of ABD fixes in raidz was *because* of the crypto changes. But it did lead us to doing something like https://github.com/openzfsonosx/zfs/commit/5995f89c1eeae7f029b3537564780458c2920f44 so we could be sure the wrong buffer was not passed into ABD.

[developer] Re: [openzfs/openzfs] fixed arc_cksum_is_equal() that doesn't take into account ABD-logic (#498)

I believe we corrected those over in the crypto PR, and I think there were a few in vdev_raidz's parity work as well, if you want to fix those too. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#124)

Closed in favour of https://github.com/openzfs/openzfs/pull/489 -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/openzfs/openzfs/pull/124#issuecomment-341079785 --

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#124)

Closed #124. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/openzfs/openzfs/pull/124#event-1320801076 -- openzfs-developer Archives:

[developer] [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

Reviewed by: Matthew Ahrens <mahr...@delphix.com> Reviewed-by: Brian Behlendorf <behlendo...@llnl.gov> Reviewed-by: Jorgen Lundman <lund...@lundman.net> Signed-off-by: Tom Caputi <tcap...@datto.com> Send / Recv Fixes following b52563 This patch fixes se

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

https://github.com/zfsonlinux/zfs/issues/6845 -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/openzfs/openzfs/pull/489#issuecomment-34795 -- openzfs-developer

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@behlendorf Thanks, now included. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/openzfs/openzfs/pull/489#issuecomment-343345787 -- openzfs-developer Archives:

Re: [developer] Re: non-linear abd_alloc_for_io vs zio_checksum_compute

Andriy Gapon wrote: On 23/11/2017 18:05, Andriy Gapon wrote: I've done a little experimentation with a version of abd_alloc_for_io() that returns a chunked abd. zio_checksum_compute() couldn't cope with that in the Another detail that's not directly related to the above is that even now we

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#124)

@ahrens This is the first PR, followed by all fixes, two more PRs and fixes. It has everything in it that ZoL has. I don't think it is so much about what I need, but what you want to happen next. -- You are receiving this because you are subscribed to this thread. Reply to this email directly

[developer] [openzfs/openzfs] Large alloc in zdb can cause trouble (#635)

16MB alloc in zdb_embedded_block() can cause cores in certain situations (clang, gcc55). You can view, comment on, or merge this pull request online at: https://github.com/openzfs/openzfs/pull/635 -- Commit Summary -- * Large alloc in zdb can cause trouble -- File Changes -- M

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@lundman pushed 1 commit. 7743d99 DMU objset should not be encrypted -- You are receiving this because you are subscribed to this thread. View it on GitHub: https://github.com/openzfs/openzfs/pull/489/files/099741e7ff87c29c3876debac50acbc99ce0ab26..7743d9912c456bce8cc6085031ef7080c8c848a3

[developer] Re: [openzfs/openzfs] Large alloc in zdb can cause trouble (#635)

https://github.com/openzfsonosx/zfs/commit/ced236a5da6e72ea7bf6d2919fe14e17cffe10f1 Boom, by 10 days! :) -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/openzfs/openzfs/pull/635#issuecomment-387029887

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@lundman pushed 1 commit. 1fa5148 Add support for decryption faults in zinject -- You are receiving this because you are subscribed to this thread. View it on GitHub:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

Not sure about that, but in that place, ZOL has the following ``` ASSERT(db->db.db_data == NULL); if (db->db_level == 0 && db->db_freed_in_flight) { /* we were freed in flight; disregard any error */ +if (buf == NULL) { +buf =

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@lundman pushed 1 commit. dad661b minor dbuf_read_done work -- You are receiving this because you are subscribed to this thread. View it on GitHub: https://github.com/openzfs/openzfs/pull/489/files/7743d9912c456bce8cc6085031ef7080c8c848a3..dad661b040d6e0d69089efc0513a450d004d61ca

[developer] [openzfs/openzfs] Add SIGINFO feature to zfs send (#638)

send load: 1.15 cmd: zfs 1523 waiting 0.04u 2.05s 16:29:30948M BOOM@send Signed-off-by: Jorgen Lundman <lund...@lundman.net> You can view, comment on, or merge this pull request online at: https://github.com/openzfs/openzfs/pull/638 -- Commit Summary -- * Add SIGINFO feature to z

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

Not sure about that one, could be the new commits in master triggering something new, is probably beyond my abilities. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

[developer] Re: [openzfs/openzfs] Add SIGINFO feature to zfs send (#638)

lundman commented on this pull request. > @@ -70,7 +70,8 @@ int lzc_get_holds(const char *, nvlist_t **); enum lzc_send_flags { LZC_SEND_FLAG_EMBED_DATA = 1 << 0, LZC_SEND_FLAG_LARGE_BLOCK = 1 << 1, - LZC_SEND_FLAG_COMPRESS = 1 << 2 + LZC_SEND_FLAG_COMPRESS = 1 <<

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

latest rebase was non-trivial due to the parallel mount commit. I had to guess a bit. But good news, we are coming up to 2 year anniversary! \o/ -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

Ah, so that is the last outstanding issue? Do you want anything squashed? Squashed-to-some, or Squashed-to-one ? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

Re: [developer] [openzfs/openzfs] Trying again with sequential scan -- this is simply (#648)

vdev_queue_io_remove+0x13c(ff03ecd7dcc0, ff03e90bb7e8) vdev_queue_io_to_issue+0x1cb(ff03ecd7dcc0) vdev_queue_io_done+0x78(ff03e90fd060) zio_vdev_io_done+0xa8(ff03e90fd060) zio_execute+0xfa(ff03e90fd060) taskq_thread+0x318(ff03ecc769e0) thread_start+8() -- Jo

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@lundman pushed 1 commit. 02dceca [PATCH] Make zvol update volsize operation synchronous. -- You are receiving this because you are subscribed to this thread. View it on GitHub:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@lundman pushed 1 commit. d9fa7a3 [PATCH] Raw receive fix and encrypted objset security fix -- You are receiving this because you are subscribed to this thread. View it on GitHub:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@lundman pushed 1 commit. d95dfde Remove test not yet ported to illumos -- You are receiving this because you are subscribed to this thread. View it on GitHub: https://github.com/openzfs/openzfs/pull/489/files/473f80e35fb25e0778ab1678fa0f4fa7ed3a48c0..d95dfdee9d978010290cfde087b25f4e0da4f56d

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@lundman pushed 2 commits. 5da0a65 Raw receive functions must not decrypt data 473f80e Update the correct abd in l2arc_read_done() -- You are receiving this because you are subscribed to this thread. View it on GitHub:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

Nah it's my fault, my VM is super slow, so its faster to rely on this build setup :) -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/openzfs/openzfs/pull/489#issuecomment-385876826

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@lundman pushed 1 commit. 77c699a Correct merge collision -- You are receiving this because you are subscribed to this thread. View it on GitHub: https://github.com/openzfs/openzfs/pull/489/files/cd8b6763fa2413d2f3950c719a6f4f7024b5a768..77c699a32931847d5557f0ebadcec07a680504b1

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@lundman pushed 1 commit. 099741e Correct ztest build -- You are receiving this because you are subscribed to this thread. View it on GitHub: https://github.com/openzfs/openzfs/pull/489/files/77c699a32931847d5557f0ebadcec07a680504b1..099741e7ff87c29c3876debac50acbc99ce0ab26

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@lundman pushed 1 commit. a381944 Fix issues found with zfs diff -- You are receiving this because you are subscribed to this thread. View it on GitHub: https://github.com/openzfs/openzfs/pull/489/files/03c0787a23cdd0fc44a47690d69ffcfae64eb72b..a3819442d8617598d8e1463f8b52ea0589fe96cf

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

rebased and up to date -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/openzfs/openzfs/pull/489#issuecomment-385841807 -- openzfs: openzfs-developer Permalink:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

Hmm odd it wasnt too big a conflict in `arc.c` but I'll go over it again -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/openzfs/openzfs/pull/489#issuecomment-385909026

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

This now includes the ZOL https://github.com/zfsonlinux/zfs/pull/6864 PR as well, so we can test `zfs send --raw` fixes. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@lundman pushed 1 commit. 3fe3c5c Update cryptv0 test pool -- You are receiving this because you are subscribed to this thread. View it on GitHub: https://github.com/openzfs/openzfs/pull/489/files/fda748267547c10116f660c7f842716d67ecaa94..3fe3c5c86306386a7acb06e153626c635fa27858

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

I was unable to replicate: ``` write: No space left on device 8245278720 bytes transferred in 1880.011589 secs (4385760 bytes/sec) ``` But that could simply be that I don't get dbuf eviction triggered. However I did find some differences between the two versions, perhaps it will help. -- You

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@lundman pushed 1 commit. d6cda08 Correct differences in arc.c for crypto -- You are receiving this because you are subscribed to this thread. View it on GitHub:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

I've run out of obvious looking diffs between ZOL and this commit. Might require actual debugging now. Seems to die pretty early in `arc_buf_remove` but it'd be nice to know where. ZOL also has a `arc_buf_access()` called from `dbuf` but the ZOL commit 0873bb63 just talks about counting stats.

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

Well, that's good.. Not sure what to do next then, perhaps @GernotS can provide a dump file, and someone who actually knows how to use mdb (I sure don't) can be coaxed into glancing at it? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

(No changes, just fixed conflicts with master) -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/openzfs/openzfs/pull/489#issuecomment-364290139 -- openzfs-developer

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

Neat, I didn't mention the commit today, since the ZOL tickets just talked about send|recv :) Still unable to reproduce dnode_destroy panic, but the tester here seems to die though -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@lundman pushed 1 commit. 7b94083 Correct l2arc_write_buffers() -- You are receiving this because you are subscribed to this thread. View it on GitHub: https://github.com/openzfs/openzfs/pull/489/files/3803144fa6df4913e94a4b35884c90fb08a6eda3..7b9408363a8a3bd53426e6b57773f43afac204ec

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

``` it should be like ZOL's l2arc_write_buffers, make it so! Done! ``` More fictionally chat logs to come, hopefully this will address @GernotS issue. Cheers! -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

On OsX I added a `cache device` to the crypt test, and I got: ``` panic(cpu 0 caller 0xff8027ec8132): Kernel trap at 0xff7fab62a0cc, type 14=page fault, registers: 0xff8087a43d60 : 0xff8027cfc200 mach_kernel : trap_from_kernel + 0x26 0xff8087a43d80 : 0xff7fab62a0cc

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@lundman pushed 1 commit. 5ebecdd Re-port l2arc_write_buffers just in case -- You are receiving this because you are subscribed to this thread. View it on GitHub:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

It is interesting that `l2arc_write_buffers` is one of the places the merge was not smooth. Re-porting the function from clean did not change much alas, adding flag `ARC_FLAG_L2_WRITING` . The IllumOS version has an extra chunk of code that the other platforms have: ```

[developer] Re: [openzfs/openzfs] 9688 aggsum_fini leaks memory (#679)

When running this commit on OsX, which is a little more strict with mutex release; we get this situation: ``` frame #4: kernel.development`panic(mutex_enter on destroyed mutex) at debug.c:605 [opt] frame #5: spl`spl_mutex_enter(mp=0xff886f557b00, file="aggsum.c", line=190) at

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

In the interest in making `dsl_crypto.c` and `zio_crypto.c` to be the same as ZOL - I added a few commits that messed around in there. [1] The 3a5bf20 commit will replace the gist I pasted in https://github.com/openzfs/openzfs/pull/489#issuecomment-404715592 [1] minus the QAT changes. --

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

Woo, I contributed! \o/ -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/openzfs/openzfs/pull/489#issuecomment-407324973 -- openzfs: openzfs-developer Permalink:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@lundman pushed 1 commit. 3972768 Do not call dmu_objset_disown twice -- You are receiving this because you are subscribed to this thread. View it on GitHub: https://github.com/openzfs/openzfs/pull/489/files/6f85552ff7a285fef0af14f31d322cc82bebd07c..3972768dc18f2f6b4b2c3c90ac9ce516f8e1cf99

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@lundman pushed 1 commit. d123482 Correct typo in zfs.1m -- You are receiving this because you are subscribed to this thread. View it on GitHub: https://github.com/openzfs/openzfs/pull/489/files/ab2d8ac94cf227207f80ca9b221ef7bd3bcd6e99..d123482538a05e9ed8dcea8f2872ee72a52b0a3f

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@lundman pushed 3 commits. 23eb923 Raw receive fix and encrypted objset security fix 58dd6e4 Add ASSERT to debug encryption key mapping issues ab2d8ac Fix coverity defects: CID 176037 -- You are receiving this because you are subscribed to this thread. View it on GitHub:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

Dropped and re-added 7632, added 7667 and 7637. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/openzfs/openzfs/pull/489#issuecomment-402581297 -- openzfs:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@lundman pushed 1 commit. 58e8434 Odd git merge decisions -- You are receiving this because you are subscribed to this thread. View it on GitHub: https://github.com/openzfs/openzfs/pull/489/files/d123482538a05e9ed8dcea8f2872ee72a52b0a3f..58e8434463cc1ffd99b4fedd8995e11ea3c4af7c

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

Hmm alas, commit https://github.com/zfsonlinux/zfs/pull/7650/commits/5248401f8bf3d8149ba6e559fa94a075bfdff9d4 depends on https://github.com/zfsonlinux/zfs/commit/a3eeab2de68670a4481eab3d086982aff23b6906 and we have a fair bit of divergence in the property parsing between the two. I'm just

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@GernotS Working on a theory to a couple of bugs over in OsX port, which might just help here as well. If you have the time/energy, could you see if this patch fixes things with "cache + crypto" and "raw receive" ? https://gist.github.com/lundman/5c269ae5582bac1adf2a58e7b9513c1b -- You are

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@lundman pushed 1 commit. 1229496 Prevent raw zfs recv -F if dataset is unencrypted -- You are receiving this because you are subscribed to this thread. View it on GitHub:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

Hmm that's defined in `zio.h` but the `cmd`'s tend not to include the sys/zfs* headers directly? Should they be in `zfs_context`? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@lundman pushed 1 commit. 097d997 Make encrypted "zfs mount -a" failures consistent -- You are receiving this because you are subscribed to this thread. View it on GitHub:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@lundman pushed 1 commit. 893c519 receive_spill does not byte swap spill contents -- You are receiving this because you are subscribed to this thread. View it on GitHub:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@lundman pushed 1 commit. ef73ffd Move enum zio_encrypt into sys/fs/zfs.h -- You are receiving this because you are subscribed to this thread. View it on GitHub:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

Not sure I'm seeing it. The difference to ZOL is that they have `zfs_main.c` add ``` #include "libzfs_impl.h" ``` But I can not add that to IllumOS, as none of the -I paths include `libzfs_impl.h` nor `sys/zio.h` - are you suggesting I change Makefiles to add the include path? As for existing

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@lundman pushed 1 commit. bf72974 Correct manpage for --raw -- You are receiving this because you are subscribed to this thread. View it on GitHub: https://github.com/openzfs/openzfs/pull/489/files/097d997308675d4d2387d5bb8a786fb09a189dfc..bf72974e371a6b1d097024f7cb567e9ea78fa65b

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

Hmm I don't even have `vdev_indirect.c:1356` ? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/openzfs/openzfs/pull/489#issuecomment-382291271 -- openzfs:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

Oppsie, my bad, one sec -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/openzfs/openzfs/pull/489#issuecomment-382289822 -- openzfs: openzfs-developer Permalink:

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

Are you hinting that you want it rebased against master maybe? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/openzfs/openzfs/pull/489#issuecomment-382294674 --

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

@lundman pushed 2 commits. bc5a90b Correct swapped keylocation error messages 5d6cd7d Fix race in dnode_check_slots_free() -- You are receiving this because you are subscribed to this thread. View it on GitHub:

  1   2   >