[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

[Jorgen Lundman]

@lundman pushed 1 commit.

e418fcb  Small bug fixes


-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/openzfs/openzfs/pull/124/files/c26140e60da6f8a885be38fd339e9c5e2781007c..e418fcbb7204fafc5f0b77881739fa0289175352

--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/T1625245905c55186-M8b134c04c20c6f7b12999f8c
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

[Jorgen Lundman]

@lundman pushed 1 commit.

ed02f22  bug fixes


-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/openzfs/openzfs/pull/124/files/2992551a9e789513fdd9a5c7868771d78c12b8bf..ed02f22933e7c5c9ae31cbd3cbaf1322c302f299

--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/T1625245905c55186-Mdbc6bb8a10216ac0a621f47a
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

[Jorgen Lundman]

A few of the tests fail that might not be related, which tests do I need to 
look at wrt to this PR ?


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/124#issuecomment-316238036
--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/T1625245905c55186-M92f3208ba1482a74cfc4512c
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] backout: 8021 ARC buf data scatter-ization (causes 8100) (4ee0199)

[Jorgen Lundman]

If you are curious:
```
 src == dst and c 2 x 2 size 0
 src == dst and c 2 x 2 size 0
 src == dst and c 2 x 2 size 0
 src == dst and c 2 x 2 size 0
```


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/commit/4ee0199ec059ca9417a72125beae8d89fbc5a9df#commitcomment-21957871
--
openzfs-developer
Archives: https://openzfs.topicbox.com/groups/developer/
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] backout: 8021 ARC buf data scatter-ization (causes 8100) (4ee0199)

[Jorgen Lundman]

The code we added to show it was happening was:
```
@@ -976,7 +976,8 @@ vdev_raidz_reconstruct_q(raidz_map_t *rm, int *tgts, int ntg
ts)
dst = rm->rm_col[x].rc_abd;

if (c == rm->rm_firstdatacol) {
-   abd_copy(dst, src, size);
+   if (dst != src)
+   abd_copy(dst, src, size);
```

```
@@ -1069,6 +1069,10 @@ abd_iterate_func2(abd_t *dabd, abd_t *sabd, size_t doff, 
size_t soff,
int ret = 0;
struct abd_iter daiter, saiter;

+   if (dabd == sabd) {
+   printf("dabd and sabd are equal\n");
+   return 0;
+   }
```

Technically both changes fixes the same problem.


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/commit/4ee0199ec059ca9417a72125beae8d89fbc5a9df#commitcomment-21956498
--
openzfs-developer
Archives: https://openzfs.topicbox-beta.com/groups/developer/
Powered by Topicbox: https://topicbox-beta.com

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

[Jorgen Lundman]

Ah volumes, so something like
```
# zpool create -f BOOM c2t3d0
# zfs create BOOM/plain
# zfs create -o encryption=on -o keyformat=passphrase BOOM/ccm
Enter passphrase:
Re-enter passphrase:

# zfs create -V 1G BOOM/vol
# zfs snapshot BOOM/vol@send
# zfs send BOOM/vol@send | zfs recv BOOM/ccm/evol
# zfs get keystatus
NAME PROPERTY   VALUESOURCE
BOOM/ccm keystatus  available-
BOOM/ccm/evolkeystatus  available-
BOOM/ccm/evol@send   keystatus  available-
BOOM/vol keystatus  --
BOOM/vol@sendkeystatus  --
```

?


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/124#issuecomment-323487498
--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/T1625245905c55186-M6c4b93d6e811815416089841
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

[Jorgen Lundman]

@GernotS  are these the commands used? I've tried it with latest commit:

```
# zpool create -f BOOM c2t3d0
# zfs create -o encryption=aes-256-ccm -o keyformat=passphrase BOOM/ccm
# zfs create BOOM/plain
# zfs snapshot BOOM/plain@send
# zfs send BOOM/plain@send | zfs recv BOOM/ccm/new
# zfs get keystatus

BOOM keystatus  --
BOOM/ccm keystatus  available-
BOOM/ccm/new keystatus  available-
BOOM/ccm/new@sendkeystatus  available-
BOOM/plain   keystatus  --
BOOM/plain@send  keystatus  --
```


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/124#issuecomment-323229283
--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/T1625245905c55186-M96a25d65e260e11fb091d8eb
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

[Jorgen Lundman]

@zettabot go


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/124#issuecomment-323239573
--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/T1625245905c55186-Mb3c0113e02ea05630ad85f36
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

[Jorgen Lundman]

I can confirm that a `send --raw plain` into encrypted panics, which shouldn't 
be allowed.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/124#issuecomment-323559993
--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/T1625245905c55186-M2a4681e03b79c37c0851fec1
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] 8100 8021 seems to cause random BAD TRAP: type=d (#gp General protection) (#375)

[Jorgen Lundman]

lundman approved this pull request.

Runs through the tester for us, which is an improvement. `abd_copy()` still 
gets called with `src==dst`, but this is a noop generally. As we added "ABD 
move" to our repo to decrease defragmentation, we will add code to skip the 
null copy, but since that is part of the "ABD move" commit, it needs not be 
addressed in the ABD commit.



-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/375#pullrequestreview-37742147
--
openzfs-developer
Archives: https://openzfs.topicbox.com/groups/developer/
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

[Jorgen Lundman]

The PR is just 3 weeks away from its first birthday, yaaay \o/


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/124#issuecomment-302028603
--
openzfs-developer
Archives: https://openzfs.topicbox.com/groups/developer/
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

[Jorgen Lundman]

@lundman pushed 1 commit.

9170e97  Correct code in raidz_checksum_error()


-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/openzfs/openzfs/pull/124/files/84fbf78cb5c75fe5027fb3edcaeacb496e25400c..9170e977ffe1f483173d85dece954b08e2ff

--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/T1625245905c55186-M5e0c764cd519e09fb966c3ea
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

[Jorgen Lundman]

`rngd` is a Linux helper for random, it can just be `echo` or similar on 
Illumos. I'll have a look at the tests in the morning.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/124#issuecomment-310043826
--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/T1625245905c55186-M41e779dd8ba25743a68680f8
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

[Jorgen Lundman]

@lundman pushed 1 commit.

1045b9c  Remove rngd from tests


-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/openzfs/openzfs/pull/124/files/7e91e1af19bceb66dcddfc8776e281e7a72d685e..1045b9c5762c2f0bad1110fe1303f461a1b96aff

--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/T1625245905c55186-Mbcb8c675665a5db1de3569c2
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

[Jorgen Lundman]

Ok I went through the FAILs I got locally

```
functional/cli_root/zpool_get/zpool_get_002_pos (run as root) [00:00] [FAIL]
```
It seems `zpool_get.cfg` is missing the `bootsize` property, which is outside 
of the scope of this PR, so I will ignore it.

```
14:17:01.36 Assertion failed: nvlist_size(nvl, , 0) == 0, file 
../common/libzfs_util.c, line 791,
 function zcmd_write_nvlist_com 
/opt/zfs-tests/tests/functional/rsend/send-c_incremental[41]: log_must[67]: 
log_pos[178]: eval: line 1: 628767: Abort(coredump)
14:17:01.36 ERROR: eval zfs receive -d -F testpool2 < /backdir-rsend/final 
exited 262
```

```
14:19:04.72 Assertion failed: 0 == nvlist_lookup_string(props, 
zfs_prop_to_name(ZFS_PROP_KEYLOCATION), _keylocation), file 
../common/libzfs_sendrecv.c, line 2670, 
function recv_fix_encryption_heirarchy 
/opt/zfs-tests/tests/functional/rsend/send_encrypted_heirarchy[60]: 
log_must[67]: log_pos[178]: eval: line 1: 636308: Abort(coredump)
```

those two needs inspection. 

Also the runfile had a few stale test names.


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/124#issuecomment-310288180
--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/T1625245905c55186-Ma17eb734fa6c5db55a321c73
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

[Jorgen Lundman]

@lundman pushed 1 commit.

a19f544  zfs-tests corrections


-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/openzfs/openzfs/pull/124/files/1045b9c5762c2f0bad1110fe1303f461a1b96aff..a19f54495262141f5a551d525c9d6179276c066e

--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/T1625245905c55186-M2730f80298ef4209e3e3558e
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

[Jorgen Lundman]

The test is the one immediate after
```
/opt/zfs-tests/tests/functional/cli_root/zpool_clear/setup 
```
ie
```
zpool_clear_001_pos.ksh
```


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/124#issuecomment-309604362
--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/T1625245905c55186-Maadf3d93ffdac7cde6e42275
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

[Jorgen Lundman]

Pushed the latest commits, I did not do the manpage changes, specifically those 
in
https://github.com/zfsonlinux/zfs/pull/5769/commits/705a2bfc5eabb45facf1e0f7a38a3ee5a69a54ea#diff-42027d93bce823d185f48bd265de4ac0
https://github.com/zfsonlinux/zfs/pull/5769/commits/369ba84201eaa730d6a4d41ef76b14e9b1df02f3#diff-42027d93bce823d185f48bd265de4ac0
https://github.com/zfsonlinux/zfs/pull/5769/commits/369ba84201eaa730d6a4d41ef76b14e9b1df02f3#diff-03c0d83ffa90bcf02b291a55c4a49874

I've only just done bare minimum compile checks, as the VM fails to build due 
to something mdb. When I find a solution, I'll push a fix if one is needed.



-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/124#issuecomment-302989850
--
openzfs-developer
Archives: https://openzfs.topicbox-beta.com/groups/developer/
Powered by Topicbox: https://topicbox-beta.com

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

[Jorgen Lundman]

@lundman pushed 2 commits.

24aa9ff  Various fixes and improvements
0f52c02  Fixes and improvements after 4th round of review


-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/openzfs/openzfs/pull/124/files/2a16cf6821ddb026d7b51181a456d9bfee2ffb93..0f52c029de5953d706bec7b3d288b5cc6d6f0072

--
openzfs-developer
Archives: https://openzfs.topicbox-beta.com/groups/developer/
Powered by Topicbox: https://topicbox-beta.com

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

[Jorgen Lundman]

@lundman pushed 1 commit.

ea53ac7  Compile fixes


-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/openzfs/openzfs/pull/124/files/0f52c029de5953d706bec7b3d288b5cc6d6f0072..ea53ac7930549f31a8d6480ab697c404ffbc05f4

--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/T1625245905c55186-M02f18c4eb1ed604b5ed81497
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

[Jorgen Lundman]

@lundman pushed 1 commit.

6c89a04  Delinting


-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/openzfs/openzfs/pull/124/files/ea53ac7930549f31a8d6480ab697c404ffbc05f4..6c89a04a59661b4bedc7e4b892c0a100204695da

--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/T1625245905c55186-Mae9a8b54c666292357e7ba98
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

[Jorgen Lundman]

@lundman pushed 1 commit.

f68ee3c  Attempt to update the manpages


-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/openzfs/openzfs/pull/124/files/79798fac7fde50d17bd90222d13ee0ac3d806be3..f68ee3ce27b2a9758ae41e0addf24b511fcd1cb4

--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/T1625245905c55186-Mb4de6012d972113ee50d2c9a
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

[Jorgen Lundman]

@lundman pushed 1 commit.

dc4e629  Fixes


-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/openzfs/openzfs/pull/124/files/6c89a04a59661b4bedc7e4b892c0a100204695da..dc4e629f2d708e7241963ddaa8b94996863075d9

--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/T1625245905c55186-Me823d3f384045457f952b378
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

[Jorgen Lundman]

It was my guess that all the zdb cores (there are many of them) made the tests 
run slow, and ended up in a timeout.. I will take a look at the zdb assert and 
see if it is something obvious. Cheers for your feedback


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/124#issuecomment-309337835
--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/T1625245905c55186-Mc92c8c8fc688c0d44784fc49
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

[Jorgen Lundman]

@lundman pushed 2 commits.

bbf9bd3  Fixes and improvements after 5th round of review
439cd9c  Fixes after rebase and more review


-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/openzfs/openzfs/pull/124/files/5556d50abd4ba1cbe70cc2599f4af32176d0a39c..439cd9c2550171097fb23c1af7b518f9de7029ed

--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/T1625245905c55186-M74b65d52096d7eddbc9212e5
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

[Jorgen Lundman]

@lundman pushed 2 commits.

27bc76c  ABD released too early in abd_hmac check
785086e  Corrections in zio_checksum for crypto


-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/openzfs/openzfs/pull/124/files/2980e8ef1b01c9a6b4abe49d67e84bb57019b9d3..785086e61d1b438c27e6b1867af187200674ed11

--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/T1625245905c55186-Md3d11126a0c72c587f474daa
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] 8648 Fix range locking in ZIL commit codepath (#462)

[Jorgen Lundman]

It is probably not related, and perhaps not even worth mentioning. But due to 
deadlocks in OsX version from zget and rangelock in `zfs_get_data()` we now 
call both zget and rangelock in `zil_lwb_commit()` before passing them into 
`zilog->zl_get_data()`. But I don't think you are dealing with deadlock 
situations here.


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/462#issuecomment-328691801
--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/T23be73c52829b560-Maaae689e014354e363bbf28c
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

[Jorgen Lundman]

This PR has been updated with latest commits up and including Oct 2nd 
"060e4cf33002c4".

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/124#issuecomment-334077937
--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/Ta17fb0edb29e7895-M9266d6889a7b0e2abf0b2b3b
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

[Jorgen Lundman]

@lundman pushed 1 commit.

0b4ee95  prevent panic when non-raidz checksum error detected


-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/openzfs/openzfs/pull/124/files/57291b074f93cd588106df1a5c0147778c600c6e..0b4ee95274efe6e3103402a8d390e9206cca2669

--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/Ta17fb0edb29e7895-Mc3c3456d80c51e051b09e37f
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

[Jorgen Lundman]

@lundman pushed 1 commit.

57291b0  de-linting


-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/openzfs/openzfs/pull/124/files/8e2c3b855a7eb155744586ec610d642c2c46a5c9..57291b074f93cd588106df1a5c0147778c600c6e

--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/T1625245905c55186-M77ee3f4745b532d33becaa0e
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

[Jorgen Lundman]

Now contains the send/recv fixes from ZOL. Please test.


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/124#issuecomment-324526413
--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/Ta17fb0edb29e7895-M308c690cb9f77387326d751f
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

[Jorgen Lundman]

@lundman pushed 1 commit.

807e3db  Minor code style changes


-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/openzfs/openzfs/pull/124/files/0b4ee95274efe6e3103402a8d390e9206cca2669..807e3dbdbda5090243d8d534c010ee6ccb34187a

--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/Ta17fb0edb29e7895-M9b3463f47a33a0f1de59c2cc
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

[Jorgen Lundman]

@lundman pushed 2 commits.

53406a2  More corrections to testers
bf0d5b0  Disable dump on encrypted zvol


-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/openzfs/openzfs/pull/124/files/14424ba0610e4ae07057a630538a1c1860dcc3a0..bf0d5b058d816c14a0934bb67bde6b9aa1b82d92

--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/Ta17fb0edb29e7895-M00763155b5b92eab3194a23b
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

[Jorgen Lundman]

@lundman pushed 1 commit.

599316e  OpenZFS repo runs delphix.run in tests, so update it to match


-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/openzfs/openzfs/pull/124/files/fba7a78847ee76f1c86db525e50e44a710575e15..599316ee7968e43c125cbd1d40bffc91099dbdea

--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/Ta17fb0edb29e7895-M9d498203c2740a0c13c6ee70
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] Native data and metadata encryption for zfs (#124)

[Jorgen Lundman]

Looks like we managed to pass all tests finally. I plan to squash everything 
into one single commit unless there is some other number of commits people 
would prefer..?


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/124#issuecomment-336278865
--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/Ta17fb0edb29e7895-M467009f7541d443069b2c2e3
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] fixed arc_cksum_is_equal() that doesn't take into account ABD-logic (#498)

[Jorgen Lundman]

Or maybe I am overselling, perhaps the hours of ABD fixes in raidz was 
*because* of the crypto changes.  But it did lead us to doing something like 
https://github.com/openzfsonosx/zfs/commit/5995f89c1eeae7f029b3537564780458c2920f44
  so we could be sure the wrong buffer was not passed into ABD.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/498#issuecomment-346494860
--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/Te84e226aa4103950-M74dccc98fb0c3c984fe4bb5e
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] fixed arc_cksum_is_equal() that doesn't take into account ABD-logic (#498)

[Jorgen Lundman]

I believe we corrected those over in the crypto PR, and I think there were a 
few in vdev_raidz's parity work as well, if you want to fix those too.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/498#issuecomment-346489048
--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/Te84e226aa4103950-Mffa272864092739c8f21c8a2
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#124)

[Jorgen Lundman]

Closed in favour of https://github.com/openzfs/openzfs/pull/489


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/124#issuecomment-341079785
--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/Ta845afb3c0e3a444-M2ecf1f02cb8c328793d27d1e
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#124)

[Jorgen Lundman]

Closed #124.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/124#event-1320801076
--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/Ta845afb3c0e3a444-Me08aafa451d24a8907e1107d
Powered by Topicbox: https://topicbox.com

[developer] [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

[Jorgen Lundman]

Re-opening the ZOL crypto commit for final reviews and testing. Original 
upstream commit is https://github.com/zfsonlinux/zfs/pull/5769 

This will replace, and close, PR https://github.com/openzfs/openzfs/pull/124

This commit contains all additional PRs, fixes and corrections from ZOL, in one 
commit.

The only difference to upstream, is userland `ztest` do not test crypto for 
IllumOS as the crypto framework is not compiled for userland. A decision should 
perhaps be made here to leave it as it is, or create and compile a library to 
link `ztest` with, or compile directly into `ztest`.

The ZOL message follows:

This change incorporates three major pieces:

The first change is a keystore that manages wrapping
and encryption keys for encrypted datasets. These
commands mostly involve manipulating the new
DSL Crypto Key ZAP Objects that live in the MOS. Each
encrypted dataset has its own DSL Crypto Key that is
protected with a user's key. This level of indirection
allows users to change their keys without re-encrypting
their entire datasets. The change implements the new
subcommands "zfs load-key", "zfs unload-key" and
"zfs change-key" which allow the user to manage their
encryption keys and settings. In addition, several new
flags and properties have been added to allow dataset
creation and to make mounting and unmounting more
convenient.

The second piece of this patch provides the ability to
encrypt, decyrpt, and authenticate protected datasets.
Each object set maintains a Merkel tree of Message
Authentication Codes that protect the lower layers,
similarly to how checksums are maintained. This part
impacts the zio layer, which handles the actual
encryption and generation of MACs, as well as the ARC
and DMU, which need to be able to handle encrypted
buffers and protected data.

The last addition is the ability to do raw, encrypted
sends and receives. The idea here is to send raw
encrypted and compressed data and receive it exactly
as is on a backup system. This means that the dataset
on the receiving system is protected using the same
user key that is in use on the sending side. By doing
so, datasets can be efficiently backed up to an
untrusted system without fear of data being
compromised.

Reviewed by: Matthew Ahrens <mahr...@delphix.com>
Reviewed-by: Brian Behlendorf <behlendo...@llnl.gov>
Reviewed-by: Jorgen Lundman <lund...@lundman.net>
Signed-off-by: Tom Caputi <tcap...@datto.com>

Send / Recv Fixes following b52563

This patch fixes several issues discovered after
the encryption patch was merged:

Fixed a bug where encrypted datasets could attempt
to receive embedded data records.

Fixed a bug where dirty records created by the recv
code wasn't properly setting the dr_raw flag.

Fixed a typo where a dmu_tx_commit() was changed to
dmu_tx_abort()

Fixed a few error handling bugs unrelated to the
encryption patch in dmu_recv_stream()

Signed-off-by: Tom Caputi <tcap...@datto.com>

Encryption patch follow-up

* HKDF implementation moved to its own file and tests added to ensure
correctness.

* Ztest can now create and test encrypted datasets. This is currently
disabled until issue ZOL #6526 is resolved, but otherwise functions as
advertised.

* Several small bug fixes discovered after enabling ztest to run on
encrypted datasets.

* Fixed coverity defects added by the encryption patch.

* Updated man pages for encrypted send / receive behavior.

* Fixed a bug where encrypted datasets could receive
  DRR_WRITE_EMBEDDED records.

* Minor code cleanups / consolidation.

* Includes fix in dmu_free_long_object_impl

IllumOS considerations:

Disable crypto tests in ztest
Unless permission is given to compile the crypto framework in userland
the crypto tests in ztest are disabled on IllumOS.
You can view, comment on, or merge this pull request online at:

  https://github.com/openzfs/openzfs/pull/489

-- Commit Summary --

  * 8727 Native data and metadata encryption for zfs

-- File Changes --

M usr/src/cmd/mdb/common/modules/zfs/zfs.c (21)
M usr/src/cmd/mdb/intel/amd64/libzpool/Makefile (3)
M usr/src/cmd/mdb/intel/amd64/zfs/Makefile (1)
M usr/src/cmd/mdb/intel/ia32/libzpool/Makefile (3)
M usr/src/cmd/mdb/intel/ia32/zfs/Makefile (1)
M usr/src/cmd/mdb/sparc/v7/libzpool/Makefile (3)
M usr/src/cmd/mdb/sparc/v9/libzpool/Makefile (3)
M usr/src/cmd/zdb/Makefile.com (3)
M usr/src/cmd/zdb/zdb.c (74)
M usr/src/cmd/zdb/zdb_il.c (11)
M usr/src/cmd/zfs/zfs_main.c (281)
M usr/src/cmd/zinject/translate.c (8)
M usr/src/cmd/zoneadm/Makefile (2)
M usr/src/cmd/zpool/zpool_main.c (64)
M usr/src/cmd/zstreamdump/zstreamdump.c (114)
M usr/src/cmd/ztest/ztest.c (175)
M usr/src/common/zfs/zfeature_common.c (10)

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

[Jorgen Lundman]

https://github.com/zfsonlinux/zfs/issues/6845

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/489#issuecomment-34795
--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/T60f1bdcc3afc4bfc-Mc6c2a4d5c2ee335716e16a71
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

[Jorgen Lundman]

@behlendorf Thanks, now included.


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/489#issuecomment-343345787
--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/T60f1bdcc3afc4bfc-Md2adf5a7fa99063dbd4cc372
Powered by Topicbox: https://topicbox.com

Re: [developer] Re: non-linear abd_alloc_for_io vs zio_checksum_compute

[Jorgen Lundman]

Andriy Gapon wrote:

On 23/11/2017 18:05, Andriy Gapon wrote:


I've done a little experimentation with a version of abd_alloc_for_io() that
returns a chunked abd. zio_checksum_compute() couldn't cope with that in the


Another detail that's not directly related to the above is that even now we do
not always send linear abd-s down to the leaf vdev layer.  For example, raid-z
creates chunked sub-ordinate abd-s.



I had to bring in some of ZOLs linear abd" work with raidz for the 
crypto PR:


https://github.com/openzfsonosx/openzfs/blob/4891718678b9da6329072e6ecc1df646fe52e0fa/usr/src/uts/common/fs/zfs/vdev_raidz.c

So some of those things are addressed in there.


--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/Tdc55c3947a3b78d0-M80fbca30ca8c36e77a4bef73
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#124)

[Jorgen Lundman]

@ahrens This is the first PR, followed by all fixes, two more PRs and fixes. It 
has everything in it that ZoL has. I don't think it is so much about what I 
need, but what you want to happen next.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/124#issuecomment-340920806
--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/Ta845afb3c0e3a444-M75e249d2ecd2301f3bb97e86
Powered by Topicbox: https://topicbox.com

[developer] [openzfs/openzfs] Large alloc in zdb can cause trouble (#635)

[Jorgen Lundman]

16MB alloc in zdb_embedded_block() can cause cores in certain situations
(clang, gcc55).
You can view, comment on, or merge this pull request online at:

  https://github.com/openzfs/openzfs/pull/635

-- Commit Summary --

  * Large alloc in zdb can cause trouble

-- File Changes --

M usr/src/cmd/zdb/zdb.c (8)

-- Patch Links --

https://github.com/openzfs/openzfs/pull/635.patch
https://github.com/openzfs/openzfs/pull/635.diff

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/635

--
openzfs: openzfs-developer
Permalink: 
https://openzfs.topicbox.com/groups/developer/discussions/T878c8282c1bf80d5-M167f5b63cad68bc7298ec2ac
Delivery options: https://openzfs.topicbox.com/groups

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

[Jorgen Lundman]

@lundman pushed 1 commit.

7743d99  DMU objset should not be encrypted


-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/openzfs/openzfs/pull/489/files/099741e7ff87c29c3876debac50acbc99ce0ab26..7743d9912c456bce8cc6085031ef7080c8c848a3

--
openzfs: openzfs-developer
Permalink: 
https://openzfs.topicbox.com/groups/developer/discussions/T3a42c8a97e84a347-Mabb1114875eda855014043d3
Delivery options: https://openzfs.topicbox.com/groups

[developer] Re: [openzfs/openzfs] Large alloc in zdb can cause trouble (#635)

[Jorgen Lundman]

https://github.com/openzfsonosx/zfs/commit/ced236a5da6e72ea7bf6d2919fe14e17cffe10f1
Boom, by 10 days!
:)


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/635#issuecomment-387029887
--
openzfs: openzfs-developer
Permalink: 
https://openzfs.topicbox.com/groups/developer/discussions/T878c8282c1bf80d5-Ma558f45dd6f6c1204da2731e
Delivery options: https://openzfs.topicbox.com/groups

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

[Jorgen Lundman]

@lundman pushed 1 commit.

1fa5148  Add support for decryption faults in zinject


-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/openzfs/openzfs/pull/489/files/31a00d3962866cf5c0f0c9755b4620a978fc312a..1fa5148fe59c8d1a7347fe5d1d5f5f4362a70f4c

--
openzfs: openzfs-developer
Permalink: 
https://openzfs.topicbox.com/groups/developer/discussions/Te9a3d86d2367f775-M7850018c280efc6765e82605
Delivery options: https://openzfs.topicbox.com/groups

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

[Jorgen Lundman]

Not sure about that, but in that place, ZOL has the following
```
ASSERT(db->db.db_data == NULL);
if (db->db_level == 0 && db->db_freed_in_flight) {
/* we were freed in flight; disregard any error */
+if (buf == NULL) {
+buf = arc_alloc_buf(db->db_objset->os_spa,
+db, DBUF_GET_BUFC_TYPE(db), db->db.db_size);
+}
arc_release(buf, db);
```

Not sure I get why they do that, but I suppose we could try it and see if it 
helps.


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/489#issuecomment-387247635
--
openzfs: openzfs-developer
Permalink: 
https://openzfs.topicbox.com/groups/developer/discussions/T3a42c8a97e84a347-M70f1942bd65a966af2a05336
Delivery options: https://openzfs.topicbox.com/groups

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

[Jorgen Lundman]

@lundman pushed 1 commit.

dad661b  minor dbuf_read_done work


-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/openzfs/openzfs/pull/489/files/7743d9912c456bce8cc6085031ef7080c8c848a3..dad661b040d6e0d69089efc0513a450d004d61ca

--
openzfs: openzfs-developer
Permalink: 
https://openzfs.topicbox.com/groups/developer/discussions/T3a42c8a97e84a347-M0df2fcae6b84d7bad4be56a9
Delivery options: https://openzfs.topicbox.com/groups

[developer] [openzfs/openzfs] Add SIGINFO feature to zfs send (#638)

[Jorgen Lundman]

Change the send_progress_thread() to always be started, both for one-second
progress, and to idle for SIGINFO signal when not. Uses pthread_sigmask() to
control which thread receives the signal, or the "main" thread sitting
in ioctl(IOC_SEND) will abort. Signal handling is only enabled if
flag_siginfo is set (always done in zfs cmd) so that other users of
libzfs can avoid it.

Currently both shell and zfs send status is printed.

 load: 0.51  cmd: zfs 1523 waiting 0.04u 0.35s
 16:29:07   26.5M   BOOM@send
 load: 0.82  cmd: zfs 1523 waiting 0.04u 1.17s
 16:29:18206M   BOOM@send
 load: 1.15  cmd: zfs 1523 waiting 0.04u 2.05s
 16:29:30948M   BOOM@send

Signed-off-by: Jorgen Lundman <lund...@lundman.net>
You can view, comment on, or merge this pull request online at:

  https://github.com/openzfs/openzfs/pull/638

-- Commit Summary --

  * Add SIGINFO feature to zfs send

-- File Changes --

M usr/src/cmd/zfs/zfs_main.c (5)
M usr/src/lib/libzfs/common/libzfs.h (3)
M usr/src/lib/libzfs/common/libzfs_sendrecv.c (117)
M usr/src/lib/libzfs_core/common/libzfs_core.c (2)
M usr/src/lib/libzfs_core/common/libzfs_core.h (3)

-- Patch Links --

https://github.com/openzfs/openzfs/pull/638.patch
https://github.com/openzfs/openzfs/pull/638.diff

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/638

--
openzfs: openzfs-developer
Permalink: 
https://openzfs.topicbox.com/groups/developer/discussions/T000e5d283b7db7e5-M91fd84c93014bcf7f392a67e
Delivery options: https://openzfs.topicbox.com/groups

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

[Jorgen Lundman]

Not sure about that one, could be the new commits in master triggering 
something new, is probably beyond my abilities.



-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/489#issuecomment-387619163
--
openzfs: openzfs-developer
Permalink: 
https://openzfs.topicbox.com/groups/developer/discussions/T3a42c8a97e84a347-M2461c53a45cbfceb265bdabc
Delivery options: https://openzfs.topicbox.com/groups

[developer] Re: [openzfs/openzfs] Add SIGINFO feature to zfs send (#638)

[Jorgen Lundman]

lundman commented on this pull request.



> @@ -70,7 +70,8 @@ int lzc_get_holds(const char *, nvlist_t **);
 enum lzc_send_flags {
LZC_SEND_FLAG_EMBED_DATA = 1 << 0,
LZC_SEND_FLAG_LARGE_BLOCK = 1 << 1,
-   LZC_SEND_FLAG_COMPRESS = 1 << 2
+   LZC_SEND_FLAG_COMPRESS = 1 << 2,
+   LZC_SEND_FLAG_SIGINFO = 1 << 3

Thought I needed to handle both versions of send-ioctl, but clearly I don't.


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/638#discussion_r186919864
--
openzfs: openzfs-developer
Permalink: 
https://openzfs.topicbox.com/groups/developer/discussions/T000e5d283b7db7e5-M6b0cafd4614c7f2da70b75ea
Delivery options: https://openzfs.topicbox.com/groups

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

[Jorgen Lundman]

latest rebase was non-trivial due to the parallel mount commit. I had to guess 
a bit. But good news, we are coming up to 2 year anniversary! \o/


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/489#issuecomment-389409212
--
openzfs: openzfs-developer
Permalink: 
https://openzfs.topicbox.com/groups/developer/discussions/Te9a3d86d2367f775-M7032b2a51e6e3378e083f5bb
Delivery options: https://openzfs.topicbox.com/groups

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

[Jorgen Lundman]

Ah, so that is the last outstanding issue? Do you want anything squashed? 
Squashed-to-some, or Squashed-to-one ?


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/489#issuecomment-391905514
--
openzfs: openzfs-developer
Permalink: 
https://openzfs.topicbox.com/groups/developer/discussions/Te9a3d86d2367f775-Mdbab0c9b973079f39c06d18f
Delivery options: https://openzfs.topicbox.com/groups

Re: [developer] [openzfs/openzfs] Trying again with sequential scan -- this is simply (#648)

[Jorgen Lundman]

There isn't much of a guide on how to work with openzfs repo, but this is
what I have guessed so far:

> contact i-0ed0812ab829e137b: java.lang.InterruptedException

This means the VM panicked.  Now click on "Artifacts" up top-right, and go
download "run-zfs-tests-crash.tar.xz"

# tar -xf run-zfs-tests-crash.tar.xz
# savecore -f vmdump.0
savecore: saving system crash dump in /var/crash/unknown/{unix,vmcore}.0
Constructing namelist /var/crash/unknown/unix.0
Constructing corefile /var/crash/unknown/vmcore.0

# mdb unix.0 vmcore.0
> $c
vpanic()
0xfbe0ec1d()
vdev_queue_io_remove+0x13c(ff03ecd7dcc0, ff03e90bb7e8)
vdev_queue_io_to_issue+0x1cb(ff03ecd7dcc0)
vdev_queue_io_done+0x78(ff03e90fd060)
zio_vdev_io_done+0xa8(ff03e90fd060)
zio_execute+0xfa(ff03e90fd060)
taskq_thread+0x318(ff03ecc769e0)
thread_start+8()


-- 
Jorgen Lundman   | 
Unix Administrator   | +81 (0)90-5578-8500
Shibuya-ku, Tokyo| Japan


--
openzfs: openzfs-developer
Permalink: 
https://openzfs.topicbox.com/groups/developer/T6e124cbb2c8562a8-Mf66e6134821d27886d9ab063
Delivery options: https://openzfs.topicbox.com/groups

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

[Jorgen Lundman]

@lundman pushed 1 commit.

02dceca  [PATCH] Make zvol update volsize operation synchronous.


-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/openzfs/openzfs/pull/489/files/ed11924afad60b692dee0cf34689e186533900a5..02dceca32e57399589f59d4f9c0fe13515ba3019

--
openzfs: openzfs-developer
Permalink: 
https://openzfs.topicbox.com/groups/developer/Te9a3d86d2367f775-M58a490ca71eb153e8c164244
Delivery options: https://openzfs.topicbox.com/groups

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

[Jorgen Lundman]

@lundman pushed 1 commit.

d9fa7a3  [PATCH] Raw receive fix and encrypted objset security fix


-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/openzfs/openzfs/pull/489/files/3fbbbf14b40989ba17b65966d77cb27abff100c3..d9fa7a3b584f33433e414fff54b426e879afd091

--
openzfs: openzfs-developer
Permalink: 
https://openzfs.topicbox.com/groups/developer/Te9a3d86d2367f775-Mb64935d1b6f0e819ac8574ec
Delivery options: https://openzfs.topicbox.com/groups

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

[Jorgen Lundman]

@lundman pushed 1 commit.

d95dfde  Remove test not yet ported to illumos


-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/openzfs/openzfs/pull/489/files/473f80e35fb25e0778ab1678fa0f4fa7ed3a48c0..d95dfdee9d978010290cfde087b25f4e0da4f56d

--
openzfs: openzfs-developer
Permalink: 
https://openzfs.topicbox.com/groups/developer/Te9a3d86d2367f775-M170b301ee4d0e2c2d47f2397
Delivery options: https://openzfs.topicbox.com/groups

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

[Jorgen Lundman]

@lundman pushed 2 commits.

5da0a65  Raw receive functions must not decrypt data
473f80e  Update the correct abd in l2arc_read_done()


-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/openzfs/openzfs/pull/489/files/9c0a780d61739bd28e15c5d564df0dbaaff284d6..473f80e35fb25e0778ab1678fa0f4fa7ed3a48c0

--
openzfs: openzfs-developer
Permalink: 
https://openzfs.topicbox.com/groups/developer/Te9a3d86d2367f775-Med99b280071328e36c880448
Delivery options: https://openzfs.topicbox.com/groups

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

[Jorgen Lundman]

Nah it's my fault, my VM is super slow, so its faster to rely on this build 
setup :)



-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/489#issuecomment-385876826
--
openzfs: openzfs-developer
Permalink: 
https://openzfs.topicbox.com/groups/developer/discussions/T91797982fdd5b7d9-Meae52625dbe9d1fb8789df02
Delivery options: https://openzfs.topicbox.com/groups

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

[Jorgen Lundman]

@lundman pushed 1 commit.

77c699a  Correct merge collision


-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/openzfs/openzfs/pull/489/files/cd8b6763fa2413d2f3950c719a6f4f7024b5a768..77c699a32931847d5557f0ebadcec07a680504b1

--
openzfs: openzfs-developer
Permalink: 
https://openzfs.topicbox.com/groups/developer/discussions/T91797982fdd5b7d9-Mbd4bb1bc1845742215be2c08
Delivery options: https://openzfs.topicbox.com/groups

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

[Jorgen Lundman]

@lundman pushed 1 commit.

099741e  Correct ztest build


-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/openzfs/openzfs/pull/489/files/77c699a32931847d5557f0ebadcec07a680504b1..099741e7ff87c29c3876debac50acbc99ce0ab26

--
openzfs: openzfs-developer
Permalink: 
https://openzfs.topicbox.com/groups/developer/discussions/T91797982fdd5b7d9-M2323f3e40b12e233938ee459
Delivery options: https://openzfs.topicbox.com/groups

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

[Jorgen Lundman]

@lundman pushed 1 commit.

a381944  Fix issues found with zfs diff


-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/openzfs/openzfs/pull/489/files/03c0787a23cdd0fc44a47690d69ffcfae64eb72b..a3819442d8617598d8e1463f8b52ea0589fe96cf

--
openzfs: openzfs-developer
Permalink: 
https://openzfs.topicbox.com/groups/developer/discussions/T91797982fdd5b7d9-M2f98ee40aa7224f7d964999f
Delivery options: https://openzfs.topicbox.com/groups

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

[Jorgen Lundman]

rebased and up to date


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/489#issuecomment-385841807
--
openzfs: openzfs-developer
Permalink: 
https://openzfs.topicbox.com/groups/developer/discussions/T91797982fdd5b7d9-Mfb2272088bf627dd167140d3
Delivery options: https://openzfs.topicbox.com/groups

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

[Jorgen Lundman]

Hmm odd it wasnt too big a conflict in `arc.c` but I'll go over it again


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/489#issuecomment-385909026
--
openzfs: openzfs-developer
Permalink: 
https://openzfs.topicbox.com/groups/developer/discussions/T91797982fdd5b7d9-M12c0cf94b69070154156f3c7
Delivery options: https://openzfs.topicbox.com/groups

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

[Jorgen Lundman]

This now includes the ZOL https://github.com/zfsonlinux/zfs/pull/6864 PR as 
well, so we can test `zfs send --raw` fixes.


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/489#issuecomment-356778890
--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/T60f1bdcc3afc4bfc-M4f17c6aee83a22ede9365a92
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

[Jorgen Lundman]

@lundman pushed 1 commit.

3fe3c5c  Update cryptv0 test pool


-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/openzfs/openzfs/pull/489/files/fda748267547c10116f660c7f842716d67ecaa94..3fe3c5c86306386a7acb06e153626c635fa27858

--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/T60f1bdcc3afc4bfc-M83442aed61f2291c62198c54
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

[Jorgen Lundman]

I was unable to replicate: 
```
write: No space left on device
8245278720 bytes transferred in 1880.011589 secs (4385760 bytes/sec)
```
But that could simply be that I don't get dbuf eviction triggered.

However I did find some differences between the two versions, perhaps it will 
help.


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/489#issuecomment-363291672
--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/T91797982fdd5b7d9-Mbe60cb124a3e18eb65474fd5
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

[Jorgen Lundman]

@lundman pushed 1 commit.

d6cda08  Correct differences in arc.c for crypto


-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/openzfs/openzfs/pull/489/files/3fe3c5c86306386a7acb06e153626c635fa27858..d6cda08bf1d56c7f960d626687818f071ab27250

--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/T91797982fdd5b7d9-M44a699071a58023f7cd06e0e
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

[Jorgen Lundman]

I've run out of obvious looking diffs between ZOL and this commit. Might 
require actual debugging now. Seems to die pretty early in `arc_buf_remove` but 
it'd be nice to know where.
ZOL also has a `arc_buf_access()` called from `dbuf` but the ZOL commit 
0873bb63 just talks about counting stats.

The zfstester runs here also do not leave any dumps, so nothing to see there


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/489#issuecomment-363979333
--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/T91797982fdd5b7d9-Mbd6c376910cd897b37331746
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

[Jorgen Lundman]

Well, that's good.. Not sure what to do next then, perhaps @GernotS  can 
provide a dump file, and someone who actually knows how to use mdb (I sure 
don't) can be coaxed into glancing at it?

 

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/489#issuecomment-364285459
--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/T91797982fdd5b7d9-Made82618ba0f3baa4f02dfc9
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

[Jorgen Lundman]

(No changes, just fixed conflicts with master)


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/489#issuecomment-364290139
--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/T91797982fdd5b7d9-Madd44bdb34b196d552cac4f3
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

[Jorgen Lundman]

Neat, I didn't mention the commit today, since the ZOL tickets just talked 
about send|recv :)

Still unable to reproduce dnode_destroy panic, but the tester here seems to die 
though


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/489#issuecomment-363728601
--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/T91797982fdd5b7d9-Mcbcbbd7458dc173b0de6e7d9
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

[Jorgen Lundman]

@lundman pushed 1 commit.

7b94083  Correct l2arc_write_buffers()


-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/openzfs/openzfs/pull/489/files/3803144fa6df4913e94a4b35884c90fb08a6eda3..7b9408363a8a3bd53426e6b57773f43afac204ec

--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/T91797982fdd5b7d9-M06232927266a9916ca315c63
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

[Jorgen Lundman]

```
 it should be like ZOL's l2arc_write_buffers, make it so!
 Done!
```
More fictionally chat logs to come, hopefully this will address @GernotS issue.

Cheers!


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/489#issuecomment-367539223
--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/T91797982fdd5b7d9-M2b60fad3b442107423fa1eda
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

[Jorgen Lundman]

On OsX I added a `cache device` to the crypt test, and I got:

```
panic(cpu 0 caller 0xff8027ec8132): Kernel trap at 0xff7fab62a0cc, type 
14=page fault, registers:
0xff8087a43d60 : 0xff8027cfc200 mach_kernel : trap_from_kernel + 0x26
0xff8087a43d80 : 0xff7fab62a0cc net.lundman.zfs : _vdev_disk_io_start + 
0x12c
0xff8087a43e90 : 0xff7fab6944af net.lundman.zfs : _zio_vdev_io_start + 0

-> 849  if (zio->io_type == ZIO_TYPE_READ) {
   850  ASSERT3S(zio->io_abd->abd_size,>=,zio->io_size);

(zio_type_t) $3 = ZIO_TYPE_WRITE

  io_abd = 0x
  io_orig_abd = 0x
  io_size = 65536
```

Wonder if we can get @tcaputi to give it a go on ZOL ?


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/489#issuecomment-367865837
--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/T91797982fdd5b7d9-Me29b69d575d4d3547e4c8a1a
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

[Jorgen Lundman]

@lundman pushed 1 commit.

5ebecdd  Re-port l2arc_write_buffers just in case


-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/openzfs/openzfs/pull/489/files/a3b2b17888259e1af5f0e99ee3eb8333ec43636f..5ebecddbb034b7ff71a75d01be1cd7b4501d9b50

--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/T91797982fdd5b7d9-Md725ae1c999ab6aa3dd055ed
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

[Jorgen Lundman]

It is interesting that `l2arc_write_buffers` is one of the places the merge was 
not smooth.  Re-porting the function from clean did not change much alas, 
adding flag `ARC_FLAG_L2_WRITING` .

The IllumOS version has an extra chunk of code that the other platforms have:
```
 * To ensure that the copy will be available for the
 * lifetime of the ZIO and be cleaned up afterwards, we
 * add it to the l2arc_free_on_write queue.
 */
if (!HDR_SHARED_DATA(hdr) && psize == asize) {
to_write = hdr->b_l1hdr.b_pabd;
} else {
to_write = abd_alloc_for_io(asize,
HDR_ISTYPE_METADATA(hdr));
abd_copy(to_write, hdr->b_l1hdr.b_pabd, psize);
if (asize != psize) {
abd_zero_off(to_write, psize,
asize - psize);
}
l2arc_free_abd_on_write(to_write, asize,
arc_buf_type(hdr));
}
wzio = zio_write_phys(pio, dev->l2ad_vdev,
```

Which perhaps needs extra thinking around it for crypto.  


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/489#issuecomment-366105904
--
openzfs-developer
Archives: 
https://openzfs.topicbox.com/groups/developer/discussions/T91797982fdd5b7d9-M5d0bda37f92ad6449cb8b7d4
Powered by Topicbox: https://topicbox.com

[developer] Re: [openzfs/openzfs] 9688 aggsum_fini leaks memory (#679)

[Jorgen Lundman]

When running this commit on OsX, which is a little more strict with mutex 
release; we get this situation:

```
frame #4:  kernel.development`panic(mutex_enter on destroyed mutex) at 
debug.c:605 [opt]
frame #5:  spl`spl_mutex_enter(mp=0xff886f557b00, file="aggsum.c", 
line=190) at spl-mutex.c:294 [opt]
frame #6:  zfs`aggsum_add(as=0xff7f8e832370, delta=-360) at 
aggsum.c:190 [opt]
frame #7:  zfs`arc_space_return(space=360, type=ARC_SPACE_HDRS) at arc.c:0 
[opt]
frame #8:  spl`kmem_magazine_destroy(cp=0xff886f546c70, 
mp=, nrounds=) at spl-kmem.c:1655 [opt]
frame #9:  spl`kmem_cache_magazine_purge(cp=0xff886f546c70) at 
spl-kmem.c:2870 [opt]
frame #10:  spl`kmem_cache_destroy(cp=0xff886f546c70) at 
spl-kmem.c:3831 [opt]
frame #11:  zfs`arc_fini [inlined] buf_fini at arc.c:1301 [opt]
  * frame #12:  zfs`arc_fini at arc.c:7822 [opt]
```

Which at a glance I _think_ is due to that the release order is:
```
frame #12: 0xff7f8e5ea342 zfs`arc_fini at arc.c:7822 [opt]
   7819 cv_destroy(_reclaim_waiters_cv);
   7820
   7821 arc_state_fini(); # aggsums released in here
-> 7822 buf_fini();
```

but `buf_fini()` can trigger release of magazines, which has callbacks to 
`arc_space_return()` which attempts to use the aggsums released in prior call 
to `arc_stat_fini()`.

Moving the aggsums release to
```
arc_state_fini();
buf_fini();

aggsum_fini(_meta_used);
aggsum_fini(_size);
aggsum_fini(_data_size);
aggsum_fini(_metadata_size);
aggsum_fini(_hdr_size);
aggsum_fini(_other_size);
aggsum_fini(_l2_hdr_size);
```

lets me unload that one time I tried it at least :)

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/679#issuecomment-414934847
--
openzfs: openzfs-developer
Permalink: 
https://openzfs.topicbox.com/groups/developer/Tc438439e218e9400-Mf819d7e4020516894b3b00fe
Delivery options: https://openzfs.topicbox.com/groups/developer/subscription

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

[Jorgen Lundman]

In the interest in making `dsl_crypto.c` and `zio_crypto.c` to be the same as 
ZOL - I added a few commits that messed around in there. [1] The 3a5bf20 commit 
will replace the gist I pasted in 
https://github.com/openzfs/openzfs/pull/489#issuecomment-404715592 

[1] minus the QAT changes.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/489#issuecomment-404990344
--
openzfs: openzfs-developer
Permalink: 
https://openzfs.topicbox.com/groups/developer/T02d6a9917e14b170-M625ae3339547adaf794cdf34
Delivery options: https://openzfs.topicbox.com/groups

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

[Jorgen Lundman]

Woo, I contributed! \o/


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/489#issuecomment-407324973
--
openzfs: openzfs-developer
Permalink: 
https://openzfs.topicbox.com/groups/developer/Te9a3d86d2367f775-Mdf13355aa69796a1a328160c
Delivery options: https://openzfs.topicbox.com/groups/developer/subscription

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

[Jorgen Lundman]

@lundman pushed 1 commit.

3972768  Do not call  dmu_objset_disown twice


-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/openzfs/openzfs/pull/489/files/6f85552ff7a285fef0af14f31d322cc82bebd07c..3972768dc18f2f6b4b2c3c90ac9ce516f8e1cf99

--
openzfs: openzfs-developer
Permalink: 
https://openzfs.topicbox.com/groups/developer/Te9a3d86d2367f775-M0a87816ff166582825149505
Delivery options: https://openzfs.topicbox.com/groups/developer/subscription

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

[Jorgen Lundman]

@lundman pushed 1 commit.

d123482  Correct typo in zfs.1m


-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/openzfs/openzfs/pull/489/files/ab2d8ac94cf227207f80ca9b221ef7bd3bcd6e99..d123482538a05e9ed8dcea8f2872ee72a52b0a3f

--
openzfs: openzfs-developer
Permalink: 
https://openzfs.topicbox.com/groups/developer/T02d6a9917e14b170-M641da47a73de997cc132671a
Delivery options: https://openzfs.topicbox.com/groups

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

[Jorgen Lundman]

@lundman pushed 3 commits.

23eb923  Raw receive fix and encrypted objset security fix
58dd6e4  Add ASSERT to debug encryption key mapping issues
ab2d8ac  Fix coverity defects: CID 176037


-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/openzfs/openzfs/pull/489/files/d9fa7a3b584f33433e414fff54b426e879afd091..ab2d8ac94cf227207f80ca9b221ef7bd3bcd6e99

--
openzfs: openzfs-developer
Permalink: 
https://openzfs.topicbox.com/groups/developer/T02d6a9917e14b170-M9d6712a0b379e9e6c55dc5bd
Delivery options: https://openzfs.topicbox.com/groups

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

[Jorgen Lundman]

Dropped and re-added 7632, added 7667 and 7637.


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/489#issuecomment-402581297
--
openzfs: openzfs-developer
Permalink: 
https://openzfs.topicbox.com/groups/developer/T02d6a9917e14b170-M54d5b8133ce9fedbcbf9509b
Delivery options: https://openzfs.topicbox.com/groups

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

[Jorgen Lundman]

@lundman pushed 1 commit.

58e8434  Odd git merge decisions


-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/openzfs/openzfs/pull/489/files/d123482538a05e9ed8dcea8f2872ee72a52b0a3f..58e8434463cc1ffd99b4fedd8995e11ea3c4af7c

--
openzfs: openzfs-developer
Permalink: 
https://openzfs.topicbox.com/groups/developer/T02d6a9917e14b170-M537d909e225f59afcdd6711b
Delivery options: https://openzfs.topicbox.com/groups

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

[Jorgen Lundman]

Hmm alas, commit 
https://github.com/zfsonlinux/zfs/pull/7650/commits/5248401f8bf3d8149ba6e559fa94a075bfdff9d4
 depends on 
https://github.com/zfsonlinux/zfs/commit/a3eeab2de68670a4481eab3d086982aff23b6906
 and we have a fair bit of divergence in the property parsing between the two. 
I'm just here to confirm that we want `-o / -x` included in this PR ?


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/489#issuecomment-403349020
--
openzfs: openzfs-developer
Permalink: 
https://openzfs.topicbox.com/groups/developer/T02d6a9917e14b170-M351ae6ca1a92ddfb347aee2b
Delivery options: https://openzfs.topicbox.com/groups

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

[Jorgen Lundman]

@GernotS Working on a theory to a couple of bugs over in OsX port, which might 
just help here as well. If you have the time/energy, could you see if this 
patch fixes things with "cache + crypto" and "raw receive" ?

https://gist.github.com/lundman/5c269ae5582bac1adf2a58e7b9513c1b



-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/489#issuecomment-404715592
--
openzfs: openzfs-developer
Permalink: 
https://openzfs.topicbox.com/groups/developer/T02d6a9917e14b170-Mc2df76b7b1eb40522db69276
Delivery options: https://openzfs.topicbox.com/groups

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

[Jorgen Lundman]

@lundman pushed 1 commit.

1229496  Prevent raw zfs recv -F if dataset is unencrypted


-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/openzfs/openzfs/pull/489/files/46f0e13723be73c7000390d30f939148ce345a54..1229496a970e3c29a6340944fc862c849474d41f

--
openzfs: openzfs-developer
Permalink: 
https://openzfs.topicbox.com/groups/developer/discussions/T91797982fdd5b7d9-M52866e49bc58fd9adb998d00
Delivery options: https://openzfs.topicbox.com/groups

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

[Jorgen Lundman]

Hmm that's defined in `zio.h` but the `cmd`'s tend not to include the sys/zfs* 
headers directly? Should they be in `zfs_context`?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/489#issuecomment-380063331
--
openzfs: openzfs-developer
Permalink: 
https://openzfs.topicbox.com/groups/developer/discussions/T91797982fdd5b7d9-M6f046015fda8b8c4f414d8af
Delivery options: https://openzfs.topicbox.com/groups

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

[Jorgen Lundman]

@lundman pushed 1 commit.

097d997  Make encrypted "zfs mount -a" failures consistent


-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/openzfs/openzfs/pull/489/files/da391edb5d7057961eda8585b6d6af08cda5d64d..097d997308675d4d2387d5bb8a786fb09a189dfc

--
openzfs: openzfs-developer
Permalink: 
https://openzfs.topicbox.com/groups/developer/discussions/T91797982fdd5b7d9-M5b00b92b2717c4e4399b1d65
Delivery options: https://openzfs.topicbox.com/groups

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

[Jorgen Lundman]

@lundman pushed 1 commit.

893c519  receive_spill does not byte swap spill contents


-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/openzfs/openzfs/pull/489/files/2b622bf65fed5df4a70f8b63f9e9b42c4792bfea..893c519c9dcb7eae00acb466c9d19c2c37b26ed3

--
openzfs: openzfs-developer
Permalink: 
https://openzfs.topicbox.com/groups/developer/discussions/T91797982fdd5b7d9-M9d5fe51b60fd8fb843fb3eaf
Delivery options: https://openzfs.topicbox.com/groups

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

[Jorgen Lundman]

@lundman pushed 1 commit.

ef73ffd  Move enum zio_encrypt into sys/fs/zfs.h


-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/openzfs/openzfs/pull/489/files/bf72974e371a6b1d097024f7cb567e9ea78fa65b..ef73ffd309419acb8389095d2887b2316cb1ba1d

--
openzfs: openzfs-developer
Permalink: 
https://openzfs.topicbox.com/groups/developer/discussions/T91797982fdd5b7d9-Mc91e3730cb4f7d3fcc8bbea0
Delivery options: https://openzfs.topicbox.com/groups

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

[Jorgen Lundman]

Not sure I'm seeing it. The difference to ZOL is that they have `zfs_main.c` add
```
#include "libzfs_impl.h"
```

But I can not add that to IllumOS, as none of the -I paths include 
`libzfs_impl.h` nor `sys/zio.h` - are you suggesting I change Makefiles to add 
the include path? As for existing precedent in `cmd/zfs` regards to property 
comparison is `canmount == ZFS_CANMOUNT_OFF` which works as it is defined in 
`sys/fs/zfs.h`  Please indicate your preference :)


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/489#issuecomment-380290582
--
openzfs: openzfs-developer
Permalink: 
https://openzfs.topicbox.com/groups/developer/discussions/T91797982fdd5b7d9-Mb89861725f93f8d800357b9d
Delivery options: https://openzfs.topicbox.com/groups

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

[Jorgen Lundman]

@lundman pushed 1 commit.

bf72974  Correct manpage for --raw


-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/openzfs/openzfs/pull/489/files/097d997308675d4d2387d5bb8a786fb09a189dfc..bf72974e371a6b1d097024f7cb567e9ea78fa65b

--
openzfs: openzfs-developer
Permalink: 
https://openzfs.topicbox.com/groups/developer/discussions/T91797982fdd5b7d9-Mfe2dd9b5e8947750676a2670
Delivery options: https://openzfs.topicbox.com/groups

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

[Jorgen Lundman]

Hmm I don't even have `vdev_indirect.c:1356` ?


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/489#issuecomment-382291271
--
openzfs: openzfs-developer
Permalink: 
https://openzfs.topicbox.com/groups/developer/discussions/T91797982fdd5b7d9-M550c100ce3fe00cb46abedb3
Delivery options: https://openzfs.topicbox.com/groups

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

[Jorgen Lundman]

Oppsie, my bad, one sec


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/489#issuecomment-382289822
--
openzfs: openzfs-developer
Permalink: 
https://openzfs.topicbox.com/groups/developer/discussions/T91797982fdd5b7d9-Mda1b5e2d6061b7e17813e5a3
Delivery options: https://openzfs.topicbox.com/groups

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

[Jorgen Lundman]

Are you hinting that you want it rebased against master maybe?


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/489#issuecomment-382294674
--
openzfs: openzfs-developer
Permalink: 
https://openzfs.topicbox.com/groups/developer/discussions/T91797982fdd5b7d9-Mbebabd10b274462570d0805f
Delivery options: https://openzfs.topicbox.com/groups

[developer] Re: [openzfs/openzfs] 8727 Native data and metadata encryption for zfs (#489)

[Jorgen Lundman]

@lundman pushed 2 commits.

bc5a90b  Correct swapped keylocation error messages
5d6cd7d  Fix race in dnode_check_slots_free()


-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/openzfs/openzfs/pull/489/files/f46f0922baeffdc1b7b8750a4fe9ed4ef202ade6..5d6cd7d203a1eb182729d2390aedbe8d09ecf7a8

--
openzfs: openzfs-developer
Permalink: 
https://openzfs.topicbox.com/groups/developer/discussions/T91797982fdd5b7d9-Ma0374a66b3b83cc73d8c9511
Delivery options: https://openzfs.topicbox.com/groups