On Jun 27, 2018, at 06:05, Mark Roper <[email protected]> wrote: > > Hi Jeremy & All, > I got a request to share the results of my SSK performance investigation with > this group from Mark Hahn, which I'm happy to do! If you're not interested > in the impact on throughput for encryption of client-to-mds and client-to-oss > communication using the SSK feature, you can stop reading now. > > The tldr is that enabling encryption reduced read throughput 81% and write > throughput 53%. To me this was large but unsurprising given that the server > and client nodes are performing software encryption, but I wanted to know the > extent of the impact.
If crypto performance is critical for you, you might consider to look at the Intel QAT PCI adapter. While I don't work at Intel anymore, I learned about this adapter and it can definitely improve performance for crypto and compression workloads, so long as you are limited by the crypto performance. It does not quite run fast enough to go full wire speed for IB/OPA networks, but at the speeds you are reporting it should be a benefit. For some benchmark results see: https://www.servethehome.com/intel-quickassist-technology-and-openssl-setup-insights-and-initial-benchmarks/ Cheers, Andreas > Details are below! > > Mark > > I set up two Lustre file systems on the same virtual machine configuration in > AWS, one OSS VM and one MDS VM in each. I enabled encryption of client and > server communication as follows: > > sudo lctl conf_param scratch.srpc.flavor.default.cli2mdt=skpi > sudo lctl conf_param scratch.srpc.flavor.default.cli2ost=skpi > > I then ran a single IOR benchmark test aimed at evaluating sustem throughput. > I ran the benchmark on a cluster of 5 clients with the following command: > srun --tasks-per-node=1 -N 5 ior -a POSIX -o /scratch/demo -z -w -r -F -B -b > 1g -t 1m -i 2 > > The IOR results for the encrypted FS were: > Summary: > api = POSIX > test filename = /lustre/demo > access = file-per-process > ordering in a file = random offsets > ordering inter file= no tasks offsets > clients = 5 (1 per node) > repetitions = 2 > xfersize = 1 MiB > blocksize = 1 GiB > aggregate filesize = 5 GiB > > access bw(MiB/s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) > total(s) iter > ------ --------- ---------- --------- -------- -------- -------- > -------- ---- > write 89.25 1048576 1024.00 0.003589 57.36 1.22 > 57.37 0 > read 95.14 1048576 1024.00 0.001952 53.81 0.860698 > 53.81 0 > remove - - - - - - > 0.002120 0 > write 88.16 1048576 1024.00 0.001738 58.08 1.23 > 58.08 1 > read 95.22 1048576 1024.00 0.001806 53.77 0.989825 > 53.77 1 > remove - - - - - - > 0.001562 1 > > Max Write: 89.25 MiB/sec (93.59 MB/sec) > Max Read: 95.22 MiB/sec (99.84 MB/sec) > > Summary of all tests: > Operation Max(MiB) Min(MiB) Mean(MiB) StdDev Mean(s) Test# #Tasks > tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize API > RefNum > write 89.25 88.16 88.70 0.55 57.72177 0 5 1 2 1 0 > 1 0 0 1 1073741824 1048576 5368709120 POSIX 0 > read 95.22 95.14 95.18 0.04 53.79227 0 5 1 2 1 0 > 1 0 0 1 1073741824 1048576 5368709120 POSIX 0 > > The IOR results of the unencrypted filesystem were: > > Summary: > api = POSIX > test filename = /scratch/demo > access = file-per-process > ordering in a file = random offsets > ordering inter file= no tasks offsets > clients = 5 (1 per node) > repetitions = 2 > xfersize = 1 MiB > blocksize = 1 GiB > aggregate filesize = 5 GiB > > access bw(MiB/s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) > total(s) iter > ------ --------- ---------- --------- -------- -------- -------- > -------- ---- > write 189.02 1048576 1024.00 0.002521 27.09 0.551086 > 27.09 0 > read 508.37 1048576 1024.00 0.001257 10.07 0.326688 > 10.07 0 > remove - - - - - - > 0.002035 0 > write 187.13 1048576 1024.00 0.001748 27.36 0.928853 > 27.36 1 > read 502.72 1048576 1024.00 0.001494 10.18 0.356007 > 10.18 1 > remove - - - - - - > 0.001705 1 > > Max Write: 189.02 MiB/sec (198.20 MB/sec) > Max Read: 508.37 MiB/sec (533.07 MB/sec) > > Summary of all tests: > Operation Max(MiB) Min(MiB) Mean(MiB) StdDev Mean(s) Test# #Tasks > tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize API > RefNum > write 189.02 187.13 188.07 0.95 27.22418 0 5 1 2 1 0 > 1 0 0 1 1073741824 1048576 5368709120 POSIX 0 > read 508.37 502.72 505.54 2.83 10.12801 0 5 1 2 1 0 > 1 0 0 1 1073741824 1048576 5368709120 POSIX 0 > > > On Mon, Jun 25, 2018 at 4:59 PM Mark Roper <[email protected]> wrote: > Thanks again Jermey. This is pretty strange but here goes: SSK encryption > works end to end if I ssh as root into the server and client nodes to mount. > If I ssh as another user (say, centos) and `sudo` or `sudo -s` the same > commands with --skpath, the client mount fails. > > So it seems like there is something going on with how user and session keys > are loaded into the linux keyring and later made available, but I haven't > gone further in my investigation than this. > > I was able to get what I needed for performance numbers which was my goal in > setting up ssk encryption: thanks again! > > Mark > > > On Sun, Jun 24, 2018 at 6:46 PM Jeremy Filizetti <[email protected]> > wrote: > I have encountered this issue before as well. Something on the system is > creating a new root user session keyring and keyctl_read fails after that > happens. For now reloading the key into the keyring is what I have done. > For the client you could mount with --skpath option so any time it's mounted > it reloads the key but there is still the issue when the session context > expires and the keys are re-established keyctl_read will fail again if a new > keyring is created. I'm not sure when I'll have time to put together a fix > for this but let me know if mounting with skpath option works. > > Jeremy > > On Sun, Jun 24, 2018 at 4:41 PM, Mark Roper <[email protected]> wrote: > Hi Jeremy, > > Thanks for taking a look at my question. I have validated that the key on the > server and the client match and that the client key has the prime generated. > > When I ssh to the client node and run > sudo mount -t lustre -o skpath=/secure_directory/scratch.client.key > 172.31.46.245@tcp:/scratch /scratch > I get the following output in /var/log/messages with verbosity turned up to > trace on the MDS node I see: > Jun 24 20:26:41 ip-172-31-44-121 lsvcgssd[23975]: keyctl_read() failed for > key 27091278: Permission denied > > Jun 24 20:26:41 ip-172-31-44-121 lsvcgssd[23975]: Failed to create sk > credentials > > As I mentioned, If I remove the option I'm able to mount the FS. I'm using > Lustre 2.11 server and clients. The server kernel is > 3.10.0-693.21.1.el7_lustre.x86_64 and the client kernel is > 3.10.0-693.21.1.el7.x86_64. > > I am wondering if this has something to do with linux keyring permissions on > CentOS. When I ssh to my server and client nodes as the user `centos` and > run `sudo lgss_sk -l /secure_directory/scratch.<server | client>.key` > followed by `keyctl show`, the lustre user key does not appear in the list of > keys. If I ssh to the client & server nodes as root and run the same two > commands, the lustre key shows up on the server as: > 772711346 --alswrv 0 0 keyring: _ses > > 1047091535 --alswrv 0 65534 \_ keyring: _uid.0 > > 27091278 --alswrv 0 0 \_ user: lustre:scratch:default > > ... and on the client as: > > Session Keyring > > 269152212 --alswrv 0 0 keyring: _ses > > 1059491764 --alswrv 0 65534 \_ keyring: _uid.0 > > > 146272009 --alswrv 0 0 \_ user: lustre:scratch > > I'm going to try setting up a 2.10.3 server and client to see if this is some > kind of regression in 2.11 and not just me fat fingering something. I'm also > going to dive deeper into keyring permissions and see if I can find anything > there. I'll update this thread for those interested if I figure it out. > > Any additional thoughts would be appreciated! > > Cheers, > > Mark > > > On Sun, Jun 24, 2018 at 4:02 PM Jeremy Filizetti <[email protected]> > wrote: > GSS error 0x60000 is GSS bad signature which would mean the HMAC was invalid. > Can you verify your key file's have the same shared key? Do you have any > logs for the server side as well? You can increase server verbosity by > adding some extra v's to LSVCGSSDARGS in /etc/sysconfig/lsvcgss. > > Jeremy > > On Fri, Jun 22, 2018 at 3:41 PM, Mark Roper <[email protected]> wrote: > Hi Lustre Admins, > > I am hoping someone can help me understand what I'm doing wrong with SSK > setup. I have set up a lustre 2.11 server and worked through the steps to use > shared secret keys (SSKs) to encrypt data in transit between client nodes and > the MDT and OSS. I followed the manual instructions here: > http://doc.lustre.org/lustre_manual.xhtml#idm140687075065344 > > Before enabling the encryption settings on the MDT, I can mount the FS on the > client node. After I turn on the encryption I get back an encryption refused > error and cannot mount: > > mount.lustre: mount 172.31.46.245@tcp:/scratch at /scratch failed: Connection > refused > The keys are definitely distributed to client nodes and server nodes and the > settings have all been made as instruct4red in the manual (I did this a few > times from scratch to make sure). I can manually load the keys into the > keyring and see them by running `keyctl show`, I can compare the key files on > client and server nodes with the command `lgss_sk --read > /secure_directory/scratch.client.key` and validate that they all match and > that the client has a prime. > > The commands I'm using to enable the encryption are: > > mdt# sudo lctl conf_param scratch.srpc.flavor.tcp.cli2mdt=skpi > mdt# sudo lctl conf_param scratch.srpc.flavor.tcp.cli2ost=skpi > > I tried tailing /var/log/messages and am not able to interpret the output, > I'm wondering - does anyone have a hypothesis about what might be wrong or > instructions to debug? > > Log output is below! Many thanks to anyone who can help! > > Mark > > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22250]:TRACE:main(): start > parsing parameters > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22250]:INFO:main(): key > 428863463, desc 0@26, ugid 0:0, sring 46159405, coinfo > 38:sk:0:0:m:p:2:0x20000ac1f2109:scratch-OST1cd0-osc-MDT0000:0x20000ac1f2ef5:1 > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: > [22250]:TRACE:parse_callout_info(): components: > 38,sk,0,0,m,p,2,0x20000ac1f2109,scratch-OST1cd0-osc-MDT0000,0x20000ac1f2ef5,1 > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: > [22250]:DEBUG:parse_callout_info(): parse call out info: secid 38, mech sk, > ugid 0:0, is_root 0, is_mdt 1, is_ost 0, svc type p, svc 2, nid > 0x20000ac1f2109, tgt scratch-OST1cd0-osc-MDT0000, self nid 0x20000ac1f2ef5, > pid 1 > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22250]:TRACE:main(): parsing > parameters OK > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: > [22250]:TRACE:lgss_mech_initialize(): initialize mech sk > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: > [22250]:TRACE:lgss_create_cred(): create a sk cred at 0x1ecc2e0 > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22250]:TRACE:main(): caller's > namespace is the same > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: > [22250]:TRACE:lgss_prepare_cred(): preparing sk cred 0x1ecc2e0 > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22250]:INFO:sk_create_cred(): > Creating credentials for target: scratch-OST1cd0-osc-MDT0000 with nodemap: > (null) > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22250]:INFO:sk_create_cred(): > Searching for key with description: lustre:scratch > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: > [22250]:TRACE:prepare_and_instantiate(): instantiated kernel key 198fefe7 > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22250]:TRACE:main(): forked > child 22251 > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: > [22251]:TRACE:lgssc_kr_negotiate(): child start on behalf of key 198fefe7: > cred 0x1ecc2e0, uid 0, svc 2, nid 20000ac1f2109, uids: 0:0/0:0 > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: > [22251]:INFO:ipv4_nid2hostname(): SOCKLND: net 0x20000, addr 0x9211fac => > ip-172-31-33-9.us-west-2.compute.internal > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: > [22251]:DEBUG:lgss_get_service_str(): constructed service string: > [email protected] > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: > [22251]:TRACE:lgss_using_cred(): using sk cred 0x1ecc2e0 > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22253]:TRACE:main(): start > parsing parameters > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22253]:INFO:main(): key > 189483693, desc 0@25, ugid 0:0, sring 46159405, coinfo > 37:sk:0:0:m:p:2:0x20000ac1f2687:scratch-OST2b9d-osc-MDT0000:0x20000ac1f2ef5:1 > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: > [22253]:TRACE:parse_callout_info(): components: > 37,sk,0,0,m,p,2,0x20000ac1f2687,scratch-OST2b9d-osc-MDT0000,0x20000ac1f2ef5,1 > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: > [22253]:DEBUG:parse_callout_info(): parse call out info: secid 37, mech sk, > ugid 0:0, is_root 0, is_mdt 1, is_ost 0, svc type p, svc 2, nid > 0x20000ac1f2687, tgt scratch-OST2b9d-osc-MDT0000, self nid 0x20000ac1f2ef5, > pid 1 > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22253]:TRACE:main(): parsing > parameters OK > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: > [22253]:TRACE:lgss_mech_initialize(): initialize mech sk > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: > [22253]:TRACE:lgss_create_cred(): create a sk cred at 0x21b02e0 > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22253]:TRACE:main(): caller's > namespace is the same > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: > [22253]:TRACE:lgss_prepare_cred(): preparing sk cred 0x21b02e0 > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22253]:INFO:sk_create_cred(): > Creating credentials for target: scratch-OST2b9d-osc-MDT0000 with nodemap: > (null) > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22253]:INFO:sk_create_cred(): > Searching for key with description: lustre:scratch > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: > [22253]:TRACE:prepare_and_instantiate(): instantiated kernel key 0b4b4aad > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22253]:TRACE:main(): forked > child 22254 > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: > [22254]:TRACE:lgssc_kr_negotiate(): child start on behalf of key 0b4b4aad: > cred 0x21b02e0, uid 0, svc 2, nid 20000ac1f2687, uids: 0:0/0:0 > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: > [22254]:INFO:ipv4_nid2hostname(): SOCKLND: net 0x20000, addr 0x87261fac => > ip-172-31-38-135.us-west-2.compute.internal > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: > [22254]:DEBUG:lgss_get_service_str(): constructed service string: > [email protected] > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: > [22254]:TRACE:lgss_using_cred(): using sk cred 0x21b02e0 > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: > [22251]:INFO:sk_encode_netstring(): Encoded netstring of 647 bytes > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: > [22251]:INFO:lgss_sk_using_cred(): Created netstring of 647 bytes > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: > [22251]:TRACE:lgssc_negotiation_manual(): starting gss negotation > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22251]:TRACE:do_nego_rpc(): > start negotiation rpc > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22251]:TRACE:gss_do_ioctl(): > to open /proc/fs/lustre/sptlrpc/gss/init_channel > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22251]:TRACE:gss_do_ioctl(): > to down-write > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: > [22254]:INFO:sk_encode_netstring(): Encoded netstring of 647 bytes > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: > [22254]:INFO:lgss_sk_using_cred(): Created netstring of 647 bytes > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: > [22254]:TRACE:lgssc_negotiation_manual(): starting gss negotation > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22254]:TRACE:do_nego_rpc(): > start negotiation rpc > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22254]:TRACE:gss_do_ioctl(): > to open /proc/fs/lustre/sptlrpc/gss/init_channel > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22254]:TRACE:gss_do_ioctl(): > to down-write > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22251]:TRACE:do_nego_rpc(): > do_nego_rpc: to parse reply > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22251]:DEBUG:do_nego_rpc(): > do_nego_rpc: receive handle len 0, token len 0, res 0 > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: > [22251]:ERROR:lgssc_negotiation_manual(): negotiation gss error 60000 > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: > [22251]:ERROR:lgssc_kr_negotiate_manual(): key 198fefe7: failed to negotiate > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: > [22251]:TRACE:error_kernel_key(): revoking kernel key 198fefe7 > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: > [22251]:INFO:error_kernel_key(): key 198fefe7: revoked > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: > [22251]:TRACE:lgss_release_cred(): releasing sk cred 0x1ecc2e0 > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22254]:TRACE:do_nego_rpc(): > do_nego_rpc: to parse reply > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22254]:DEBUG:do_nego_rpc(): > do_nego_rpc: receive handle len 0, token len 0, res 0 > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: > [22254]:ERROR:lgssc_negotiation_manual(): negotiation gss error 60000 > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: > [22254]:ERROR:lgssc_kr_negotiate_manual(): key 0b4b4aad: failed to negotiate > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: > [22254]:TRACE:error_kernel_key(): revoking kernel key 0b4b4aad > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: > [22254]:INFO:error_kernel_key(): key 0b4b4aad: revoked > Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: > [22254]:TRACE:lgss_release_cred(): releasing sk cred 0x21b02e0 > > _______________________________________________ > lustre-discuss mailing list > [email protected] > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org > > > > _______________________________________________ > lustre-discuss mailing list > [email protected] > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org Cheers, Andreas --- Andreas Dilger Principal Lustre Architect Whamcloud
signature.asc
Description: Message signed with OpenPGP
_______________________________________________ lustre-discuss mailing list [email protected] http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
