Hi Lustre Admins, I am hoping someone can help me understand what I'm doing wrong with SSK setup. I have set up a lustre 2.11 server and worked through the steps to use shared secret keys (SSKs) to encrypt data in transit between client nodes and the MDT and OSS. I followed the manual instructions here: http://doc.lustre.org/lustre_manual.xhtml#idm140687075065344
Before enabling the encryption settings on the MDT, I can mount the FS on the client node. After I turn on the encryption I get back an encryption refused error and cannot mount: mount.lustre: mount 172.31.46.245@tcp:/scratch at /scratch failed: Connection refused The keys are definitely distributed to client nodes and server nodes and the settings have all been made as instruct4red in the manual (I did this a few times from scratch to make sure). I can manually load the keys into the keyring and see them by running `keyctl show`, I can compare the key files on client and server nodes with the command `lgss_sk --read /secure_directory/scratch.client.key` and validate that they all match and that the client has a prime. The commands I'm using to enable the encryption are: mdt# sudo lctl conf_param scratch.srpc.flavor.tcp.cli2mdt=skpi mdt# sudo lctl conf_param scratch.srpc.flavor.tcp.cli2ost=skpi I tried tailing /var/log/messages and am not able to interpret the output, I'm wondering - does anyone have a hypothesis about what might be wrong or instructions to debug? Log output is below! Many thanks to anyone who can help! Mark Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22250]:TRACE:main(): start parsing parameters Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22250]:INFO:main(): key 428863463, desc 0@26, ugid 0:0, sring 46159405, coinfo 38:sk:0:0:m:p:2:0x20000ac1f2109:scratch-OST1cd0-osc-MDT0000:0x20000ac1f2ef5:1 Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22250]:TRACE:parse_callout_info(): components: 38,sk,0,0,m,p,2,0x20000ac1f2109,scratch-OST1cd0-osc-MDT0000,0x20000ac1f2ef5,1 Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22250]:DEBUG:parse_callout_info(): parse call out info: secid 38, mech sk, ugid 0:0, is_root 0, is_mdt 1, is_ost 0, svc type p, svc 2, nid 0x20000ac1f2109, tgt scratch-OST1cd0-osc-MDT0000, self nid 0x20000ac1f2ef5, pid 1 Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22250]:TRACE:main(): parsing parameters OK Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22250]:TRACE:lgss_mech_initialize(): initialize mech sk Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22250]:TRACE:lgss_create_cred(): create a sk cred at 0x1ecc2e0 Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22250]:TRACE:main(): caller's namespace is the same Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22250]:TRACE:lgss_prepare_cred(): preparing sk cred 0x1ecc2e0 Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22250]:INFO:sk_create_cred(): Creating credentials for target: scratch-OST1cd0-osc-MDT0000 with nodemap: (null) Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22250]:INFO:sk_create_cred(): Searching for key with description: lustre:scratch Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22250]:TRACE:prepare_and_instantiate(): instantiated kernel key 198fefe7 Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22250]:TRACE:main(): forked child 22251 Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22251]:TRACE:lgssc_kr_negotiate(): child start on behalf of key 198fefe7: cred 0x1ecc2e0, uid 0, svc 2, nid 20000ac1f2109, uids: 0:0/0:0 Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22251]:INFO:ipv4_nid2hostname(): SOCKLND: net 0x20000, addr 0x9211fac => ip-172-31-33-9.us-west-2.compute.internal Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22251]:DEBUG:lgss_get_service_str(): constructed service string: [email protected] Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22251]:TRACE:lgss_using_cred(): using sk cred 0x1ecc2e0 Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22253]:TRACE:main(): start parsing parameters Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22253]:INFO:main(): key 189483693, desc 0@25, ugid 0:0, sring 46159405, coinfo 37:sk:0:0:m:p:2:0x20000ac1f2687:scratch-OST2b9d-osc-MDT0000:0x20000ac1f2ef5:1 Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22253]:TRACE:parse_callout_info(): components: 37,sk,0,0,m,p,2,0x20000ac1f2687,scratch-OST2b9d-osc-MDT0000,0x20000ac1f2ef5,1 Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22253]:DEBUG:parse_callout_info(): parse call out info: secid 37, mech sk, ugid 0:0, is_root 0, is_mdt 1, is_ost 0, svc type p, svc 2, nid 0x20000ac1f2687, tgt scratch-OST2b9d-osc-MDT0000, self nid 0x20000ac1f2ef5, pid 1 Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22253]:TRACE:main(): parsing parameters OK Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22253]:TRACE:lgss_mech_initialize(): initialize mech sk Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22253]:TRACE:lgss_create_cred(): create a sk cred at 0x21b02e0 Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22253]:TRACE:main(): caller's namespace is the same Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22253]:TRACE:lgss_prepare_cred(): preparing sk cred 0x21b02e0 Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22253]:INFO:sk_create_cred(): Creating credentials for target: scratch-OST2b9d-osc-MDT0000 with nodemap: (null) Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22253]:INFO:sk_create_cred(): Searching for key with description: lustre:scratch Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22253]:TRACE:prepare_and_instantiate(): instantiated kernel key 0b4b4aad Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22253]:TRACE:main(): forked child 22254 Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22254]:TRACE:lgssc_kr_negotiate(): child start on behalf of key 0b4b4aad: cred 0x21b02e0, uid 0, svc 2, nid 20000ac1f2687, uids: 0:0/0:0 Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22254]:INFO:ipv4_nid2hostname(): SOCKLND: net 0x20000, addr 0x87261fac => ip-172-31-38-135.us-west-2.compute.internal Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22254]:DEBUG:lgss_get_service_str(): constructed service string: [email protected] Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22254]:TRACE:lgss_using_cred(): using sk cred 0x21b02e0 Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22251]:INFO:sk_encode_netstring(): Encoded netstring of 647 bytes Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22251]:INFO:lgss_sk_using_cred(): Created netstring of 647 bytes Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22251]:TRACE:lgssc_negotiation_manual(): starting gss negotation Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22251]:TRACE:do_nego_rpc(): start negotiation rpc Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22251]:TRACE:gss_do_ioctl(): to open /proc/fs/lustre/sptlrpc/gss/init_channel Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22251]:TRACE:gss_do_ioctl(): to down-write Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22254]:INFO:sk_encode_netstring(): Encoded netstring of 647 bytes Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22254]:INFO:lgss_sk_using_cred(): Created netstring of 647 bytes Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22254]:TRACE:lgssc_negotiation_manual(): starting gss negotation Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22254]:TRACE:do_nego_rpc(): start negotiation rpc Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22254]:TRACE:gss_do_ioctl(): to open /proc/fs/lustre/sptlrpc/gss/init_channel Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22254]:TRACE:gss_do_ioctl(): to down-write Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22251]:TRACE:do_nego_rpc(): do_nego_rpc: to parse reply Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22251]:DEBUG:do_nego_rpc(): do_nego_rpc: receive handle len 0, token len 0, res 0 Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22251]:ERROR:lgssc_negotiation_manual(): negotiation gss error 60000 Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22251]:ERROR:lgssc_kr_negotiate_manual(): key 198fefe7: failed to negotiate Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22251]:TRACE:error_kernel_key(): revoking kernel key 198fefe7 Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22251]:INFO:error_kernel_key(): key 198fefe7: revoked Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22251]:TRACE:lgss_release_cred(): releasing sk cred 0x1ecc2e0 Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22254]:TRACE:do_nego_rpc(): do_nego_rpc: to parse reply Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22254]:DEBUG:do_nego_rpc(): do_nego_rpc: receive handle len 0, token len 0, res 0 Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22254]:ERROR:lgssc_negotiation_manual(): negotiation gss error 60000 Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22254]:ERROR:lgssc_kr_negotiate_manual(): key 0b4b4aad: failed to negotiate Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22254]:TRACE:error_kernel_key(): revoking kernel key 0b4b4aad Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22254]:INFO:error_kernel_key(): key 0b4b4aad: revoked Jun 22 19:22:02 ip-172-31-46-245 lgss_keyring: [22254]:TRACE:lgss_release_cred(): releasing sk cred 0x21b02e0
_______________________________________________ lustre-discuss mailing list [email protected] http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
