gaborgsomogyi edited a comment on issue #24204: [SPARK-27270][SS] Add Kafka dynamic JAAS authentication debug possibility URL: https://github.com/apache/spark/pull/24204#issuecomment-478485095 Maybe my explanation was not enough/clean. Let me give a little bit more details. I've created a small standalone application where these things can be tested easily. The application connect to a secure Kafka cluster and tries to do authentication with dynamic JAAS configuration (where `debug=true` can be set). I've re-tested everything to give exact logs. Test with `debug=false`: Command: `java -Dsun.security.krb5.debug=true -Dsun.security.spnego.debug=true ...` Output: ``` Java config name: null Native config name: /etc/krb5.conf >>> KeyTabInputStream, readName(): GCE.CLOUDERA.COM >>> KeyTabInputStream, readName(): systest >>> KeyTab: load() entry length: 70; type: 16 >>> KeyTabInputStream, readName(): GCE.CLOUDERA.COM >>> KeyTabInputStream, readName(): systest >>> KeyTab: load() entry length: 62; type: 23 >>> KeyTabInputStream, readName(): GCE.CLOUDERA.COM >>> KeyTabInputStream, readName(): systest >>> KeyTab: load() entry length: 54; type: 8 >>> KeyTabInputStream, readName(): GCE.CLOUDERA.COM >>> KeyTabInputStream, readName(): systest >>> KeyTab: load() entry length: 54; type: 3 >>> KeyTabInputStream, readName(): GCE.CLOUDERA.COM >>> KeyTabInputStream, readName(): systest >>> KeyTab: load() entry length: 54; type: 1 >>> KeyTabInputStream, readName(): GCE.CLOUDERA.COM >>> KeyTabInputStream, readName(): systest >>> KeyTab: load() entry length: 54; type: 1 Looking for keys for: [email protected] Found unsupported keytype (1) for [email protected] Found unsupported keytype (1) for [email protected] Found unsupported keytype (3) for [email protected] Found unsupported keytype (8) for [email protected] Added key: 23version: 1 Added key: 16version: 1 >>> KdcAccessibility: reset Looking for keys for: [email protected] Found unsupported keytype (1) for [email protected] Found unsupported keytype (1) for [email protected] Found unsupported keytype (3) for [email protected] Found unsupported keytype (8) for [email protected] Added key: 23version: 1 Added key: 16version: 1 Using builtin default etypes for default_tkt_enctypes default etypes for default_tkt_enctypes: 17 16 23. >>> KrbAsReq creating message getKDCFromDNS using UDP getKDCFromDNS using TCP >>> KrbKdcReq send: kdc=gsomogyi-cdh6x-secure-1.gce.cloudera.com UDP:88, timeout=30000, number of retries =3, #bytes=148 >>> KDCCommunication: kdc=gsomogyi-cdh6x-secure-1.gce.cloudera.com UDP:88, timeout=30000,Attempt =1, #bytes=148 >>> KrbKdcReq send: #bytes read=775 >>> KdcAccessibility: remove gsomogyi-cdh6x-secure-1.gce.cloudera.com Looking for keys for: [email protected] Found unsupported keytype (1) for [email protected] Found unsupported keytype (1) for [email protected] Found unsupported keytype (3) for [email protected] Found unsupported keytype (8) for [email protected] Added key: 23version: 1 Added key: 16version: 1 >>> EType: sun.security.krb5.internal.crypto.Des3CbcHmacSha1KdEType >>> KrbAsRep cons in KrbAsReq.getReply systest >>> 19/04/01 10:08:56 INFO authenticator.AbstractLogin: Successfully logged in. >>> 19/04/01 10:08:56 INFO kerberos.KerberosLogin: [[email protected]]: TGT refresh thread started. >>> 19/04/01 10:08:56 INFO kerberos.KerberosLogin: [[email protected]]: TGT valid starting at: Mon Apr 01 10:08:55 CEST 2019 >>> 19/04/01 10:08:56 INFO kerberos.KerberosLogin: [[email protected]]: TGT expires: Wed May 01 10:08:55 CEST 2019 >>> 19/04/01 10:08:56 INFO kerberos.KerberosLogin: [[email protected]]: TGT refresh sleeping until: Fri Apr 26 05:03:45 CEST 2019 >>> 19/04/01 10:08:56 INFO utils.AppInfoParser: Kafka version : 2.0.0 >>> 19/04/01 10:08:56 INFO utils.AppInfoParser: Kafka commitId : 3402a8361b734732 ``` Test with `debug=true`: Command: `java ...` Output: ``` Java config name: null Native config name: /etc/krb5.conf Debug is true storeKey true useTicketCache false useKeyTab true doNotPrompt false ticketCache is null isInitiator true KeyTab is /Users/gaborsomogyi/kafka-delegation-token/kafka-consumer/systest.keytab refreshKrb5Config is false principal is [email protected] tryFirstPass is false useFirstPass is false storePass is false clearPass is false >>> KeyTabInputStream, readName(): GCE.CLOUDERA.COM >>> KeyTabInputStream, readName(): systest >>> KeyTab: load() entry length: 70; type: 16 >>> KeyTabInputStream, readName(): GCE.CLOUDERA.COM >>> KeyTabInputStream, readName(): systest >>> KeyTab: load() entry length: 62; type: 23 >>> KeyTabInputStream, readName(): GCE.CLOUDERA.COM >>> KeyTabInputStream, readName(): systest >>> KeyTab: load() entry length: 54; type: 8 >>> KeyTabInputStream, readName(): GCE.CLOUDERA.COM >>> KeyTabInputStream, readName(): systest >>> KeyTab: load() entry length: 54; type: 3 >>> KeyTabInputStream, readName(): GCE.CLOUDERA.COM >>> KeyTabInputStream, readName(): systest >>> KeyTab: load() entry length: 54; type: 1 >>> KeyTabInputStream, readName(): GCE.CLOUDERA.COM >>> KeyTabInputStream, readName(): systest >>> KeyTab: load() entry length: 54; type: 1 Looking for keys for: [email protected] Found unsupported keytype (1) for [email protected] Found unsupported keytype (1) for [email protected] Found unsupported keytype (3) for [email protected] Found unsupported keytype (8) for [email protected] Added key: 23version: 1 Added key: 16version: 1 >>> KdcAccessibility: reset Looking for keys for: [email protected] Found unsupported keytype (1) for [email protected] Found unsupported keytype (1) for [email protected] Found unsupported keytype (3) for [email protected] Found unsupported keytype (8) for [email protected] Added key: 23version: 1 Added key: 16version: 1 Using builtin default etypes for default_tkt_enctypes default etypes for default_tkt_enctypes: 17 16 23. >>> KrbAsReq creating message getKDCFromDNS using UDP getKDCFromDNS using TCP >>> KrbKdcReq send: kdc=gsomogyi-cdh6x-secure-1.gce.cloudera.com UDP:88, timeout=30000, number of retries =3, #bytes=148 >>> KDCCommunication: kdc=gsomogyi-cdh6x-secure-1.gce.cloudera.com UDP:88, timeout=30000,Attempt =1, #bytes=148 >>> KrbKdcReq send: #bytes read=775 >>> KdcAccessibility: remove gsomogyi-cdh6x-secure-1.gce.cloudera.com Looking for keys for: [email protected] Found unsupported keytype (1) for [email protected] Found unsupported keytype (1) for [email protected] Found unsupported keytype (3) for [email protected] Found unsupported keytype (8) for [email protected] Added key: 23version: 1 Added key: 16version: 1 >>> EType: sun.security.krb5.internal.crypto.Des3CbcHmacSha1KdEType >>> KrbAsRep cons in KrbAsReq.getReply systest principal is [email protected] Will use keytab Commit Succeeded >>> 19/04/01 10:11:25 INFO authenticator.AbstractLogin: Successfully logged in. >>> 19/04/01 10:11:25 INFO kerberos.KerberosLogin: [[email protected]]: TGT refresh thread started. >>> 19/04/01 10:11:25 INFO kerberos.KerberosLogin: [[email protected]]: TGT valid starting at: Mon Apr 01 10:11:24 CEST 2019 >>> 19/04/01 10:11:25 INFO kerberos.KerberosLogin: [[email protected]]: TGT expires: Wed May 01 10:11:24 CEST 2019 >>> 19/04/01 10:11:25 INFO kerberos.KerberosLogin: [[email protected]]: TGT refresh sleeping until: Fri Apr 26 02:37:27 CEST 2019 >>> 19/04/01 10:11:25 INFO utils.AppInfoParser: Kafka version : 2.0.0 >>> 19/04/01 10:11:25 INFO utils.AppInfoParser: Kafka commitId : 3402a8361b734732 ``` As a final conclusion even if global krb debug flag is enabled `Krb5LoginModule` debug messages not shown with dynamic JAAS configuration. Please see from the first execution the following message is missing: ``` Debug is true storeKey true useTicketCache false useKeyTab true doNotPrompt false ticketCache is null isInitiator true KeyTab is /Users/gaborsomogyi/kafka-delegation-token/kafka-consumer/systest.keytab refreshKrb5Config is false principal is [email protected] tryFirstPass is false useFirstPass is false storePass is false clearPass is false ... principal is [email protected] Will use keytab Commit Succeeded ... ``` These messages is just a sample and `Krb5LoginModule` provides much more debug information which may be helpful for debugging.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
