koertkuipers commented on issue #22598: [SPARK-25501][SS] Add kafka delegation token support. URL: https://github.com/apache/spark/pull/22598#issuecomment-497974098 i have been testing long running structured streaming jobs from and to kafka using delegation tokens. the driver is launched by a user with a kerberos login and keytab, principal and keytab are provided to spark-submit. trigger is hourly. i see no errors in executor logs. in driver i see this starting 24 hours after job was launched (i assume this is when kerberos ticket expires?): ``` 2019-05-31 17:05:03 WARN internals.AbstractCoordinator: [Consumer clientId=consumer-1, groupId=spark-kafka-source-84afdf97-b7e0-468e-9ee1-d7ab50e81b85-1237283319-driver-0] This member will leave the group because consumer poll timeout has expired. This means the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time processing messages. You can address this either by increasing max.poll.interval.ms or by reducing the maximum size of batches returned in poll() with max.poll.records. 2019-05-31 17:05:03 INFO internals.AbstractCoordinator: [Consumer clientId=consumer-1, groupId=spark-kafka-source-84afdf97-b7e0-468e-9ee1-d7ab50e81b85-1237283319-driver-0] Member consumer-1-3f19c416-c17a-41f6-b710-67f50dd7a568 sending LeaveGroup request to coordinator node05.company.com:9092 (id: 2147483592 rack: null) 2019-05-31 18:00:00 INFO internals.ConsumerCoordinator: [Consumer clientId=consumer-1, groupId=spark-kafka-source-84afdf97-b7e0-468e-9ee1-d7ab50e81b85-1237283319-driver-0] Revoking previously assigned partitions [twitter-0] 2019-05-31 18:00:00 INFO internals.AbstractCoordinator: [Consumer clientId=consumer-1, groupId=spark-kafka-source-84afdf97-b7e0-468e-9ee1-d7ab50e81b85-1237283319-driver-0] (Re-)joining group 2019-05-31 18:00:00 INFO internals.AbstractCoordinator: [Consumer clientId=consumer-1, groupId=spark-kafka-source-84afdf97-b7e0-468e-9ee1-d7ab50e81b85-1237283319-driver-0] Group coordinator node05.company.com:9092 (id: 2147483592 rack: null) is unavailable or invalid, will attempt rediscovery 2019-05-31 18:00:00 INFO network.Selector: [Consumer clientId=consumer-1, groupId=spark-kafka-source-84afdf97-b7e0-468e-9ee1-d7ab50e81b85-1237283319-driver-0] Failed authentication with node06.company.com/10.0.0.56 (Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-512) 2019-05-31 18:00:00 ERROR clients.NetworkClient: [Consumer clientId=consumer-1, groupId=spark-kafka-source-84afdf97-b7e0-468e-9ee1-d7ab50e81b85-1237283319-driver-0] Connection to node 56 (node06.company.com/10.0.0.56:9092) failed authentication due to: Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-512 2019-05-31 18:00:00 INFO network.Selector: [Consumer clientId=consumer-1, groupId=spark-kafka-source-84afdf97-b7e0-468e-9ee1-d7ab50e81b85-1237283319-driver-0] Failed authentication with node10.company.com/10.0.0.60 (Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-512) 2019-05-31 18:00:00 ERROR clients.NetworkClient: [Consumer clientId=consumer-1, groupId=spark-kafka-source-84afdf97-b7e0-468e-9ee1-d7ab50e81b85-1237283319-driver-0] Connection to node 60 (node10.company.com/10.0.0.60:9092) failed authentication due to: Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-512 2019-05-31 18:00:00 WARN kafka010.KafkaOffsetReader: Error in attempt 1 getting Kafka offsets: org.apache.kafka.common.errors.SaslAuthenticationException: Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-512 2019-05-31 18:00:01 INFO consumer.ConsumerConfig: ConsumerConfig values: <redacted> 2019-05-31 18:00:01 INFO authenticator.AbstractLogin: Successfully logged in. 2019-05-31 18:00:01 INFO utils.AppInfoParser: Kafka version: 2.2.0 2019-05-31 18:00:01 INFO utils.AppInfoParser: Kafka commitId: 05fcfde8f69b0349 2019-05-31 18:00:01 INFO consumer.KafkaConsumer: [Consumer clientId=consumer-2, groupId=spark-kafka-source-84afdf97-b7e0-468e-9ee1-d7ab50e81b85-1237283319-driver-1] Subscribed to topic(s): <redacted> 2019-05-31 18:00:01 INFO clients.Metadata: Cluster ID: <redacted> 2019-05-31 18:00:01 INFO internals.AbstractCoordinator: [Consumer clientId=consumer-2, groupId=spark-kafka-source-84afdf97-b7e0-468e-9ee1-d7ab50e81b85-1237283319-driver-1] Discovered group coordinator node06.company.com:9092 (id: 2147483591 rack: null) 2019-05-31 18:00:01 INFO internals.ConsumerCoordinator: [Consumer clientId=consumer-2, groupId=spark-kafka-source-84afdf97-b7e0-468e-9ee1-d7ab50e81b85-1237283319-driver-1] Revoking previously assigned partitions [] 2019-05-31 18:00:01 INFO internals.AbstractCoordinator: [Consumer clientId=consumer-2, groupId=spark-kafka-source-84afdf97-b7e0-468e-9ee1-d7ab50e81b85-1237283319-driver-1] (Re-)joining group ``` it seems like login with SCRAM fails a few times and then succeeds again. is this expected?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
