gaborgsomogyi opened a new pull request #24305: [SPARK-27294][SS] Add multi-cluster Kafka delegation token URL: https://github.com/apache/spark/pull/24305 ## What changes were proposed in this pull request? The actual implementation doesn't support multi-cluster Kafka connection with delegation token. In this PR I've added this functionality. What this PR contains: * New way of configuration * Multiple delegation token obtain/store/use functionality * Temporary turn off DStreams delegation token usage not to create horror big PR What this PR doesn't contain: * DStreams implementation * Documentation ## How was this patch tested? Existing + additional unit tests. Additionally tested on cluster. Test scenario: * 2 * 4 node clusters * The 4-4 nodes are in different kerberos realms * Cross-Realm trust between the 2 realms * Yarn * Kafka broker version 2.1.0 * security.protocol = SASL_SSL * sasl.mechanism = SCRAM-SHA-512 * Artificial exceptions during processing * Source reads from realm1 sink writes to realm2 Kafka broker settings: * delegation.token.expiry.time.ms=600000 (10 min) * delegation.token.max.lifetime.ms=1200000 (20 min) * delegation.token.expiry.check.interval.ms=300000 (5 min)
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
