GitHub user gaborgsomogyi opened a pull request:
https://github.com/apache/spark/pull/22598
[SPARK-25501][SS] Add kafka delegation token support.
## What changes were proposed in this pull request?
It adds kafka delegation token support for structured streaming. Please see
the relevant
[SPIP](https://docs.google.com/document/d/1ouRayzaJf_N5VQtGhVq9FURXVmRpXzEEWYHob0ne3NY/edit?usp=sharing)
What this PR contains:
* Configuration parameters for the feature
* Delegation token fetching from broker
* Usage of token through dynamic JAAS configuration
* Minor refactoring in the existing code
What this PR doesn't contain:
* Documentation changes because design can change
## How was this patch tested?
Existing tests + added small amount of additional unit tests.
Because it's an external service integration mainly tested on cluster.
* 4 node cluster
* Kafka broker version 1.1.0
* Topic with 4 partitions
* security.protocol = SASL_SSL
* sasl.mechanism = SCRAM-SHA-256
An example of obtaining a token:
```
18/10/01 01:07:49 INFO kafka010.TokenUtil: TOKENID HMAC
OWNER RENEWERS ISSUEDATE
EXPIRYDATE MAXDATE
18/10/01 01:07:49 INFO kafka010.TokenUtil: D1-v__Q5T_uHx55rW16Jwg [hidden]
User:user [] 2018-10-01T01:07 2018-10-02T01:07
2018-10-08T01:07
18/10/01 01:07:49 INFO security.KafkaDelegationTokenProvider: Get token
from Kafka: Kind: KAFKA_DELEGATION_TOKEN, Service:
kafka.server.delegation.token, Ident: 44 31 2d 76 5f 5f 51 35 54 5f 75 48 78 35
35 72 57 31 36 4a 77 67
```
An example token usage:
```
18/10/01 01:08:07 INFO kafka010.KafkaSecurityHelper: Scram JAAS params:
org.apache.kafka.common.security.scram.ScramLoginModule required tokenauth=true
serviceName="kafka" username="D1-v__Q5T_uHx55rW16Jwg" password="[hidden]";
18/10/01 01:08:07 INFO kafka010.KafkaSourceProvider: Delegation token
detected, using it for login.
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gaborgsomogyi/spark SPARK-25501
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22598.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22598
----
commit f9b46855b3dd4ef525ab1bb75a3e461d4074d8e6
Author: Gabor Somogyi <gabor.g.somogyi@...>
Date: 2018-10-01T09:07:10Z
[SPARK-25501][SS] Add kafka delegation token support.
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
