[
https://issues.apache.org/jira/browse/FLINK-8981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16550360#comment-16550360
]
ASF GitHub Bot commented on FLINK-8981:
---------------------------------------
GitHub user aljoscha opened a pull request:
https://github.com/apache/flink/pull/6377
[FLINK-8981] Add end-to-end test for running on YARN with Kerberos
This adds a complete Docker container setup and Docker Compose file for
starting a kerberized Hadoop cluster on Docker.
The test script does the following:
* package "build-target" Flink dist into a tarball
* build docker container
* start cluster using docker compose
* upload tarball and unpack
* modify flink-conf.yaml to use Kerberos keytab for hadoop-user
* Run Streaming WordCount Job
* verify results
We set an exit trap before to ensure that we shut down the docker
compose cluster at the end.
As a prerequisite, this also fixes how we resolve directories in the
end-to-end scripts.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/aljoscha/flink
jira-8981-kerberos-end-to-end-test
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/6377.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #6377
----
commit 5aec051a76089f623ebc21418ec5751f9fcad780
Author: Aljoscha Krettek <aljoscha.krettek@...>
Date: 2018-07-18T09:51:27Z
[hotfix] Resolve symbolic links in test scripts
commit 634426b096a36147c3180f9c732efef51155e5bb
Author: Aljoscha Krettek <aljoscha.krettek@...>
Date: 2018-07-18T11:46:29Z
[FLINK-8981] Add end-to-end test for running on YARN with Kerberos
This adds a complete Docker container setup and Docker Compose file for
starting a kerberized Hadoop cluster on Docker.
The test script does the following:
* package "build-target" Flink dist into a tarball
* build docker container
* start cluster using docker compose
* upload tarball and unpack
* modify flink-conf.yaml to use Kerberos keytab for hadoop-user
* Run Streaming WordCount Job
* verify results
We set an exit trap before to ensure that we shut down the docker
compose cluster at the end.
----
> Add end-to-end test for running on YARN with Kerberos
> -----------------------------------------------------
>
> Key: FLINK-8981
> URL: https://issues.apache.org/jira/browse/FLINK-8981
> Project: Flink
> Issue Type: Sub-task
> Components: Security, Tests
> Affects Versions: 1.5.0
> Reporter: Till Rohrmann
> Assignee: Aljoscha Krettek
> Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.6.0
>
>
> We should add an end-to-end test which verifies Flink's integration with
> Kerberos security. In order to do this, we should start a Kerberos secured
> Hadoop, ZooKeeper and Kafka cluster. Then we should start a Flink cluster
> with HA enabled and run a job which reads from and writes to Kafka. We could
> use a simple pipe job for that purpose which has some state for checkpointing
> to HDFS.
> See [security docs|
> https://ci.apache.org/projects/flink/flink-docs-master/ops/security-kerberos.html]
> for how more information about Flink's Kerberos integration.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
