I am attempting to run flink / YARN on a secure MapR 5.2 cluster. The
cluster is secured using "MapR Native Security", not kerberos.
I did include the MapR zookeeper when building:
opt/apache-maven-3.3.9/bin/mvn clean install -DskipTests -Pvendor-repos
-Dhadoop.version=2.7.0-mapr-1607 -Dzookeeper.version=3.4.5-mapr-1604
When I try to launch yarn-session.sh, I receive the following error:
[mapr@maprdemo flink]$ ./bin/yarn-session.sh -n 1
2017-03-08 14:47:24,223 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: jobmanager.rpc.address, localhost
2017-03-08 14:47:24,224 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: jobmanager.rpc.port, 6123
2017-03-08 14:47:24,224 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: jobmanager.heap.mb, 256
2017-03-08 14:47:24,224 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: taskmanager.heap.mb, 512
2017-03-08 14:47:24,225 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: taskmanager.numberOfTaskSlots, 1
2017-03-08 14:47:24,225 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: taskmanager.memory.preallocate, false
2017-03-08 14:47:24,225 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: parallelism.default, 1
2017-03-08 14:47:24,225 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: jobmanager.web.port, 8081
2017-03-08 14:47:24,292 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: jobmanager.rpc.address, localhost
2017-03-08 14:47:24,292 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: jobmanager.rpc.port, 6123
2017-03-08 14:47:24,293 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: jobmanager.heap.mb, 256
2017-03-08 14:47:24,293 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: taskmanager.heap.mb, 512
2017-03-08 14:47:24,293 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: taskmanager.numberOfTaskSlots, 1
2017-03-08 14:47:24,293 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: taskmanager.memory.preallocate, false
2017-03-08 14:47:24,293 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: parallelism.default, 1
2017-03-08 14:47:24,294 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: jobmanager.web.port, 8081
2017-03-08 14:47:26,107 WARN org.apache.hadoop.util.NativeCodeLoader
- Unable to load native-hadoop library for your platform... using
builtin-java classes where applicable
2017-03-08 14:47:26,390 WARN
org.apache.flink.runtime.security.modules.HadoopModule - Hadoop
security is enabled but current login user does not have Kerberos
credentials
2017-03-08 14:47:26,390 INFO
org.apache.flink.runtime.security.modules.HadoopModule - Hadoop user
set to mapr (auth:CUSTOM)
2017-03-08 14:47:26,431 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: jobmanager.rpc.address, localhost
2017-03-08 14:47:26,431 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: jobmanager.rpc.port, 6123
2017-03-08 14:47:26,431 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: jobmanager.heap.mb, 256
2017-03-08 14:47:26,431 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: taskmanager.heap.mb, 512
2017-03-08 14:47:26,432 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: taskmanager.numberOfTaskSlots, 1
2017-03-08 14:47:26,432 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: taskmanager.memory.preallocate, false
2017-03-08 14:47:26,433 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: parallelism.default, 1
2017-03-08 14:47:26,433 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: jobmanager.web.port, 8081
2017-03-08 14:47:26,440 ERROR org.apache.flink.yarn.YarnClusterDescriptor
- Hadoop security is enabled but the login user does not have Kerberos
credentials
Error while deploying YARN cluster: Couldn't deploy Yarn cluster
java.lang.RuntimeException: Couldn't deploy Yarn cluster
at
org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploy(AbstractYarnClusterDescriptor.java:421)
at
org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:620)
at
org.apache.flink.yarn.cli.FlinkYarnSessionCli$1.call(FlinkYarnSessionCli.java:476)
at
org.apache.flink.yarn.cli.FlinkYarnSessionCli$1.call(FlinkYarnSessionCli.java:473)
at
org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595)
at
org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
at
org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:473)
Caused by: java.lang.RuntimeException: Hadoop security is enabled but the
login user does not have Kerberos credentials
at
org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploy(AbstractYarnClusterDescriptor.java:415)
The 'mapr' user does have a ticket:
[mapr@maprdemo flink]$ maprlogin print
Opening keyfile /tmp/maprticket_2000
demo.mapr.com: user = mapr, created = 'Wed Mar 01 07:26:02 PST 2017',
expires = 'Wed Mar 15 08:26:02 PDT 2017', RenewalTill = 'Fri Mar 31 08:26:02
PDT 2017', uid = 2000, gids = 2000, 2001, CanImpersonate = false
I can launch other YARN jobs without error. Most of the MapR environment
variables are picked up in /opt/mapr/conf/env.sh and
/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/hadoop-env.sh.
I have set the following environment variables:
export FLINK_HOME=/opt/flink-1.2
export YARN_CONF_DIR=/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop
export HADOOP_HOME=/opt/mapr/hadoop/hadoop-2.7.0
export HADOOP_CONF_DIR=/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop
--
View this message in context:
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Flink-1-2-YARN-on-Secure-MapR-Cluster-tp16399.html
Sent from the Apache Flink Mailing List archive. mailing list archive at
Nabble.com.
