richard.lee created ZEPPELIN-3186:
-------------------------------------
Summary: zeppelin cannot config hadoop.rpc.protection
Key: ZEPPELIN-3186
URL: https://issues.apache.org/jira/browse/ZEPPELIN-3186
Project: Zeppelin
Issue Type: Improvement
Components: Interpreters
Affects Versions: 0.7.2
Reporter: richard.lee
When using zeppelin's with a kerberozied hadoop/spark cluster, spark
interpreter cannot start with following exception:
{code:java}
INFO [2018-01-23 18:46:05,013] ({pool-2-thread-2} Logging.scala[logInfo]:54) -
Attempting to login to the Kerberos using principal: hadoop/[email protected]
and keytab: /etc/hadoop/hadoop.service.keytab INFO [2018-01-23 18:46:05,048]
({pool-2-thread-2} RMProxy.java[createRMProxy]:98) - Connecting to
ResourceManager at l-opsdev1.dev.p1/10.64.16.34:8032 WARN [2018-01-23
18:46:11,927] ({pool-2-thread-2} Client.java[run]:678) - Couldn't setup
connection for hadoop/[email protected] to kdc/10.64.xx.xx:8032
javax.security.sasl.SaslException: No common protection layer between client
and server at
com.sun.security.sasl.gsskerb.GssKrb5Client.doFinalHandshake(GssKrb5Client.java:251)
at
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:186)
at
org.apache.hadoop.security.SaslRpcClient.saslEvaluateToken(SaslRpcClient.java:483)
at
org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:427) at
org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:560) at
org.apache.hadoop.ipc.Client$Connection.access$1900(Client.java:375) at
org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:729) at
org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:725) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:422) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:725) at
org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:375) at
org.apache.hadoop.ipc.Client.getConnection(Client.java:1528) at
org.apache.hadoop.ipc.Client.call(Client.java:1451) at
org.apache.hadoop.ipc.Client.call(Client.java:1412) at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy15.getClusterMetrics(Unknown Source) at
org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getClusterMetrics(ApplicationClientProtocolPBClientImpl.java:206)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497) at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy16.getClusterMetrics(Unknown Source) at
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getYarnClusterMetrics(YarnClientImpl.java:487)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:725) at
org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:375) at
org.apache.hadoop.ipc.Client.getConnection(Client.java:1528) at
org.apache.hadoop.ipc.Client.call(Client.java:1451) at
org.apache.hadoop.ipc.Client.call(Client.java:1412) at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy15.getClusterMetrics(Unknown Source) at
org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getClusterMetrics(ApplicationClientProtocolPBClientImpl.java:206)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497) at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy16.getClusterMetrics(Unknown Source) at
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getYarnClusterMetrics(YarnClientImpl.java:487)
at
org.apache.spark.deploy.yarn.Client$$anonfun$submitApplication$1.apply(Client.scala:156)
at
org.apache.spark.deploy.yarn.Client$$anonfun$submitApplication$1.apply(Client.scala:156)
at org.apache.spark.internal.Logging$class.logInfo(Logging.scala:54) at
org.apache.spark.deploy.yarn.Client.logInfo(Client.scala:59) at
org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:155) at
org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:56)
at
org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:173)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:509) at
org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2516) at
org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:918)
at
org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:910)
at scala.Option.getOrElse(Option.scala:121) at
org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:910)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497) at
org.apache.zeppelin.spark.Utils.invokeMethod(Utils.java:38) at
org.apache.zeppelin.spark.Utils.invokeMethod(Utils.java:33) at
org.apache.zeppelin.spark.SparkInterpreter.createSparkSession(SparkInterpreter.java:367)
at
org.apache.zeppelin.spark.SparkInterpreter.getSparkSession(SparkInterpreter.java:233)
at org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:826)
at
org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:491)
at org.apache.zeppelin.scheduler.Job.run(Job.java:175) at
org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139) at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at
java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
After google I think there should be a way to set "hadoop.rpc.protection"
before execute
UserGroupInformation.loginUserFromKeytab(principal, keytab) in
SparkInterpreter.open(). My hdfs cluster's configuration is :
<property>
<name>hadoop.rpc.protection</name>
<value>privacy</value>
</property>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
