[
https://issues.apache.org/jira/browse/IGNITE-2525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15144461#comment-15144461
]
Ivan Veselovsky edited comment on IGNITE-2525 at 2/12/16 11:29 AM:
-------------------------------------------------------------------
Hi, Luca,
I tried to reproduce the issue you describe (single-node kerberized HDP used),
and the only problem I see is an exception in the node manager log indicating
that Log Aggregation subsystem has failed to initialize (please see below). At
the same time the yarn app itself works without errors, the Ignite yarn
application is able to run Ignite node(s), and the app logs are visible through
the Yarn web console, as the doc page prescribes. It would be nice to clarify
the following questions:
1) Is the exception you initially reported the same as below? If not, can you
please provide the full stack trace of your exception?
2) Does any other yarn or mapreduce application (e.g. hadoop wordcount example)
run okay in your kerberized cluster? (Just trying to separate some common
configuration issues from Ignite yarn app specific problems.)
3) Is the problem you mention specific to long running applications only (that
is, related to ticket renewal problem only), or it prevents Ignite yarn module
from running at all?
{code}
Caused by: java.io.IOException: Failed on local exception: java.io.IOException:
org.apache.hadoop.security.AccessControlException: Client cannot authenticate
via:[TOKEN, KERBEROS]; Host Details : local host is:
"sandbox.hortonworks.com/10.0.2.15"; destination host is:
"sandbox.hortonworks.com":8020;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:773)
at org.apache.hadoop.ipc.Client.call(Client.java:1431)
at org.apache.hadoop.ipc.Client.call(Client.java:1358)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy13.getFileInfo(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy14.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2116)
at
org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
at
org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1301)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.checkExists(LogAggregationService.java:248)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.access$100(LogAggregationService.java:67)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService$1.run(LogAggregationService.java:276)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.createAppDir(LogAggregationService.java:261)
... 7 more
{code}
was (Author: iveselovskiy):
Hi, Luca,
I tried to reproduce the issue you describe (single-node kerberized HDP used),
and the only problem I see is an exception in the node manager log indicating
that Log Aggregation subsystem has failed to initialize (please see below). At
the same time the yarn app itself works without errors, the Ignite yarn
application is able to run Ignite node(s), and the app logs are visible through
the Yarn web console, as the doc page prescribes. It would be nice to clarify
the following questions:
1) Is the exception you initially reported the same as below?
2) Does any other yarn or mapreduce application (e.g. hadoop wordcount example)
run okay in your kerberized cluster?
3) Is the problem you mention specific to long running applications only (that
is, related to ticket renewal problem only), or it prevents Ignite yarn module
from running at all?
{code}
Caused by: java.io.IOException: Failed on local exception: java.io.IOException:
org.apache.hadoop.security.AccessControlException: Client cannot authenticate
via:[TOKEN, KERBEROS]; Host Details : local host is:
"sandbox.hortonworks.com/10.0.2.15"; destination host is:
"sandbox.hortonworks.com":8020;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:773)
at org.apache.hadoop.ipc.Client.call(Client.java:1431)
at org.apache.hadoop.ipc.Client.call(Client.java:1358)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy13.getFileInfo(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy14.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2116)
at
org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
at
org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1301)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.checkExists(LogAggregationService.java:248)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.access$100(LogAggregationService.java:67)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService$1.run(LogAggregationService.java:276)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.createAppDir(LogAggregationService.java:261)
... 7 more
{code}
> Yarn - Kerberos delegation token support for long running applications
> ----------------------------------------------------------------------
>
> Key: IGNITE-2525
> URL: https://issues.apache.org/jira/browse/IGNITE-2525
> Project: Ignite
> Issue Type: Improvement
> Affects Versions: 1.5.0.final
> Reporter: Luca Rea
> Assignee: Ivan Veselovsky
> Fix For: 1.6
>
>
> Hi,
> now Yarn returns an authentication error when kerberos is enabled:
> java.io.IOException: Failed on local exception: java.io.IOException:
> org.apache.hadoop.security.AccessControlException: Client cannot authenticate
> via:[TOKEN, KERBEROS];
> I think Apache Ignite needs somethink like "AMDelegationTokenRenewer.scala"
> and "ExecutorDelegationTokenUpdater.scala" of Apache Spark to support long
> running applications in clusters with kerberos authantication enabled.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
