Error message is unreadable. Please attach the stack trace snapshot image. It will be better.
qq <987626...@qq.com> 于2020年7月17日周五 下午9:25写道： > Hello: > > The following error occurred while running the shell task using > Oozie: > > W] ACTION[0000001-200717100610211-oozie-root-W@shell-node] Adding > default credentials for action: hdfs, yarn and jhs > 2020-07-17 10:12:58,520 INFO ShellActionExecutor:520 - > SERVER[master] USER[oozie] GROUP[-] TOKEN[] APP[shell-wf] > JOB[0000001-200717100610211-oozie-root-W] > ACTION[0000001-200717100610211-oozie-root-W@shell-node] Adding default > credentials for action: hdfs, yarn and jhs > 2020-07-17 10:12:58,522 INFO deprecation:1395 - SERVER[master] > fs.default.name is deprecated. Instead, use fs.defaultFS > 2020-07-17 10:12:58,522 INFO deprecation:1395 - SERVER[master] > fs.default.name is deprecated. Instead, use fs.defaultFS > 2020-07-17 10:12:58,523 WARN HadoopTokenHelper:523 - SERVER[master] > USER[oozie] GROUP[-] TOKEN[] APP[shell-wf] > JOB[0000001-200717100610211-oozie-root-W] > ACTION[0000001-200717100610211-oozie-root-W@shell-node] An error happened > while trying to get server principal. Getting it from service principal > anyway. > 2020-07-17 10:12:58,523 WARN HadoopTokenHelper:523 - SERVER[master] > USER[oozie] GROUP[-] TOKEN[] APP[shell-wf] > JOB[0000001-200717100610211-oozie-root-W] > ACTION[0000001-200717100610211-oozie-root-W@shell-node] An error happened > while trying to get server principal. Getting it from service principal > anyway. > 2020-07-17 10:12:58,523 INFO HadoopTokenHelper:520 - SERVER[master] > USER[oozie] GROUP[-] TOKEN[] APP[shell-wf] > JOB[0000001-200717100610211-oozie-root-W] > ACTION[0000001-200717100610211-oozie-root-W@shell-node] Delegation Token > Renewer for yarncluster is yarn > 2020-07-17 10:12:58,523 INFO HadoopTokenHelper:520 - SERVER[master] > USER[oozie] GROUP[-] TOKEN[] APP[shell-wf] > JOB[0000001-200717100610211-oozie-root-W] > ACTION[0000001-200717100610211-oozie-root-W@shell-node] Delegation Token > Renewer for yarncluster is yarn > 2020-07-17 10:12:58,525 WARN Client:765 - SERVER[master] Exception > encountered while connecting to the server : > org.apache.hadoop.security.AccessControlException: Client cannot > authenticate via:[TOKEN, KERBEROS] > 2020-07-17 10:12:58,525 WARN Client:765 - SERVER[master] Exception > encountered while connecting to the server : > org.apache.hadoop.security.AccessControlException: Client cannot > authenticate via:[TOKEN, KERBEROS] > 2020-07-17 10:12:58,525 WARN ActionStartXCommand:523 - > SERVER[master] USER[oozie] GROUP[-] TOKEN[] APP[shell-wf] > JOB[0000001-200717100610211-oozie-root-W] > ACTION[0000001-200717100610211-oozie-root-W@shell-node] Error starting > action [shell-node]. ErrorType [TRANSIENT], ErrorCode [JA009], Message > [JA009: DestHost:destPort master:10020 , LocalHost:localPort master/ > 28.11.1.14:0. Failed on local exception: java.io.IOException: > org.apache.hadoop.security.AccessControlException: Client cannot > authenticate via:[TOKEN, KERBEROS]] > org.apache.oozie.action.ActionExecutorException: JA009: DestHost:destPort > master:10020 , LocalHost:localPort master/28.11.1.14:0. Failed on local > exception: java.io.IOException: > org.apache.hadoop.security.AccessControlException: Client cannot > authenticate via:[TOKEN, KERBEROS] > at > org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:463) > at > org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:443) > at > org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1134) > at > org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1644) > at org.apache.oozie.command.wf > .ActionStartXCommand.execute(ActionStartXCommand.java:243) > at org.apache.oozie.command.wf > .ActionStartXCommand.execute(ActionStartXCommand.java:68) > at > org.apache.oozie.command.XCommand.call(XCommand.java:290) > at > org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:363) > at > org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:292) > at > java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:210) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.io.IOException: DestHost:destPort master:10020 , > LocalHost:localPort master/28.11.1.14:0. Failed on local exception: > java.io.IOException: org.apache.hadoop.security.AccessControlException: > Client cannot authenticate via:[TOKEN, KERBEROS] > at > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at > java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at org.apache.hadoop.net > .NetUtils.wrapWithMessage(NetUtils.java:833) > at org.apache.hadoop.net > .NetUtils.wrapException(NetUtils.java:808) > at > org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1549) > at > org.apache.hadoop.ipc.Client.call(Client.java:1491) > at > org.apache.hadoop.ipc.Client.call(Client.java:1388) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) > at > com.sun.proxy.$Proxy42.getDelegationToken(Unknown Source) > at > org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getDelegationToken(MRClientProtocolPBClientImpl.java:211) > at > org.apache.oozie.action.hadoop.JHSCredentials.getDelegationTokenFromJHS(JHSCredentials.java:88) > at > org.apache.oozie.action.hadoop.JHSCredentials.updateCredentials(JHSCredentials.java:65) > at > org.apache.oozie.action.hadoop.JavaActionExecutor.setCredentialTokens(JavaActionExecutor.java:1546) > at > org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1082) > ... 11 more > Caused by: java.io.IOException: > org.apache.hadoop.security.AccessControlException: Client cannot > authenticate via:[TOKEN, KERBEROS] > at > org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:770) > at > java.security.AccessController.doPrivileged(Native Method) > at > javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) > at > org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:733) > at > org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:827) > at > org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:421) > at > org.apache.hadoop.ipc.Client.getConnection(Client.java:1606) > at > org.apache.hadoop.ipc.Client.call(Client.java:1435) > ... 20 more > Caused by: org.apache.hadoop.security.AccessControlException: Client > cannot authenticate via:[TOKEN, KERBEROS] > at > org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:173) > at > org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:390) > at > org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:627) > at > org.apache.hadoop.ipc.Client$Connection.access$2300(Client.java:421) > at > org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:814) > at > org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:810) > at > java.security.AccessController.doPrivileged(Native Method) > at > javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) > at > org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:810) > ... 23 more > 2020-07-17 10:12:58,525 WARN ActionStartXCommand:523 - > SERVER[master] USER[oozie] GROUP[-] TOKEN[] APP[shell-wf] > JOB[0000001-200717100610211-oozie-root-W] > ACTION[0000001-200717100610211-oozie-root-W@shell-node] Error starting > action [shell-node]. ErrorType [TRANSIENT], ErrorCode [JA009], Message > [JA009: DestHost:destPort master:10020 , LocalHost:localPort master/ > 28.11.1.14:0. Failed on local exception: java.io.IOException: > org.apache.hadoop.security.AccessControlException: Client cannot > authenticate via:[TOKEN, KERBEROS]] > org.apache.oozie.action.ActionExecutorException: JA009: DestHost:destPort > master:10020 , LocalHost:localPort master/28.11.1.14:0. Failed on local > exception: java.io.IOException: > org.apache.hadoop.security.AccessControlException: Client cannot > authenticate via:[TOKEN, KERBEROS] > at > org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:463) > at > org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:443) > at > org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1134) > at > org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1644) > at org.apache.oozie.command.wf > .ActionStartXCommand.execute(ActionStartXCommand.java:243) > at org.apache.oozie.command.wf > .ActionStartXCommand.execute(ActionStartXCommand.java:68) > at > org.apache.oozie.command.XCommand.call(XCommand.java:290) > at > org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:363) > at > org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:292) > at > java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:210) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.io.IOException: DestHost:destPort master:10020 , > LocalHost:localPort master/28.11.1.14:0. Failed on local exception: > java.io.IOException: org.apache.hadoop.security.AccessControlException: > Client cannot authenticate via:[TOKEN, KERBEROS] > at > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at > java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at org.apache.hadoop.net > .NetUtils.wrapWithMessage(NetUtils.java:833) > at org.apache.hadoop.net > .NetUtils.wrapException(NetUtils.java:808) > at > org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1549) > at > org.apache.hadoop.ipc.Client.call(Client.java:1491) > at > org.apache.hadoop.ipc.Client.call(Client.java:1388) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) > at > com.sun.proxy.$Proxy42.getDelegationToken(Unknown Source) > at > org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getDelegationToken(MRClientProtocolPBClientImpl.java:211) > at > org.apache.oozie.action.hadoop.JHSCredentials.getDelegationTokenFromJHS(JHSCredentials.java:88) > at > org.apache.oozie.action.hadoop.JHSCredentials.updateCredentials(JHSCredentials.java:65) > at > org.apache.oozie.action.hadoop.JavaActionExecutor.setCredentialTokens(JavaActionExecutor.java:1546) > at > org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1082) > ... 11 more > Caused by: java.io.IOException: > org.apache.hadoop.security.AccessControlException: Client cannot > authenticate via:[TOKEN, KERBEROS] > at > org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:770) > at > java.security.AccessController.doPrivileged(Native Method) > at > javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) > at > org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:733) > at > org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:827) > at > org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:421) > at > org.apache.hadoop.ipc.Client.getConnection(Client.java:1606) > at > org.apache.hadoop.ipc.Client.call(Client.java:1435) > ... 20 more > Caused by: org.apache.hadoop.security.AccessControlException: Client > cannot authenticate via:[TOKEN, KERBEROS] > at > org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:173) > at > org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:390) > at > org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:627) > at > org.apache.hadoop.ipc.Client$Connection.access$2300(Client.java:421) > at > org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:814) > at > org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:810) > at > java.security.AccessController.doPrivileged(Native Method) > at > javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) > at > org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:810) > ... 23 more > 2020-07-17 10:12:58,527 INFO ActionStartXCommand:520 - > SERVER[master] USER[oozie] GROUP[-] TOKEN[] APP[shell-wf] > JOB[0000001-200717100610211-oozie-root-W] > ACTION[0000001-200717100610211-oozie-root-W@shell-node] Next Retry, > Attempt Number [1] in [10,000] milliseconds > 2020-07-17 10:12:58,527 INFO ActionStartXCommand:520 - > SERVER[master] USER[oozie] GROUP[-] TOKEN[] APP[shell-wf] > JOB[0000001-200717100610211-oozie-root-W] > ACTION[0000001-200717100610211-oozie-root-W@shell-node] Next Retry, > Attempt Number [1] in [10,000] milliseconds > > > The cluster environment information is as follows: > oozie version is 5.2.0 > hadoop version is 3.2.1 > the authentication mode is kerberos. > > > This problem also occurs when running other types of > tasks, such as MapReduce. > Can anyone tell me other ways to solve this problem? > > thinks. > I am looking forward to your reply！
