各位好,
我发现从flink1.16及以上版本开始,引入了HadoopFSDelegationTokenProvider
,此时在hadoop2.7.x开kerberos安全的情况下,flink on
yarn模式存在认证问题,报错如下。请问,有什么解决办法吗?或者flink高版本已经不再支持低版本hadoop了?2025-02-05
11:56:15,770 INFO org.apache.flink.yarn.YarnClusterDescriptor
[] - Adding delegation tokens to the AM container.
2025-02-05 11:56:15,773 INFO
org.apache.flink.runtime.security.token.KerberosDelegationTokenManager [] -
Loading delegation token providers
2025-02-05 11:56:15,782 INFO
org.apache.flink.runtime.security.token.KerberosDelegationTokenManager [] -
Delegation token provider hadoopfs loaded and initialized
2025-02-05 11:56:15,784 INFO
org.apache.flink.runtime.security.token.KerberosDelegationTokenManager [] -
Delegation token providers loaded successfully
2025-02-05 11:56:15,784 INFO
org.apache.flink.runtime.security.token.KerberosDelegationTokenManager [] -
Obtaining delegation tokens
2025-02-05 11:56:15,784 INFO
org.apache.flink.runtime.security.token.KerberosLoginProvider [] - Attempting
to login to KDC using principal: yarn@TDH keytab: /home/transwarper/yarn.keytab
2025-02-05 11:56:15,799 INFO
org.apache.flink.runtime.security.token.KerberosLoginProvider [] - Successfully
logged into KDC
2025-02-05 11:56:15,877 ERROR
org.apache.flink.runtime.security.token.HadoopFSDelegationTokenProvider [] -
Failed to obtain delegation token for
DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-225487396_1, ugi=yarn@TDH
(auth:KERBEROS)]]
java.lang.NullPointerException: null
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getDelegationToken(ClientNamenodeProtocolTranslatorPB.java:935)
~[flink-shaded-hadoop-2-uber-2.7.5-10.0.jar:2.7.5-10.0]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_262]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_262]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_262]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_262]
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
~[flink-shaded-hadoop-2-uber-2.7.5-10.0.jar:2.7.5-10.0]
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
~[flink-shaded-hadoop-2-uber-2.7.5-10.0.jar:2.7.5-10.0]
at com.sun.proxy.$Proxy31.getDelegationToken(Unknown Source) ~[?:?]