Hi, AFAIK, Flink does exclude the HDFS_DELEGATION_TOKEN in the HadoopModule when user provides the keytab and principal. I'll try to do a deeper investigation to figure out is there any HDFS access before the HadoopModule installed.
Best, Yangze Guo On Tue, Nov 17, 2020 at 4:36 PM Kien Truong <duckientru...@gmail.com> wrote: > > Hi, > > Yes, I did. There're also logs about logging in using keytab successfully in > both Job Manager and Task Manager. > > I found some YARN docs about token renewal on AM restart > > > > Therefore, to survive AM restart after token expiry, your AM has to get the > > NMs to localize the keytab or make no HDFS accesses until (somehow) a new > > token has been passed to them from a client. > > Maybe Flink did access HDFS with an expired token, before switching to use > the localized keytab ? > > Regards, > Kien > > > > On 17 Nov 2020 at 15:14, Yangze Guo <karma...@gmail.com> wrote: > > Hi, Kien, > > > > Do you config the "security.kerberos.login.principal" and the > > "security.kerberos.login.keytab" together? If you only set the keytab, > > it will not take effect. > > > > Best, > > Yangze Guo > > > > On Tue, Nov 17, 2020 at 3:03 PM Kien Truong <duckientru...@gmail.com> wrote: > > > > > > Hi all, > > > > > > We are having an issue where Flink Application Master is unable to > > automatically restart Flink job after its delegation token has expired. > > > > > > We are using Flink 1.11 with YARN 3.1.1 in single job per yarn-cluster > > mode. We have also add valid keytab configuration and taskmanagers are able > > to login with keytabs correctly. However, it seems YARN Application Master > > still use delegation tokens instead of the keytab. > > > > > > Any idea how to resolve this would be much appreciated. > > > > > > Thanks > > > Kien > > > > > > > > > > > > >
