[jira] [Commented] (HADOOP-17996) UserGroupInformation#unprotectedRelogin sets the last login time before logging in
[ https://issues.apache.org/jira/browse/HADOOP-17996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17655822#comment-17655822 ] Surendra Singh Lilhore commented on HADOOP-17996: - Re-login in server handled as part of HADOOP-18581. > UserGroupInformation#unprotectedRelogin sets the last login time before > logging in > -- > > Key: HADOOP-17996 > URL: https://issues.apache.org/jira/browse/HADOOP-17996 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 3.3.1 >Reporter: Prabhu Joseph >Assignee: Ravuri Sushma sree >Priority: Major > Attachments: HADOOP-17996.001.patch > > > UserGroupInformation#unprotectedRelogin sets the last login time before > logging in. IPC#Client does reloginFromKeytab when there is a connection > reset failure from AD which does logout and set the last login time to now > and then tries to login. The login also fails as not able to connect to AD. > Then the reattempts does not happen as kerberosMinSecondsBeforeRelogin check > fails. All Client and Server operations fails with *GSS initiate failed* > {code} > 2021-10-31 09:50:53,546 WARN ha.EditLogTailer - Unable to trigger a roll of > the active NN > java.util.concurrent.ExecutionException: > org.apache.hadoop.security.KerberosAuthException: DestHost:destPort > namenode0:8020 , LocalHost:localPort namenode1/1.2.3.4:0. Failed on local > exception: org.apache.hadoop.security.KerberosAuthException: Login failure > for user: nn/nameno...@example.com javax.security.auth.login.LoginException: > Connection reset > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:206) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll(EditLogTailer.java:382) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:441) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$400(EditLogTailer.java:410) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:427) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:360) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1712) > at > org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:480) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:423) > Caused by: org.apache.hadoop.security.KerberosAuthException: > DestHost:destPort namenode0:8020 , LocalHost:localPort namenode1/1.2.3.4:0. > Failed on local exception: org.apache.hadoop.security.KerberosAuthException: > Login failure for user: nn/nameno...@example.com > javax.security.auth.login.LoginException: Connection reset > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831) > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:806) > at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1501) > at org.apache.hadoop.ipc.Client.call(Client.java:1443) > at org.apache.hadoop.ipc.Client.call(Client.java:1353) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) > at com.sun.proxy.$Proxy21.rollEditLog(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(NamenodeProtocolTranslatorPB.java:150) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$2.doWork(EditLogTailer.java:367) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$2.doWork(EditLogTailer.java:364) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$MultipleNameNodeProxy.call(EditLogTailer.java:514) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by:
[jira] [Commented] (HADOOP-17996) UserGroupInformation#unprotectedRelogin sets the last login time before logging in
[ https://issues.apache.org/jira/browse/HADOOP-17996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17655258#comment-17655258 ] Bhavik Patel commented on HADOOP-17996: --- any plan to merge this Jira? > UserGroupInformation#unprotectedRelogin sets the last login time before > logging in > -- > > Key: HADOOP-17996 > URL: https://issues.apache.org/jira/browse/HADOOP-17996 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 3.3.1 >Reporter: Prabhu Joseph >Assignee: Ravuri Sushma sree >Priority: Major > Attachments: HADOOP-17996.001.patch > > > UserGroupInformation#unprotectedRelogin sets the last login time before > logging in. IPC#Client does reloginFromKeytab when there is a connection > reset failure from AD which does logout and set the last login time to now > and then tries to login. The login also fails as not able to connect to AD. > Then the reattempts does not happen as kerberosMinSecondsBeforeRelogin check > fails. All Client and Server operations fails with *GSS initiate failed* > {code} > 2021-10-31 09:50:53,546 WARN ha.EditLogTailer - Unable to trigger a roll of > the active NN > java.util.concurrent.ExecutionException: > org.apache.hadoop.security.KerberosAuthException: DestHost:destPort > namenode0:8020 , LocalHost:localPort namenode1/1.2.3.4:0. Failed on local > exception: org.apache.hadoop.security.KerberosAuthException: Login failure > for user: nn/nameno...@example.com javax.security.auth.login.LoginException: > Connection reset > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:206) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll(EditLogTailer.java:382) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:441) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$400(EditLogTailer.java:410) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:427) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:360) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1712) > at > org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:480) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:423) > Caused by: org.apache.hadoop.security.KerberosAuthException: > DestHost:destPort namenode0:8020 , LocalHost:localPort namenode1/1.2.3.4:0. > Failed on local exception: org.apache.hadoop.security.KerberosAuthException: > Login failure for user: nn/nameno...@example.com > javax.security.auth.login.LoginException: Connection reset > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831) > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:806) > at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1501) > at org.apache.hadoop.ipc.Client.call(Client.java:1443) > at org.apache.hadoop.ipc.Client.call(Client.java:1353) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) > at com.sun.proxy.$Proxy21.rollEditLog(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(NamenodeProtocolTranslatorPB.java:150) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$2.doWork(EditLogTailer.java:367) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$2.doWork(EditLogTailer.java:364) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$MultipleNameNodeProxy.call(EditLogTailer.java:514) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: org.apache.hadoop.security.KerberosAuthException: Login
[jira] [Commented] (HADOOP-17996) UserGroupInformation#unprotectedRelogin sets the last login time before logging in
[ https://issues.apache.org/jira/browse/HADOOP-17996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17608968#comment-17608968 ] Ashutosh Gupta commented on HADOOP-17996: - Thanks [~surendralilhore]. I will change the title and raise the PR for this change. Please help in reviewing it in your free slots. > UserGroupInformation#unprotectedRelogin sets the last login time before > logging in > -- > > Key: HADOOP-17996 > URL: https://issues.apache.org/jira/browse/HADOOP-17996 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 3.3.1 >Reporter: Prabhu Joseph >Assignee: Ravuri Sushma sree >Priority: Major > Attachments: HADOOP-17996.001.patch > > > UserGroupInformation#unprotectedRelogin sets the last login time before > logging in. IPC#Client does reloginFromKeytab when there is a connection > reset failure from AD which does logout and set the last login time to now > and then tries to login. The login also fails as not able to connect to AD. > Then the reattempts does not happen as kerberosMinSecondsBeforeRelogin check > fails. All Client and Server operations fails with *GSS initiate failed* > {code} > 2021-10-31 09:50:53,546 WARN ha.EditLogTailer - Unable to trigger a roll of > the active NN > java.util.concurrent.ExecutionException: > org.apache.hadoop.security.KerberosAuthException: DestHost:destPort > namenode0:8020 , LocalHost:localPort namenode1/1.2.3.4:0. Failed on local > exception: org.apache.hadoop.security.KerberosAuthException: Login failure > for user: nn/nameno...@example.com javax.security.auth.login.LoginException: > Connection reset > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:206) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll(EditLogTailer.java:382) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:441) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$400(EditLogTailer.java:410) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:427) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:360) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1712) > at > org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:480) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:423) > Caused by: org.apache.hadoop.security.KerberosAuthException: > DestHost:destPort namenode0:8020 , LocalHost:localPort namenode1/1.2.3.4:0. > Failed on local exception: org.apache.hadoop.security.KerberosAuthException: > Login failure for user: nn/nameno...@example.com > javax.security.auth.login.LoginException: Connection reset > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831) > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:806) > at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1501) > at org.apache.hadoop.ipc.Client.call(Client.java:1443) > at org.apache.hadoop.ipc.Client.call(Client.java:1353) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) > at com.sun.proxy.$Proxy21.rollEditLog(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(NamenodeProtocolTranslatorPB.java:150) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$2.doWork(EditLogTailer.java:367) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$2.doWork(EditLogTailer.java:364) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$MultipleNameNodeProxy.call(EditLogTailer.java:514) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at
[jira] [Commented] (HADOOP-17996) UserGroupInformation#unprotectedRelogin sets the last login time before logging in
[ https://issues.apache.org/jira/browse/HADOOP-17996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17608967#comment-17608967 ] Surendra Singh Lilhore commented on HADOOP-17996: - [~groot], Yes we can. Please change the Jira title. > UserGroupInformation#unprotectedRelogin sets the last login time before > logging in > -- > > Key: HADOOP-17996 > URL: https://issues.apache.org/jira/browse/HADOOP-17996 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 3.3.1 >Reporter: Prabhu Joseph >Assignee: Ravuri Sushma sree >Priority: Major > Attachments: HADOOP-17996.001.patch > > > UserGroupInformation#unprotectedRelogin sets the last login time before > logging in. IPC#Client does reloginFromKeytab when there is a connection > reset failure from AD which does logout and set the last login time to now > and then tries to login. The login also fails as not able to connect to AD. > Then the reattempts does not happen as kerberosMinSecondsBeforeRelogin check > fails. All Client and Server operations fails with *GSS initiate failed* > {code} > 2021-10-31 09:50:53,546 WARN ha.EditLogTailer - Unable to trigger a roll of > the active NN > java.util.concurrent.ExecutionException: > org.apache.hadoop.security.KerberosAuthException: DestHost:destPort > namenode0:8020 , LocalHost:localPort namenode1/1.2.3.4:0. Failed on local > exception: org.apache.hadoop.security.KerberosAuthException: Login failure > for user: nn/nameno...@example.com javax.security.auth.login.LoginException: > Connection reset > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:206) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll(EditLogTailer.java:382) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:441) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$400(EditLogTailer.java:410) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:427) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:360) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1712) > at > org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:480) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:423) > Caused by: org.apache.hadoop.security.KerberosAuthException: > DestHost:destPort namenode0:8020 , LocalHost:localPort namenode1/1.2.3.4:0. > Failed on local exception: org.apache.hadoop.security.KerberosAuthException: > Login failure for user: nn/nameno...@example.com > javax.security.auth.login.LoginException: Connection reset > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831) > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:806) > at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1501) > at org.apache.hadoop.ipc.Client.call(Client.java:1443) > at org.apache.hadoop.ipc.Client.call(Client.java:1353) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) > at com.sun.proxy.$Proxy21.rollEditLog(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(NamenodeProtocolTranslatorPB.java:150) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$2.doWork(EditLogTailer.java:367) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$2.doWork(EditLogTailer.java:364) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$MultipleNameNodeProxy.call(EditLogTailer.java:514) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by:
[jira] [Commented] (HADOOP-17996) UserGroupInformation#unprotectedRelogin sets the last login time before logging in
[ https://issues.apache.org/jira/browse/HADOOP-17996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17608711#comment-17608711 ] Ashutosh Gupta commented on HADOOP-17996: - [~surendralilhore] - I think we can address handling Server side re-login in this same Jira cc:[~prabhujoseph] > UserGroupInformation#unprotectedRelogin sets the last login time before > logging in > -- > > Key: HADOOP-17996 > URL: https://issues.apache.org/jira/browse/HADOOP-17996 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 3.3.1 >Reporter: Prabhu Joseph >Assignee: Ravuri Sushma sree >Priority: Major > Attachments: HADOOP-17996.001.patch > > > UserGroupInformation#unprotectedRelogin sets the last login time before > logging in. IPC#Client does reloginFromKeytab when there is a connection > reset failure from AD which does logout and set the last login time to now > and then tries to login. The login also fails as not able to connect to AD. > Then the reattempts does not happen as kerberosMinSecondsBeforeRelogin check > fails. All Client and Server operations fails with *GSS initiate failed* > {code} > 2021-10-31 09:50:53,546 WARN ha.EditLogTailer - Unable to trigger a roll of > the active NN > java.util.concurrent.ExecutionException: > org.apache.hadoop.security.KerberosAuthException: DestHost:destPort > namenode0:8020 , LocalHost:localPort namenode1/1.2.3.4:0. Failed on local > exception: org.apache.hadoop.security.KerberosAuthException: Login failure > for user: nn/nameno...@example.com javax.security.auth.login.LoginException: > Connection reset > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:206) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll(EditLogTailer.java:382) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:441) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$400(EditLogTailer.java:410) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:427) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:360) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1712) > at > org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:480) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:423) > Caused by: org.apache.hadoop.security.KerberosAuthException: > DestHost:destPort namenode0:8020 , LocalHost:localPort namenode1/1.2.3.4:0. > Failed on local exception: org.apache.hadoop.security.KerberosAuthException: > Login failure for user: nn/nameno...@example.com > javax.security.auth.login.LoginException: Connection reset > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831) > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:806) > at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1501) > at org.apache.hadoop.ipc.Client.call(Client.java:1443) > at org.apache.hadoop.ipc.Client.call(Client.java:1353) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) > at com.sun.proxy.$Proxy21.rollEditLog(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(NamenodeProtocolTranslatorPB.java:150) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$2.doWork(EditLogTailer.java:367) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$2.doWork(EditLogTailer.java:364) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$MultipleNameNodeProxy.call(EditLogTailer.java:514) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at
[jira] [Commented] (HADOOP-17996) UserGroupInformation#unprotectedRelogin sets the last login time before logging in
[ https://issues.apache.org/jira/browse/HADOOP-17996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17463589#comment-17463589 ] Surendra Singh Lilhore commented on HADOOP-17996: - [~Sushma_28] and [~prabhujoseph]. Looks like this patch is trying to handle two scenario. # Set last login time after re-login in *UserGroupInformation#unprotectedRelogin().* # Handle re-login in Server when client and server running in same JVM and client trying to re-login but it failed. This impacted server also. #1 is absolutely not required and for this already configuration available if you want to reduce the time. #2 is different scenario and I tried reproducing it by adding some extra code in namenode. I added new thread which will logout in a 2 minute after namenode start and login again after waiting 2 minute. {code:java} new Thread() { public void run() { try { LOG.info("Logout from UGI"); Thread.sleep(12); UserGroupInformation.getLoginUser().getLogin().logout(); LOG.info("Waiting got 2 min"); Thread.sleep(12); LOG.info("Login again"); UserGroupInformation.getLoginUser().getLogin().login(); LOG.info("Relogin success.."); } catch (LoginException | IOException | InterruptedException e) { LOG.error("Failed log out thread ", e); } } }.start(); {code} For the 2 minute namenode not able to handle any client operation and keep on printing below exception. {code:java} Auth failed for x.x.x.x:42199:null (GSS initiate failed) with true cause: (GSS initiate failed) Auth failed for x.x.x.x:42199:null (GSS initiate failed) with true cause: (GSS initiate failed) Auth failed for x.x.x.x:42199:null (GSS initiate failed) with true cause: (GSS initiate failed) {code} I feel raise new Jira to handle Server side re-login and close this as Invalid. > UserGroupInformation#unprotectedRelogin sets the last login time before > logging in > -- > > Key: HADOOP-17996 > URL: https://issues.apache.org/jira/browse/HADOOP-17996 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 3.3.1 >Reporter: Prabhu Joseph >Assignee: Ravuri Sushma sree >Priority: Major > Attachments: HADOOP-17996.001.patch > > > UserGroupInformation#unprotectedRelogin sets the last login time before > logging in. IPC#Client does reloginFromKeytab when there is a connection > reset failure from AD which does logout and set the last login time to now > and then tries to login. The login also fails as not able to connect to AD. > Then the reattempts does not happen as kerberosMinSecondsBeforeRelogin check > fails. All Client and Server operations fails with *GSS initiate failed* > {code} > 2021-10-31 09:50:53,546 WARN ha.EditLogTailer - Unable to trigger a roll of > the active NN > java.util.concurrent.ExecutionException: > org.apache.hadoop.security.KerberosAuthException: DestHost:destPort > namenode0:8020 , LocalHost:localPort namenode1/1.2.3.4:0. Failed on local > exception: org.apache.hadoop.security.KerberosAuthException: Login failure > for user: nn/nameno...@example.com javax.security.auth.login.LoginException: > Connection reset > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:206) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll(EditLogTailer.java:382) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:441) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$400(EditLogTailer.java:410) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:427) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:360) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1712) > at > org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:480) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:423) > Caused by: org.apache.hadoop.security.KerberosAuthException: > DestHost:destPort namenode0:8020 , LocalHost:localPort namenode1/1.2.3.4:0. > Failed on local exception: org.apache.hadoop.security.KerberosAuthException: > Login failure for user: nn/nameno...@example.com > javax.security.auth.login.LoginException: Connection reset > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at >
[jira] [Commented] (HADOOP-17996) UserGroupInformation#unprotectedRelogin sets the last login time before logging in
[ https://issues.apache.org/jira/browse/HADOOP-17996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17448161#comment-17448161 ] Surendra Singh Lilhore commented on HADOOP-17996: - >> Yes it can be workaround by setting re-login attempt time to a lower value. >>Every user has to modify this value after facing this issue. Instead this >>patch improves that by reattempting if a previous login failed. This is not workaround. This property added to avoid load on KDC server. If you feel your clusters are not putting enough load on KDC then change default value to 0. Changing it to 0 is same as your patch. >>This Jira is an improvement. Do you see any problem/impact with this patch. yes, it will impact the KDC server where is shared by multiple cluster. All the processes will start re-login immediately and load will increase. >> Don't we immediately login into our laptop if the previous login failed? This is single user scenario, not for distributed system. :) > UserGroupInformation#unprotectedRelogin sets the last login time before > logging in > -- > > Key: HADOOP-17996 > URL: https://issues.apache.org/jira/browse/HADOOP-17996 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 3.3.1 >Reporter: Prabhu Joseph >Assignee: Ravuri Sushma sree >Priority: Major > Attachments: HADOOP-17996.001.patch > > > UserGroupInformation#unprotectedRelogin sets the last login time before > logging in. IPC#Client does reloginFromKeytab when there is a connection > reset failure from AD which does logout and set the last login time to now > and then tries to login. The login also fails as not able to connect to AD. > Then the reattempts does not happen as kerberosMinSecondsBeforeRelogin check > fails. All Client and Server operations fails with *GSS initiate failed* > {code} > 2021-10-31 09:50:53,546 WARN ha.EditLogTailer - Unable to trigger a roll of > the active NN > java.util.concurrent.ExecutionException: > org.apache.hadoop.security.KerberosAuthException: DestHost:destPort > namenode0:8020 , LocalHost:localPort namenode1/1.2.3.4:0. Failed on local > exception: org.apache.hadoop.security.KerberosAuthException: Login failure > for user: nn/nameno...@example.com javax.security.auth.login.LoginException: > Connection reset > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:206) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll(EditLogTailer.java:382) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:441) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$400(EditLogTailer.java:410) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:427) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:360) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1712) > at > org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:480) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:423) > Caused by: org.apache.hadoop.security.KerberosAuthException: > DestHost:destPort namenode0:8020 , LocalHost:localPort namenode1/1.2.3.4:0. > Failed on local exception: org.apache.hadoop.security.KerberosAuthException: > Login failure for user: nn/nameno...@example.com > javax.security.auth.login.LoginException: Connection reset > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831) > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:806) > at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1501) > at org.apache.hadoop.ipc.Client.call(Client.java:1443) > at org.apache.hadoop.ipc.Client.call(Client.java:1353) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) > at com.sun.proxy.$Proxy21.rollEditLog(Unknown Source) > at >
[jira] [Commented] (HADOOP-17996) UserGroupInformation#unprotectedRelogin sets the last login time before logging in
[ https://issues.apache.org/jira/browse/HADOOP-17996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17448083#comment-17448083 ] Prabhu Joseph commented on HADOOP-17996: [~surendralilhore] The issue in existing code is if a re-login failed for some reason then the retries to re-login will be skipped for next configured re-login attempt time. Yes it can be workaround by setting re-login attempt time to a lower value. Every user has to modify this value after facing this issue. Instead this patch improves that by reattempting if a previous login failed. Don't we immediately login into our laptop if the previous login failed? Do we wait for configured re-login attempt time after every login failure. If so, what is the use in waiting for that period? >> One question here, even after 60s second login was not successful ? Is this >> going in unnecessary loop ? It will be successful if AD is available. But for 60s, the HDFS Service is unavailable. All IPC Server and Client Operations will be failed with *GSS initiate failed*. This Jira is an improvement. Do you see any problem/impact with this patch. > UserGroupInformation#unprotectedRelogin sets the last login time before > logging in > -- > > Key: HADOOP-17996 > URL: https://issues.apache.org/jira/browse/HADOOP-17996 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 3.3.1 >Reporter: Prabhu Joseph >Assignee: Ravuri Sushma sree >Priority: Major > Attachments: HADOOP-17996.001.patch > > > UserGroupInformation#unprotectedRelogin sets the last login time before > logging in. IPC#Client does reloginFromKeytab when there is a connection > reset failure from AD which does logout and set the last login time to now > and then tries to login. The login also fails as not able to connect to AD. > Then the reattempts does not happen as kerberosMinSecondsBeforeRelogin check > fails. All Client and Server operations fails with *GSS initiate failed* > {code} > 2021-10-31 09:50:53,546 WARN ha.EditLogTailer - Unable to trigger a roll of > the active NN > java.util.concurrent.ExecutionException: > org.apache.hadoop.security.KerberosAuthException: DestHost:destPort > namenode0:8020 , LocalHost:localPort namenode1/1.2.3.4:0. Failed on local > exception: org.apache.hadoop.security.KerberosAuthException: Login failure > for user: nn/nameno...@example.com javax.security.auth.login.LoginException: > Connection reset > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:206) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll(EditLogTailer.java:382) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:441) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$400(EditLogTailer.java:410) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:427) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:360) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1712) > at > org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:480) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:423) > Caused by: org.apache.hadoop.security.KerberosAuthException: > DestHost:destPort namenode0:8020 , LocalHost:localPort namenode1/1.2.3.4:0. > Failed on local exception: org.apache.hadoop.security.KerberosAuthException: > Login failure for user: nn/nameno...@example.com > javax.security.auth.login.LoginException: Connection reset > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831) > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:806) > at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1501) > at org.apache.hadoop.ipc.Client.call(Client.java:1443) > at org.apache.hadoop.ipc.Client.call(Client.java:1353) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) > at >
[jira] [Commented] (HADOOP-17996) UserGroupInformation#unprotectedRelogin sets the last login time before logging in
[ https://issues.apache.org/jira/browse/HADOOP-17996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17447931#comment-17447931 ] Surendra Singh Lilhore commented on HADOOP-17996: - [~Sushma_28] , last login time is not successful login time, it is just time which indicate when login attempted. So I don't thing setting it after login make any sense. HADOOP-7930 allow you to change relogin attempt time if you need, by default it is 60 sec. One question here, even after 60s second login was not successful ? Is this going in unnecessary loop ? > UserGroupInformation#unprotectedRelogin sets the last login time before > logging in > -- > > Key: HADOOP-17996 > URL: https://issues.apache.org/jira/browse/HADOOP-17996 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 3.3.1 >Reporter: Prabhu Joseph >Assignee: Ravuri Sushma sree >Priority: Major > Attachments: HADOOP-17996.001.patch > > > UserGroupInformation#unprotectedRelogin sets the last login time before > logging in. IPC#Client does reloginFromKeytab when there is a connection > reset failure from AD which does logout and set the last login time to now > and then tries to login. The login also fails as not able to connect to AD. > Then the reattempts does not happen as kerberosMinSecondsBeforeRelogin check > fails. All Client and Server operations fails with *GSS initiate failed* > {code} > 2021-10-31 09:50:53,546 WARN ha.EditLogTailer - Unable to trigger a roll of > the active NN > java.util.concurrent.ExecutionException: > org.apache.hadoop.security.KerberosAuthException: DestHost:destPort > namenode0:8020 , LocalHost:localPort namenode1/1.2.3.4:0. Failed on local > exception: org.apache.hadoop.security.KerberosAuthException: Login failure > for user: nn/nameno...@example.com javax.security.auth.login.LoginException: > Connection reset > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:206) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll(EditLogTailer.java:382) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:441) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$400(EditLogTailer.java:410) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:427) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:360) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1712) > at > org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:480) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:423) > Caused by: org.apache.hadoop.security.KerberosAuthException: > DestHost:destPort namenode0:8020 , LocalHost:localPort namenode1/1.2.3.4:0. > Failed on local exception: org.apache.hadoop.security.KerberosAuthException: > Login failure for user: nn/nameno...@example.com > javax.security.auth.login.LoginException: Connection reset > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831) > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:806) > at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1501) > at org.apache.hadoop.ipc.Client.call(Client.java:1443) > at org.apache.hadoop.ipc.Client.call(Client.java:1353) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) > at com.sun.proxy.$Proxy21.rollEditLog(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(NamenodeProtocolTranslatorPB.java:150) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$2.doWork(EditLogTailer.java:367) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$2.doWork(EditLogTailer.java:364) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$MultipleNameNodeProxy.call(EditLogTailer.java:514) > at
[jira] [Commented] (HADOOP-17996) UserGroupInformation#unprotectedRelogin sets the last login time before logging in
[ https://issues.apache.org/jira/browse/HADOOP-17996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17446001#comment-17446001 ] Prabhu Joseph commented on HADOOP-17996: [~brahmareddy] If you are fine, we will go and commit this patch. > UserGroupInformation#unprotectedRelogin sets the last login time before > logging in > -- > > Key: HADOOP-17996 > URL: https://issues.apache.org/jira/browse/HADOOP-17996 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 3.3.1 >Reporter: Prabhu Joseph >Assignee: Ravuri Sushma sree >Priority: Major > Attachments: HADOOP-17996.001.patch > > > UserGroupInformation#unprotectedRelogin sets the last login time before > logging in. IPC#Client does reloginFromKeytab when there is a connection > reset failure from AD which does logout and set the last login time to now > and then tries to login. The login also fails as not able to connect to AD. > Then the reattempts does not happen as kerberosMinSecondsBeforeRelogin check > fails. All Client and Server operations fails with *GSS initiate failed* > {code} > 2021-10-31 09:50:53,546 WARN ha.EditLogTailer - Unable to trigger a roll of > the active NN > java.util.concurrent.ExecutionException: > org.apache.hadoop.security.KerberosAuthException: DestHost:destPort > namenode0:8020 , LocalHost:localPort namenode1/1.2.3.4:0. Failed on local > exception: org.apache.hadoop.security.KerberosAuthException: Login failure > for user: nn/nameno...@example.com javax.security.auth.login.LoginException: > Connection reset > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:206) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll(EditLogTailer.java:382) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:441) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$400(EditLogTailer.java:410) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:427) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:360) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1712) > at > org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:480) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:423) > Caused by: org.apache.hadoop.security.KerberosAuthException: > DestHost:destPort namenode0:8020 , LocalHost:localPort namenode1/1.2.3.4:0. > Failed on local exception: org.apache.hadoop.security.KerberosAuthException: > Login failure for user: nn/nameno...@example.com > javax.security.auth.login.LoginException: Connection reset > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831) > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:806) > at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1501) > at org.apache.hadoop.ipc.Client.call(Client.java:1443) > at org.apache.hadoop.ipc.Client.call(Client.java:1353) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) > at com.sun.proxy.$Proxy21.rollEditLog(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(NamenodeProtocolTranslatorPB.java:150) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$2.doWork(EditLogTailer.java:367) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$2.doWork(EditLogTailer.java:364) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$MultipleNameNodeProxy.call(EditLogTailer.java:514) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by:
[jira] [Commented] (HADOOP-17996) UserGroupInformation#unprotectedRelogin sets the last login time before logging in
[ https://issues.apache.org/jira/browse/HADOOP-17996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444021#comment-17444021 ] Prabhu Joseph commented on HADOOP-17996: Thanks [~brahmareddy] for reviewing the patch. {quote}this was just to track the re-login attempt so that so many retries can be avoided.? {quote} There are two issues the patch tries to address 1. When IPC#Client fails during {{{}saslConnect{}}}, it does re-login from {{{}handleSaslConnectionFailure{}}}. The re-login sets the last login time to current time irrespective of the login status, followed by logout and then login. When login fails for some reason like intermittent issue in connecting to AD, then all subsequent Client and Server operations will fail with GSS Initiate Failed for next configured {{kerberosMinSecondsBeforeLogin}} (60 seconds). {code:java} // try re-login if (UserGroupInformation.isLoginKeytabBased()) { UserGroupInformation.getLoginUser().reloginFromKeytab(); } else if (UserGroupInformation.isLoginTicketBased()) { UserGroupInformation.getLoginUser().reloginFromTicketCache(); } {code} This issue is addressed by setting the last login time to current time after the login succeeds. 2. Currently the re-login happens only from IPC#Client during {{{}handleSaslConnectionFailure(){}}}. Have observed cases where Client has logged out and have failed to login back leading to all IPC#Server operations failing in {{processSaslMessage}} with below error. {code:java} 2021-11-02 13:28:08,750 WARN ipc.Server - Auth failed for 10.25.35.45:37849:null (GSS initiate failed) with true cause: (GSS initiate failed) 2021-11-02 13:28:08,767 WARN ipc.Server - Auth failed for 10.25.35.46:35919:null (GSS initiate failed) with true cause: (GSS initiate failed) {code} This patch adds re-login from Server side as well during any Authentication Failure. bq. Configuring kerberosMinSecondsBeforeRelogin with low value will not work here if it's needed.? This will workaround the first issue. {quote} {quote}After this fix , on failure it will continuously retry..? {quote} IPC#Client does re-login during Connection Failure. This patch adds at IPC#Server side as well. Retries are based on the retry mechanism of IPC#Client and IPC#Server. The real kerberos login will happen for every retry from IPC#Client and IPC#Server till the login succeeds. > UserGroupInformation#unprotectedRelogin sets the last login time before > logging in > -- > > Key: HADOOP-17996 > URL: https://issues.apache.org/jira/browse/HADOOP-17996 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 3.3.1 >Reporter: Prabhu Joseph >Assignee: Ravuri Sushma sree >Priority: Major > Attachments: HADOOP-17996.001.patch > > > UserGroupInformation#unprotectedRelogin sets the last login time before > logging in. IPC#Client does reloginFromKeytab when there is a connection > reset failure from AD which does logout and set the last login time to now > and then tries to login. The login also fails as not able to connect to AD. > Then the reattempts does not happen as kerberosMinSecondsBeforeRelogin check > fails. All Client and Server operations fails with *GSS initiate failed* > {code} > 2021-10-31 09:50:53,546 WARN ha.EditLogTailer - Unable to trigger a roll of > the active NN > java.util.concurrent.ExecutionException: > org.apache.hadoop.security.KerberosAuthException: DestHost:destPort > namenode0:8020 , LocalHost:localPort namenode1/1.2.3.4:0. Failed on local > exception: org.apache.hadoop.security.KerberosAuthException: Login failure > for user: nn/nameno...@example.com javax.security.auth.login.LoginException: > Connection reset > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:206) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll(EditLogTailer.java:382) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:441) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$400(EditLogTailer.java:410) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:427) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:360) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1712) > at > org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:480) > at >
[jira] [Commented] (HADOOP-17996) UserGroupInformation#unprotectedRelogin sets the last login time before logging in
[ https://issues.apache.org/jira/browse/HADOOP-17996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17443809#comment-17443809 ] Brahma Reddy Battula commented on HADOOP-17996: --- [~prabhujoseph] and [~Sushma_28] thanks for reporting and working on this..IMO, this was just to track the re-login attempt so that so many retries can be avoided.? Configuring *kerberosMinSecondsBeforeRelogin* with low value will not work here if it's needed.? After this fix , on failure it will continuously retry..? > UserGroupInformation#unprotectedRelogin sets the last login time before > logging in > -- > > Key: HADOOP-17996 > URL: https://issues.apache.org/jira/browse/HADOOP-17996 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 3.3.1 >Reporter: Prabhu Joseph >Assignee: Ravuri Sushma sree >Priority: Major > Attachments: HADOOP-17996.001.patch > > > UserGroupInformation#unprotectedRelogin sets the last login time before > logging in. IPC#Client does reloginFromKeytab when there is a connection > reset failure from AD which does logout and set the last login time to now > and then tries to login. The login also fails as not able to connect to AD. > Then the reattempts does not happen as kerberosMinSecondsBeforeRelogin check > fails. All Client and Server operations fails with *GSS initiate failed* > {code} > 2021-10-31 09:50:53,546 WARN ha.EditLogTailer - Unable to trigger a roll of > the active NN > java.util.concurrent.ExecutionException: > org.apache.hadoop.security.KerberosAuthException: DestHost:destPort > namenode0:8020 , LocalHost:localPort namenode1/1.2.3.4:0. Failed on local > exception: org.apache.hadoop.security.KerberosAuthException: Login failure > for user: nn/nameno...@example.com javax.security.auth.login.LoginException: > Connection reset > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:206) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll(EditLogTailer.java:382) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:441) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$400(EditLogTailer.java:410) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:427) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:360) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1712) > at > org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:480) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:423) > Caused by: org.apache.hadoop.security.KerberosAuthException: > DestHost:destPort namenode0:8020 , LocalHost:localPort namenode1/1.2.3.4:0. > Failed on local exception: org.apache.hadoop.security.KerberosAuthException: > Login failure for user: nn/nameno...@example.com > javax.security.auth.login.LoginException: Connection reset > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831) > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:806) > at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1501) > at org.apache.hadoop.ipc.Client.call(Client.java:1443) > at org.apache.hadoop.ipc.Client.call(Client.java:1353) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) > at com.sun.proxy.$Proxy21.rollEditLog(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(NamenodeProtocolTranslatorPB.java:150) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$2.doWork(EditLogTailer.java:367) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$2.doWork(EditLogTailer.java:364) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$MultipleNameNodeProxy.call(EditLogTailer.java:514) > at
[jira] [Commented] (HADOOP-17996) UserGroupInformation#unprotectedRelogin sets the last login time before logging in
[ https://issues.apache.org/jira/browse/HADOOP-17996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17443619#comment-17443619 ] Prabhu Joseph commented on HADOOP-17996: Thanks [~Sushma_28] for the patch. The patch looks good to me, +1. Will commit it tomorrow if no other comments. > UserGroupInformation#unprotectedRelogin sets the last login time before > logging in > -- > > Key: HADOOP-17996 > URL: https://issues.apache.org/jira/browse/HADOOP-17996 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 3.3.1 >Reporter: Prabhu Joseph >Assignee: Ravuri Sushma sree >Priority: Major > Attachments: HADOOP-17996.001.patch > > > UserGroupInformation#unprotectedRelogin sets the last login time before > logging in. IPC#Client does reloginFromKeytab when there is a connection > reset failure from AD which does logout and set the last login time to now > and then tries to login. The login also fails as not able to connect to AD. > Then the reattempts does not happen as kerberosMinSecondsBeforeRelogin check > fails. All Client and Server operations fails with *GSS initiate failed* > {code} > 2021-10-31 09:50:53,546 WARN ha.EditLogTailer - Unable to trigger a roll of > the active NN > java.util.concurrent.ExecutionException: > org.apache.hadoop.security.KerberosAuthException: DestHost:destPort > namenode0:8020 , LocalHost:localPort namenode1/1.2.3.4:0. Failed on local > exception: org.apache.hadoop.security.KerberosAuthException: Login failure > for user: nn/nameno...@example.com javax.security.auth.login.LoginException: > Connection reset > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:206) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll(EditLogTailer.java:382) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:441) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$400(EditLogTailer.java:410) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:427) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:360) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1712) > at > org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:480) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:423) > Caused by: org.apache.hadoop.security.KerberosAuthException: > DestHost:destPort namenode0:8020 , LocalHost:localPort namenode1/1.2.3.4:0. > Failed on local exception: org.apache.hadoop.security.KerberosAuthException: > Login failure for user: nn/nameno...@example.com > javax.security.auth.login.LoginException: Connection reset > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831) > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:806) > at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1501) > at org.apache.hadoop.ipc.Client.call(Client.java:1443) > at org.apache.hadoop.ipc.Client.call(Client.java:1353) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) > at com.sun.proxy.$Proxy21.rollEditLog(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(NamenodeProtocolTranslatorPB.java:150) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$2.doWork(EditLogTailer.java:367) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$2.doWork(EditLogTailer.java:364) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$MultipleNameNodeProxy.call(EditLogTailer.java:514) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at
[jira] [Commented] (HADOOP-17996) UserGroupInformation#unprotectedRelogin sets the last login time before logging in
[ https://issues.apache.org/jira/browse/HADOOP-17996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17442674#comment-17442674 ] Hadoop QA commented on HADOOP-17996: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Logfile || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 44s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red}{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} trunk Compile Tests {color} || || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m 11s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 22m 22s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m 47s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 15s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 34s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 17m 19s{color} | {color:green}{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 39s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 22m 28s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are enabled, using SpotBugs. {color} | | {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 2m 23s{color} | {color:green}{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 56s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 23m 14s{color} | {color:green}{color} | {color:green} the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 23m 13s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 22m 21s{color} | {color:green}{color} | {color:green} the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 22m 21s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 10s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 44s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 17m 21s{color} | {color:green}{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 13s{color} | {color:green}{color} | {color:green} the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} | |
[jira] [Commented] (HADOOP-17996) UserGroupInformation#unprotectedRelogin sets the last login time before logging in
[ https://issues.apache.org/jira/browse/HADOOP-17996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17441761#comment-17441761 ] Ravuri Sushma sree commented on HADOOP-17996: - Uploaded a patch correcting the set last login time and also added a logic of re-login in Server.java similar to that of Client.java. Please review > UserGroupInformation#unprotectedRelogin sets the last login time before > logging in > -- > > Key: HADOOP-17996 > URL: https://issues.apache.org/jira/browse/HADOOP-17996 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 3.3.1 >Reporter: Prabhu Joseph >Assignee: Ravuri Sushma sree >Priority: Major > Attachments: HADOOP-17996.001.patch > > > UserGroupInformation#unprotectedRelogin sets the last login time before > logging in. IPC#Client does reloginFromKeytab when there is a connection > reset failure from AD which does logout and set the last login time to now > and then tries to login. The login also fails as not able to connect to AD. > Then the reattempts does not happen as kerberosMinSecondsBeforeRelogin check > fails. All Client and Server operations fails with *GSS initiate failed* > {code} > 2021-10-31 09:50:53,546 WARN ha.EditLogTailer - Unable to trigger a roll of > the active NN > java.util.concurrent.ExecutionException: > org.apache.hadoop.security.KerberosAuthException: DestHost:destPort > namenode0:8020 , LocalHost:localPort namenode1/1.2.3.4:0. Failed on local > exception: org.apache.hadoop.security.KerberosAuthException: Login failure > for user: nn/nameno...@example.com javax.security.auth.login.LoginException: > Connection reset > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:206) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll(EditLogTailer.java:382) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:441) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$400(EditLogTailer.java:410) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:427) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:360) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1712) > at > org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:480) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:423) > Caused by: org.apache.hadoop.security.KerberosAuthException: > DestHost:destPort namenode0:8020 , LocalHost:localPort namenode1/1.2.3.4:0. > Failed on local exception: org.apache.hadoop.security.KerberosAuthException: > Login failure for user: nn/nameno...@example.com > javax.security.auth.login.LoginException: Connection reset > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831) > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:806) > at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1501) > at org.apache.hadoop.ipc.Client.call(Client.java:1443) > at org.apache.hadoop.ipc.Client.call(Client.java:1353) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) > at com.sun.proxy.$Proxy21.rollEditLog(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(NamenodeProtocolTranslatorPB.java:150) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$2.doWork(EditLogTailer.java:367) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$2.doWork(EditLogTailer.java:364) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$MultipleNameNodeProxy.call(EditLogTailer.java:514) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at >
[jira] [Commented] (HADOOP-17996) UserGroupInformation#unprotectedRelogin sets the last login time before logging in
[ https://issues.apache.org/jira/browse/HADOOP-17996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17441316#comment-17441316 ] Ravuri Sushma sree commented on HADOOP-17996: - Thanks for filing this JIRA [~prabhujoseph] Yes, I think it makes sense to update the set last login time only after the login is successful instead of updating it irrespective of status of login. Will provide a fix soon > UserGroupInformation#unprotectedRelogin sets the last login time before > logging in > -- > > Key: HADOOP-17996 > URL: https://issues.apache.org/jira/browse/HADOOP-17996 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 3.3.1 >Reporter: Prabhu Joseph >Assignee: Ravuri Sushma sree >Priority: Major > > UserGroupInformation#unprotectedRelogin sets the last login time before > logging in. IPC#Client does reloginFromKeytab when there is a connection > reset failure from AD which does logout and set the last login time to now > and then tries to login. The login also fails as not able to connect to AD. > Then the reattempts does not happen as kerberosMinSecondsBeforeRelogin check > fails. All Client and Server operations fails with *GSS initiate failed* > {code} > 2021-10-31 09:50:53,546 WARN ha.EditLogTailer - Unable to trigger a roll of > the active NN > java.util.concurrent.ExecutionException: > org.apache.hadoop.security.KerberosAuthException: DestHost:destPort > namenode0:8020 , LocalHost:localPort namenode1/1.2.3.4:0. Failed on local > exception: org.apache.hadoop.security.KerberosAuthException: Login failure > for user: nn/nameno...@example.com javax.security.auth.login.LoginException: > Connection reset > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:206) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll(EditLogTailer.java:382) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:441) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$400(EditLogTailer.java:410) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:427) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:360) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1712) > at > org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:480) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:423) > Caused by: org.apache.hadoop.security.KerberosAuthException: > DestHost:destPort namenode0:8020 , LocalHost:localPort namenode1/1.2.3.4:0. > Failed on local exception: org.apache.hadoop.security.KerberosAuthException: > Login failure for user: nn/nameno...@example.com > javax.security.auth.login.LoginException: Connection reset > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831) > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:806) > at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1501) > at org.apache.hadoop.ipc.Client.call(Client.java:1443) > at org.apache.hadoop.ipc.Client.call(Client.java:1353) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) > at com.sun.proxy.$Proxy21.rollEditLog(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(NamenodeProtocolTranslatorPB.java:150) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$2.doWork(EditLogTailer.java:367) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$2.doWork(EditLogTailer.java:364) > at > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$MultipleNameNodeProxy.call(EditLogTailer.java:514) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at >