Hello Matthew, I think Subroto is correct that HADOOP-10786 is the likely root cause. Note that even though the issue was filed originally against JDK 8, it appears that the JDK changes that broke Hadoop's relogin functionality eventually trickled down to JDK 7 too. Here is a comment on the JIRA from me, describing my experiences with the bug on JDK 7.
https://issues.apache.org/jira/browse/HADOOP-10786?focusedCommentId=14694182&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14694182 Just like you, my observation is that running with JDK 1.7.0_79 does not have the problem, so it must be the _80 revision that introduced the regression. The known fixes for this problem are either to keep running with JDK 1.7.0_79 or upgrade to Apache Hadoop 2.6.1 or 2.7.0 to pick up the code change we did on our side to work around it. I hope this helps. --Chris Nauroth From: Subroto Sanyal <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Thursday, October 8, 2015 at 8:33 AM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: Secured Hadoop Cluster with Java 1.7.0_80 - Failure re-login with keytabs Hi Matthew You can check if you are hitting into: https://issues.apache.org/jira/browse/HADOOP-10786 Cheers, Subroto Sanyal On Thu, Oct 8, 2015 at 5:11 PM, Matthew Bruce <[email protected]<mailto:[email protected]>> wrote: Hello Hadoop Users, We have been doing java upgrade testing in one of our Hadoop lab environments and have run into issues using Oracle java 1.7.0_80 with a secured Hadoop cluster (Since the initial issue, we've verified this in a second environment too). Basically all the Hadoop components once the initial Kerberos ticket has expired fail to re-login using their keytab files (note that I've found nothing in the logs indicating why this happens, it seems like the components don't even attempt to re-login). Moreover I've verified that this behavior does not occur with java 1.7.0_79. The only thing I've been able to find that might be realted/cause this is this blurb in the u80 release notes: Issues with Third party's JCE Providers The fix for JDK-8023069 updated both the SunJSSE and and SunJCE providers, including some internal interfaces. Some third party JCE providers (such as RSA JSAFE) are using some sun.* internal interfaces, and therefore will not work with the updated SunJSSE provider. Such providers will need to be updated in order for them to work with the updated SunJSSE provider. If you have been impacted by this issue, contact your JCE vendor for an update. See 8133502<http://bugs.java.com/view_bug.do?bug_id=8133502>. I'm wondering, has anyone else run into issues running a secured Hadoop cluster with java 1.7.0_80? Thanks, Matthew Bruce [email protected]<mailto:[email protected]>
