The RM issue should be fixed by YARN-1600 which I just committed this
morning.
Jason
On 01/29/2014 10:33 AM, Steve Loughran wrote:
I'm just switching over to use the 2.4-SNAPSHOT in a secured pseudo-dist
cluster, and now the services are failing to come up because the web
principals haven't been defined. Example
2014-01-29 15:42:58,558 INFO org.apache.hadoop.http.HttpServer2: Added
filter static_user_filter (class=org.apache.hadoop.ht
tp.lib.StaticUserWebFilter$StaticUserFilter) to context hdfs
2014-01-29 15:42:58,559 INFO org.apache.hadoop.http.HttpServer2: Added
filter static_user_filter (class=org.apache.hadoop.ht
tp.lib.StaticUserWebFilter$StaticUserFilter) to context static
2014-01-29 15:42:58,559 INFO org.apache.hadoop.http.HttpServer2: Added
filter static_user_filter (class=org.apache.hadoop.ht
tp.lib.StaticUserWebFilter$StaticUserFilter) to context logs
2014-01-29 15:42:58,630 ERROR org.apache.hadoop.http.HttpServer2: *WebHDFS
and security are enabled, but configuration proper*
*ty 'dfs.web.authentication.kerberos.principal' is not set.*
2014-01-29 15:42:58,630 INFO org.apache.hadoop.http.HttpServer2: Added
filter 'SPNEGO' (class=org.apache.hadoop.hdfs.web.Aut
hFilter)
2014-01-29 15:42:58,631 INFO org.apache.hadoop.http.HttpServer2:
addJerseyResourcePackage: packageName=org.apache.hadoop.hdf
s.server.namenode.web.resources;org.apache.hadoop.hdfs.web.resources,
pathSpec=/webhdfs/v1/*
2014-01-29 15:42:58,658 INFO org.apache.hadoop.http.HttpServer2: Adding
Kerberos (SPNEGO) filter to getDelegationToken
2014-01-29 15:42:58,662 INFO org.apache.hadoop.http.HttpServer2: Adding
Kerberos (SPNEGO) filter to renewDelegationToken
2014-01-29 15:42:58,663 INFO org.apache.hadoop.http.HttpServer2: Adding
Kerberos (SPNEGO) filter to cancelDelegationToken
2014-01-29 15:42:58,663 INFO org.apache.hadoop.http.HttpServer2: Adding
Kerberos (SPNEGO) filter to fsck
2014-01-29 15:42:58,671 INFO org.apache.hadoop.http.HttpServer2: Adding
Kerberos (SPNEGO) filter to getimage
2014-01-29 15:42:58,748 INFO org.apache.hadoop.http.HttpServer2: Jetty
bound to port 50070
2014-01-29 15:42:58,748 INFO org.mortbay.log: jetty-6.1.26
2014-01-29 15:42:58,941 INFO
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler:
Login using key
tab /home/stevel/conf/hdfs.keytab, for principal HTTP/ubuntu@COTHAM
2014-01-29 15:42:58,981 INFO
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler:
Initialized, pr
incipal [HTTP/ubuntu@COTHAM] from keytab [/home/stevel/conf/hdfs.keytab]
2014-01-29 15:42:58,981 WARN
org.apache.hadoop.security.authentication.server.AuthenticationFilter:
'signature.secret' confi
guration not set, using a random value as secret
2014-01-29 15:42:58,982 WARN org.mortbay.log: failed SPNEGO:
javax.servlet.ServletException: javax.servlet.ServletException:
Principal not defined in configuration
2014-01-29 15:42:58,982 WARN org.mortbay.log: Failed startup of context
org.mortbay.jetty.webapp.WebAppContext@167a465{/,fil
e:/home/stevel/hadoop/share/hadoop/hdfs/webapps/hdfs}
javax.servlet.ServletException: javax.servlet.ServletException: Principal
not defined in configuration
at
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.init(KerberosAuthenticationHandler
.java:203)
:
YARN is the same but without the text telling me what config option I have
to set (i.e no equivalent of https://issues.apache.org/jira/browse/HDFS-3813)
-29 16:04:33,908 INFO
org.apache.hadoop.yarn.util.AbstractLivelinessMonitor: AMLivelinessMonitor
thread interrupted
2014-01-29 16:04:33,908 INFO
org.apache.hadoop.yarn.util.AbstractLivelinessMonitor:
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.ContainerAllocationExpirer
thread interrupted
2014-01-29 16:04:33,908 ERROR
org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
InterruptedExcpetion recieved for ExpiredTokenRemover thread
java.lang.InterruptedException: sleep interrupted
2014-01-29 16:04:33,909 INFO
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Transitioned
to standby state
2014-01-29 16:04:33,909 FATAL
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error
starting ResourceManager
org.apache.hadoop.yarn.webapp.WebAppException: Error starting http server
at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:250)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startWepApp(ResourceManager.java:775)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:866)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:995)
Caused by: java.io.IOException: Unable to initialize WebAppContext
at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:809)
at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:245)
... 4 more
Caused by: javax.servlet.ServletException: javax.servlet.ServletException:
Principal not defined in configuration
at
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.init(KerberosAuthenticationHandler.java:203)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.init(AuthenticationFilter.java:146)
at org.mortbay.jetty.servlet.FilterHolder.doStart(FilterHolder.java:97)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at
org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:713)
at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
at
org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1282)
at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:518)
at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:499)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at
org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
at
org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
at org.mortbay.jetty.Server.doStart(Server.java:224)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:787)
... 5 more
Caused by: javax.servlet.ServletException: Principal not defined in
configuration
at
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.init(KerberosAuthenticationHandler.java:164)
... 21 more
This cluster config (
https://github.com/hortonworks/hoya/tree/master/hoya-funtest/src/test/configs/ubuntu-secure)
did work on Hadoop-2.2, which implies that there is some change in
Hadoop
2.4 that is incompatible to the extent that you now need to declare some
principals you didn't need to set now need to be
1. it Looks like HDFS-3604, enable webhdfs is the issue that stops the NN
and DN -IMO it should be marked as an incompatible change, as it stops
clusters that used to start, from starting.
2. what's causing this in YARN?
