[
https://issues.apache.org/jira/browse/AMBARI-12155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14617143#comment-14617143
]
Zack Marsh commented on AMBARI-12155:
-------------------------------------
The falcon.application.log contains the following exceptions:
{code}
2015-07-07 14:10:18,476 ERROR - [LaterunHandler:] ~ Error getting the message
from ActiveMQ (DelayedQueue:88)
javax.jms.JMSException: java.io.EOFException
at
org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:62)
at
org.apache.activemq.ActiveMQMessageConsumer.dequeue(ActiveMQMessageConsumer.java:458)
at
org.apache.activemq.ActiveMQMessageConsumer.receive(ActiveMQMessageConsumer.java:504)
at org.apache.falcon.rerun.queue.ActiveMQueue.take(ActiveMQueue.java:81)
at
org.apache.falcon.rerun.handler.AbstractRerunHandler.takeFromQueue(AbstractRerunHandler.java:66)
at
org.apache.falcon.rerun.handler.AbstractRerunConsumer.run(AbstractRerunConsumer.java:57)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:227)
at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:219)
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:202)
... 1 more
{code}
{code}
2015-07-07 14:10:18,483 ERROR - [LaterunHandler:] ~ Error while reading message
from the queue (AbstractRerunConsumer:60)
org.apache.falcon.FalconException: Error getting the message from ActiveMQ:
at org.apache.falcon.rerun.queue.ActiveMQueue.take(ActiveMQueue.java:89)
at
org.apache.falcon.rerun.handler.AbstractRerunHandler.takeFromQueue(AbstractRerunHandler.java:66)
at
org.apache.falcon.rerun.handler.AbstractRerunConsumer.run(AbstractRerunConsumer.java:57)
at java.lang.Thread.run(Thread.java:745)
Caused by: javax.jms.JMSException: java.io.EOFException
at
org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:62)
at
org.apache.activemq.ActiveMQMessageConsumer.dequeue(ActiveMQMessageConsumer.java:458)
at
org.apache.activemq.ActiveMQMessageConsumer.receive(ActiveMQMessageConsumer.java:504)
at org.apache.falcon.rerun.queue.ActiveMQueue.take(ActiveMQueue.java:81)
... 3 more
Caused by: java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:227)
at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:219)
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:202)
... 1 more
{code}
{code}
2015-07-07 14:19:09,308 ERROR - [main:] ~ Nested in
javax.servlet.ServletException: java.lang.IllegalArgumentException: Invalid
rule: \: (log:87)
java.lang.IllegalArgumentException: Invalid rule: \
at
org.apache.hadoop.security.authentication.util.KerberosName.parseRules(KerberosName.java:331)
at
org.apache.hadoop.security.authentication.util.KerberosName.setRules(KerberosName.java:397)
at
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.init(KerberosAuthenticationHandler.java:210)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.initializeAuthHandler(AuthenticationFilter.java:238)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.init(AuthenticationFilter.java:227)
at
org.apache.falcon.security.FalconAuthenticationFilter.init(FalconAuthenticationFilter.java:82)
at org.mortbay.jetty.servlet.FilterHolder.doStart(FilterHolder.java:97)
at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at
org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:713)
at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
at
org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1282)
at
org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:519)
at
org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:499)
at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at
org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
at org.mortbay.jetty.Server.doStart(Server.java:224)
at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.apache.falcon.util.EmbeddedServer.start(EmbeddedServer.java:57)
at org.apache.falcon.Main.main(Main.java:83)
{code}
> Falcon Service Check fails after enabling Kerberos, disabling Kerberos, and
> re-enabling Kerberos
> ------------------------------------------------------------------------------------------------
>
> Key: AMBARI-12155
> URL: https://issues.apache.org/jira/browse/AMBARI-12155
> Project: Ambari
> Issue Type: Bug
> Environment: ambari-2.1.0-1249, hdp-2.3.0.0-2469 , sles11sp3
> Reporter: Zack Marsh
> Priority: Critical
> Attachments: falcon.application.log, falcon.metric.log,
> falcon.out.2015070703371436254644, falcon.out.2015070703471436255250,
> falcon.out.2015070713211436289672, falcon.out.2015070713421436290933,
> falcon.out.2015070714041436292295, falcon.out.2015070714191436293145,
> falcon.security.audit.log
>
>
> When Kerberos is enabled, disabled, and re-enabled on an HDP-2.3 cluster, the
> Falcon Service Check is failing during the final "Start and Test All
> Services" step of the Enable Kerberos Wizard. The Falcon Service Check is
> also failing consistently after the wizard is complete.
> Error output as seen in Ambari
> stderr:
> {code}
> Traceback (most recent call last):
> File
> "/var/lib/ambari-agent/cache/common-services/FALCON/0.5.0.2.1/package/scripts/service_check.py",
> line 53, in <module>
> FalconServiceCheck().execute()
> File
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
> line 216, in execute
> method(env)
> File
> "/var/lib/ambari-agent/cache/common-services/FALCON/0.5.0.2.1/package/scripts/service_check.py",
> line 40, in service_check
> try_sleep = 20
> File "/usr/lib/python2.6/site-packages/resource_management/core/base.py",
> line 157, in __init__
> self.env.run()
> File
> "/usr/lib/python2.6/site-packages/resource_management/core/environment.py",
> line 152, in run
> self.run_action(resource, action)
> File
> "/usr/lib/python2.6/site-packages/resource_management/core/environment.py",
> line 118, in run_action
> provider_action()
> File
> "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py",
> line 254, in action_run
> tries=self.resource.tries, try_sleep=self.resource.try_sleep)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line 70, in inner
> result = function(command, **kwargs)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line 92, in checked_call
> tries=tries, try_sleep=try_sleep)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line 140, in _call_wrapper
> result = _call(command, **kwargs_copy)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line 291, in _call
> raise Fail(err_msg)
> resource_management.core.exceptions.Fail: Execution of
> '/usr/hdp/current/falcon-client/bin/falcon admin -version' returned 255.
> ERROR: Unable to initialize Falcon Client object
> {code}
> There is also a persistent Warning Alert for the Falcon Service in Ambari:
> {code}
> Falcon Server Web UI
> HTTP 503 response from http://jolokia1.labs.teradata.com:15000 in 0.000s ( %
> Total % Received % Xferd Average Speed Time Time Time Current Dload Upload
> Total Spent Left Speed 107 1288 107 1288 0 0 933k 0 --:--:-- --:--:--
> --:--:-- 933k 107 1288 107 1288 0 0 244k 0 --:--:-- --:--:-- --:--:-- 0)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
