Hey Maxim, I forgot to mention that yesterday I killed all the running jobs in an attempt to get it working again. When I run aurora.pex job list <cluster>/<role>/<env> it just prints this out before handing me back a terminal:
INFO] Retrieving jobs for role produser Cheers, Josh On Tue, Mar 31, 2015 at 12:54 PM, Maxim Khutornenko <[email protected]> wrote: > Hi Josh, > > How many jobs do you have under that role? Any chance you could run > "aurora job status" for every one of them to see if you have any tasks > with abnormal event sequences? Every task must have at least one event > with a timestamp. Here is a reference example: > > events: > 2015-03-23 01:01:28 PENDING: Rescheduled > 2015-03-23 01:01:28 ASSIGNED: None > 2015-03-23 01:01:30 STARTING: Initializing sandbox. > 2015-03-23 01:01:31 RUNNING: None > > On Tue, Mar 31, 2015 at 12:17 PM, Josh Adams <[email protected]> wrote: > > Hi there, > > > > I'm getting this error when clicking on a Role on the /scheduler page. We > > only have a single role and this is our staging cluster. It seems like > > other operations like `aurora.pex job list ...` and similar commands work > > fine, but I can't seem to figure out how to get the scheduler out of this > > state so I can view the Job summary page. > > > > I've upgraded from Mesos/0.20.1 & Aurora/0.6.1 to Mesos/0.21.1 and > > Aurora/0.7.1@3ab8a9e and the problem persists. Below is the stack trace > > from the scheduler process. > > > > Thank you! > > Josh > > > > D0331 19:14:31.229 THREAD2827 org.eclipse.jetty.server.Server.handle: > > REQUEST /api on AsyncHttpConnection@6fa15c80 > > ,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=2,l=362,c=43},r=4 > > W0331 19:14:31.229 THREAD2828 > > org.apache.aurora.scheduler.thrift.aop.LoggingInterceptor.invoke: > Uncaught > > exception while handling getJobSummary(produser) > > com.google.common.collect.Ordering$IncomparableValueException: Cannot > > compare value: null > > at > > com.google.common.collect.ExplicitOrdering.rank(ExplicitOrdering.java:46) > > at > > > com.google.common.collect.ExplicitOrdering.compare(ExplicitOrdering.java:40) > > at > > > com.google.common.collect.ByFunctionOrdering.compare(ByFunctionOrdering.java:46) > > at > > > com.google.common.collect.CompoundOrdering.compare(CompoundOrdering.java:43) > > at com.google.common.collect.Ordering.max(Ordering.java:572) > > at com.google.common.collect.Ordering.max(Ordering.java:539) > > at com.google.common.collect.Ordering.max(Ordering.java:555) > > at > > > org.apache.aurora.scheduler.base.Tasks.getLatestActiveTask(Tasks.java:215) > > at > > > org.apache.aurora.scheduler.thrift.ReadOnlySchedulerImpl$8.transformEntry(ReadOnlySchedulerImpl.java:382) > > at > > > org.apache.aurora.scheduler.thrift.ReadOnlySchedulerImpl$8.transformEntry(ReadOnlySchedulerImpl.java:374) > > at com.google.common.collect.Maps$10.getValue(Maps.java:1855) > > at java.util.HashMap.putAll(HashMap.java:647) > > at > > > org.apache.aurora.scheduler.thrift.ReadOnlySchedulerImpl.getJobs(ReadOnlySchedulerImpl.java:373) > > at > > > org.apache.aurora.scheduler.thrift.ReadOnlySchedulerImpl.getJobSummary(ReadOnlySchedulerImpl.java:257) > > at > > > org.apache.aurora.scheduler.thrift.SchedulerThriftInterface.getJobSummary(SchedulerThriftInterface.java:461) > > at > > > org.apache.aurora.scheduler.thrift.aop.ThriftStatsExporterInterceptor.invoke(ThriftStatsExporterInterceptor.java:47) > > at > > > org.apache.aurora.scheduler.thrift.aop.FeatureToggleInterceptor.invoke(FeatureToggleInterceptor.java:38) > > at > > > org.apache.aurora.scheduler.thrift.aop.LoggingInterceptor.invoke(LoggingInterceptor.java:102) > > at > > > org.apache.aurora.scheduler.thrift.aop.ServerInfoInterceptor.invoke(ServerInfoInterceptor.java:30) > > at > > > org.apache.aurora.gen.ReadOnlyScheduler$Processor$getJobSummary.getResult(ReadOnlyScheduler.java:906) > > at > > > org.apache.aurora.gen.ReadOnlyScheduler$Processor$getJobSummary.getResult(ReadOnlyScheduler.java:891) > > at > > org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) > > at > org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) > > at org.apache.thrift.server.TServlet.doPost(TServlet.java:83) > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:727) > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) > > at > > > com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263) > > at > > > com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178) > > at > > > com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91) > > at > > > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62) > > at > > > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > > at > > > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > > at > > > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > > at > > > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > > at > > > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > > at > > > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > > at > > > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > > at > > > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > > at > > > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > > at > > > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > > at > > > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > > at > > > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > > at > > > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > > at > > > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > > at > > > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > > at > > > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > > at > > > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > > at > > > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > > at > > > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > > at > > > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > > at > > > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > > at > > > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > > at > > > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > > at > > > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > > at > > > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > > at > > > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > > at > > > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > > at > > > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > > at > > > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > > at > > > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > > at > > > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > > at > > > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > > at > > > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > > at > > > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > > at > > > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > > at > > > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > > at > > > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > > at > > > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > > at > > > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > > at > > > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > > at > > > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > > at > > > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > > at > > > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > > at > > > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > > at > > > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > > at > > > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > > at > > > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > > at > > > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > > at > > > org.apache.aurora.scheduler.http.HttpStatsFilter.doFilter(HttpStatsFilter.java:69) > > at > > > org.apache.aurora.scheduler.http.AbstractFilter.doFilter(AbstractFilter.java:44) > > at > > > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) > > at > > > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > > at > > > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > > at > > > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > > at > > > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > > at > > > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > > at > > > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > > at > > > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > > at > > > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > > at > > > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > > at > > > org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:82) > > at > > org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:294) > > at > > > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) > > at > > > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > > at > > > com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118) > > at > > com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113) > > at > > > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1288) > > at > > > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:443) > > at > > > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1044) > > at > > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:372) > > at > > > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:978) > > at > > > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) > > at > > > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154) > > at > > > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) > > at > > > org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:317) > > at > > > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) > > at org.eclipse.jetty.server.Server.handle(Server.java:365) > > at > > > org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:486) > > at > > > org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:944) > > at > > > org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1005) > > at > org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865) > > at > > org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240) > > at > > > org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) > > at > > > org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667) > > at > > > org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) > > at > > > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) > > at > > > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) > > at java.lang.Thread.run(Thread.java:744) >
