David McLaughlin created AURORA-759:
---------------------------------------
Summary: Scheduler updater stuck in ROLLING_BACK
Key: AURORA-759
URL: https://issues.apache.org/jira/browse/AURORA-759
Project: Aurora
Issue Type: Story
Components: Scheduler
Reporter: David McLaughlin
When using the new updater, I ran into a situation where my update was stuck in
ROLLING_BACK state.
My sequence of events:
1) Create a job where your package is being fetched from a local directory.
2) Have that local directory wiped clean.
3) Update your job with a new config, with a bad path.
Stack traces from scheduler log:
{code}
I0926 18:50:56.888 THREAD278
org.apache.aurora.scheduler.updater.JobUpdateControllerImpl$4.execute:
Forwarding task change for www-data/devel/hello_world/1
E0926 18:50:56.888 THREAD278
org.apache.aurora.scheduler.updater.JobUpdateEventSubscriber.taskChangedState:
Failed to handle state change: java.lang.IllegalStateException: Attempted
to evaluate an inactive job updater.
java.lang.IllegalStateException: Attempted to evaluate an inactive job updater.
at
com.google.common.base.Preconditions.checkState(Preconditions.java:176)
at
org.apache.aurora.scheduler.updater.OneWayJobUpdater.evaluate(OneWayJobUpdater.java:145)
at
org.apache.aurora.scheduler.updater.JobUpdateControllerImpl.evaluateUpdater(JobUpdateControllerImpl.java:451)
at
org.apache.aurora.scheduler.updater.JobUpdateControllerImpl.access$700(JobUpdateControllerImpl.java:92)
at
org.apache.aurora.scheduler.updater.JobUpdateControllerImpl$4.execute(JobUpdateControllerImpl.java:239)
at
org.apache.aurora.scheduler.storage.Storage$MutateWork$NoResult.apply(Storage.java:132)
at
org.apache.aurora.scheduler.storage.Storage$MutateWork$NoResult$Quiet.apply(Storage.java:149)
at
org.apache.aurora.scheduler.storage.log.LogStorage.write(LogStorage.java:551)
at
org.apache.aurora.scheduler.storage.CallOrderEnforcingStorage.write(CallOrderEnforcingStorage.java:130)
at
org.apache.aurora.scheduler.updater.JobUpdateControllerImpl.instanceChanged(JobUpdateControllerImpl.java:231)
at
org.apache.aurora.scheduler.updater.JobUpdateControllerImpl.instanceChangedState(JobUpdateControllerImpl.java:216)
at
org.apache.aurora.scheduler.updater.JobUpdateEventSubscriber.taskChangedState(JobUpdateEventSubscriber.java:55)
at sun.reflect.GeneratedMethodAccessor39.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
com.google.common.eventbus.EventSubscriber.handleEvent(EventSubscriber.java:74)
at
com.google.common.eventbus.SynchronizedEventSubscriber.handleEvent(SynchronizedEventSubscriber.java:47)
at com.google.common.eventbus.EventBus.dispatch(EventBus.java:322)
at
com.google.common.eventbus.EventBus.dispatchQueuedEvents(EventBus.java:304)
at com.google.common.eventbus.EventBus.post(EventBus.java:275)
at
org.apache.aurora.scheduler.events.PubsubEventModule$2.post(PubsubEventModule.java:68)
at
org.apache.aurora.scheduler.state.StateManagerImpl.updateTaskAndExternalState(StateManagerImpl.java:443)
at
org.apache.aurora.scheduler.state.StateManagerImpl.access$100(StateManagerImpl.java:80)
at
org.apache.aurora.scheduler.state.StateManagerImpl$5.apply(StateManagerImpl.java:273)
at
org.apache.aurora.scheduler.state.StateManagerImpl$5.apply(StateManagerImpl.java:259)
at
org.apache.aurora.scheduler.storage.log.LogStorage$7.apply(LogStorage.java:559)
at
org.apache.aurora.scheduler.storage.log.LogStorage$7.apply(LogStorage.java:556)
at
org.apache.aurora.scheduler.storage.mem.MemStorage.doWork(MemStorage.java:175)
at
org.apache.aurora.scheduler.storage.mem.MemStorage.access$400(MemStorage.java:59)
at
org.apache.aurora.scheduler.storage.mem.MemStorage$4.apply(MemStorage.java:200)
at
org.apache.aurora.scheduler.storage.mem.MemStorage$4.apply(MemStorage.java:197)
{code}
{code}
java.lang.IllegalArgumentException: A terminal state should not specify
actions: EvaluationResult{status=FAILED,
sideEffects={4=SideEffect{action=Optional.absent(), statusChanges=[W
ORKING, FAILED]}, 3=SideEffect{action=Optional.absent(),
statusChanges=[WORKING, FAILED]}, 2=SideEffect{action=Optional.of(WATCH_TASK),
statusChanges=[WORKING]}}}
at
org.apache.aurora.scheduler.updater.JobUpdateControllerImpl.evaluateUpdater(JobUpdateControllerImpl.java:490)
at
org.apache.aurora.scheduler.updater.JobUpdateControllerImpl.changeJobUpdateStatus(JobUpdateControllerImpl.java:402)
at
org.apache.aurora.scheduler.updater.JobUpdateControllerImpl.recordAndChangeJobUpdateStatus(JobUpdateControllerImpl.java:334)
at
org.apache.aurora.scheduler.updater.JobUpdateControllerImpl.changeUpdateStatus(JobUpdateControllerImpl.java:319)
at
org.apache.aurora.scheduler.updater.JobUpdateControllerImpl.evaluateUpdater(JobUpdateControllerImpl.java:497)
at
org.apache.aurora.scheduler.storage.db.DbStorage.write(DbStorage.java:150)
at
org.apache.aurora.scheduler.storage.db.DbStorage$$EnhancerByGuice$$44df2497.CGLIB$write$4(<generated>)
at
org.apache.aurora.scheduler.storage.db.DbStorage$$EnhancerByGuice$$44df2497$$FastClassByGuice$$d23df519.invoke(<generated>)
at
com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228)
at
com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:72)
at
org.mybatis.guice.transactional.TransactionalMethodInterceptor.invoke(TransactionalMethodInterceptor.java:101)
at
com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:72)
at
com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:52)
at
org.apache.aurora.scheduler.storage.db.DbStorage$$EnhancerByGuice$$44df2497.write(<generated>)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
