[
https://issues.apache.org/jira/browse/AURORA-699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bill Farner resolved AURORA-699.
--------------------------------
Resolution: Won't Fix
This is a result of {{[5154f61 |
https://github.com/apache/incubator-aurora/commit/5154f61e7534f7f9c22d4760a6e12302b89c7dc5]}}.
This issue occurs in the rare overlap between: maintenance commands being
used on hosts before being 'introduced' to the scheduler by the master, and
hosts being fully committed (tasks consuming all CPU/RAM/disk) on a build >=
{{5154f61}}.
Given the extreme rarity of the case, i suggest we answer this with
course-correction advice rather than code. The simplest fix in this case is to
kill and restart a job encountering the issue.
> Stack trace in AttributeAggregate
> ---------------------------------
>
> Key: AURORA-699
> URL: https://issues.apache.org/jira/browse/AURORA-699
> Project: Aurora
> Issue Type: Bug
> Components: Scheduler
> Reporter: Maxim Khutornenko
>
> This does not look right:
> {noformat}
> W0910 23:34:47.149 THREAD145
> org.apache.aurora.scheduler.async.TaskScheduler$TaskSchedulerImpl.schedule:
> Task scheduling unexpectedly failed, will be retried
> java.lang.IllegalStateException: Optional.get() cannot be called on an absent
> value
> at com.google.common.base.Absent.get(Absent.java:47)
> at
> org.apache.aurora.scheduler.filter.AttributeAggregate$1.apply(AttributeAggregate.java:73)
> at
> org.apache.aurora.scheduler.filter.AttributeAggregate$1.apply(AttributeAggregate.java:67)
> at com.google.common.collect.Iterators$8.transform(Iterators.java:794)
> at
> com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
> at
> com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
> at com.google.common.collect.Iterators$5.hasNext(Iterators.java:543)
> at
> org.apache.aurora.scheduler.filter.AttributeAggregate$2.get(AttributeAggregate.java:83)
> at
> org.apache.aurora.scheduler.filter.AttributeAggregate$2.get(AttributeAggregate.java:77)
> at
> com.google.common.base.Suppliers$MemoizingSupplier.get(Suppliers.java:125)
> at
> org.apache.aurora.scheduler.filter.AttributeAggregate.getNumTasksWithAttribute(AttributeAggregate.java:112)
> at
> org.apache.aurora.scheduler.filter.AttributeFilter.matches(AttributeFilter.java:59)
> at
> org.apache.aurora.scheduler.filter.ConstraintFilter.getVeto(ConstraintFilter.java:110)
> at
> org.apache.aurora.scheduler.filter.SchedulingFilterImpl$6$1.apply(SchedulingFilterImpl.java:215)
> at
> org.apache.aurora.scheduler.filter.SchedulingFilterImpl$6$1.apply(SchedulingFilterImpl.java:207)
> at
> org.apache.aurora.scheduler.storage.mem.MemStorage$5.apply(MemStorage.java:218)
> at
> org.apache.aurora.scheduler.storage.db.DbStorage.weaklyConsistentRead(DbStorage.java:140)
> at
> org.mybatis.guice.transactional.TransactionalMethodInterceptor.invoke(TransactionalMethodInterceptor.java:101)
> at
> org.apache.aurora.scheduler.storage.mem.MemStorage.weaklyConsistentRead(MemStorage.java:215)
> at
> com.twitter.common.inject.TimedInterceptor.invoke(TimedInterceptor.java:87)
> at
> org.apache.aurora.scheduler.storage.log.LogStorage.weaklyConsistentRead(LogStorage.java:587)
> at
> org.apache.aurora.scheduler.storage.CallOrderEnforcingStorage.weaklyConsistentRead(CallOrderEnforcingStorage.java:123)
> at
> org.apache.aurora.scheduler.filter.SchedulingFilterImpl$6.apply(SchedulingFilterImpl.java:207)
> at
> org.apache.aurora.scheduler.filter.SchedulingFilterImpl$6.apply(SchedulingFilterImpl.java:196)
> at
> org.apache.aurora.scheduler.filter.SchedulingFilterImpl.filter(SchedulingFilterImpl.java:271)
> at
> org.apache.aurora.scheduler.events.NotifyingSchedulingFilter.filter(NotifyingSchedulingFilter.java:68)
> at
> org.apache.aurora.scheduler.state.TaskAssigner$TaskAssignerImpl.maybeAssign(TaskAssigner.java:94)
> at
> org.apache.aurora.scheduler.async.TaskScheduler$TaskSchedulerImpl$1.apply(TaskScheduler.java:147)
> at
> org.apache.aurora.scheduler.async.TaskScheduler$TaskSchedulerImpl$1.apply(TaskScheduler.java:133)
> at
> org.apache.aurora.scheduler.async.OfferQueue$OfferQueueImpl.launchFirst(OfferQueue.java:336)
> at
> org.apache.aurora.scheduler.async.TaskScheduler$TaskSchedulerImpl$3.apply(TaskScheduler.java:194)
> at
> org.apache.aurora.scheduler.async.TaskScheduler$TaskSchedulerImpl$3.apply(TaskScheduler.java:181)
> at
> org.apache.aurora.scheduler.storage.log.LogStorage$7.apply(LogStorage.java:560)
> at
> org.apache.aurora.scheduler.storage.log.LogStorage$7.apply(LogStorage.java:557)
> at
> org.apache.aurora.scheduler.storage.mem.MemStorage.doWork(MemStorage.java:175)
> at
> org.apache.aurora.scheduler.storage.mem.MemStorage.access$400(MemStorage.java:59)
> at
> org.apache.aurora.scheduler.storage.mem.MemStorage$4.apply(MemStorage.java:200)
> at
> org.apache.aurora.scheduler.storage.mem.MemStorage$4.apply(MemStorage.java:197)
> at
> org.apache.aurora.scheduler.storage.db.DbStorage.write(DbStorage.java:150)
> at
> org.mybatis.guice.transactional.TransactionalMethodInterceptor.invoke(TransactionalMethodInterceptor.java:101)
> at
> org.apache.aurora.scheduler.storage.mem.MemStorage.write(MemStorage.java:197)
> at
> org.apache.aurora.scheduler.storage.log.LogStorage.write(LogStorage.java:557)
> at
> org.apache.aurora.scheduler.storage.CallOrderEnforcingStorage.write(CallOrderEnforcingStorage.java:130)
> at
> org.apache.aurora.scheduler.async.TaskScheduler$TaskSchedulerImpl.schedule(TaskScheduler.java:181)
> at
> com.twitter.common.inject.TimedInterceptor.invoke(TimedInterceptor.java:87)
> at
> org.apache.aurora.scheduler.async.TaskGroups$1.schedule(TaskGroups.java:116)
> at
> org.apache.aurora.scheduler.async.TaskGroups$2.run(TaskGroups.java:138)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
