For posterity: https://issues.apache.org/jira/browse/MESOS-9619

On Thu, Feb 28, 2019 at 6:02 PM Meng Zhu <m...@mesosphere.com> wrote:

> Hi Nimi:
>
> Thanks for reporting this.
>
> From the log snippet, looks like, when de-allocating resources, the agent
> does not have the port resources that is supposed to have been allocated.
> Can you provide the master log (which at least covers the period from when
> the resources on the agent is offered to the crash point)? Also, can you
> create a JIRA ticket and upload the log to there? (
> https://issues.apache.org/jira/projects/MESOS/issues)
>
> -Meng
>
> On Thu, Feb 28, 2019 at 1:58 PM Nimi W <psnim2...@gmail.com> wrote:
>
>> Hi,
>>
>> Mesos: 1.7.1
>>
>> I'm trying to debug an issue where if I launch a task using the
>> LAUNCH_GROUP method,
>> and the task fails to start, the mesos master will crash. I am using a
>> custom framework
>> I've built using the HTTP Scheduler API.
>>
>> When my framework received an offer - I return with an ACCEPT with this
>> JSON:
>>
>> https://gist.github.com/nemosupremo/3b23c4e1ca0ab241376aa5b975993270
>>
>> I then receive the following UPDATE events:
>>
>> TASK_STARTING
>> TASK_RUNNING
>> TASK_FAILED
>>
>> My framework then immediately tries to relaunch the task on the next
>> OFFERS:
>>
>> https://gist.github.com/nemosupremo/2b02443241c3bd002f04be034d8e64f7
>>
>> But between sometime when I get that event and try to acknowledge the
>> TASK_FAILED event,
>> the mesos master crashes with:
>>
>> Feb 28 21:34:02 master03 mesos-master[7124]: F0228 21:34:02.118693  7142
>> sorter.hpp:357] Check failed: resources.at(slaveId).contains(toRemove)
>> Resources disk(allocated: faust)(reservations: [(STATIC,faust)]):1;
>> cpus(allocated: faust)(reservations: [(STATIC,faust)]):0.1; mem(allocated:
>> faust)(reservations: [(STATIC,faust)]):64 at agent
>> 643078ba-8cb8-4582-b9c3-345d602506c8-S0 does not contain cpus(allocated:
>> faust)(reservations: [(STATIC,faust)]):0.1; mem(allocated:
>> faust)(reservations: [(STATIC,faust)]):64; disk(allocated:
>> faust)(reservations: [(STATIC,faust)]):1; ports(allocated:
>> faust)(reservations: [(STATIC,faust)]):[7777-7777]
>> Feb 28 21:34:02 master03 mesos-master[7124]: *** Check failure stack
>> trace: ***
>> Feb 28 21:34:02 master03 mesos-master[7124]:     @     0x7f1fd935e48d
>> google::LogMessage::Fail()
>> Feb 28 21:34:02 master03 mesos-master[7124]:     @     0x7f1fd9360240
>> google::LogMessage::SendToLog()
>> Feb 28 21:34:02 master03 mesos-master[7124]:     @     0x7f1fd935e073
>> google::LogMessage::Flush()
>> Feb 28 21:34:02 master03 mesos-master[7124]:     @     0x7f1fd9360c69
>> google::LogMessageFatal::~LogMessageFatal()
>> Feb 28 21:34:02 master03 mesos-master[7124]:     @     0x7f1fd83d85f8
>> mesos::internal::master::allocator::DRFSorter::unallocated()
>> Feb 28 21:34:02 master03 mesos-master[7124]:     @     0x7f1fd83a78af
>> mesos::internal::master::allocator::internal::HierarchicalAllocatorProcess::untrackAllocatedResources()
>> Feb 28 21:34:02 master03 mesos-master[7124]:     @     0x7f1fd83ba281
>> mesos::internal::master::allocator::internal::HierarchicalAllocatorProcess::recoverResources()
>> Feb 28 21:34:02 master03 mesos-master[7124]:     @     0x7f1fd92a6631
>> process::ProcessBase::consume()
>> Feb 28 21:34:02 master03 mesos-master[7124]:     @     0x7f1fd92c878a
>> process::ProcessManager::resume()
>> Feb 28 21:34:02 master03 mesos-master[7124]:     @     0x7f1fd92cc4d6
>> _ZNSt6thread5_ImplISt12_Bind_simpleIFZN7process14ProcessManager12init_threadsEvEUlvE_vEEE6_M_runEv
>> Feb 28 21:34:02 master03 mesos-master[7124]:     @     0x7f1fd6289c80
>> (unknown)
>> Feb 28 21:34:02 master03 mesos-master[7124]:     @     0x7f1fd5da56ba
>> start_thread
>> Feb 28 21:34:02 master03 mesos-master[7124]:     @     0x7f1fd5adb41d
>> (unknown)
>> Feb 28 21:34:02 master03 systemd[1]: mesos-master.service: Main process
>> exited, code=killed, status=6/ABRT
>> Feb 28 21:34:02 master03 systemd[1]: mesos-master.service: Unit entered
>> failed state.
>> Feb 28 21:34:02 master03 systemd[1]: mesos-master.service: Failed with
>> result 'signal'.
>>
>> The entire process works with the older LAUNCH API (for some reason the
>> docker task crashes with filesystem permission issues when using
>> LAUNCH_GROUPS)
>>
>

Reply via email to