Thanks for the info. I did find the "Removed agent" line as you suspected, but not much else in logging looked promising. I opened a JIRA to track from here on out https://issues.apache.org/jira/browse/MESOS-9555.
On Tue, Feb 5, 2019 at 2:03 PM Joseph Wu <jos...@mesosphere.io> wrote: > From the stack, it looks like the master is attempting to remove an agent > from the master's in-memory state. In the master's logs you should find a > line shortly before the exit, like: > > <timestamp> master.cpp:nnnn] Removed agent <ID of agent>: <reason> > > The agent's ID should at least give you some pointer to which agent is > causing the problem. Feel free to create a JIRA ( > https://issues.apache.org/jira/) with any information you can glean. > This particular type of failure, a CHECK-failure, means some invariant has > been violated and usually means we missed a corner case. > > On Tue, Feb 5, 2019 at 12:04 PM Jeff Pollard <jeff.poll...@gmail.com> > wrote: > >> We recently upgraded our Mesos cluster from version 1.3 to 1.5, and >> since then have been getting periodic master crashes due to this error: >> >> Feb 5 15:53:57 ip-10-0-16-140 mesos-master[8414]: F0205 15:53:57.385118 >> 8434 hierarchical.cpp:2630] Check failed: >> reservationScalarQuantities.contains(role) >> >> Full stack trace is at the end of this email. When the master fails, we >> automatically restart it and it rejoins the cluster just fine. I did some >> initial searching and was unable to find any existing bug reports or other >> people experiencing this issue. We run a cluster of 3 masters, and see >> crashes on all 3 instances. >> >> Hope to get some guidance on what is going on and/or where to start >> looking for more information. >> >> Feb 5 15:53:57 ip-10-0-16-140 mesos-master[8414]: @ >> 0x7f87e9170a7d google::LogMessage::Fail() >> Feb 5 15:53:57 ip-10-0-16-140 mesos-master[8414]: @ >> 0x7f87e9172830 google::LogMessage::SendToLog() >> Feb 5 15:53:57 ip-10-0-16-140 mesos-master[8414]: @ >> 0x7f87e9170663 google::LogMessage::Flush() >> Feb 5 15:53:57 ip-10-0-16-140 mesos-master[8414]: @ >> 0x7f87e9173259 google::LogMessageFatal::~LogMessageFatal() >> Feb 5 15:53:57 ip-10-0-16-140 mesos-master[8414]: @ >> 0x7f87e8443cbd >> mesos::internal::master::allocator::internal::HierarchicalAllocatorProcess::untrackReservations() >> Feb 5 15:53:57 ip-10-0-16-140 mesos-master[8414]: @ >> 0x7f87e8448fcd >> mesos::internal::master::allocator::internal::HierarchicalAllocatorProcess::removeSlave() >> Feb 5 15:53:57 ip-10-0-16-140 mesos-master[8414]: @ >> 0x7f87e90c4f11 process::ProcessBase::consume() >> Feb 5 15:53:57 ip-10-0-16-140 mesos-master[8414]: @ >> 0x7f87e90dea4a process::ProcessManager::resume() >> Feb 5 15:53:57 ip-10-0-16-140 mesos-master[8414]: @ >> 0x7f87e90e25d6 >> _ZNSt6thread5_ImplISt12_Bind_simpleIFZN7process14ProcessManager12init_threadsEvEUlvE_vEEE6_M_runEv >> Feb 5 15:53:57 ip-10-0-16-140 mesos-master[8414]: @ >> 0x7f87e6700c80 (unknown) >> Feb 5 15:53:57 ip-10-0-16-140 mesos-master[8414]: @ >> 0x7f87e5f136ba start_thread >> Feb 5 15:53:57 ip-10-0-16-140 mesos-master[8414]: @ >> 0x7f87e5c4941d (unknown) >> >
