Re: [ceph-users] FAILED assert(i.first <= i.last)

[Peter Rosell]

That sound like an easy rule to follow. thanks again for you reply.

/Peter

mån 19 juni 2017 kl 10:19 skrev Wido den Hollander :

>
> > Op 19 juni 2017 om 9:55 schreef Peter Rosell :
> >
> >
> > I have my servers on UPS and shutdown them manually the way I use to turn
> > them off. There where enough power in the UPS after the servers were
> > shutdown because is continued to beep. Anyway, I will wipe it and re-add
> > it. Thanks for your reply.
> >
>
> Ok, you didn't mention that in the first post. I assumed a sudden power
> failure.
>
> My general recommendation is to wipe a single OSD if it has issues. The
> reason is that I've seen many cases where people ran XFS repair, played
> with the files on the disk and then had data corruption.
>
> That's why I'd say that you should try to avoid fixing single OSDs when
> you don't need to.
>
> Wido
>
> > /Peter
> >
> > mån 19 juni 2017 kl 09:11 skrev Wido den Hollander :
> >
> > >
> > > > Op 18 juni 2017 om 16:21 schreef Peter Rosell <
> peter.ros...@gmail.com>:
> > > >
> > > >
> > > > Hi,
> > > > I have a small cluster with only three nodes, 4 OSDs + 3 OSDs. I have
> > > been
> > > > running version 0.87.2 (Giant) for over 2.5 year, but a couple of day
> > > ago I
> > > > upgraded to 0.94.10 (Hammer) and then up to 10.2.7 (Jewel). Both the
> > > > upgrades went great. Started with monitors, osd and finally mds. The
> log
> > > > shows all 448 pgs active+clean. I'm running all daemons inside
> docker and
> > > > ceph version 10.2.7 (50e863e0f4bc8f4b9e31156de690d765af245185)
> > > >
> > > > Today I had a power outage and I had to take down the servers. When
> I now
> > > > start the servers again one OSD daemon doesn't start properly. It
> keeps
> > > > crashing.
> > > >
> > > > I noticed that the two first restarts of the osd daemon crashed with
> this
> > > > error:
> > > > FAILED assert(rollback_info_trimmed_to_riter == log.rbegin())
> > > >
> > > > After that it always fails with "FAILED assert(i.first <= i.last)"
> > > >
> > > > I have 15 logs like this one:
> > > > Jun 18 08:56:18 island sh[27991]: 2017-06-18 08:56:18.300641
> 7f5c5e0ff8c0
> > > > -1 log_channel(cluster) log [ERR] : 2.38 log bound mismatch, info
> > > > (19544'666742,19691'671046] actual [19499'665843,19691'671046]
> > > >
> > > > I removed the directories _head, but that just removed these
> error
> > > > logs. It crashes anyway.
> > > >
> > > > Anyone has any suggestions what to do to make it start up correct. Of
> > > > course I can remove the OSD from the cluster and re-add it, but it
> feels
> > > > like a bug.
> > >
> > > Are you sure? Since you had a power failure it could be that certain
> parts
> > > weren't committed to disk/FS properly when the power failed. That
> really
> > > depends on the hardware and configuration.
> > >
> > > Please, do not try to repair this OSD. Wipe it and re-add it to the
> > > cluster.
> > >
> > > Wido
> > >
> > > > A small snippet from the logs is added below. I didn't include the
> event
> > > > list. If it will help I can send it too.
> > > >
> > > > Jun 18 13:52:23 island sh[7068]: osd/osd_types.cc: In function
> 'static
> > > bool
> > > > pg_interval_t::check_new_interval(int, int, const std::vector&,
> > > const
> > > > std::vector&, int, int, const std::vector&, const
> > > > std::vector&, epoch_t, epoch_t, OSDMapRef, OSDMapRef, pg_t,
> > > > IsPGRecoverablePredicate*, std::map*,
> > > > std::ostream*)' thread 7f4fc2500700 time 2017-06-18 13:52:23.593991
> > > > Jun 18 13:52:23 island sh[7068]: osd/osd_types.cc: 3132: FAILED
> > > > assert(i.first <= i.last)
> > > > Jun 18 13:52:23 island sh[7068]:  ceph version 10.2.7
> > > > (50e863e0f4bc8f4b9e31156de690d765af245185)
> > > > Jun 18 13:52:23 island sh[7068]:  1: (ceph::__ceph_assert_fail(char
> > > const*,
> > > > char const*, int, char const*)+0x80) [0x559fe4c14360]
> > > > Jun 18 13:52:23 island sh[7068]:  2:
> > > > (pg_interval_t::check_new_interval(int, int, std::vector > > > std::allocator > const&, std::vector
> > > > const&, int, int, std::vector const&,
> > > > std::vector const&, unsigned int, unsigned
> > > int,
> > > > std::shared_ptr, std::shared_ptr, pg_t,
> > > > IsPGRecoverablePredicate*, std::map > > > std::less, std::allocator > > pg_interval_t> > >*, std::ostream*)+0x72c) [0x559fe47f723c]
> > > > Jun 18 13:52:23 island sh[7068]:  3:
> > > > (PG::start_peering_interval(std::shared_ptr,
> > > std::vector > > > std::allocator > const&, int, std::vector std::allocator >
> > > > const&, int, ObjectStore::Transaction*)+0x3ff) [0x559fe461439f]
> > > > Jun 18 13:52:23 island sh[7068]:  4:
> > > > (PG::RecoveryState::Reset::react(PG::AdvMap const&)+0x478)
> > > [0x559fe4615828]
> > > > Jun 18 13:52:23 island sh[7068]:  5:
> > > > (boost::statechart::simple_state > > > PG::RecoveryState::RecoveryMachine, 

Re: [ceph-users] FAILED assert(i.first <= i.last)

[Wido den Hollander]

> Op 19 juni 2017 om 9:55 schreef Peter Rosell :
> 
> 
> I have my servers on UPS and shutdown them manually the way I use to turn
> them off. There where enough power in the UPS after the servers were
> shutdown because is continued to beep. Anyway, I will wipe it and re-add
> it. Thanks for your reply.
> 

Ok, you didn't mention that in the first post. I assumed a sudden power failure.

My general recommendation is to wipe a single OSD if it has issues. The reason 
is that I've seen many cases where people ran XFS repair, played with the files 
on the disk and then had data corruption.

That's why I'd say that you should try to avoid fixing single OSDs when you 
don't need to.

Wido

> /Peter
> 
> mån 19 juni 2017 kl 09:11 skrev Wido den Hollander :
> 
> >
> > > Op 18 juni 2017 om 16:21 schreef Peter Rosell :
> > >
> > >
> > > Hi,
> > > I have a small cluster with only three nodes, 4 OSDs + 3 OSDs. I have
> > been
> > > running version 0.87.2 (Giant) for over 2.5 year, but a couple of day
> > ago I
> > > upgraded to 0.94.10 (Hammer) and then up to 10.2.7 (Jewel). Both the
> > > upgrades went great. Started with monitors, osd and finally mds. The log
> > > shows all 448 pgs active+clean. I'm running all daemons inside docker and
> > > ceph version 10.2.7 (50e863e0f4bc8f4b9e31156de690d765af245185)
> > >
> > > Today I had a power outage and I had to take down the servers. When I now
> > > start the servers again one OSD daemon doesn't start properly. It keeps
> > > crashing.
> > >
> > > I noticed that the two first restarts of the osd daemon crashed with this
> > > error:
> > > FAILED assert(rollback_info_trimmed_to_riter == log.rbegin())
> > >
> > > After that it always fails with "FAILED assert(i.first <= i.last)"
> > >
> > > I have 15 logs like this one:
> > > Jun 18 08:56:18 island sh[27991]: 2017-06-18 08:56:18.300641 7f5c5e0ff8c0
> > > -1 log_channel(cluster) log [ERR] : 2.38 log bound mismatch, info
> > > (19544'666742,19691'671046] actual [19499'665843,19691'671046]
> > >
> > > I removed the directories _head, but that just removed these error
> > > logs. It crashes anyway.
> > >
> > > Anyone has any suggestions what to do to make it start up correct. Of
> > > course I can remove the OSD from the cluster and re-add it, but it feels
> > > like a bug.
> >
> > Are you sure? Since you had a power failure it could be that certain parts
> > weren't committed to disk/FS properly when the power failed. That really
> > depends on the hardware and configuration.
> >
> > Please, do not try to repair this OSD. Wipe it and re-add it to the
> > cluster.
> >
> > Wido
> >
> > > A small snippet from the logs is added below. I didn't include the event
> > > list. If it will help I can send it too.
> > >
> > > Jun 18 13:52:23 island sh[7068]: osd/osd_types.cc: In function 'static
> > bool
> > > pg_interval_t::check_new_interval(int, int, const std::vector&,
> > const
> > > std::vector&, int, int, const std::vector&, const
> > > std::vector&, epoch_t, epoch_t, OSDMapRef, OSDMapRef, pg_t,
> > > IsPGRecoverablePredicate*, std::map*,
> > > std::ostream*)' thread 7f4fc2500700 time 2017-06-18 13:52:23.593991
> > > Jun 18 13:52:23 island sh[7068]: osd/osd_types.cc: 3132: FAILED
> > > assert(i.first <= i.last)
> > > Jun 18 13:52:23 island sh[7068]:  ceph version 10.2.7
> > > (50e863e0f4bc8f4b9e31156de690d765af245185)
> > > Jun 18 13:52:23 island sh[7068]:  1: (ceph::__ceph_assert_fail(char
> > const*,
> > > char const*, int, char const*)+0x80) [0x559fe4c14360]
> > > Jun 18 13:52:23 island sh[7068]:  2:
> > > (pg_interval_t::check_new_interval(int, int, std::vector > > std::allocator > const&, std::vector
> > > const&, int, int, std::vector const&,
> > > std::vector const&, unsigned int, unsigned
> > int,
> > > std::shared_ptr, std::shared_ptr, pg_t,
> > > IsPGRecoverablePredicate*, std::map > > std::less, std::allocator > pg_interval_t> > >*, std::ostream*)+0x72c) [0x559fe47f723c]
> > > Jun 18 13:52:23 island sh[7068]:  3:
> > > (PG::start_peering_interval(std::shared_ptr,
> > std::vector > > std::allocator > const&, int, std::vector
> > > const&, int, ObjectStore::Transaction*)+0x3ff) [0x559fe461439f]
> > > Jun 18 13:52:23 island sh[7068]:  4:
> > > (PG::RecoveryState::Reset::react(PG::AdvMap const&)+0x478)
> > [0x559fe4615828]
> > > Jun 18 13:52:23 island sh[7068]:  5:
> > > (boost::statechart::simple_state > > PG::RecoveryState::RecoveryMachine, boost::mpl::list > > mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na,
> > > mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na,
> > > mpl_::na, mpl_::na, mpl_::na, mpl_::na>,
> > >
> > (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base
> > > const&, void const*)+0x176) [0x559fe4645b86]
> > > 

Re: [ceph-users] FAILED assert(i.first <= i.last)

[Peter Rosell]

I have my servers on UPS and shutdown them manually the way I use to turn
them off. There where enough power in the UPS after the servers were
shutdown because is continued to beep. Anyway, I will wipe it and re-add
it. Thanks for your reply.

/Peter

mån 19 juni 2017 kl 09:11 skrev Wido den Hollander :

>
> > Op 18 juni 2017 om 16:21 schreef Peter Rosell :
> >
> >
> > Hi,
> > I have a small cluster with only three nodes, 4 OSDs + 3 OSDs. I have
> been
> > running version 0.87.2 (Giant) for over 2.5 year, but a couple of day
> ago I
> > upgraded to 0.94.10 (Hammer) and then up to 10.2.7 (Jewel). Both the
> > upgrades went great. Started with monitors, osd and finally mds. The log
> > shows all 448 pgs active+clean. I'm running all daemons inside docker and
> > ceph version 10.2.7 (50e863e0f4bc8f4b9e31156de690d765af245185)
> >
> > Today I had a power outage and I had to take down the servers. When I now
> > start the servers again one OSD daemon doesn't start properly. It keeps
> > crashing.
> >
> > I noticed that the two first restarts of the osd daemon crashed with this
> > error:
> > FAILED assert(rollback_info_trimmed_to_riter == log.rbegin())
> >
> > After that it always fails with "FAILED assert(i.first <= i.last)"
> >
> > I have 15 logs like this one:
> > Jun 18 08:56:18 island sh[27991]: 2017-06-18 08:56:18.300641 7f5c5e0ff8c0
> > -1 log_channel(cluster) log [ERR] : 2.38 log bound mismatch, info
> > (19544'666742,19691'671046] actual [19499'665843,19691'671046]
> >
> > I removed the directories _head, but that just removed these error
> > logs. It crashes anyway.
> >
> > Anyone has any suggestions what to do to make it start up correct. Of
> > course I can remove the OSD from the cluster and re-add it, but it feels
> > like a bug.
>
> Are you sure? Since you had a power failure it could be that certain parts
> weren't committed to disk/FS properly when the power failed. That really
> depends on the hardware and configuration.
>
> Please, do not try to repair this OSD. Wipe it and re-add it to the
> cluster.
>
> Wido
>
> > A small snippet from the logs is added below. I didn't include the event
> > list. If it will help I can send it too.
> >
> > Jun 18 13:52:23 island sh[7068]: osd/osd_types.cc: In function 'static
> bool
> > pg_interval_t::check_new_interval(int, int, const std::vector&,
> const
> > std::vector&, int, int, const std::vector&, const
> > std::vector&, epoch_t, epoch_t, OSDMapRef, OSDMapRef, pg_t,
> > IsPGRecoverablePredicate*, std::map*,
> > std::ostream*)' thread 7f4fc2500700 time 2017-06-18 13:52:23.593991
> > Jun 18 13:52:23 island sh[7068]: osd/osd_types.cc: 3132: FAILED
> > assert(i.first <= i.last)
> > Jun 18 13:52:23 island sh[7068]:  ceph version 10.2.7
> > (50e863e0f4bc8f4b9e31156de690d765af245185)
> > Jun 18 13:52:23 island sh[7068]:  1: (ceph::__ceph_assert_fail(char
> const*,
> > char const*, int, char const*)+0x80) [0x559fe4c14360]
> > Jun 18 13:52:23 island sh[7068]:  2:
> > (pg_interval_t::check_new_interval(int, int, std::vector > std::allocator > const&, std::vector
> > const&, int, int, std::vector const&,
> > std::vector const&, unsigned int, unsigned
> int,
> > std::shared_ptr, std::shared_ptr, pg_t,
> > IsPGRecoverablePredicate*, std::map > std::less, std::allocator pg_interval_t> > >*, std::ostream*)+0x72c) [0x559fe47f723c]
> > Jun 18 13:52:23 island sh[7068]:  3:
> > (PG::start_peering_interval(std::shared_ptr,
> std::vector > std::allocator > const&, int, std::vector
> > const&, int, ObjectStore::Transaction*)+0x3ff) [0x559fe461439f]
> > Jun 18 13:52:23 island sh[7068]:  4:
> > (PG::RecoveryState::Reset::react(PG::AdvMap const&)+0x478)
> [0x559fe4615828]
> > Jun 18 13:52:23 island sh[7068]:  5:
> > (boost::statechart::simple_state > PG::RecoveryState::RecoveryMachine, boost::mpl::list > mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na,
> > mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na,
> > mpl_::na, mpl_::na, mpl_::na, mpl_::na>,
> >
> (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base
> > const&, void const*)+0x176) [0x559fe4645b86]
> > Jun 18 13:52:23 island sh[7068]:  6:
> > (boost::statechart::state_machine > PG::RecoveryState::Initial, std::allocator,
> >
> boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base
> > const&)+0x69) [0x559fe4626d49]
> > Jun 18 13:52:23 island sh[7068]:  7:
> > (PG::handle_advance_map(std::shared_ptr,
> > std::shared_ptr, std::vector&,
> > int, std::vector&, int,
> PG::RecoveryCtx*)+0x49e)
> > [0x559fe45fa5ae]
> > Jun 18 13:52:23 island sh[7068]:  8: (OSD::advance_pg(unsigned int, PG*,
> > ThreadPool::TPHandle&, PG::RecoveryCtx*,
> 

Re: [ceph-users] FAILED assert(i.first <= i.last)

[Wido den Hollander]

> Op 18 juni 2017 om 16:21 schreef Peter Rosell :
> 
> 
> Hi,
> I have a small cluster with only three nodes, 4 OSDs + 3 OSDs. I have been
> running version 0.87.2 (Giant) for over 2.5 year, but a couple of day ago I
> upgraded to 0.94.10 (Hammer) and then up to 10.2.7 (Jewel). Both the
> upgrades went great. Started with monitors, osd and finally mds. The log
> shows all 448 pgs active+clean. I'm running all daemons inside docker and
> ceph version 10.2.7 (50e863e0f4bc8f4b9e31156de690d765af245185)
> 
> Today I had a power outage and I had to take down the servers. When I now
> start the servers again one OSD daemon doesn't start properly. It keeps
> crashing.
> 
> I noticed that the two first restarts of the osd daemon crashed with this
> error:
> FAILED assert(rollback_info_trimmed_to_riter == log.rbegin())
> 
> After that it always fails with "FAILED assert(i.first <= i.last)"
> 
> I have 15 logs like this one:
> Jun 18 08:56:18 island sh[27991]: 2017-06-18 08:56:18.300641 7f5c5e0ff8c0
> -1 log_channel(cluster) log [ERR] : 2.38 log bound mismatch, info
> (19544'666742,19691'671046] actual [19499'665843,19691'671046]
> 
> I removed the directories _head, but that just removed these error
> logs. It crashes anyway.
> 
> Anyone has any suggestions what to do to make it start up correct. Of
> course I can remove the OSD from the cluster and re-add it, but it feels
> like a bug.

Are you sure? Since you had a power failure it could be that certain parts 
weren't committed to disk/FS properly when the power failed. That really 
depends on the hardware and configuration.

Please, do not try to repair this OSD. Wipe it and re-add it to the cluster.

Wido

> A small snippet from the logs is added below. I didn't include the event
> list. If it will help I can send it too.
> 
> Jun 18 13:52:23 island sh[7068]: osd/osd_types.cc: In function 'static bool
> pg_interval_t::check_new_interval(int, int, const std::vector&, const
> std::vector&, int, int, const std::vector&, const
> std::vector&, epoch_t, epoch_t, OSDMapRef, OSDMapRef, pg_t,
> IsPGRecoverablePredicate*, std::map*,
> std::ostream*)' thread 7f4fc2500700 time 2017-06-18 13:52:23.593991
> Jun 18 13:52:23 island sh[7068]: osd/osd_types.cc: 3132: FAILED
> assert(i.first <= i.last)
> Jun 18 13:52:23 island sh[7068]:  ceph version 10.2.7
> (50e863e0f4bc8f4b9e31156de690d765af245185)
> Jun 18 13:52:23 island sh[7068]:  1: (ceph::__ceph_assert_fail(char const*,
> char const*, int, char const*)+0x80) [0x559fe4c14360]
> Jun 18 13:52:23 island sh[7068]:  2:
> (pg_interval_t::check_new_interval(int, int, std::vector std::allocator > const&, std::vector
> const&, int, int, std::vector const&,
> std::vector const&, unsigned int, unsigned int,
> std::shared_ptr, std::shared_ptr, pg_t,
> IsPGRecoverablePredicate*, std::map std::less, std::allocator > >*, std::ostream*)+0x72c) [0x559fe47f723c]
> Jun 18 13:52:23 island sh[7068]:  3:
> (PG::start_peering_interval(std::shared_ptr, std::vector std::allocator > const&, int, std::vector
> const&, int, ObjectStore::Transaction*)+0x3ff) [0x559fe461439f]
> Jun 18 13:52:23 island sh[7068]:  4:
> (PG::RecoveryState::Reset::react(PG::AdvMap const&)+0x478) [0x559fe4615828]
> Jun 18 13:52:23 island sh[7068]:  5:
> (boost::statechart::simple_state PG::RecoveryState::RecoveryMachine, boost::mpl::list mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na,
> mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na,
> mpl_::na, mpl_::na, mpl_::na, mpl_::na>,
> (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base
> const&, void const*)+0x176) [0x559fe4645b86]
> Jun 18 13:52:23 island sh[7068]:  6:
> (boost::statechart::state_machine PG::RecoveryState::Initial, std::allocator,
> boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base
> const&)+0x69) [0x559fe4626d49]
> Jun 18 13:52:23 island sh[7068]:  7:
> (PG::handle_advance_map(std::shared_ptr,
> std::shared_ptr, std::vector&,
> int, std::vector&, int, PG::RecoveryCtx*)+0x49e)
> [0x559fe45fa5ae]
> Jun 18 13:52:23 island sh[7068]:  8: (OSD::advance_pg(unsigned int, PG*,
> ThreadPool::TPHandle&, PG::RecoveryCtx*, std::set std::less,
> std::allocator >*)+0x2f2) [0x559fe452c042]
> Jun 18 13:52:23 island sh[7068]:  9:
> (OSD::process_peering_events(std::__cxx11::list >
> const&, ThreadPool::TPHandle&)+0x214) [0x559fe4546d34]
> Jun 18 13:52:23 island sh[7068]:  10:
> (ThreadPool::BatchWorkQueue::_void_process(void*,
> ThreadPool::TPHandle&)+0x25) [0x559fe458f8e5]
> Jun 18 13:52:23 island sh[7068]:  11:
> 

[ceph-users] FAILED assert(i.first <= i.last)

[Peter Rosell]

Hi,
I have a small cluster with only three nodes, 4 OSDs + 3 OSDs. I have been
running version 0.87.2 (Giant) for over 2.5 year, but a couple of day ago I
upgraded to 0.94.10 (Hammer) and then up to 10.2.7 (Jewel). Both the
upgrades went great. Started with monitors, osd and finally mds. The log
shows all 448 pgs active+clean. I'm running all daemons inside docker and
ceph version 10.2.7 (50e863e0f4bc8f4b9e31156de690d765af245185)

Today I had a power outage and I had to take down the servers. When I now
start the servers again one OSD daemon doesn't start properly. It keeps
crashing.

I noticed that the two first restarts of the osd daemon crashed with this
error:
FAILED assert(rollback_info_trimmed_to_riter == log.rbegin())

After that it always fails with "FAILED assert(i.first <= i.last)"

I have 15 logs like this one:
Jun 18 08:56:18 island sh[27991]: 2017-06-18 08:56:18.300641 7f5c5e0ff8c0
-1 log_channel(cluster) log [ERR] : 2.38 log bound mismatch, info
(19544'666742,19691'671046] actual [19499'665843,19691'671046]

I removed the directories _head, but that just removed these error
logs. It crashes anyway.

Anyone has any suggestions what to do to make it start up correct. Of
course I can remove the OSD from the cluster and re-add it, but it feels
like a bug.
A small snippet from the logs is added below. I didn't include the event
list. If it will help I can send it too.

Jun 18 13:52:23 island sh[7068]: osd/osd_types.cc: In function 'static bool
pg_interval_t::check_new_interval(int, int, const std::vector&, const
std::vector&, int, int, const std::vector&, const
std::vector&, epoch_t, epoch_t, OSDMapRef, OSDMapRef, pg_t,
IsPGRecoverablePredicate*, std::map*,
std::ostream*)' thread 7f4fc2500700 time 2017-06-18 13:52:23.593991
Jun 18 13:52:23 island sh[7068]: osd/osd_types.cc: 3132: FAILED
assert(i.first <= i.last)
Jun 18 13:52:23 island sh[7068]:  ceph version 10.2.7
(50e863e0f4bc8f4b9e31156de690d765af245185)
Jun 18 13:52:23 island sh[7068]:  1: (ceph::__ceph_assert_fail(char const*,
char const*, int, char const*)+0x80) [0x559fe4c14360]
Jun 18 13:52:23 island sh[7068]:  2:
(pg_interval_t::check_new_interval(int, int, std::vector const&, std::vector
const&, int, int, std::vector const&,
std::vector const&, unsigned int, unsigned int,
std::shared_ptr, std::shared_ptr, pg_t,
IsPGRecoverablePredicate*, std::map, std::allocator >*, std::ostream*)+0x72c) [0x559fe47f723c]
Jun 18 13:52:23 island sh[7068]:  3:
(PG::start_peering_interval(std::shared_ptr, std::vector const&, int, std::vector
const&, int, ObjectStore::Transaction*)+0x3ff) [0x559fe461439f]
Jun 18 13:52:23 island sh[7068]:  4:
(PG::RecoveryState::Reset::react(PG::AdvMap const&)+0x478) [0x559fe4615828]
Jun 18 13:52:23 island sh[7068]:  5:
(boost::statechart::simple_state,
(boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base
const&, void const*)+0x176) [0x559fe4645b86]
Jun 18 13:52:23 island sh[7068]:  6:
(boost::statechart::state_machine::process_event(boost::statechart::event_base
const&)+0x69) [0x559fe4626d49]
Jun 18 13:52:23 island sh[7068]:  7:
(PG::handle_advance_map(std::shared_ptr,
std::shared_ptr, std::vector&,
int, std::vector&, int, PG::RecoveryCtx*)+0x49e)
[0x559fe45fa5ae]
Jun 18 13:52:23 island sh[7068]:  8: (OSD::advance_pg(unsigned int, PG*,
ThreadPool::TPHandle&, PG::RecoveryCtx*, std::set,
std::allocator >*)+0x2f2) [0x559fe452c042]
Jun 18 13:52:23 island sh[7068]:  9:
(OSD::process_peering_events(std::__cxx11::list >
const&, ThreadPool::TPHandle&)+0x214) [0x559fe4546d34]
Jun 18 13:52:23 island sh[7068]:  10:
(ThreadPool::BatchWorkQueue::_void_process(void*,
ThreadPool::TPHandle&)+0x25) [0x559fe458f8e5]
Jun 18 13:52:23 island sh[7068]:  11:
(ThreadPool::worker(ThreadPool::WorkThread*)+0xdb1) [0x559fe4c06531]
Jun 18 13:52:23 island sh[7068]:  12:
(ThreadPool::WorkThread::entry()+0x10) [0x559fe4c07630]
Jun 18 13:52:23 island sh[7068]:  13: (()+0x76fa) [0x7f4fe256b6fa]
Jun 18 13:52:23 island sh[7068]:  14: (clone()+0x6d) [0x7f4fe05e3b5d]
Jun 18 13:52:23 island sh[7068]:  NOTE: a copy of the executable, or
`objdump -rdS ` is needed to interpret this.
Jun 18 13:52:23 island sh[7068]: 2017-06-18 13:52:23.599558 7f4fc2500700 -1
osd/osd_types.cc: In function 'static bool
pg_interval_t::check_new_interval(int, int, const