Hi Sage
I find the osd assert due to the wrongly generated past_intervals, could you
give me some advice and solutions to this problem?
Here is the detail:
Ceph version: 0.94.5
HOST-A HOST-B HOST-C
osd 7 osd 21 osd11
1. osdmap epoch95, pg 1.20f on osd acting set [11,7]/ up set[11,7],then
shutdown HOST-C
2. for a long time, cluster has only HOST A and HOST B, write data
3. shutdown HOST-A , then start HOST-C, restart HOST-B about 4 times
4. start HOST-A, osd 21 assert
Analysis:
when osd 11 start, it generate past_intervals wrongly, make [92~1000] in the
same interval
pg map 1673,osd11 become the primary,and pg 1.20f change from peering to
activating+undersized+degraded , modified last_epoch_start;
osd7 start, find_best_info will choose out bigger last_epoch_start，althought
osd7 has the latest data;
past_intervals on osd 7:
~95 [11,7]/[11,7]
96~100 [7]/[7]
101 [7,21]/[7,21]
102~178 [7,21]/[7]
179~1663 [7,21]/[7,21]
1664~1672 [21]/[21]
1673~1692 [11]/[11]
past_intervals on osd11:
92~1000 [11,7]/[11,7]----------------the wrong pi
1001~1663 [7,21]/[7,21] no rw
1664~1672 [21]/[21] no rw
1673~1692 [11]/[11]
Logs:
Assert on osd7:
2017-07-10 16:08:29.836722 7f4fac24a700 -1 osd/PGLog.cc: In function 'void
PGLog::rewind_divergent_log(ObjectStore::Transaction&, eversion_t, pg_info_t&,
PGLog::LogEntryHandler*, bool&, bool&)' thread 7f4fac24a700 time 2017-07-10
16:08:29.833699
osd/PGLog.cc: 503: FAILED assert(newhead >= log.tail)
ceph version 0.94.5 (664cc0b54fdb496233a81ab19d42df3f46dcda50)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b)
[0xbd1ebb]
2: (PGLog::rewind_divergent_log(ObjectStore::Transaction&, eversion_t,
pg_info_t&, PGLog::LogEntryHandler*, bool&, bool&)+0x60b) [0x7840fb]
3: (PG::rewind_divergent_log(ObjectStore::Transaction&, eversion_t)+0x97)
[0x7df4b7]
4: (PG::RecoveryState::Stray::react(PG::MInfoRec const&)+0x22f) [0x80109f]
5: (boost::statechart::simple_state<PG::RecoveryState::Stray,
PG::RecoveryState::Started, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na,
mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na,
mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na,
mpl_::na>,
(boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base
const&, void const*)+0x216) [0x83c8d6]
6: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine,
PG::RecoveryState::Initial, std::allocator<void>,
boost::statechart::null_exception_translator>::send_event(boost::statechart::event_base
const&)+0x5b) [0x827edb]
7: (PG::handle_peering_event(std::tr1::shared_ptr<PG::CephPeeringEvt>,
PG::RecoveryCtx*)+0x1ce) [0x7d5dce]
8: (OSD::process_peering_events(std::list<PG*, std::allocator<PG*> > const&,
ThreadPool::TPHandle&)+0x2c0) [0x6b5930]
9: (OSD::PeeringWQ::_process(std::list<PG*, std::allocator<PG*> > const&,
ThreadPool::TPHandle&)+0x18) [0x70ef18]
10: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa56) [0xbc2aa6]
11: (ThreadPool::WorkThread::entry()+0x10) [0xbc3b50]
12: (()+0x8182) [0x7f4fc6458182]
13: (clone()+0x6d) [0x7f4fc49c347d]
osd 11 log:
2017-07-10 15:40:43.742547 7f8b1567a740 10 osd.11 95 pgid 1.20f coll 1.20f_head
2017-07-10 15:40:43.742580 7f8b1567a740 10 osd.11 95 _open_lock_pg 1.20f
2017-07-10 15:40:43.742592 7f8b1567a740 5 osd.11 pg_epoch: 95
pg[1.20f(unlocked)] enter Initial
2017-07-10 15:40:43.743207 7f8b1567a740 10 osd.11 pg_epoch: 95 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93 92/92/92) [11,7] r=0
lpr=0 crt=95'394 lcod 0'0 mlcod 0'0 inactive] handle_loaded
2017-07-10 15:40:43.743214 7f8b1567a740 5 osd.11 pg_epoch: 95 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93 92/92/92) [11,7] r=0
lpr=0 crt=95'394 lcod 0'0 mlcod 0'0 inactive] exit Initial 0.000622 0 0.000000
2017-07-10 15:40:43.743220 7f8b1567a740 5 osd.11 pg_epoch: 95 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93 92/92/92) [11,7] r=0
lpr=0 crt=95'394 lcod 0'0 mlcod 0'0 inactive] enter Reset
2017-07-10 15:40:43.743224 7f8b1567a740 10 osd.11 pg_epoch: 95 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93 92/92/92) [11,7] r=0
lpr=95 crt=95'394 lcod 0'0 mlcod 0'0 inactive] Clearing blocked outgoing
recovery messages
2017-07-10 15:40:43.743228 7f8b1567a740 10 osd.11 pg_epoch: 95 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93 92/92/92) [11,7] r=0
lpr=95 crt=95'394 lcod 0'0 mlcod 0'0 inactive] Not blocking outgoing recovery
messages
2017-07-10 15:40:43.743232 7f8b1567a740 10 osd.11 95 load_pgs loaded pg[1.20f(
v 95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93 92/92/92) [11,7]
r=0 lpr=95 crt=95'394 lcod 0'0 mlcod 0'0 inactive] log((95'300,95'397],
crt=95'394)
2017-07-10 15:40:43.829867 7f8b1567a740 10 osd.11 pg_epoch: 95
pg[1.20f(unlocked)] _calc_past_interval_range start epoch 93 >= end epoch 92,
nothing to do
2017-07-10 15:42:40.899157 7f8b1567a740 10 osd.11 pg_epoch: 95 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93 92/92/92) [11,7] r=0
lpr=95 crt=95'394 lcod 0'0 mlcod 0'0 inactive] null
2017-07-10 15:42:40.902520 7f8afa1bd700 10 osd.11 pg_epoch: 95 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93 92/92/92) [11,7] r=0
lpr=95 crt=95'394 lcod 0'0 mlcod 0'0 inactive] handle_peering_event:
epoch_sent: 95 epoch_requested: 95 NullEvt
2017-07-10 15:42:40.917668 7f8b031cf700 10 osd.11 pg_epoch: 95 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93 92/92/92) [11,7] r=0
lpr=95 crt=95'394 lcod 0'0 mlcod 0'0 inactive] null
2017-07-10 15:42:41.846755 7f8af89ba700 10 osd.11 pg_epoch: 95 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93 92/92/92) [11,7] r=0
lpr=95 crt=95'394 lcod 0'0 mlcod 0'0 inactive] handle_advance_map [7,21]/[7,21]
-- 7/7
2017-07-10 15:42:41.846771 7f8af89ba700 10 osd.11 pg_epoch: 1001 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93 92/92/92) [11,7] r=0
lpr=95 crt=95'394 lcod 0'0 mlcod 0'0 inactive] state<Reset>: Reset advmap
2017-07-10 15:42:41.846781 7f8af89ba700 10 osd.11 pg_epoch: 1001 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93 92/92/92) [11,7] r=0
lpr=95 crt=95'394 lcod 0'0 mlcod 0'0 inactive] _calc_past_interval_range start
epoch 1001 >= end epoch 92, nothing to do
2017-07-10 15:42:41.846790 7f8af89ba700 10 osd.11 pg_epoch: 1001 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93 92/92/92) [11,7] r=0
lpr=95 crt=95'394 lcod 0'0 mlcod 0'0 inactive] state<Reset>: should restart
peering, calling start_peering_interval again
2017-07-10 15:42:41.846798 7f8af89ba700 10 osd.11 pg_epoch: 1001 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93 92/92/92) [11,7] r=0
lpr=1001 crt=95'394 lcod 0'0 mlcod 0'0 inactive] Clearing blocked outgoing
recovery messages
2017-07-10 15:42:41.846807 7f8af89ba700 10 osd.11 pg_epoch: 1001 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93 92/92/92) [11,7] r=0
lpr=1001 crt=95'394 lcod 0'0 mlcod 0'0 inactive] Not blocking outgoing recovery
messages
2017-07-10 15:42:41.846829 7f8af89ba700 10 osd.11 pg_epoch: 1001 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93 92/92/92) [7,21]
r=-1 lpr=1001 pi=92-1000/1 crt=95'394 lcod 0'0 inactive]
start_peering_interval: check_new_interval output: generate_past_intervals
interval(92-1000 up [11,7](11) acting [11,7](11)): not rw, up_thru 94 up_from
92 last_epoch_clean 93
2017-07-10 15:42:41.846839 7f8af89ba700 10 osd.11 pg_epoch: 1001 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93 92/92/92) [7,21]
r=-1 lpr=1001 pi=92-1000/1 crt=95'394 lcod 0'0 inactive] noting past
interval(92-1000 up [11,7](11) acting [11,7](11) maybe_went_rw)
...
2017-07-10 15:42:52.600376 7f8af91bb700 5 osd.11 pg_epoch: 1673 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93 1673/1673/1673) [11]
r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0 inactive] enter Started
2017-07-10 15:42:52.600387 7f8af91bb700 5 osd.11 pg_epoch: 1673 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93 1673/1673/1673) [11]
r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0 inactive] enter Start
2017-07-10 15:42:52.600397 7f8af91bb700 1 osd.11 pg_epoch: 1673 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93 1673/1673/1673) [11]
r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0 inactive] state<Start>:
transitioning to Primary
2017-07-10 15:42:52.600410 7f8af91bb700 5 osd.11 pg_epoch: 1673 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93 1673/1673/1673) [11]
r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0 inactive] exit Start
0.000023 0 0.000000
2017-07-10 15:42:52.600424 7f8af91bb700 5 osd.11 pg_epoch: 1673 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93 1673/1673/1673) [11]
r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0 inactive] enter
Started/Primary
2017-07-10 15:42:52.600435 7f8af91bb700 5 osd.11 pg_epoch: 1673 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93 1673/1673/1673) [11]
r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0 inactive] enter
Started/Primary/Peering
2017-07-10 15:42:52.600446 7f8af91bb700 5 osd.11 pg_epoch: 1673 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93 1673/1673/1673) [11]
r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0 peering] enter
Started/Primary/Peering/GetInfo
2017-07-10 15:42:52.600460 7f8af91bb700 10 osd.11 pg_epoch: 1673 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93 1673/1673/1673) [11]
r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0 peering]
_calc_past_interval_range: already have past intervals back to 93
2017-07-10 15:42:52.600474 7f8af91bb700 10 osd.11 pg_epoch: 1673 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93 1673/1673/1673) [11]
r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0 peering] PriorSet:
build_prior interval(1664-1672 up [21](21) acting [21](21))
2017-07-10 15:42:52.608241 7f8af91bb700 10 osd.11 pg_epoch: 1673 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93 1673/1673/1673) [11]
r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0 peering] PriorSet:
build_prior interval(1001-1663 up [7,21](7) acting [7,21](7))
2017-07-10 15:42:52.608256 7f8af91bb700 10 osd.11 pg_epoch: 1673 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93 1673/1673/1673) [11]
r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0 peering] PriorSet:
build_prior interval(92-1000 up [11,7](11) acting [11,7](11) maybe_went_rw)
2017-07-10 15:42:52.608265 7f8af91bb700 10 osd.11 pg_epoch: 1673 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93 1673/1673/1673) [11]
r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0 peering] PriorSet:
build_prior prior osd.7 is down
2017-07-10 15:42:52.608273 7f8af91bb700 10 osd.11 pg_epoch: 1673 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93 1673/1673/1673) [11]
r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0 peering] PriorSet:
build_prior final: probe 11 down 7 blocked_by {}
2017-07-10 15:42:52.608281 7f8af91bb700 10 osd.11 pg_epoch: 1673 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93 1673/1673/1673) [11]
r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0 peering] up_thru 94 <
same_since 1673, must notify monitor
2017-07-10 15:42:52.608294 7f8af91bb700 5 osd.11 pg_epoch: 1673 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93 1673/1673/1673) [11]
r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0 peering] exit
Started/Primary/Peering/GetInfo 0.007847 0 0.000000
2017-07-10 15:42:52.608306 7f8af91bb700 5 osd.11 pg_epoch: 1673 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93 1673/1673/1673) [11]
r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0 peering] enter
Started/Primary/Peering/GetLog
2017-07-10 15:42:52.608318 7f8af91bb700 10 osd.11 pg_epoch: 1673 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93 1673/1673/1673) [11]
r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0 peering] calc_acting
osd.11 1.20f( v 95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93
1673/1673/1673)
2017-07-10 15:42:52.608338 7f8af91bb700 10 osd.11 pg_epoch: 1673 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93 1673/1673/1673) [11]
r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0 peering] calc_acting
newest update on osd.11 with 1.20f( v 95'397 (95'300,95'397] local-les=93 n=69
ec=75 les/c 93/93 1673/1673/1673)
calc_acting primary is osd.11 with 1.20f( v 95'397 (95'300,95'397] local-les=93
n=69 ec=75 les/c 93/93 1673/1673/1673)
2017-07-10 15:42:52.608348 7f8af91bb700 10 osd.11 pg_epoch: 1673 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93 1673/1673/1673) [11]
r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0 peering] actingbackfill
is 11
2017-07-10 15:42:52.608357 7f8af91bb700 10 osd.11 pg_epoch: 1673 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93 1673/1673/1673) [11]
r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0 peering] choose_acting
want [11] (== acting) backfill_targets
2017-07-10 15:42:52.608367 7f8af91bb700 10 osd.11 pg_epoch: 1673 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93 1673/1673/1673) [11]
r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0 peering]
state<Started/Primary/Peering/GetLog>: leaving GetLog
2017-07-10 15:42:52.608380 7f8af91bb700 5 osd.11 pg_epoch: 1673 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93 1673/1673/1673) [11]
r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0 peering] exit
Started/Primary/Peering/GetLog 0.000073 0 0.000000
2017-07-10 15:42:52.608391 7f8af91bb700 5 osd.11 pg_epoch: 1673 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93 1673/1673/1673) [11]
r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0 peering] enter
Started/Primary/Peering/GetMissing
2017-07-10 15:42:52.608400 7f8af91bb700 10 osd.11 pg_epoch: 1673 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93 1673/1673/1673) [11]
r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0 peering]
state<Started/Primary/Peering/GetMissing>: still need up_thru update before
going active
2017-07-10 15:42:52.608409 7f8af91bb700 5 osd.11 pg_epoch: 1673 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93 1673/1673/1673) [11]
r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0 peering] exit
Started/Primary/Peering/GetMissing 0.000018 0 0.000000
2017-07-10 15:42:52.608417 7f8af91bb700 5 osd.11 pg_epoch: 1673 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93 1673/1673/1673) [11]
r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0 peering] enter
Started/Primary/Peering/WaitUpThru
2017-07-10 15:42:52.608424 7f8af91bb700 10 osd.11 pg_epoch: 1673 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93 1673/1673/1673) [11]
r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0 peering]
handle_peering_event: epoch_sent: 1673 epoch_requested: 1673 NullEvt
2017-07-10 15:42:52.610190 7f8b06334700 10 osd.11 pg_epoch: 1673 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93 1673/1673/1673) [11]
r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0 peering] flushed
2017-07-10 15:42:52.640813 7f8af91bb700 10 osd.11 pg_epoch: 1673 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93 1673/1673/1673) [11]
r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0 peering]
handle_peering_event: epoch_sent: 1673 epoch_requested: 1673 MNotifyRec from 21
notify: (query_epoch:1673, epoch_sent:1673, info:1.20f( v 1663'655
(1663'600,1663'655] local-les=180 n=69 ec=75 les/c 180/180 1673/1673/1673))
features: 0x483ffffffffffff
2017-07-10 15:42:52.640821 7f8af91bb700 7 osd.11 pg_epoch: 1673 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93 1673/1673/1673) [11]
r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0 peering]
state<Started/Primary>: handle_pg_notify from osd.21
2017-07-10 15:42:52.640829 7f8af91bb700 10 osd.11 pg_epoch: 1673 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93 1673/1673/1673) [11]
r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0 peering] got osd.21
1.20f( v 1663'655 (1663'600,1663'655] local-les=180 n=69 ec=75 les/c 180/180
1673/1673/1673)
2017-07-10 15:42:52.640840 7f8af91bb700 10 osd.11 pg_epoch: 1673 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 180/180 1673/1673/1673)
[11] r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0 peering] osd.21
has stray content: 1.20f( v 1663'655 (1663'600,1663'655] local-les=180 n=69
ec=75 les/c 180/180 1673/1673/1673)
2017-07-10 15:42:52.640852 7f8af91bb700 10 osd.11 pg_epoch: 1673 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 180/180 1673/1673/1673)
[11] r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0 peering]
update_heartbeat_peers 11 -> 11,21
2017-07-10 15:42:52.642542 7f8af89ba700 10 osd.11 pg_epoch: 1673 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 180/180 1673/1673/1673)
[11] r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0 peering]
handle_peering_event: epoch_sent: 1673 epoch_requested: 1673 FlushedEvt
2017-07-10 15:42:53.549732 7f8b031cf700 10 osd.11 pg_epoch: 1673 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 180/180 1673/1673/1673)
[11] r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0 peering] null
2017-07-10 15:42:53.556132 7f8af99bc700 10 osd.11 pg_epoch: 1673 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 180/180 1673/1673/1673)
[11] r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0 peering]
handle_advance_map [11]/[11] -- 11/11
2017-07-10 15:42:53.556142 7f8af99bc700 10 osd.11 pg_epoch: 1674 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 180/180 1673/1673/1673)
[11] r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0 peering]
state<Started/Primary/Peering>: Peering advmap
2017-07-10 15:42:53.556149 7f8af99bc700 10 osd.11 pg_epoch: 1674 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 180/180 1673/1673/1673)
[11] r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0 peering]
adjust_need_up_thru now 1673, need_up_thru now false
2017-07-10 15:42:53.556157 7f8af99bc700 10 osd.11 pg_epoch: 1674 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 180/180 1673/1673/1673)
[11] r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0 peering]
state<Started>: Started advmap
2017-07-10 15:42:53.556164 7f8af99bc700 10 osd.11 pg_epoch: 1674 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 180/180 1673/1673/1673)
[11] r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0 peering]
check_recovery_sources no source osds () went down
2017-07-10 15:42:53.556173 7f8af99bc700 10 osd.11 pg_epoch: 1674 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 180/180 1673/1673/1673)
[11] r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0 peering]
handle_activate_map
2017-07-10 15:42:53.556179 7f8af99bc700 7 osd.11 pg_epoch: 1674 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 180/180 1673/1673/1673)
[11] r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0 peering]
state<Started/Primary>: handle ActMap primary
2017-07-10 15:42:53.556187 7f8af99bc700 10 osd.11 pg_epoch: 1674 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 180/180 1673/1673/1673)
[11] r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0 peering]
take_waiters
2017-07-10 15:42:53.556194 7f8af99bc700 5 osd.11 pg_epoch: 1674 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 180/180 1673/1673/1673)
[11] r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0 peering] exit
Started/Primary/Peering/WaitUpThru 0.947776 5 0.008159
2017-07-10 15:42:53.556203 7f8af99bc700 10 osd.11 pg_epoch: 1674 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 180/180 1673/1673/1673)
[11] r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0 peering]
state<Started/Primary/Peering>: Leaving Peering
2017-07-10 15:42:53.556211 7f8af99bc700 5 osd.11 pg_epoch: 1674 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 180/180 1673/1673/1673)
[11] r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0 peering] exit
Started/Primary/Peering 0.955776 0 0.000000
2017-07-10 15:42:53.556221 7f8af99bc700 5 osd.11 pg_epoch: 1674 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 180/180 1673/1673/1673)
[11] r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0 inactive] enter
Started/Primary/Active
2017-07-10 15:42:53.556232 7f8af99bc700 10 osd.11 pg_epoch: 1674 pg[1.20f( v
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 180/180 1673/1673/1673)
[11] r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0 inactive]
state<Started/Primary/Active>: In Active, about to call activate
2017-07-10 15:42:53.556241 7f8af99bc700 10 osd.11 pg_epoch: 1674 pg[1.20f( v
95'397 (95'300,95'397] local-les=1674 n=69 ec=75 les/c 180/180 1673/1673/1673)
[11] r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0 inactive] activate
- snap_trimq []
2017-07-10 15:42:53.556249 7f8af99bc700 10 osd.11 pg_epoch: 1674 pg[1.20f( v
95'397 (95'300,95'397] local-les=1674 n=69 ec=75 les/c 180/180 1673/1673/1673)
[11] r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0 inactive] activate
- no missing, moving last_complete 95'397 -> 95'397
2017-07-10 15:42:53.556261 7f8af99bc700 10 osd.11 pg_epoch: 1674 pg[1.20f( v
95'397 (95'300,95'397] local-les=1674 n=69 ec=75 les/c 180/180 1673/1673/1673)
[11] r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0 inactive] 1.20f
primary missing_loc.add_active_missing peer 11 missing: {}
2017-07-10 15:42:53.556268 7f8af99bc700 10 osd.11 pg_epoch: 1674 pg[1.20f( v
95'397 (95'300,95'397] local-les=1674 n=69 ec=75 les/c 180/180 1673/1673/1673)
[11] r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0 inactive]
needs_recovery is recovered
2017-07-10 15:42:53.556275 7f8af99bc700 10 osd.11 pg_epoch: 1674 pg[1.20f( v
95'397 (95'300,95'397] local-les=1674 n=69 ec=75 les/c 180/180 1673/1673/1673)
[11] r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0
activating+undersized+degraded] state<Started/Primary/Active>: Activate Finished
2017-07-10 15:42:53.556283 7f8af99bc700 5 osd.11 pg_epoch: 1674 pg[1.20f( v
95'397 (95'300,95'397] local-les=1674 n=69 ec=75 les/c 180/180 1673/1673/1673)
[11] r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0
activating+undersized+degraded] enter Started/Primary/Active/Activating
2017-07-10 15:42:53.556290 7f8af99bc700 10 osd.11 pg_epoch: 1674 pg[1.20f( v
95'397 (95'300,95'397] local-les=1674 n=69 ec=75 les/c 180/180 1673/1673/1673)
[11] r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0
activating+undersized+degraded] handle_peering_event: epoch_sent: 1674
epoch_requested: 1674 NullEvt
2017-07-10 15:42:53.564228 7f8b06334700 10 osd.11 pg_epoch: 1674 pg[1.20f( v
95'397 (95'300,95'397] local-les=1674 n=69 ec=75 les/c 180/180 1673/1673/1673)
[11] r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0
activating+undersized+degraded] _activate_committed 1674 peer_activated now 11
last_epoch_started 180 same_interval_since 1673
2017-07-10 15:42:53.564250 7f8b06334700 10 osd.11 pg_epoch: 1674 pg[1.20f( v
95'397 (95'300,95'397] local-les=1674 n=69 ec=75 les/c 180/180 1673/1673/1673)
[11] r=0 lpr=1673 pi=92-1672/3 crt=95'394 lcod 0'0 mlcod 0'0
activating+undersized+degraded] all_activated_and_committed
osd 7 log:
2017-07-10 16:08:29.812470 7f4fb425a700 7 osd.7 2123 handle_pg_info pg_info(1
pgs e2123:1.20f(6)) v4 from osd.11
2017-07-10 16:08:29.833669 7f4fac24a700 10 osd.7 pg_epoch: 2123 pg[1.20f( v
1663'655 (1663'600,1663'655] local-les=180 n=69 ec=75 les/c 1898/1898
2122/2122/1673) [11,7] r=1 lpr=2122 pi=179-2121/15 crt=1663'652 lcod 0'0
inactive NOTIFY] handle_peering_event: epoch_sent: 2123 epoch_requested: 2123
MInfoRec from 11 info: 1.20f( v 95'397 (95'300,95'397] local-les=2123 n=69
ec=75 les/c 1898/1898 2122/2122/1673)
2017-07-10 16:08:29.833684 7f4fac24a700 10 osd.7 pg_epoch: 2123 pg[1.20f( v
1663'655 (1663'600,1663'655] local-les=180 n=69 ec=75 les/c 1898/1898
2122/2122/1673) [11,7] r=1 lpr=2122 pi=179-2121/15 crt=1663'652 lcod 0'0
inactive NOTIFY] state<Started/Stray>: got info from osd.11 1.20f( v 95'397
(95'300,95'397] local-les=2123 n=69 ec=75 les/c 1898/1898 2122/2122/1673)
-------------------------------------------------------------------------------------------------------------------------------------
本邮件及其附件含有新华三技术有限公司的保密信息，仅限于发送给上面地址中列出
的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、
或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本
邮件！
This e-mail and its attachments contain confidential information from New H3C,
which is
intended only for the person or entity whose address is listed above. Any use
of the
information contained herein in any way (including, but not limited to, total
or partial
disclosure, reproduction, or dissemination) by persons other than the intended
recipient(s) is prohibited. If you receive this e-mail in error, please notify
the sender
by phone or email immediately and delete it!
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
