Hi List, FYI
I tripped across a bug when adding a new monitor into an existing cluster
(see attached). I was on GIT commit 4b3bb5ab37a05fa001d59f24da7d9c30d650321b.I found that the function 'bool Paxos::is_consistent()' was added in commit
0bd545f5d6beb790e67fe2ee61f57d2ed6332531. I reset back to commit 3a7bb9999b49641ee28cac91e493aa4c231ede54 (it's parent) and it has gone away. -- Thanks, Dyweni
2012-02-17 11:39:07.797739 b6efc710 store(/ceph/mon/disk7dc86aa3e88a9a2def2153a3f6c127e44deec449/mondata) mount 2012-02-17 11:39:07.797785 b6efc710 store(/ceph/mon/disk7dc86aa3e88a9a2def2153a3f6c127e44deec449/mondata) reading at off 0 of 21 2012-02-17 11:39:07.797792 b6efc710 store(/ceph/mon/disk7dc86aa3e88a9a2def2153a3f6c127e44deec449/mondata) get_bl magic = 21 bytes 2012-02-17 11:39:07.797800 b6efc710 store(/ceph/mon/disk7dc86aa3e88a9a2def2153a3f6c127e44deec449/mondata) reading at off 0 of 75 2012-02-17 11:39:07.797803 b6efc710 store(/ceph/mon/disk7dc86aa3e88a9a2def2153a3f6c127e44deec449/mondata) get_bl feature_set = 75 bytes 2012-02-17 11:39:07.797818 b6efc710 store(/ceph/mon/disk7dc86aa3e88a9a2def2153a3f6c127e44deec449/mondata) get_bl monmap/latest No such file or directory 2012-02-17 11:39:07.797823 b6efc710 store(/ceph/mon/disk7dc86aa3e88a9a2def2153a3f6c127e44deec449/mondata) reading at off 0 of 588 2012-02-17 11:39:07.797826 b6efc710 store(/ceph/mon/disk7dc86aa3e88a9a2def2153a3f6c127e44deec449/mondata) get_bl mkfs/monmap = 588 bytes 2012-02-17 11:39:07.797851 b6efc710 mon.2a597b6ca798c9cd55a43279598b81a328df7a3c_3 does not exist in monmap, will attempt to join an existing cluster [0;33m ** WARNING: Ceph is still under development. Any feedback can be directed **[0m [0;33m ** at [email protected] or http://ceph.newdream.net/. **[0m 2012-02-17 11:39:07.798028 b6efc710 ceph version 0.41-369-g4b3bb5a (commit:4b3bb5ab37a05fa001d59f24da7d9c30d650321b), process ceph-mon, pid 1125 starting mon.2a597b6ca798c9cd55a43279598b81a328df7a3c_3 rank -1 at 192.168.0.4:6789/0 mon_data /ceph/mon/disk7dc86aa3e88a9a2def2153a3f6c127e44deec449/mondata fsid 73d6c057-d1dd-4e30-85b3-e87a2f97fa3f 2012-02-17 11:39:07.798256 b6efc710 mon.2a597b6ca798c9cd55a43279598b81a328df7a3c_3@-1(probing).pg v0 update_full_ratios full 0.95 nearfull 0.85 2012-02-17 11:39:07.798313 b6efc710 mon.2a597b6ca798c9cd55a43279598b81a328df7a3c_3@-1(probing) e1 init fsid 73d6c057-d1dd-4e30-85b3-e87a2f97fa3f 2012-02-17 11:39:07.798336 b6efc710 store(/ceph/mon/disk7dc86aa3e88a9a2def2153a3f6c127e44deec449/mondata) get_bl pgmap/latest No such file or directory 2012-02-17 11:39:07.798350 b6efc710 store(/ceph/mon/disk7dc86aa3e88a9a2def2153a3f6c127e44deec449/mondata) get_int mdsmap/last_committed = 6 2012-02-17 11:39:07.798355 b6efc710 store(/ceph/mon/disk7dc86aa3e88a9a2def2153a3f6c127e44deec449/mondata) get_int mdsmap/first_committed = 1 2012-02-17 11:39:07.798361 b6efc710 store(/ceph/mon/disk7dc86aa3e88a9a2def2153a3f6c127e44deec449/mondata) reading at off 0 of 1036 2012-02-17 11:39:07.798365 b6efc710 store(/ceph/mon/disk7dc86aa3e88a9a2def2153a3f6c127e44deec449/mondata) get_bl mdsmap/latest = 1036 bytes 2012-02-17 11:39:07.798373 b6efc710 store(/ceph/mon/disk7dc86aa3e88a9a2def2153a3f6c127e44deec449/mondata) get_int mdsmap/slurping = 0 2012-02-17 11:39:07.798379 b6efc710 mon.2a597b6ca798c9cd55a43279598b81a328df7a3c_3@-1(probing).mds e0 update_from_paxos paxosv 6, my e 0 2012-02-17 11:39:07.798385 b6efc710 store(/ceph/mon/disk7dc86aa3e88a9a2def2153a3f6c127e44deec449/mondata) reading at off 0 of 1024 2012-02-17 11:39:07.798389 b6efc710 store(/ceph/mon/disk7dc86aa3e88a9a2def2153a3f6c127e44deec449/mondata) get_bl mdsmap/6 = 1024 bytes 2012-02-17 11:39:07.798392 b6efc710 mon.2a597b6ca798c9cd55a43279598b81a328df7a3c_3@-1(probing).mds e0 update_from_paxos got 6 2012-02-17 11:39:07.798429 b6efc710 mon.2a597b6ca798c9cd55a43279598b81a328df7a3c_3@-1(probing).mds e6 new map 2012-02-17 11:39:07.798463 b6efc710 mon.2a597b6ca798c9cd55a43279598b81a328df7a3c_3@-1(probing).mds e6 print_map epoch 6 flags 0 created 2012-02-17 11:19:48.979439 modified 2012-02-17 11:20:14.357436 tableserver 0 root 0 session_timeout 60 session_autoclose 300 last_failure 0 last_failure_osd_epoch 0 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object} max_mds 1 in 0 up {0=4098} failed stopped data_pools [0] metadata_pool 1 4098: 192.168.0.2:6800/1228 '24c94c97f0d6d131ee1ad7173d0f78b350d41ac1' mds.0.1 up:active seq 4 4099: 192.168.0.3:6800/1223 '3be17740a935bde47911960f1a6bd28f947b2391' mds.-1.0 up:standby seq 1 4105: 192.168.0.1:6800/1866 '85ffeac8368302141b36f2500e0e96289bd9e6cd' mds.-1.0 up:standby seq 1 2012-02-17 11:39:07.798469 b6efc710 mon.2a597b6ca798c9cd55a43279598b81a328df7a3c_3@-1(probing).mds e6 update_logger 2012-02-17 11:39:07.798480 b6efc710 store(/ceph/mon/disk7dc86aa3e88a9a2def2153a3f6c127e44deec449/mondata) get_bl osdmap/latest No such file or directory 2012-02-17 11:39:07.798490 b6efc710 store(/ceph/mon/disk7dc86aa3e88a9a2def2153a3f6c127e44deec449/mondata) get_int logm/last_committed = 149 2012-02-17 11:39:07.798495 b6efc710 store(/ceph/mon/disk7dc86aa3e88a9a2def2153a3f6c127e44deec449/mondata) get_int logm/first_committed = 1 2012-02-17 11:39:07.798501 b6efc710 store(/ceph/mon/disk7dc86aa3e88a9a2def2153a3f6c127e44deec449/mondata) reading at off 0 of 9328 2012-02-17 11:39:07.798509 b6efc710 store(/ceph/mon/disk7dc86aa3e88a9a2def2153a3f6c127e44deec449/mondata) get_bl logm/latest = 9328 bytes 2012-02-17 11:39:07.798515 b6efc710 store(/ceph/mon/disk7dc86aa3e88a9a2def2153a3f6c127e44deec449/mondata) get_int logm/slurping = 0 2012-02-17 11:39:07.798521 b6efc710 store(/ceph/mon/disk7dc86aa3e88a9a2def2153a3f6c127e44deec449/mondata) reading at off 0 of 9328 2012-02-17 11:39:07.798525 b6efc710 store(/ceph/mon/disk7dc86aa3e88a9a2def2153a3f6c127e44deec449/mondata) get_bl logm/latest = 9328 bytes 2012-02-17 11:39:07.798530 b6efc710 mon.2a597b6ca798c9cd55a43279598b81a328df7a3c_3@-1(probing).log v149 update_from_paxos loading summary e149 2012-02-17 11:39:07.798722 b6efc710 store(/ceph/mon/disk7dc86aa3e88a9a2def2153a3f6c127e44deec449/mondata) get_int monmap/last_committed = 1 2012-02-17 11:39:07.798727 b6efc710 store(/ceph/mon/disk7dc86aa3e88a9a2def2153a3f6c127e44deec449/mondata) get_int monmap/first_committed = 1 2012-02-17 11:39:07.798731 b6efc710 store(/ceph/mon/disk7dc86aa3e88a9a2def2153a3f6c127e44deec449/mondata) get_bl monmap/latest No such file or directory 2012-02-17 11:39:07.798736 b6efc710 store(/ceph/mon/disk7dc86aa3e88a9a2def2153a3f6c127e44deec449/mondata) get_int monmap/slurping = 0 mon/Paxos.cc: In function 'bool Paxos::is_consistent()' thread b6efc710 time 2012-02-17 11:39:07.798742 mon/Paxos.cc: 1081: FAILED assert(consistent || (slurping == 1)) ceph version 0.41-369-g4b3bb5a (commit:4b3bb5ab37a05fa001d59f24da7d9c30d650321b) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0xa4) [0x84377e5] 2: (Paxos::is_consistent()+0xd8) [0x82e066e] 3: (Monitor::init()+0x616) [0x82a4bd0] 4: (main()+0x1edf) [0x82969da] 5: (__libc_start_main()+0xe6) [0xb6f18296] 6: ceph-mon() [0x8294931] ceph version 0.41-369-g4b3bb5a (commit:4b3bb5ab37a05fa001d59f24da7d9c30d650321b) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0xa4) [0x84377e5] 2: (Paxos::is_consistent()+0xd8) [0x82e066e] 3: (Monitor::init()+0x616) [0x82a4bd0] 4: (main()+0x1edf) [0x82969da] 5: (__libc_start_main()+0xe6) [0xb6f18296] 6: ceph-mon() [0x8294931] terminate called after throwing an instance of 'ceph::FailedAssertion' *** Caught signal (Aborted) ** in thread b6efc710 ceph version 0.41-369-g4b3bb5a (commit:4b3bb5ab37a05fa001d59f24da7d9c30d650321b) 1: (ceph::BackTrace::BackTrace(int)+0x30) [0x8437b44] 2: ceph-mon() [0x8484796] 3: [0xb7785400] 4: [0xb7785424] 5: (gsignal()+0x51) [0xb6f2c171] 6: (abort()+0x17e) [0xb6f2d94e] 7: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0xb7150d85] 8: (()+0xafc85) [0xb714ec85] 9: (()+0xafcc2) [0xb714ecc2] 10: (()+0xafe31) [0xb714ee31] 11: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x21e) [0x843795f] 12: (Paxos::is_consistent()+0xd8) [0x82e066e] 13: (Monitor::init()+0x616) [0x82a4bd0] 14: (main()+0x1edf) [0x82969da] 15: (__libc_start_main()+0xe6) [0xb6f18296] 16: ceph-mon() [0x8294931]
