Hi,
I have been very impressed with the BlueStore test environment I made,
which is build on the Ubuntu 16.04 using the Ceph development master
repository.
But now I have run into some self inflicted problems.
Yesterday I accidentally updated the OSD while they were being heavily
used. Then the OSD started to go down one by one, and when they all had
did I ended up with pgs in practically every possible state.
/:~# ceph health //
//2016-09-30 09:50:39.044987 7f27e4ee2700 -1 WARNING: the following
dangerous and experimental features are enabled: bluestore,rocksdb//
//2016-09-30 09:50:39.052592 7f27e4ee2700 -1 WARNING: the following
dangerous and experimental features are enabled: bluestore,rocksdb//
//HEALTH_ERR 243 pgs are stuck inactive for more than 300 seconds;
130 pgs backfill_wait; 488 pgs degraded; 55 pgs down; 49 pgs
incomplete; 63 pgs peering; 2 pgs recovering; 281 pgs recovery_wait;
600 pgs stale; 243 pgs stuck inactive; 357 pgs stuck unclean; 488
pgs undersized; recovery 1240205/1848822 objects degraded (67.081%);
recovery 397635/1848822 objects misplaced (21.507%); recovery
57149/616274 unfound (9.273%); mds cluster is degraded; 8/8 in osds
are down/
As mentioned all OSD's are now down and refuse to come back up.
From the osd log file I see this error message:
//srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-11.0.0-2808-g76e120c/src/os/bluestore/StupidAllocator.cc:
317: FAILED assert(rm.empty())/
Of course the data was not important in this test environment and the
easiest would properly be to start over, but I am considering building a
production environment build on Bluestore as soon as it becomes stable,
so for the sport of it I would like to see if I can actually recover the
OSD's. Just to get some deeper insight into Ceph recovery.
I have been though:
http://docs.ceph.com/docs/jewel/rados/troubleshooting/troubleshooting-osd/
without any luck.
What would be the next steps to try?
Thanks!
//Jesper
2016-09-30 08:51:23.464389 7f17985528c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
2016-09-30 08:51:23.464409 7f17985528c0 0 set uid:gid to 64045:64045 (ceph:ceph)
2016-09-30 08:51:23.464430 7f17985528c0 0 ceph version v11.0.0-2808-g76e120c (76e120c705b77d2d2cef1b94cacdd11c14460a3f), process ceph-osd, pid 16590
2016-09-30 08:51:23.464479 7f17985528c0 5 object store type is bluestore
2016-09-30 08:51:23.464504 7f17985528c0 -1 WARNING: experimental feature 'bluestore' is enabled
Please be aware that this feature is experimental, untested,
unsupported, and may result in data corruption, data loss,
and/or irreparable damage to your cluster. Do not use
feature with important data.
2016-09-30 08:51:23.466344 7f17985528c0 0 pidfile_write: ignore empty --pid-file
2016-09-30 08:51:23.468345 7f17985528c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
2016-09-30 08:51:23.473859 7f17985528c0 10 ErasureCodePluginSelectJerasure: load: jerasure_sse4
2016-09-30 08:51:23.475249 7f17985528c0 10 load: jerasure load: lrc load: isa
2016-09-30 08:51:23.475657 7f17985528c0 2 osd.0 0 mounting /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal
2016-09-30 08:51:23.475669 7f17985528c0 1 bluestore(/var/lib/ceph/osd/ceph-0) mount path /var/lib/ceph/osd/ceph-0
2016-09-30 08:51:23.475707 7f17985528c0 1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel
2016-09-30 08:51:23.476182 7f17985528c0 1 bdev(/var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
2016-09-30 08:51:23.476545 7f17985528c0 1 bdev(/var/lib/ceph/osd/ceph-0/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) non-rotational
2016-09-30 08:51:23.476814 7f17985528c0 1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel
2016-09-30 08:51:23.477256 7f17985528c0 1 bdev(/var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
2016-09-30 08:51:23.477501 7f17985528c0 1 bdev(/var/lib/ceph/osd/ceph-0/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) non-rotational
2016-09-30 08:51:23.477509 7f17985528c0 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-0/block size 3725 GB
2016-09-30 08:51:23.477533 7f17985528c0 1 bluefs mount
2016-09-30 08:51:23.553415 7f17985528c0 -1 /srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-11.0.0-2808-g76e120c/src/os/bluestore/StupidAllocator.cc: In function 'virtual void StupidAllocator::init_rm_free(uint64_t, uint64_t)' thread 7f17985528c0 time 2016-09-30 08:51:23.550071
/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-11.0.0-2808-g76e120c/src/os/bluestore/StupidAllocator.cc: 317: FAILED assert(rm.empty())
ceph version v11.0.0-2808-g76e120c (76e120c705b77d2d2cef1b94cacdd11c14460a3f)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x80) [0x55d4c31a6240]
2: (StupidAllocator::init_rm_free(unsigned long, unsigned long)+0x1b8c) [0x55d4c2ff36cc]
3: (BlueFS::mount()+0x40a) [0x55d4c2fcf2ba]
4: (BlueStore::_open_db(bool)+0xc45) [0x55d4c2efd825]
5: (BlueStore::mount()+0x38d) [0x55d4c2f18aad]
6: (OSD::init()+0x273) [0x55d4c2b17ad3]
7: (main()+0x2ea9) [0x55d4c2a65cd9]
8: (__libc_start_main()+0xf0) [0x7f1794f93830]
9: (_start()+0x29) [0x55d4c2ac3899]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- begin dump of recent events ---
-69> 2016-09-30 08:51:23.459785 7f17985528c0 5 asok(0x55d4cc850f00) register_command perfcounters_dump hook 0x55d4cc814030
-68> 2016-09-30 08:51:23.459810 7f17985528c0 5 asok(0x55d4cc850f00) register_command 1 hook 0x55d4cc814030
-67> 2016-09-30 08:51:23.459815 7f17985528c0 5 asok(0x55d4cc850f00) register_command perf dump hook 0x55d4cc814030
-66> 2016-09-30 08:51:23.459822 7f17985528c0 5 asok(0x55d4cc850f00) register_command perfcounters_schema hook 0x55d4cc814030
-65> 2016-09-30 08:51:23.459829 7f17985528c0 5 asok(0x55d4cc850f00) register_command 2 hook 0x55d4cc814030
-64> 2016-09-30 08:51:23.459834 7f17985528c0 5 asok(0x55d4cc850f00) register_command perf schema hook 0x55d4cc814030
-63> 2016-09-30 08:51:23.459841 7f17985528c0 5 asok(0x55d4cc850f00) register_command perf reset hook 0x55d4cc814030
-62> 2016-09-30 08:51:23.459845 7f17985528c0 5 asok(0x55d4cc850f00) register_command config show hook 0x55d4cc814030
-61> 2016-09-30 08:51:23.459852 7f17985528c0 5 asok(0x55d4cc850f00) register_command config set hook 0x55d4cc814030
-60> 2016-09-30 08:51:23.459855 7f17985528c0 5 asok(0x55d4cc850f00) register_command config get hook 0x55d4cc814030
-59> 2016-09-30 08:51:23.459862 7f17985528c0 5 asok(0x55d4cc850f00) register_command config diff hook 0x55d4cc814030
-58> 2016-09-30 08:51:23.459867 7f17985528c0 5 asok(0x55d4cc850f00) register_command log flush hook 0x55d4cc814030
-57> 2016-09-30 08:51:23.459872 7f17985528c0 5 asok(0x55d4cc850f00) register_command log dump hook 0x55d4cc814030
-56> 2016-09-30 08:51:23.459876 7f17985528c0 5 asok(0x55d4cc850f00) register_command log reopen hook 0x55d4cc814030
-55> 2016-09-30 08:51:23.464140 7f17985528c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
-54> 2016-09-30 08:51:23.464389 7f17985528c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
-53> 2016-09-30 08:51:23.464409 7f17985528c0 0 set uid:gid to 64045:64045 (ceph:ceph)
-52> 2016-09-30 08:51:23.464430 7f17985528c0 0 ceph version v11.0.0-2808-g76e120c (76e120c705b77d2d2cef1b94cacdd11c14460a3f), process ceph-osd, pid 16590
-51> 2016-09-30 08:51:23.464479 7f17985528c0 5 object store type is bluestore
-50> 2016-09-30 08:51:23.464504 7f17985528c0 -1 WARNING: experimental feature 'bluestore' is enabled
Please be aware that this feature is experimental, untested,
unsupported, and may result in data corruption, data loss,
and/or irreparable damage to your cluster. Do not use
feature with important data.
-49> 2016-09-30 08:51:23.465448 7f179309f700 2 Event(0x55d4cc851160 nevent=5000 time_id=1).set_owner idx=0 owner=139739227879168
-48> 2016-09-30 08:51:23.465482 7f179209d700 2 Event(0x55d4cc851520 nevent=5000 time_id=1).set_owner idx=2 owner=139739211093760
-47> 2016-09-30 08:51:23.465523 7f179289e700 2 Event(0x55d4cc851340 nevent=5000 time_id=1).set_owner idx=1 owner=139739219486464
-46> 2016-09-30 08:51:23.465740 7f17985528c0 2 Event(0x55d4cc851160 nevent=5000 time_id=1).wakeup
-45> 2016-09-30 08:51:23.465854 7f17985528c0 1 Processor -- bind bind my_inst.addr is 0.0.0.0:6800/16590
-44> 2016-09-30 08:51:23.465873 7f17985528c0 2 Event(0x55d4cc851160 nevent=5000 time_id=1).wakeup
-43> 2016-09-30 08:51:23.465949 7f17985528c0 2 Event(0x55d4cc851160 nevent=5000 time_id=1).wakeup
-42> 2016-09-30 08:51:23.465996 7f17985528c0 1 Processor -- bind bind my_inst.addr is 0.0.0.0:6801/16590
-41> 2016-09-30 08:51:23.466009 7f17985528c0 2 Event(0x55d4cc851160 nevent=5000 time_id=1).wakeup
-40> 2016-09-30 08:51:23.466053 7f17985528c0 2 Event(0x55d4cc851160 nevent=5000 time_id=1).wakeup
-39> 2016-09-30 08:51:23.466104 7f17985528c0 2 Event(0x55d4cc851160 nevent=5000 time_id=1).wakeup
-38> 2016-09-30 08:51:23.466166 7f17985528c0 1 Processor -- bind bind my_inst.addr is 0.0.0.0:6802/16590
-37> 2016-09-30 08:51:23.466181 7f17985528c0 2 Event(0x55d4cc851160 nevent=5000 time_id=1).wakeup
-36> 2016-09-30 08:51:23.466220 7f17985528c0 2 Event(0x55d4cc851160 nevent=5000 time_id=1).wakeup
-35> 2016-09-30 08:51:23.466257 7f17985528c0 2 Event(0x55d4cc851160 nevent=5000 time_id=1).wakeup
-34> 2016-09-30 08:51:23.466295 7f17985528c0 2 Event(0x55d4cc851160 nevent=5000 time_id=1).wakeup
-33> 2016-09-30 08:51:23.466330 7f17985528c0 1 Processor -- bind bind my_inst.addr is 0.0.0.0:6803/16590
-32> 2016-09-30 08:51:23.466344 7f17985528c0 0 pidfile_write: ignore empty --pid-file
-31> 2016-09-30 08:51:23.468345 7f17985528c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
-30> 2016-09-30 08:51:23.468362 7f17985528c0 5 asok(0x55d4cc850f00) init /var/run/ceph/ceph-osd.0.asok
-29> 2016-09-30 08:51:23.468374 7f17985528c0 5 asok(0x55d4cc850f00) bind_and_listen /var/run/ceph/ceph-osd.0.asok
-28> 2016-09-30 08:51:23.468536 7f17985528c0 5 asok(0x55d4cc850f00) register_command 0 hook 0x55d4cc810108
-27> 2016-09-30 08:51:23.468551 7f17985528c0 5 asok(0x55d4cc850f00) register_command version hook 0x55d4cc810108
-26> 2016-09-30 08:51:23.468561 7f17985528c0 5 asok(0x55d4cc850f00) register_command git_version hook 0x55d4cc810108
-25> 2016-09-30 08:51:23.468571 7f17985528c0 5 asok(0x55d4cc850f00) register_command help hook 0x55d4cc8143c0
-24> 2016-09-30 08:51:23.468581 7f17985528c0 5 asok(0x55d4cc850f00) register_command get_command_descriptions hook 0x55d4cc8143d0
-23> 2016-09-30 08:51:23.468643 7f1790900700 5 asok(0x55d4cc850f00) entry start
-22> 2016-09-30 08:51:23.468684 7f17985528c0 10 monclient(hunting): build_initial_monmap
-21> 2016-09-30 08:51:23.473859 7f17985528c0 10 ErasureCodePluginSelectJerasure: load: jerasure_sse4
-20> 2016-09-30 08:51:23.475249 7f17985528c0 10 load: jerasure load: lrc load: isa
-19> 2016-09-30 08:51:23.475320 7f17985528c0 5 adding auth protocol: cephx
-18> 2016-09-30 08:51:23.475325 7f17985528c0 5 adding auth protocol: cephx
-17> 2016-09-30 08:51:23.475508 7f17985528c0 5 asok(0x55d4cc850f00) register_command objecter_requests hook 0x55d4cc814470
-16> 2016-09-30 08:51:23.475571 7f17985528c0 1 -- 0.0.0.0:6800/16590 start start
-15> 2016-09-30 08:51:23.475581 7f17985528c0 1 -- :/0 start start
-14> 2016-09-30 08:51:23.475583 7f17985528c0 1 -- 0.0.0.0:6803/16590 start start
-13> 2016-09-30 08:51:23.475586 7f17985528c0 1 -- 0.0.0.0:6802/16590 start start
-12> 2016-09-30 08:51:23.475589 7f17985528c0 1 -- 0.0.0.0:6801/16590 start start
-11> 2016-09-30 08:51:23.475591 7f17985528c0 1 -- :/0 start start
-10> 2016-09-30 08:51:23.475657 7f17985528c0 2 osd.0 0 mounting /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal
-9> 2016-09-30 08:51:23.475669 7f17985528c0 1 bluestore(/var/lib/ceph/osd/ceph-0) mount path /var/lib/ceph/osd/ceph-0
-8> 2016-09-30 08:51:23.475707 7f17985528c0 1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel
-7> 2016-09-30 08:51:23.476182 7f17985528c0 1 bdev(/var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
-6> 2016-09-30 08:51:23.476545 7f17985528c0 1 bdev(/var/lib/ceph/osd/ceph-0/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) non-rotational
-5> 2016-09-30 08:51:23.476814 7f17985528c0 1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel
-4> 2016-09-30 08:51:23.477256 7f17985528c0 1 bdev(/var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
-3> 2016-09-30 08:51:23.477501 7f17985528c0 1 bdev(/var/lib/ceph/osd/ceph-0/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) non-rotational
-2> 2016-09-30 08:51:23.477509 7f17985528c0 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-0/block size 3725 GB
-1> 2016-09-30 08:51:23.477533 7f17985528c0 1 bluefs mount
0> 2016-09-30 08:51:23.553415 7f17985528c0 -1 /srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-11.0.0-2808-g76e120c/src/os/bluestore/StupidAllocator.cc: In function 'virtual void StupidAllocator::init_rm_free(uint64_t, uint64_t)' thread 7f17985528c0 time 2016-09-30 08:51:23.550071
/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-11.0.0-2808-g76e120c/src/os/bluestore/StupidAllocator.cc: 317: FAILED assert(rm.empty())
ceph version v11.0.0-2808-g76e120c (76e120c705b77d2d2cef1b94cacdd11c14460a3f)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x80) [0x55d4c31a6240]
2: (StupidAllocator::init_rm_free(unsigned long, unsigned long)+0x1b8c) [0x55d4c2ff36cc]
3: (BlueFS::mount()+0x40a) [0x55d4c2fcf2ba]
4: (BlueStore::_open_db(bool)+0xc45) [0x55d4c2efd825]
5: (BlueStore::mount()+0x38d) [0x55d4c2f18aad]
6: (OSD::init()+0x273) [0x55d4c2b17ad3]
7: (main()+0x2ea9) [0x55d4c2a65cd9]
8: (__libc_start_main()+0xf0) [0x7f1794f93830]
9: (_start()+0x29) [0x55d4c2ac3899]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
20/20 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 newstore
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
4/ 5 memdb
1/ 5 kinetic
1/ 5 fuse
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-osd.0.log
--- end dump of recent events ---
2016-09-30 08:51:23.556963 7f17985528c0 -1 *** Caught signal (Aborted) **
in thread 7f17985528c0 thread_name:ceph-osd
ceph version v11.0.0-2808-g76e120c (76e120c705b77d2d2cef1b94cacdd11c14460a3f)
1: (()+0x8d8c1e) [0x55d4c3005c1e]
2: (()+0x113d0) [0x7f179679a3d0]
3: (gsignal()+0x38) [0x7f1794fa8418]
4: (abort()+0x16a) [0x7f1794faa01a]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26b) [0x55d4c31a642b]
6: (StupidAllocator::init_rm_free(unsigned long, unsigned long)+0x1b8c) [0x55d4c2ff36cc]
7: (BlueFS::mount()+0x40a) [0x55d4c2fcf2ba]
8: (BlueStore::_open_db(bool)+0xc45) [0x55d4c2efd825]
9: (BlueStore::mount()+0x38d) [0x55d4c2f18aad]
10: (OSD::init()+0x273) [0x55d4c2b17ad3]
11: (main()+0x2ea9) [0x55d4c2a65cd9]
12: (__libc_start_main()+0xf0) [0x7f1794f93830]
13: (_start()+0x29) [0x55d4c2ac3899]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- begin dump of recent events ---
0> 2016-09-30 08:51:23.556963 7f17985528c0 -1 *** Caught signal (Aborted) **
in thread 7f17985528c0 thread_name:ceph-osd
ceph version v11.0.0-2808-g76e120c (76e120c705b77d2d2cef1b94cacdd11c14460a3f)
1: (()+0x8d8c1e) [0x55d4c3005c1e]
2: (()+0x113d0) [0x7f179679a3d0]
3: (gsignal()+0x38) [0x7f1794fa8418]
4: (abort()+0x16a) [0x7f1794faa01a]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26b) [0x55d4c31a642b]
6: (StupidAllocator::init_rm_free(unsigned long, unsigned long)+0x1b8c) [0x55d4c2ff36cc]
7: (BlueFS::mount()+0x40a) [0x55d4c2fcf2ba]
8: (BlueStore::_open_db(bool)+0xc45) [0x55d4c2efd825]
9: (BlueStore::mount()+0x38d) [0x55d4c2f18aad]
10: (OSD::init()+0x273) [0x55d4c2b17ad3]
11: (main()+0x2ea9) [0x55d4c2a65cd9]
12: (__libc_start_main()+0xf0) [0x7f1794f93830]
13: (_start()+0x29) [0x55d4c2ac3899]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
20/20 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 newstore
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
4/ 5 memdb
1/ 5 kinetic
1/ 5 fuse
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-osd.0.log
--- end dump of recent events ---
2016-09-30 08:51:24.040032 7fe98b6148c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
2016-09-30 08:51:24.040044 7fe98b6148c0 0 set uid:gid to 64045:64045 (ceph:ceph)
2016-09-30 08:51:24.040055 7fe98b6148c0 0 ceph version v11.0.0-2808-g76e120c (76e120c705b77d2d2cef1b94cacdd11c14460a3f), process ceph-osd, pid 16661
2016-09-30 08:51:24.040085 7fe98b6148c0 5 object store type is bluestore
2016-09-30 08:51:24.040100 7fe98b6148c0 -1 WARNING: experimental feature 'bluestore' is enabled
Please be aware that this feature is experimental, untested,
unsupported, and may result in data corruption, data loss,
and/or irreparable damage to your cluster. Do not use
feature with important data.
2016-09-30 08:51:24.041250 7fe98b6148c0 0 pidfile_write: ignore empty --pid-file
2016-09-30 08:51:24.042414 7fe98b6148c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
2016-09-30 08:51:24.046019 7fe98b6148c0 10 ErasureCodePluginSelectJerasure: load: jerasure_sse4
2016-09-30 08:51:24.047218 7fe98b6148c0 10 load: jerasure load: lrc load: isa
2016-09-30 08:51:24.047598 7fe98b6148c0 2 osd.0 0 mounting /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal
2016-09-30 08:51:24.047608 7fe98b6148c0 1 bluestore(/var/lib/ceph/osd/ceph-0) mount path /var/lib/ceph/osd/ceph-0
2016-09-30 08:51:24.047651 7fe98b6148c0 1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel
2016-09-30 08:51:24.048033 7fe98b6148c0 1 bdev(/var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
2016-09-30 08:51:24.048354 7fe98b6148c0 1 bdev(/var/lib/ceph/osd/ceph-0/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) non-rotational
2016-09-30 08:51:24.048637 7fe98b6148c0 1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel
2016-09-30 08:51:24.048927 7fe98b6148c0 1 bdev(/var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
2016-09-30 08:51:24.049169 7fe98b6148c0 1 bdev(/var/lib/ceph/osd/ceph-0/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) non-rotational
2016-09-30 08:51:24.049177 7fe98b6148c0 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-0/block size 3725 GB
2016-09-30 08:51:24.049200 7fe98b6148c0 1 bluefs mount
2016-09-30 08:51:24.116575 7fe98b6148c0 -1 /srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-11.0.0-2808-g76e120c/src/os/bluestore/StupidAllocator.cc: In function 'virtual void StupidAllocator::init_rm_free(uint64_t, uint64_t)' thread 7fe98b6148c0 time 2016-09-30 08:51:24.114653
/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-11.0.0-2808-g76e120c/src/os/bluestore/StupidAllocator.cc: 317: FAILED assert(rm.empty())
ceph version v11.0.0-2808-g76e120c (76e120c705b77d2d2cef1b94cacdd11c14460a3f)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x80) [0x558f57423240]
2: (StupidAllocator::init_rm_free(unsigned long, unsigned long)+0x1b8c) [0x558f572706cc]
3: (BlueFS::mount()+0x40a) [0x558f5724c2ba]
4: (BlueStore::_open_db(bool)+0xc45) [0x558f5717a825]
5: (BlueStore::mount()+0x38d) [0x558f57195aad]
6: (OSD::init()+0x273) [0x558f56d94ad3]
7: (main()+0x2ea9) [0x558f56ce2cd9]
8: (__libc_start_main()+0xf0) [0x7fe988055830]
9: (_start()+0x29) [0x558f56d40899]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- begin dump of recent events ---
-69> 2016-09-30 08:51:24.036666 7fe98b6148c0 5 asok(0x558f62502f00) register_command perfcounters_dump hook 0x558f624c6030
-68> 2016-09-30 08:51:24.036692 7fe98b6148c0 5 asok(0x558f62502f00) register_command 1 hook 0x558f624c6030
-67> 2016-09-30 08:51:24.036699 7fe98b6148c0 5 asok(0x558f62502f00) register_command perf dump hook 0x558f624c6030
-66> 2016-09-30 08:51:24.036706 7fe98b6148c0 5 asok(0x558f62502f00) register_command perfcounters_schema hook 0x558f624c6030
-65> 2016-09-30 08:51:24.036712 7fe98b6148c0 5 asok(0x558f62502f00) register_command 2 hook 0x558f624c6030
-64> 2016-09-30 08:51:24.036714 7fe98b6148c0 5 asok(0x558f62502f00) register_command perf schema hook 0x558f624c6030
-63> 2016-09-30 08:51:24.036718 7fe98b6148c0 5 asok(0x558f62502f00) register_command perf reset hook 0x558f624c6030
-62> 2016-09-30 08:51:24.036723 7fe98b6148c0 5 asok(0x558f62502f00) register_command config show hook 0x558f624c6030
-61> 2016-09-30 08:51:24.036727 7fe98b6148c0 5 asok(0x558f62502f00) register_command config set hook 0x558f624c6030
-60> 2016-09-30 08:51:24.036732 7fe98b6148c0 5 asok(0x558f62502f00) register_command config get hook 0x558f624c6030
-59> 2016-09-30 08:51:24.036734 7fe98b6148c0 5 asok(0x558f62502f00) register_command config diff hook 0x558f624c6030
-58> 2016-09-30 08:51:24.036737 7fe98b6148c0 5 asok(0x558f62502f00) register_command log flush hook 0x558f624c6030
-57> 2016-09-30 08:51:24.036741 7fe98b6148c0 5 asok(0x558f62502f00) register_command log dump hook 0x558f624c6030
-56> 2016-09-30 08:51:24.036744 7fe98b6148c0 5 asok(0x558f62502f00) register_command log reopen hook 0x558f624c6030
-55> 2016-09-30 08:51:24.039856 7fe98b6148c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
-54> 2016-09-30 08:51:24.040032 7fe98b6148c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
-53> 2016-09-30 08:51:24.040044 7fe98b6148c0 0 set uid:gid to 64045:64045 (ceph:ceph)
-52> 2016-09-30 08:51:24.040055 7fe98b6148c0 0 ceph version v11.0.0-2808-g76e120c (76e120c705b77d2d2cef1b94cacdd11c14460a3f), process ceph-osd, pid 16661
-51> 2016-09-30 08:51:24.040085 7fe98b6148c0 5 object store type is bluestore
-50> 2016-09-30 08:51:24.040100 7fe98b6148c0 -1 WARNING: experimental feature 'bluestore' is enabled
Please be aware that this feature is experimental, untested,
unsupported, and may result in data corruption, data loss,
and/or irreparable damage to your cluster. Do not use
feature with important data.
-49> 2016-09-30 08:51:24.040658 7fe986161700 2 Event(0x558f62503160 nevent=5000 time_id=1).set_owner idx=0 owner=140640953702144
-48> 2016-09-30 08:51:24.040707 7fe985960700 2 Event(0x558f62503340 nevent=5000 time_id=1).set_owner idx=1 owner=140640945309440
-47> 2016-09-30 08:51:24.040735 7fe98515f700 2 Event(0x558f62503520 nevent=5000 time_id=1).set_owner idx=2 owner=140640936916736
-46> 2016-09-30 08:51:24.040869 7fe98b6148c0 2 Event(0x558f62503160 nevent=5000 time_id=1).wakeup
-45> 2016-09-30 08:51:24.040953 7fe98b6148c0 1 Processor -- bind bind my_inst.addr is 0.0.0.0:6800/16661
-44> 2016-09-30 08:51:24.040975 7fe98b6148c0 2 Event(0x558f62503160 nevent=5000 time_id=1).wakeup
-43> 2016-09-30 08:51:24.041009 7fe98b6148c0 2 Event(0x558f62503160 nevent=5000 time_id=1).wakeup
-42> 2016-09-30 08:51:24.041050 7fe98b6148c0 1 Processor -- bind bind my_inst.addr is 0.0.0.0:6801/16661
-41> 2016-09-30 08:51:24.041057 7fe98b6148c0 2 Event(0x558f62503160 nevent=5000 time_id=1).wakeup
-40> 2016-09-30 08:51:24.041093 7fe98b6148c0 2 Event(0x558f62503160 nevent=5000 time_id=1).wakeup
-39> 2016-09-30 08:51:24.041118 7fe98b6148c0 2 Event(0x558f62503160 nevent=5000 time_id=1).wakeup
-38> 2016-09-30 08:51:24.041143 7fe98b6148c0 1 Processor -- bind bind my_inst.addr is 0.0.0.0:6802/16661
-37> 2016-09-30 08:51:24.041149 7fe98b6148c0 2 Event(0x558f62503160 nevent=5000 time_id=1).wakeup
-36> 2016-09-30 08:51:24.041171 7fe98b6148c0 2 Event(0x558f62503160 nevent=5000 time_id=1).wakeup
-35> 2016-09-30 08:51:24.041196 7fe98b6148c0 2 Event(0x558f62503160 nevent=5000 time_id=1).wakeup
-34> 2016-09-30 08:51:24.041221 7fe98b6148c0 2 Event(0x558f62503160 nevent=5000 time_id=1).wakeup
-33> 2016-09-30 08:51:24.041244 7fe98b6148c0 1 Processor -- bind bind my_inst.addr is 0.0.0.0:6803/16661
-32> 2016-09-30 08:51:24.041250 7fe98b6148c0 0 pidfile_write: ignore empty --pid-file
-31> 2016-09-30 08:51:24.042414 7fe98b6148c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
-30> 2016-09-30 08:51:24.042426 7fe98b6148c0 5 asok(0x558f62502f00) init /var/run/ceph/ceph-osd.0.asok
-29> 2016-09-30 08:51:24.042433 7fe98b6148c0 5 asok(0x558f62502f00) bind_and_listen /var/run/ceph/ceph-osd.0.asok
-28> 2016-09-30 08:51:24.042514 7fe98b6148c0 5 asok(0x558f62502f00) register_command 0 hook 0x558f624c2108
-27> 2016-09-30 08:51:24.042520 7fe98b6148c0 5 asok(0x558f62502f00) register_command version hook 0x558f624c2108
-26> 2016-09-30 08:51:24.042523 7fe98b6148c0 5 asok(0x558f62502f00) register_command git_version hook 0x558f624c2108
-25> 2016-09-30 08:51:24.042527 7fe98b6148c0 5 asok(0x558f62502f00) register_command help hook 0x558f624c63c0
-24> 2016-09-30 08:51:24.042530 7fe98b6148c0 5 asok(0x558f62502f00) register_command get_command_descriptions hook 0x558f624c63d0
-23> 2016-09-30 08:51:24.042565 7fe9839c2700 5 asok(0x558f62502f00) entry start
-22> 2016-09-30 08:51:24.042574 7fe98b6148c0 10 monclient(hunting): build_initial_monmap
-21> 2016-09-30 08:51:24.046019 7fe98b6148c0 10 ErasureCodePluginSelectJerasure: load: jerasure_sse4
-20> 2016-09-30 08:51:24.047218 7fe98b6148c0 10 load: jerasure load: lrc load: isa
-19> 2016-09-30 08:51:24.047248 7fe98b6148c0 5 adding auth protocol: cephx
-18> 2016-09-30 08:51:24.047251 7fe98b6148c0 5 adding auth protocol: cephx
-17> 2016-09-30 08:51:24.047462 7fe98b6148c0 5 asok(0x558f62502f00) register_command objecter_requests hook 0x558f624c6470
-16> 2016-09-30 08:51:24.047527 7fe98b6148c0 1 -- 0.0.0.0:6800/16661 start start
-15> 2016-09-30 08:51:24.047537 7fe98b6148c0 1 -- :/0 start start
-14> 2016-09-30 08:51:24.047539 7fe98b6148c0 1 -- 0.0.0.0:6803/16661 start start
-13> 2016-09-30 08:51:24.047541 7fe98b6148c0 1 -- 0.0.0.0:6802/16661 start start
-12> 2016-09-30 08:51:24.047543 7fe98b6148c0 1 -- 0.0.0.0:6801/16661 start start
-11> 2016-09-30 08:51:24.047545 7fe98b6148c0 1 -- :/0 start start
-10> 2016-09-30 08:51:24.047598 7fe98b6148c0 2 osd.0 0 mounting /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal
-9> 2016-09-30 08:51:24.047608 7fe98b6148c0 1 bluestore(/var/lib/ceph/osd/ceph-0) mount path /var/lib/ceph/osd/ceph-0
-8> 2016-09-30 08:51:24.047651 7fe98b6148c0 1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel
-7> 2016-09-30 08:51:24.048033 7fe98b6148c0 1 bdev(/var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
-6> 2016-09-30 08:51:24.048354 7fe98b6148c0 1 bdev(/var/lib/ceph/osd/ceph-0/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) non-rotational
-5> 2016-09-30 08:51:24.048637 7fe98b6148c0 1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel
-4> 2016-09-30 08:51:24.048927 7fe98b6148c0 1 bdev(/var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
-3> 2016-09-30 08:51:24.049169 7fe98b6148c0 1 bdev(/var/lib/ceph/osd/ceph-0/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) non-rotational
-2> 2016-09-30 08:51:24.049177 7fe98b6148c0 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-0/block size 3725 GB
-1> 2016-09-30 08:51:24.049200 7fe98b6148c0 1 bluefs mount
0> 2016-09-30 08:51:24.116575 7fe98b6148c0 -1 /srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-11.0.0-2808-g76e120c/src/os/bluestore/StupidAllocator.cc: In function 'virtual void StupidAllocator::init_rm_free(uint64_t, uint64_t)' thread 7fe98b6148c0 time 2016-09-30 08:51:24.114653
/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-11.0.0-2808-g76e120c/src/os/bluestore/StupidAllocator.cc: 317: FAILED assert(rm.empty())
ceph version v11.0.0-2808-g76e120c (76e120c705b77d2d2cef1b94cacdd11c14460a3f)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x80) [0x558f57423240]
2: (StupidAllocator::init_rm_free(unsigned long, unsigned long)+0x1b8c) [0x558f572706cc]
3: (BlueFS::mount()+0x40a) [0x558f5724c2ba]
4: (BlueStore::_open_db(bool)+0xc45) [0x558f5717a825]
5: (BlueStore::mount()+0x38d) [0x558f57195aad]
6: (OSD::init()+0x273) [0x558f56d94ad3]
7: (main()+0x2ea9) [0x558f56ce2cd9]
8: (__libc_start_main()+0xf0) [0x7fe988055830]
9: (_start()+0x29) [0x558f56d40899]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
20/20 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 newstore
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
4/ 5 memdb
1/ 5 kinetic
1/ 5 fuse
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-osd.0.log
--- end dump of recent events ---
2016-09-30 08:51:24.118766 7fe98b6148c0 -1 *** Caught signal (Aborted) **
in thread 7fe98b6148c0 thread_name:ceph-osd
ceph version v11.0.0-2808-g76e120c (76e120c705b77d2d2cef1b94cacdd11c14460a3f)
1: (()+0x8d8c1e) [0x558f57282c1e]
2: (()+0x113d0) [0x7fe98985c3d0]
3: (gsignal()+0x38) [0x7fe98806a418]
4: (abort()+0x16a) [0x7fe98806c01a]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26b) [0x558f5742342b]
6: (StupidAllocator::init_rm_free(unsigned long, unsigned long)+0x1b8c) [0x558f572706cc]
7: (BlueFS::mount()+0x40a) [0x558f5724c2ba]
8: (BlueStore::_open_db(bool)+0xc45) [0x558f5717a825]
9: (BlueStore::mount()+0x38d) [0x558f57195aad]
10: (OSD::init()+0x273) [0x558f56d94ad3]
11: (main()+0x2ea9) [0x558f56ce2cd9]
12: (__libc_start_main()+0xf0) [0x7fe988055830]
13: (_start()+0x29) [0x558f56d40899]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- begin dump of recent events ---
0> 2016-09-30 08:51:24.118766 7fe98b6148c0 -1 *** Caught signal (Aborted) **
in thread 7fe98b6148c0 thread_name:ceph-osd
ceph version v11.0.0-2808-g76e120c (76e120c705b77d2d2cef1b94cacdd11c14460a3f)
1: (()+0x8d8c1e) [0x558f57282c1e]
2: (()+0x113d0) [0x7fe98985c3d0]
3: (gsignal()+0x38) [0x7fe98806a418]
4: (abort()+0x16a) [0x7fe98806c01a]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26b) [0x558f5742342b]
6: (StupidAllocator::init_rm_free(unsigned long, unsigned long)+0x1b8c) [0x558f572706cc]
7: (BlueFS::mount()+0x40a) [0x558f5724c2ba]
8: (BlueStore::_open_db(bool)+0xc45) [0x558f5717a825]
9: (BlueStore::mount()+0x38d) [0x558f57195aad]
10: (OSD::init()+0x273) [0x558f56d94ad3]
11: (main()+0x2ea9) [0x558f56ce2cd9]
12: (__libc_start_main()+0xf0) [0x7fe988055830]
13: (_start()+0x29) [0x558f56d40899]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
20/20 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 newstore
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
4/ 5 memdb
1/ 5 kinetic
1/ 5 fuse
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-osd.0.log
--- end dump of recent events ---
2016-09-30 08:51:24.541635 7fe252eb28c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
2016-09-30 08:51:24.541644 7fe252eb28c0 0 set uid:gid to 64045:64045 (ceph:ceph)
2016-09-30 08:51:24.541653 7fe252eb28c0 0 ceph version v11.0.0-2808-g76e120c (76e120c705b77d2d2cef1b94cacdd11c14460a3f), process ceph-osd, pid 16736
2016-09-30 08:51:24.541678 7fe252eb28c0 5 object store type is bluestore
2016-09-30 08:51:24.541690 7fe252eb28c0 -1 WARNING: experimental feature 'bluestore' is enabled
Please be aware that this feature is experimental, untested,
unsupported, and may result in data corruption, data loss,
and/or irreparable damage to your cluster. Do not use
feature with important data.
2016-09-30 08:51:24.542708 7fe252eb28c0 0 pidfile_write: ignore empty --pid-file
2016-09-30 08:51:24.543843 7fe252eb28c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
2016-09-30 08:51:24.546781 7fe252eb28c0 10 ErasureCodePluginSelectJerasure: load: jerasure_sse4
2016-09-30 08:51:24.547655 7fe252eb28c0 10 load: jerasure load: lrc load: isa
2016-09-30 08:51:24.547889 7fe252eb28c0 2 osd.0 0 mounting /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal
2016-09-30 08:51:24.547897 7fe252eb28c0 1 bluestore(/var/lib/ceph/osd/ceph-0) mount path /var/lib/ceph/osd/ceph-0
2016-09-30 08:51:24.547923 7fe252eb28c0 1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel
2016-09-30 08:51:24.548198 7fe252eb28c0 1 bdev(/var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
2016-09-30 08:51:24.548437 7fe252eb28c0 1 bdev(/var/lib/ceph/osd/ceph-0/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) non-rotational
2016-09-30 08:51:24.548636 7fe252eb28c0 1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel
2016-09-30 08:51:24.548890 7fe252eb28c0 1 bdev(/var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
2016-09-30 08:51:24.549025 7fe252eb28c0 1 bdev(/var/lib/ceph/osd/ceph-0/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) non-rotational
2016-09-30 08:51:24.549030 7fe252eb28c0 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-0/block size 3725 GB
2016-09-30 08:51:24.549047 7fe252eb28c0 1 bluefs mount
2016-09-30 08:51:24.614917 7fe252eb28c0 -1 /srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-11.0.0-2808-g76e120c/src/os/bluestore/StupidAllocator.cc: In function 'virtual void StupidAllocator::init_rm_free(uint64_t, uint64_t)' thread 7fe252eb28c0 time 2016-09-30 08:51:24.612943
/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-11.0.0-2808-g76e120c/src/os/bluestore/StupidAllocator.cc: 317: FAILED assert(rm.empty())
ceph version v11.0.0-2808-g76e120c (76e120c705b77d2d2cef1b94cacdd11c14460a3f)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x80) [0x55d2d96d1240]
2: (StupidAllocator::init_rm_free(unsigned long, unsigned long)+0x1b8c) [0x55d2d951e6cc]
3: (BlueFS::mount()+0x40a) [0x55d2d94fa2ba]
4: (BlueStore::_open_db(bool)+0xc45) [0x55d2d9428825]
5: (BlueStore::mount()+0x38d) [0x55d2d9443aad]
6: (OSD::init()+0x273) [0x55d2d9042ad3]
7: (main()+0x2ea9) [0x55d2d8f90cd9]
8: (__libc_start_main()+0xf0) [0x7fe24f8f3830]
9: (_start()+0x29) [0x55d2d8fee899]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- begin dump of recent events ---
-69> 2016-09-30 08:51:24.539155 7fe252eb28c0 5 asok(0x55d2e287af00) register_command perfcounters_dump hook 0x55d2e283e030
-68> 2016-09-30 08:51:24.539166 7fe252eb28c0 5 asok(0x55d2e287af00) register_command 1 hook 0x55d2e283e030
-67> 2016-09-30 08:51:24.539169 7fe252eb28c0 5 asok(0x55d2e287af00) register_command perf dump hook 0x55d2e283e030
-66> 2016-09-30 08:51:24.539171 7fe252eb28c0 5 asok(0x55d2e287af00) register_command perfcounters_schema hook 0x55d2e283e030
-65> 2016-09-30 08:51:24.539174 7fe252eb28c0 5 asok(0x55d2e287af00) register_command 2 hook 0x55d2e283e030
-64> 2016-09-30 08:51:24.539181 7fe252eb28c0 5 asok(0x55d2e287af00) register_command perf schema hook 0x55d2e283e030
-63> 2016-09-30 08:51:24.539184 7fe252eb28c0 5 asok(0x55d2e287af00) register_command perf reset hook 0x55d2e283e030
-62> 2016-09-30 08:51:24.539186 7fe252eb28c0 5 asok(0x55d2e287af00) register_command config show hook 0x55d2e283e030
-61> 2016-09-30 08:51:24.539189 7fe252eb28c0 5 asok(0x55d2e287af00) register_command config set hook 0x55d2e283e030
-60> 2016-09-30 08:51:24.539192 7fe252eb28c0 5 asok(0x55d2e287af00) register_command config get hook 0x55d2e283e030
-59> 2016-09-30 08:51:24.539194 7fe252eb28c0 5 asok(0x55d2e287af00) register_command config diff hook 0x55d2e283e030
-58> 2016-09-30 08:51:24.539197 7fe252eb28c0 5 asok(0x55d2e287af00) register_command log flush hook 0x55d2e283e030
-57> 2016-09-30 08:51:24.539199 7fe252eb28c0 5 asok(0x55d2e287af00) register_command log dump hook 0x55d2e283e030
-56> 2016-09-30 08:51:24.539202 7fe252eb28c0 5 asok(0x55d2e287af00) register_command log reopen hook 0x55d2e283e030
-55> 2016-09-30 08:51:24.541487 7fe252eb28c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
-54> 2016-09-30 08:51:24.541635 7fe252eb28c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
-53> 2016-09-30 08:51:24.541644 7fe252eb28c0 0 set uid:gid to 64045:64045 (ceph:ceph)
-52> 2016-09-30 08:51:24.541653 7fe252eb28c0 0 ceph version v11.0.0-2808-g76e120c (76e120c705b77d2d2cef1b94cacdd11c14460a3f), process ceph-osd, pid 16736
-51> 2016-09-30 08:51:24.541678 7fe252eb28c0 5 object store type is bluestore
-50> 2016-09-30 08:51:24.541690 7fe252eb28c0 -1 WARNING: experimental feature 'bluestore' is enabled
Please be aware that this feature is experimental, untested,
unsupported, and may result in data corruption, data loss,
and/or irreparable damage to your cluster. Do not use
feature with important data.
-49> 2016-09-30 08:51:24.542285 7fe24d9ff700 2 Event(0x55d2e287b160 nevent=5000 time_id=1).set_owner idx=0 owner=140609941665536
-48> 2016-09-30 08:51:24.542323 7fe24d1fe700 2 Event(0x55d2e287b340 nevent=5000 time_id=1).set_owner idx=1 owner=140609933272832
-47> 2016-09-30 08:51:24.542337 7fe24c9fd700 2 Event(0x55d2e287b520 nevent=5000 time_id=1).set_owner idx=2 owner=140609924880128
-46> 2016-09-30 08:51:24.542439 7fe252eb28c0 2 Event(0x55d2e287b160 nevent=5000 time_id=1).wakeup
-45> 2016-09-30 08:51:24.542487 7fe252eb28c0 1 Processor -- bind bind my_inst.addr is 0.0.0.0:6800/16736
-44> 2016-09-30 08:51:24.542496 7fe252eb28c0 2 Event(0x55d2e287b160 nevent=5000 time_id=1).wakeup
-43> 2016-09-30 08:51:24.542525 7fe252eb28c0 2 Event(0x55d2e287b160 nevent=5000 time_id=1).wakeup
-42> 2016-09-30 08:51:24.542547 7fe252eb28c0 1 Processor -- bind bind my_inst.addr is 0.0.0.0:6801/16736
-41> 2016-09-30 08:51:24.542554 7fe252eb28c0 2 Event(0x55d2e287b160 nevent=5000 time_id=1).wakeup
-40> 2016-09-30 08:51:24.542575 7fe252eb28c0 2 Event(0x55d2e287b160 nevent=5000 time_id=1).wakeup
-39> 2016-09-30 08:51:24.542597 7fe252eb28c0 2 Event(0x55d2e287b160 nevent=5000 time_id=1).wakeup
-38> 2016-09-30 08:51:24.542617 7fe252eb28c0 1 Processor -- bind bind my_inst.addr is 0.0.0.0:6802/16736
-37> 2016-09-30 08:51:24.542623 7fe252eb28c0 2 Event(0x55d2e287b160 nevent=5000 time_id=1).wakeup
-36> 2016-09-30 08:51:24.542642 7fe252eb28c0 2 Event(0x55d2e287b160 nevent=5000 time_id=1).wakeup
-35> 2016-09-30 08:51:24.542662 7fe252eb28c0 2 Event(0x55d2e287b160 nevent=5000 time_id=1).wakeup
-34> 2016-09-30 08:51:24.542682 7fe252eb28c0 2 Event(0x55d2e287b160 nevent=5000 time_id=1).wakeup
-33> 2016-09-30 08:51:24.542702 7fe252eb28c0 1 Processor -- bind bind my_inst.addr is 0.0.0.0:6803/16736
-32> 2016-09-30 08:51:24.542708 7fe252eb28c0 0 pidfile_write: ignore empty --pid-file
-31> 2016-09-30 08:51:24.543843 7fe252eb28c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
-30> 2016-09-30 08:51:24.543854 7fe252eb28c0 5 asok(0x55d2e287af00) init /var/run/ceph/ceph-osd.0.asok
-29> 2016-09-30 08:51:24.543861 7fe252eb28c0 5 asok(0x55d2e287af00) bind_and_listen /var/run/ceph/ceph-osd.0.asok
-28> 2016-09-30 08:51:24.543937 7fe252eb28c0 5 asok(0x55d2e287af00) register_command 0 hook 0x55d2e283a108
-27> 2016-09-30 08:51:24.543944 7fe252eb28c0 5 asok(0x55d2e287af00) register_command version hook 0x55d2e283a108
-26> 2016-09-30 08:51:24.543947 7fe252eb28c0 5 asok(0x55d2e287af00) register_command git_version hook 0x55d2e283a108
-25> 2016-09-30 08:51:24.543950 7fe252eb28c0 5 asok(0x55d2e287af00) register_command help hook 0x55d2e283e3c0
-24> 2016-09-30 08:51:24.543953 7fe252eb28c0 5 asok(0x55d2e287af00) register_command get_command_descriptions hook 0x55d2e283e3d0
-23> 2016-09-30 08:51:24.543986 7fe24b260700 5 asok(0x55d2e287af00) entry start
-22> 2016-09-30 08:51:24.543996 7fe252eb28c0 10 monclient(hunting): build_initial_monmap
-21> 2016-09-30 08:51:24.546781 7fe252eb28c0 10 ErasureCodePluginSelectJerasure: load: jerasure_sse4
-20> 2016-09-30 08:51:24.547655 7fe252eb28c0 10 load: jerasure load: lrc load: isa
-19> 2016-09-30 08:51:24.547669 7fe252eb28c0 5 adding auth protocol: cephx
-18> 2016-09-30 08:51:24.547671 7fe252eb28c0 5 adding auth protocol: cephx
-17> 2016-09-30 08:51:24.547786 7fe252eb28c0 5 asok(0x55d2e287af00) register_command objecter_requests hook 0x55d2e283e470
-16> 2016-09-30 08:51:24.547832 7fe252eb28c0 1 -- 0.0.0.0:6800/16736 start start
-15> 2016-09-30 08:51:24.547839 7fe252eb28c0 1 -- :/0 start start
-14> 2016-09-30 08:51:24.547840 7fe252eb28c0 1 -- 0.0.0.0:6803/16736 start start
-13> 2016-09-30 08:51:24.547842 7fe252eb28c0 1 -- 0.0.0.0:6802/16736 start start
-12> 2016-09-30 08:51:24.547844 7fe252eb28c0 1 -- 0.0.0.0:6801/16736 start start
-11> 2016-09-30 08:51:24.547846 7fe252eb28c0 1 -- :/0 start start
-10> 2016-09-30 08:51:24.547889 7fe252eb28c0 2 osd.0 0 mounting /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal
-9> 2016-09-30 08:51:24.547897 7fe252eb28c0 1 bluestore(/var/lib/ceph/osd/ceph-0) mount path /var/lib/ceph/osd/ceph-0
-8> 2016-09-30 08:51:24.547923 7fe252eb28c0 1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel
-7> 2016-09-30 08:51:24.548198 7fe252eb28c0 1 bdev(/var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
-6> 2016-09-30 08:51:24.548437 7fe252eb28c0 1 bdev(/var/lib/ceph/osd/ceph-0/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) non-rotational
-5> 2016-09-30 08:51:24.548636 7fe252eb28c0 1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel
-4> 2016-09-30 08:51:24.548890 7fe252eb28c0 1 bdev(/var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
-3> 2016-09-30 08:51:24.549025 7fe252eb28c0 1 bdev(/var/lib/ceph/osd/ceph-0/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) non-rotational
-2> 2016-09-30 08:51:24.549030 7fe252eb28c0 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-0/block size 3725 GB
-1> 2016-09-30 08:51:24.549047 7fe252eb28c0 1 bluefs mount
0> 2016-09-30 08:51:24.614917 7fe252eb28c0 -1 /srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-11.0.0-2808-g76e120c/src/os/bluestore/StupidAllocator.cc: In function 'virtual void StupidAllocator::init_rm_free(uint64_t, uint64_t)' thread 7fe252eb28c0 time 2016-09-30 08:51:24.612943
/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-11.0.0-2808-g76e120c/src/os/bluestore/StupidAllocator.cc: 317: FAILED assert(rm.empty())
ceph version v11.0.0-2808-g76e120c (76e120c705b77d2d2cef1b94cacdd11c14460a3f)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x80) [0x55d2d96d1240]
2: (StupidAllocator::init_rm_free(unsigned long, unsigned long)+0x1b8c) [0x55d2d951e6cc]
3: (BlueFS::mount()+0x40a) [0x55d2d94fa2ba]
4: (BlueStore::_open_db(bool)+0xc45) [0x55d2d9428825]
5: (BlueStore::mount()+0x38d) [0x55d2d9443aad]
6: (OSD::init()+0x273) [0x55d2d9042ad3]
7: (main()+0x2ea9) [0x55d2d8f90cd9]
8: (__libc_start_main()+0xf0) [0x7fe24f8f3830]
9: (_start()+0x29) [0x55d2d8fee899]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
20/20 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 newstore
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
4/ 5 memdb
1/ 5 kinetic
1/ 5 fuse
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-osd.0.log
--- end dump of recent events ---
2016-09-30 08:51:24.617118 7fe252eb28c0 -1 *** Caught signal (Aborted) **
in thread 7fe252eb28c0 thread_name:ceph-osd
ceph version v11.0.0-2808-g76e120c (76e120c705b77d2d2cef1b94cacdd11c14460a3f)
1: (()+0x8d8c1e) [0x55d2d9530c1e]
2: (()+0x113d0) [0x7fe2510fa3d0]
3: (gsignal()+0x38) [0x7fe24f908418]
4: (abort()+0x16a) [0x7fe24f90a01a]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26b) [0x55d2d96d142b]
6: (StupidAllocator::init_rm_free(unsigned long, unsigned long)+0x1b8c) [0x55d2d951e6cc]
7: (BlueFS::mount()+0x40a) [0x55d2d94fa2ba]
8: (BlueStore::_open_db(bool)+0xc45) [0x55d2d9428825]
9: (BlueStore::mount()+0x38d) [0x55d2d9443aad]
10: (OSD::init()+0x273) [0x55d2d9042ad3]
11: (main()+0x2ea9) [0x55d2d8f90cd9]
12: (__libc_start_main()+0xf0) [0x7fe24f8f3830]
13: (_start()+0x29) [0x55d2d8fee899]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- begin dump of recent events ---
0> 2016-09-30 08:51:24.617118 7fe252eb28c0 -1 *** Caught signal (Aborted) **
in thread 7fe252eb28c0 thread_name:ceph-osd
ceph version v11.0.0-2808-g76e120c (76e120c705b77d2d2cef1b94cacdd11c14460a3f)
1: (()+0x8d8c1e) [0x55d2d9530c1e]
2: (()+0x113d0) [0x7fe2510fa3d0]
3: (gsignal()+0x38) [0x7fe24f908418]
4: (abort()+0x16a) [0x7fe24f90a01a]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26b) [0x55d2d96d142b]
6: (StupidAllocator::init_rm_free(unsigned long, unsigned long)+0x1b8c) [0x55d2d951e6cc]
7: (BlueFS::mount()+0x40a) [0x55d2d94fa2ba]
8: (BlueStore::_open_db(bool)+0xc45) [0x55d2d9428825]
9: (BlueStore::mount()+0x38d) [0x55d2d9443aad]
10: (OSD::init()+0x273) [0x55d2d9042ad3]
11: (main()+0x2ea9) [0x55d2d8f90cd9]
12: (__libc_start_main()+0xf0) [0x7fe24f8f3830]
13: (_start()+0x29) [0x55d2d8fee899]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
20/20 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 newstore
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
4/ 5 memdb
1/ 5 kinetic
1/ 5 fuse
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-osd.0.log
--- end dump of recent events ---
2016-09-30 08:51:25.033591 7f0f52f8c8c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
2016-09-30 08:51:25.033599 7f0f52f8c8c0 0 set uid:gid to 64045:64045 (ceph:ceph)
2016-09-30 08:51:25.033607 7f0f52f8c8c0 0 ceph version v11.0.0-2808-g76e120c (76e120c705b77d2d2cef1b94cacdd11c14460a3f), process ceph-osd, pid 16762
2016-09-30 08:51:25.033630 7f0f52f8c8c0 5 object store type is bluestore
2016-09-30 08:51:25.033643 7f0f52f8c8c0 -1 WARNING: experimental feature 'bluestore' is enabled
Please be aware that this feature is experimental, untested,
unsupported, and may result in data corruption, data loss,
and/or irreparable damage to your cluster. Do not use
feature with important data.
2016-09-30 08:51:25.034470 7f0f52f8c8c0 0 pidfile_write: ignore empty --pid-file
2016-09-30 08:51:25.035546 7f0f52f8c8c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
2016-09-30 08:51:25.038375 7f0f52f8c8c0 10 ErasureCodePluginSelectJerasure: load: jerasure_sse4
2016-09-30 08:51:25.039178 7f0f52f8c8c0 10 load: jerasure load: lrc load: isa
2016-09-30 08:51:25.039432 7f0f52f8c8c0 2 osd.0 0 mounting /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal
2016-09-30 08:51:25.039441 7f0f52f8c8c0 1 bluestore(/var/lib/ceph/osd/ceph-0) mount path /var/lib/ceph/osd/ceph-0
2016-09-30 08:51:25.039465 7f0f52f8c8c0 1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel
2016-09-30 08:51:25.039757 7f0f52f8c8c0 1 bdev(/var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
2016-09-30 08:51:25.039987 7f0f52f8c8c0 1 bdev(/var/lib/ceph/osd/ceph-0/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) non-rotational
2016-09-30 08:51:25.040181 7f0f52f8c8c0 1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel
2016-09-30 08:51:25.040429 7f0f52f8c8c0 1 bdev(/var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
2016-09-30 08:51:25.040563 7f0f52f8c8c0 1 bdev(/var/lib/ceph/osd/ceph-0/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) non-rotational
2016-09-30 08:51:25.040568 7f0f52f8c8c0 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-0/block size 3725 GB
2016-09-30 08:51:25.040584 7f0f52f8c8c0 1 bluefs mount
2016-09-30 08:51:25.106090 7f0f52f8c8c0 -1 /srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-11.0.0-2808-g76e120c/src/os/bluestore/StupidAllocator.cc: In function 'virtual void StupidAllocator::init_rm_free(uint64_t, uint64_t)' thread 7f0f52f8c8c0 time 2016-09-30 08:51:25.104252
/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-11.0.0-2808-g76e120c/src/os/bluestore/StupidAllocator.cc: 317: FAILED assert(rm.empty())
ceph version v11.0.0-2808-g76e120c (76e120c705b77d2d2cef1b94cacdd11c14460a3f)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x80) [0x555e8167e240]
2: (StupidAllocator::init_rm_free(unsigned long, unsigned long)+0x1b8c) [0x555e814cb6cc]
3: (BlueFS::mount()+0x40a) [0x555e814a72ba]
4: (BlueStore::_open_db(bool)+0xc45) [0x555e813d5825]
5: (BlueStore::mount()+0x38d) [0x555e813f0aad]
6: (OSD::init()+0x273) [0x555e80fefad3]
7: (main()+0x2ea9) [0x555e80f3dcd9]
8: (__libc_start_main()+0xf0) [0x7f0f4f9cd830]
9: (_start()+0x29) [0x555e80f9b899]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- begin dump of recent events ---
-69> 2016-09-30 08:51:25.031050 7f0f52f8c8c0 5 asok(0x555e8c256f00) register_command perfcounters_dump hook 0x555e8c21a030
-68> 2016-09-30 08:51:25.031061 7f0f52f8c8c0 5 asok(0x555e8c256f00) register_command 1 hook 0x555e8c21a030
-67> 2016-09-30 08:51:25.031064 7f0f52f8c8c0 5 asok(0x555e8c256f00) register_command perf dump hook 0x555e8c21a030
-66> 2016-09-30 08:51:25.031068 7f0f52f8c8c0 5 asok(0x555e8c256f00) register_command perfcounters_schema hook 0x555e8c21a030
-65> 2016-09-30 08:51:25.031074 7f0f52f8c8c0 5 asok(0x555e8c256f00) register_command 2 hook 0x555e8c21a030
-64> 2016-09-30 08:51:25.031077 7f0f52f8c8c0 5 asok(0x555e8c256f00) register_command perf schema hook 0x555e8c21a030
-63> 2016-09-30 08:51:25.031080 7f0f52f8c8c0 5 asok(0x555e8c256f00) register_command perf reset hook 0x555e8c21a030
-62> 2016-09-30 08:51:25.031084 7f0f52f8c8c0 5 asok(0x555e8c256f00) register_command config show hook 0x555e8c21a030
-61> 2016-09-30 08:51:25.031086 7f0f52f8c8c0 5 asok(0x555e8c256f00) register_command config set hook 0x555e8c21a030
-60> 2016-09-30 08:51:25.031089 7f0f52f8c8c0 5 asok(0x555e8c256f00) register_command config get hook 0x555e8c21a030
-59> 2016-09-30 08:51:25.031091 7f0f52f8c8c0 5 asok(0x555e8c256f00) register_command config diff hook 0x555e8c21a030
-58> 2016-09-30 08:51:25.031094 7f0f52f8c8c0 5 asok(0x555e8c256f00) register_command log flush hook 0x555e8c21a030
-57> 2016-09-30 08:51:25.031098 7f0f52f8c8c0 5 asok(0x555e8c256f00) register_command log dump hook 0x555e8c21a030
-56> 2016-09-30 08:51:25.031100 7f0f52f8c8c0 5 asok(0x555e8c256f00) register_command log reopen hook 0x555e8c21a030
-55> 2016-09-30 08:51:25.033408 7f0f52f8c8c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
-54> 2016-09-30 08:51:25.033591 7f0f52f8c8c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
-53> 2016-09-30 08:51:25.033599 7f0f52f8c8c0 0 set uid:gid to 64045:64045 (ceph:ceph)
-52> 2016-09-30 08:51:25.033607 7f0f52f8c8c0 0 ceph version v11.0.0-2808-g76e120c (76e120c705b77d2d2cef1b94cacdd11c14460a3f), process ceph-osd, pid 16762
-51> 2016-09-30 08:51:25.033630 7f0f52f8c8c0 5 object store type is bluestore
-50> 2016-09-30 08:51:25.033643 7f0f52f8c8c0 -1 WARNING: experimental feature 'bluestore' is enabled
Please be aware that this feature is experimental, untested,
unsupported, and may result in data corruption, data loss,
and/or irreparable damage to your cluster. Do not use
feature with important data.
-49> 2016-09-30 08:51:25.034030 7f0f4dad9700 2 Event(0x555e8c257160 nevent=5000 time_id=1).set_owner idx=0 owner=139703704459008
-48> 2016-09-30 08:51:25.034045 7f0f4d2d8700 2 Event(0x555e8c257340 nevent=5000 time_id=1).set_owner idx=1 owner=139703696066304
-47> 2016-09-30 08:51:25.034087 7f0f4cad7700 2 Event(0x555e8c257520 nevent=5000 time_id=1).set_owner idx=2 owner=139703687673600
-46> 2016-09-30 08:51:25.034187 7f0f52f8c8c0 2 Event(0x555e8c257160 nevent=5000 time_id=1).wakeup
-45> 2016-09-30 08:51:25.034237 7f0f52f8c8c0 1 Processor -- bind bind my_inst.addr is 0.0.0.0:6800/16762
-44> 2016-09-30 08:51:25.034245 7f0f52f8c8c0 2 Event(0x555e8c257160 nevent=5000 time_id=1).wakeup
-43> 2016-09-30 08:51:25.034273 7f0f52f8c8c0 2 Event(0x555e8c257160 nevent=5000 time_id=1).wakeup
-42> 2016-09-30 08:51:25.034297 7f0f52f8c8c0 1 Processor -- bind bind my_inst.addr is 0.0.0.0:6801/16762
-41> 2016-09-30 08:51:25.034303 7f0f52f8c8c0 2 Event(0x555e8c257160 nevent=5000 time_id=1).wakeup
-40> 2016-09-30 08:51:25.034324 7f0f52f8c8c0 2 Event(0x555e8c257160 nevent=5000 time_id=1).wakeup
-39> 2016-09-30 08:51:25.034347 7f0f52f8c8c0 2 Event(0x555e8c257160 nevent=5000 time_id=1).wakeup
-38> 2016-09-30 08:51:25.034374 7f0f52f8c8c0 1 Processor -- bind bind my_inst.addr is 0.0.0.0:6802/16762
-37> 2016-09-30 08:51:25.034381 7f0f52f8c8c0 2 Event(0x555e8c257160 nevent=5000 time_id=1).wakeup
-36> 2016-09-30 08:51:25.034402 7f0f52f8c8c0 2 Event(0x555e8c257160 nevent=5000 time_id=1).wakeup
-35> 2016-09-30 08:51:25.034423 7f0f52f8c8c0 2 Event(0x555e8c257160 nevent=5000 time_id=1).wakeup
-34> 2016-09-30 08:51:25.034445 7f0f52f8c8c0 2 Event(0x555e8c257160 nevent=5000 time_id=1).wakeup
-33> 2016-09-30 08:51:25.034465 7f0f52f8c8c0 1 Processor -- bind bind my_inst.addr is 0.0.0.0:6803/16762
-32> 2016-09-30 08:51:25.034470 7f0f52f8c8c0 0 pidfile_write: ignore empty --pid-file
-31> 2016-09-30 08:51:25.035546 7f0f52f8c8c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
-30> 2016-09-30 08:51:25.035556 7f0f52f8c8c0 5 asok(0x555e8c256f00) init /var/run/ceph/ceph-osd.0.asok
-29> 2016-09-30 08:51:25.035562 7f0f52f8c8c0 5 asok(0x555e8c256f00) bind_and_listen /var/run/ceph/ceph-osd.0.asok
-28> 2016-09-30 08:51:25.035632 7f0f52f8c8c0 5 asok(0x555e8c256f00) register_command 0 hook 0x555e8c216108
-27> 2016-09-30 08:51:25.035638 7f0f52f8c8c0 5 asok(0x555e8c256f00) register_command version hook 0x555e8c216108
-26> 2016-09-30 08:51:25.035642 7f0f52f8c8c0 5 asok(0x555e8c256f00) register_command git_version hook 0x555e8c216108
-25> 2016-09-30 08:51:25.035646 7f0f52f8c8c0 5 asok(0x555e8c256f00) register_command help hook 0x555e8c21a3c0
-24> 2016-09-30 08:51:25.035648 7f0f52f8c8c0 5 asok(0x555e8c256f00) register_command get_command_descriptions hook 0x555e8c21a3d0
-23> 2016-09-30 08:51:25.035675 7f0f4b33a700 5 asok(0x555e8c256f00) entry start
-22> 2016-09-30 08:51:25.035690 7f0f52f8c8c0 10 monclient(hunting): build_initial_monmap
-21> 2016-09-30 08:51:25.038375 7f0f52f8c8c0 10 ErasureCodePluginSelectJerasure: load: jerasure_sse4
-20> 2016-09-30 08:51:25.039178 7f0f52f8c8c0 10 load: jerasure load: lrc load: isa
-19> 2016-09-30 08:51:25.039190 7f0f52f8c8c0 5 adding auth protocol: cephx
-18> 2016-09-30 08:51:25.039194 7f0f52f8c8c0 5 adding auth protocol: cephx
-17> 2016-09-30 08:51:25.039333 7f0f52f8c8c0 5 asok(0x555e8c256f00) register_command objecter_requests hook 0x555e8c21a470
-16> 2016-09-30 08:51:25.039376 7f0f52f8c8c0 1 -- 0.0.0.0:6800/16762 start start
-15> 2016-09-30 08:51:25.039383 7f0f52f8c8c0 1 -- :/0 start start
-14> 2016-09-30 08:51:25.039385 7f0f52f8c8c0 1 -- 0.0.0.0:6803/16762 start start
-13> 2016-09-30 08:51:25.039387 7f0f52f8c8c0 1 -- 0.0.0.0:6802/16762 start start
-12> 2016-09-30 08:51:25.039388 7f0f52f8c8c0 1 -- 0.0.0.0:6801/16762 start start
-11> 2016-09-30 08:51:25.039390 7f0f52f8c8c0 1 -- :/0 start start
-10> 2016-09-30 08:51:25.039432 7f0f52f8c8c0 2 osd.0 0 mounting /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal
-9> 2016-09-30 08:51:25.039441 7f0f52f8c8c0 1 bluestore(/var/lib/ceph/osd/ceph-0) mount path /var/lib/ceph/osd/ceph-0
-8> 2016-09-30 08:51:25.039465 7f0f52f8c8c0 1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel
-7> 2016-09-30 08:51:25.039757 7f0f52f8c8c0 1 bdev(/var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
-6> 2016-09-30 08:51:25.039987 7f0f52f8c8c0 1 bdev(/var/lib/ceph/osd/ceph-0/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) non-rotational
-5> 2016-09-30 08:51:25.040181 7f0f52f8c8c0 1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel
-4> 2016-09-30 08:51:25.040429 7f0f52f8c8c0 1 bdev(/var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
-3> 2016-09-30 08:51:25.040563 7f0f52f8c8c0 1 bdev(/var/lib/ceph/osd/ceph-0/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) non-rotational
-2> 2016-09-30 08:51:25.040568 7f0f52f8c8c0 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-0/block size 3725 GB
-1> 2016-09-30 08:51:25.040584 7f0f52f8c8c0 1 bluefs mount
0> 2016-09-30 08:51:25.106090 7f0f52f8c8c0 -1 /srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-11.0.0-2808-g76e120c/src/os/bluestore/StupidAllocator.cc: In function 'virtual void StupidAllocator::init_rm_free(uint64_t, uint64_t)' thread 7f0f52f8c8c0 time 2016-09-30 08:51:25.104252
/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-11.0.0-2808-g76e120c/src/os/bluestore/StupidAllocator.cc: 317: FAILED assert(rm.empty())
ceph version v11.0.0-2808-g76e120c (76e120c705b77d2d2cef1b94cacdd11c14460a3f)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x80) [0x555e8167e240]
2: (StupidAllocator::init_rm_free(unsigned long, unsigned long)+0x1b8c) [0x555e814cb6cc]
3: (BlueFS::mount()+0x40a) [0x555e814a72ba]
4: (BlueStore::_open_db(bool)+0xc45) [0x555e813d5825]
5: (BlueStore::mount()+0x38d) [0x555e813f0aad]
6: (OSD::init()+0x273) [0x555e80fefad3]
7: (main()+0x2ea9) [0x555e80f3dcd9]
8: (__libc_start_main()+0xf0) [0x7f0f4f9cd830]
9: (_start()+0x29) [0x555e80f9b899]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
20/20 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 newstore
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
4/ 5 memdb
1/ 5 kinetic
1/ 5 fuse
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-osd.0.log
--- end dump of recent events ---
2016-09-30 08:51:25.108243 7f0f52f8c8c0 -1 *** Caught signal (Aborted) **
in thread 7f0f52f8c8c0 thread_name:ceph-osd
ceph version v11.0.0-2808-g76e120c (76e120c705b77d2d2cef1b94cacdd11c14460a3f)
1: (()+0x8d8c1e) [0x555e814ddc1e]
2: (()+0x113d0) [0x7f0f511d43d0]
3: (gsignal()+0x38) [0x7f0f4f9e2418]
4: (abort()+0x16a) [0x7f0f4f9e401a]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26b) [0x555e8167e42b]
6: (StupidAllocator::init_rm_free(unsigned long, unsigned long)+0x1b8c) [0x555e814cb6cc]
7: (BlueFS::mount()+0x40a) [0x555e814a72ba]
8: (BlueStore::_open_db(bool)+0xc45) [0x555e813d5825]
9: (BlueStore::mount()+0x38d) [0x555e813f0aad]
10: (OSD::init()+0x273) [0x555e80fefad3]
11: (main()+0x2ea9) [0x555e80f3dcd9]
12: (__libc_start_main()+0xf0) [0x7f0f4f9cd830]
13: (_start()+0x29) [0x555e80f9b899]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- begin dump of recent events ---
0> 2016-09-30 08:51:25.108243 7f0f52f8c8c0 -1 *** Caught signal (Aborted) **
in thread 7f0f52f8c8c0 thread_name:ceph-osd
ceph version v11.0.0-2808-g76e120c (76e120c705b77d2d2cef1b94cacdd11c14460a3f)
1: (()+0x8d8c1e) [0x555e814ddc1e]
2: (()+0x113d0) [0x7f0f511d43d0]
3: (gsignal()+0x38) [0x7f0f4f9e2418]
4: (abort()+0x16a) [0x7f0f4f9e401a]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26b) [0x555e8167e42b]
6: (StupidAllocator::init_rm_free(unsigned long, unsigned long)+0x1b8c) [0x555e814cb6cc]
7: (BlueFS::mount()+0x40a) [0x555e814a72ba]
8: (BlueStore::_open_db(bool)+0xc45) [0x555e813d5825]
9: (BlueStore::mount()+0x38d) [0x555e813f0aad]
10: (OSD::init()+0x273) [0x555e80fefad3]
11: (main()+0x2ea9) [0x555e80f3dcd9]
12: (__libc_start_main()+0xf0) [0x7f0f4f9cd830]
13: (_start()+0x29) [0x555e80f9b899]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
20/20 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 newstore
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
4/ 5 memdb
1/ 5 kinetic
1/ 5 fuse
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-osd.0.log
--- end dump of recent events ---
[global]
fsid = 34ba1a46-0936-4da5-9b8f-299c5e973ca3
mon_initial_members = dementor, banshee
mon_host = 10.14.65.176,10.14.65.79
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
filestore_xattr_use_omap = true
osd_pool_default_size = 3
enable_experimental_unrecoverable_data_corrupting_features = bluestore rocksdb
bluestore_default_buffered_read = true
mds cache size = 4000000
mon_clock_drift_allowed = .10
osd crush chooseleaf type = 3
osd op threads = 4
[mon]
debug mon = 20
[osd]
debug osd = 20
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
