Hi,
Are erasure coded pools suitable for use with MDS?
I tried to give it a go by creating two new pools like so:
# ceph osd pool create ecdata 128 128 erasure
# ceph osd pool create ecmetadata 128 128 erasure
Then looked up their id's:
# ceph osd lspools
..., 6 ecdata,7 ecmetadata
# ceph mds newfs 7 6 --yes-i-really-mean-it
But then when I start MDS, it crashes horribly. I did notice that MDS
created a couple of objects in the ecmetadata pool:
# rados ls -p ecmetadata
mds0_sessionmap
mds0_inotable
1.00000000.inode
200.00000000
mds_anchortable
mds_snaptable
100.00000000.inode
However it crashes immediately after. I started mds manually to try and
see what's up:
# ceph-mds -i 0 -d
This spews out so much information that I saved it in a logfile, added
as an attachment.
Kind regards,
Erik.
2014-06-19 22:07:34.492328 7f3572f6e7c0 0 ceph version 0.80.1 (a38fe1169b6d2ac98b427334c12d7cf81f809b74), process ceph-mds, pid 2943
starting mds.0 at :/0
2014-06-19 22:07:35.793309 7f356dd88700 1 mds.-1.0 handle_mds_map standby
2014-06-19 22:07:35.876689 7f356dd88700 1 mds.0.15 handle_mds_map i am now mds.0.15
2014-06-19 22:07:35.876695 7f356dd88700 1 mds.0.15 handle_mds_map state change up:standby --> up:creating
2014-06-19 22:07:35.876931 7f356dd88700 0 mds.0.cache creating system inode with ino:1
2014-06-19 22:07:35.877204 7f356dd88700 0 mds.0.cache creating system inode with ino:100
2014-06-19 22:07:35.877209 7f356dd88700 0 mds.0.cache creating system inode with ino:600
2014-06-19 22:07:35.877369 7f356dd88700 0 mds.0.cache creating system inode with ino:601
2014-06-19 22:07:35.877455 7f356dd88700 0 mds.0.cache creating system inode with ino:602
2014-06-19 22:07:35.877519 7f356dd88700 0 mds.0.cache creating system inode with ino:603
2014-06-19 22:07:35.877566 7f356dd88700 0 mds.0.cache creating system inode with ino:604
2014-06-19 22:07:35.877606 7f356dd88700 0 mds.0.cache creating system inode with ino:605
2014-06-19 22:07:35.877683 7f356dd88700 0 mds.0.cache creating system inode with ino:606
2014-06-19 22:07:35.877723 7f356dd88700 0 mds.0.cache creating system inode with ino:607
2014-06-19 22:07:35.877780 7f356dd88700 0 mds.0.cache creating system inode with ino:608
2014-06-19 22:07:35.877819 7f356dd88700 0 mds.0.cache creating system inode with ino:609
2014-06-19 22:07:35.877858 7f356dd88700 0 mds.0.cache creating system inode with ino:200
mds/CDir.cc: In function 'virtual void C_Dir_Committed::finish(int)' thread 7f356dd88700 time 2014-06-19 22:07:35.881337
mds/CDir.cc: 1809: FAILED assert(r == 0)
ceph version 0.80.1 (a38fe1169b6d2ac98b427334c12d7cf81f809b74)
1: ceph-mds() [0x75c6f1]
2: (Context::complete(int)+0x9) [0x56cff9]
3: (C_Gather::sub_finish(Context*, int)+0x1f7) [0x56e9a7]
4: (C_Gather::C_GatherSub::finish(int)+0x12) [0x56eab2]
5: (Context::complete(int)+0x9) [0x56cff9]
6: (Objecter::handle_osd_op_reply(MOSDOpReply*)+0xf4e) [0x7d26ee]
7: (MDS::handle_core_message(Message*)+0xb1f) [0x58e5ef]
8: (MDS::_dispatch(Message*)+0x32) [0x58e7f2]
9: (MDS::ms_dispatch(Message*)+0xa3) [0x5901d3]
10: (DispatchQueue::entry()+0x57a) [0x99d9da]
11: (DispatchQueue::DispatchThread::entry()+0xd) [0x8be63d]
12: (()+0x7c53) [0x7f3572366c53]
13: (clone()+0x6d) [0x7f3571257dbd]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2014-06-19 22:07:35.883239 7f356dd88700 -1 mds/CDir.cc: In function 'virtual void C_Dir_Committed::finish(int)' thread 7f356dd88700 time 2014-06-19 22:07:35.881337
mds/CDir.cc: 1809: FAILED assert(r == 0)
ceph version 0.80.1 (a38fe1169b6d2ac98b427334c12d7cf81f809b74)
1: ceph-mds() [0x75c6f1]
2: (Context::complete(int)+0x9) [0x56cff9]
3: (C_Gather::sub_finish(Context*, int)+0x1f7) [0x56e9a7]
4: (C_Gather::C_GatherSub::finish(int)+0x12) [0x56eab2]
5: (Context::complete(int)+0x9) [0x56cff9]
6: (Objecter::handle_osd_op_reply(MOSDOpReply*)+0xf4e) [0x7d26ee]
7: (MDS::handle_core_message(Message*)+0xb1f) [0x58e5ef]
8: (MDS::_dispatch(Message*)+0x32) [0x58e7f2]
9: (MDS::ms_dispatch(Message*)+0xa3) [0x5901d3]
10: (DispatchQueue::entry()+0x57a) [0x99d9da]
11: (DispatchQueue::DispatchThread::entry()+0xd) [0x8be63d]
12: (()+0x7c53) [0x7f3572366c53]
13: (clone()+0x6d) [0x7f3571257dbd]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- begin dump of recent events ---
-144> 2014-06-19 22:07:34.489920 7f3572f6e7c0 5 asok(0x1e00000) register_command perfcounters_dump hook 0x1dc8010
-143> 2014-06-19 22:07:34.489992 7f3572f6e7c0 5 asok(0x1e00000) register_command 1 hook 0x1dc8010
-142> 2014-06-19 22:07:34.490003 7f3572f6e7c0 5 asok(0x1e00000) register_command perf dump hook 0x1dc8010
-141> 2014-06-19 22:07:34.490015 7f3572f6e7c0 5 asok(0x1e00000) register_command perfcounters_schema hook 0x1dc8010
-140> 2014-06-19 22:07:34.490027 7f3572f6e7c0 5 asok(0x1e00000) register_command 2 hook 0x1dc8010
-139> 2014-06-19 22:07:34.490035 7f3572f6e7c0 5 asok(0x1e00000) register_command perf schema hook 0x1dc8010
-138> 2014-06-19 22:07:34.490046 7f3572f6e7c0 5 asok(0x1e00000) register_command config show hook 0x1dc8010
-137> 2014-06-19 22:07:34.490055 7f3572f6e7c0 5 asok(0x1e00000) register_command config set hook 0x1dc8010
-136> 2014-06-19 22:07:34.490062 7f3572f6e7c0 5 asok(0x1e00000) register_command config get hook 0x1dc8010
-135> 2014-06-19 22:07:34.490070 7f3572f6e7c0 5 asok(0x1e00000) register_command log flush hook 0x1dc8010
-134> 2014-06-19 22:07:34.490078 7f3572f6e7c0 5 asok(0x1e00000) register_command log dump hook 0x1dc8010
-133> 2014-06-19 22:07:34.490086 7f3572f6e7c0 5 asok(0x1e00000) register_command log reopen hook 0x1dc8010
-132> 2014-06-19 22:07:34.492328 7f3572f6e7c0 0 ceph version 0.80.1 (a38fe1169b6d2ac98b427334c12d7cf81f809b74), process ceph-mds, pid 2943
-131> 2014-06-19 22:07:34.492601 7f3572f6e7c0 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6800/2943 need_addr=1
-130> 2014-06-19 22:07:34.494286 7f3572f6e7c0 5 asok(0x1e00000) init /var/run/ceph/ceph-mds.0.asok
-129> 2014-06-19 22:07:34.494302 7f3572f6e7c0 5 asok(0x1e00000) bind_and_listen /var/run/ceph/ceph-mds.0.asok
-128> 2014-06-19 22:07:34.495066 7f3572f6e7c0 5 asok(0x1e00000) register_command 0 hook 0x1dc00b8
-127> 2014-06-19 22:07:34.495080 7f3572f6e7c0 5 asok(0x1e00000) register_command version hook 0x1dc00b8
-126> 2014-06-19 22:07:34.495086 7f3572f6e7c0 5 asok(0x1e00000) register_command git_version hook 0x1dc00b8
-125> 2014-06-19 22:07:34.495092 7f3572f6e7c0 5 asok(0x1e00000) register_command help hook 0x1dc80c0
-124> 2014-06-19 22:07:34.495097 7f3572f6e7c0 5 asok(0x1e00000) register_command get_command_descriptions hook 0x1dc80b0
-123> 2014-06-19 22:07:34.495132 7f356ed8a700 5 asok(0x1e00000) entry start
-122> 2014-06-19 22:07:34.495143 7f3572f6e7c0 10 monclient(hunting): build_initial_monmap
-121> 2014-06-19 22:07:34.495204 7f3572f6e7c0 1 -- 0.0.0.0:6800/2943 messenger.start
-120> 2014-06-19 22:07:34.495263 7f3572f6e7c0 5 adding auth protocol: cephx
-119> 2014-06-19 22:07:34.495269 7f3572f6e7c0 5 adding auth protocol: cephx
-118> 2014-06-19 22:07:34.495398 7f3572f6e7c0 1 accepter.accepter.start
-117> 2014-06-19 22:07:34.495422 7f3572f6e7c0 10 monclient(hunting): init
-116> 2014-06-19 22:07:34.495431 7f3572f6e7c0 5 adding auth protocol: cephx
-115> 2014-06-19 22:07:34.495433 7f3572f6e7c0 10 monclient(hunting): auth_supported 2 method cephx
-114> 2014-06-19 22:07:34.495653 7f3572f6e7c0 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/mds.0/mds.0.keyring
-113> 2014-06-19 22:07:34.495720 7f3572f6e7c0 10 monclient(hunting): _reopen_session rank -1 name
-112> 2014-06-19 22:07:34.495773 7f3572f6e7c0 10 monclient(hunting): picked mon.0 con 0x1e38f20 addr 192.168.1.16:6789/0
-111> 2014-06-19 22:07:34.495797 7f3572f6e7c0 10 monclient(hunting): _send_mon_message to mon.0 at 192.168.1.16:6789/0
-110> 2014-06-19 22:07:34.495810 7f3572f6e7c0 1 -- 0.0.0.0:6800/2943 --> 192.168.1.16:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- ?+0 0x1df8000 con 0x1e38f20
-109> 2014-06-19 22:07:34.495832 7f3572f6e7c0 10 monclient(hunting): renew_subs
-108> 2014-06-19 22:07:34.496312 7f3572f6c700 1 -- 192.168.1.15:6800/2943 learned my addr 192.168.1.15:6800/2943
-107> 2014-06-19 22:07:34.496611 7f356dd88700 5 mds.-1.0 ms_handle_connect on 192.168.1.16:6789/0
-106> 2014-06-19 22:07:34.497149 7f356dd88700 1 -- 192.168.1.15:6800/2943 <== mon.0 192.168.1.16:6789/0 1 ==== mon_map v1 ==== 191+0+0 (3308226674 0 0) 0x1ef8000 con 0x1e38f20
-105> 2014-06-19 22:07:34.497179 7f356dd88700 10 monclient(hunting): handle_monmap mon_map v1
-104> 2014-06-19 22:07:34.497196 7f356dd88700 10 monclient(hunting): got monmap 1, mon.0 is now rank 0
-103> 2014-06-19 22:07:34.497204 7f356dd88700 10 monclient(hunting): dump:
epoch 1
fsid 306b3903-f13a-4a21-bf21-e185c0357925
last_changed 2013-07-31 00:46:59.410455
created 2013-07-31 00:46:59.410455
0: 192.168.1.16:6789/0 mon.0
-102> 2014-06-19 22:07:34.497245 7f356dd88700 1 -- 192.168.1.15:6800/2943 <== mon.0 192.168.1.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (3492799188 0 0) 0x1e21400 con 0x1e38f20
-101> 2014-06-19 22:07:34.497288 7f356dd88700 10 monclient(hunting): my global_id is 11715
-100> 2014-06-19 22:07:34.497527 7f356dd88700 10 monclient(hunting): _send_mon_message to mon.0 at 192.168.1.16:6789/0
-99> 2014-06-19 22:07:34.497534 7f356dd88700 1 -- 192.168.1.15:6800/2943 --> 192.168.1.16:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 0x1df8480 con 0x1e38f20
-98> 2014-06-19 22:07:34.498016 7f356dd88700 1 -- 192.168.1.15:6800/2943 <== mon.0 192.168.1.16:6789/0 3 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (3509936303 0 0) 0x1e21800 con 0x1e38f20
-97> 2014-06-19 22:07:34.498133 7f356dd88700 10 monclient(hunting): _send_mon_message to mon.0 at 192.168.1.16:6789/0
-96> 2014-06-19 22:07:34.498144 7f356dd88700 1 -- 192.168.1.15:6800/2943 --> 192.168.1.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- ?+0 0x1df8900 con 0x1e38f20
-95> 2014-06-19 22:07:34.498721 7f356dd88700 1 -- 192.168.1.15:6800/2943 <== mon.0 192.168.1.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (212679146 0 0) 0x1e21600 con 0x1e38f20
-94> 2014-06-19 22:07:34.498839 7f356dd88700 1 monclient(hunting): found mon.0
-93> 2014-06-19 22:07:34.498847 7f356dd88700 10 monclient: _send_mon_message to mon.0 at 192.168.1.16:6789/0
-92> 2014-06-19 22:07:34.498855 7f356dd88700 1 -- 192.168.1.15:6800/2943 --> 192.168.1.16:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0x1e98380 con 0x1e38f20
-91> 2014-06-19 22:07:34.498885 7f356dd88700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2014-06-19 22:07:04.498885)
-90> 2014-06-19 22:07:34.498900 7f356dd88700 10 monclient: _send_mon_message to mon.0 at 192.168.1.16:6789/0
-89> 2014-06-19 22:07:34.498910 7f356dd88700 1 -- 192.168.1.15:6800/2943 --> 192.168.1.16:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- ?+0 0x1df86c0 con 0x1e38f20
-88> 2014-06-19 22:07:34.498935 7f3572f6e7c0 5 monclient: authenticate success, global_id 11715
-87> 2014-06-19 22:07:34.498941 7f3572f6e7c0 10 monclient: wait_auth_rotating waiting (until 2014-06-19 22:08:04.498941)
-86> 2014-06-19 22:07:34.499217 7f356dd88700 1 -- 192.168.1.15:6800/2943 <== mon.0 192.168.1.16:6789/0 5 ==== mon_map v1 ==== 191+0+0 (3308226674 0 0) 0x1ef83c0 con 0x1e38f20
-85> 2014-06-19 22:07:34.499235 7f356dd88700 10 monclient: handle_monmap mon_map v1
-84> 2014-06-19 22:07:34.499241 7f356dd88700 10 monclient: got monmap 1, mon.0 is now rank 0
-83> 2014-06-19 22:07:34.499247 7f356dd88700 10 monclient: dump:
epoch 1
fsid 306b3903-f13a-4a21-bf21-e185c0357925
last_changed 2013-07-31 00:46:59.410455
created 2013-07-31 00:46:59.410455
0: 192.168.1.16:6789/0 mon.0
-82> 2014-06-19 22:07:34.499314 7f356dd88700 1 -- 192.168.1.15:6800/2943 <== mon.0 192.168.1.16:6789/0 6 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (1074470259 0 0) 0x1e98540 con 0x1e38f20
-81> 2014-06-19 22:07:34.499330 7f356dd88700 10 monclient: handle_subscribe_ack sent 2014-06-19 22:07:34.495837 renew after 2014-06-19 22:10:04.495837
-80> 2014-06-19 22:07:34.499412 7f356dd88700 1 -- 192.168.1.15:6800/2943 <== mon.0 192.168.1.16:6789/0 7 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (1043900712 0 0) 0x1e21a00 con 0x1e38f20
-79> 2014-06-19 22:07:34.499484 7f356dd88700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2014-06-19 22:07:04.499483)
-78> 2014-06-19 22:07:34.499495 7f3572f6e7c0 10 monclient: wait_auth_rotating done
-77> 2014-06-19 22:07:34.499509 7f3572f6e7c0 5 asok(0x1e00000) register_command objecter_requests hook 0x1dc8110
-76> 2014-06-19 22:07:34.499520 7f3572f6e7c0 10 monclient: renew_subs
-75> 2014-06-19 22:07:34.499522 7f3572f6e7c0 10 monclient: _send_mon_message to mon.0 at 192.168.1.16:6789/0
-74> 2014-06-19 22:07:34.499525 7f3572f6e7c0 1 -- 192.168.1.15:6800/2943 --> 192.168.1.16:6789/0 -- mon_subscribe({monmap=2+,osdmap=0}) v2 -- ?+0 0x1e981c0 con 0x1e38f20
-73> 2014-06-19 22:07:34.499537 7f3572f6e7c0 10 monclient: renew_subs
-72> 2014-06-19 22:07:34.499539 7f3572f6e7c0 10 monclient: _send_mon_message to mon.0 at 192.168.1.16:6789/0
-71> 2014-06-19 22:07:34.499542 7f3572f6e7c0 1 -- 192.168.1.15:6800/2943 --> 192.168.1.16:6789/0 -- mon_subscribe({mdsmap=0+,monmap=2+,osdmap=0}) v2 -- ?+0 0x1e98a80 con 0x1e38f20
-70> 2014-06-19 22:07:34.500136 7f356dd88700 1 -- 192.168.1.15:6800/2943 <== mon.0 192.168.1.16:6789/0 8 ==== osd_map(346..346 src has 1..346) v3 ==== 6466+0+0 (1629858625 0 0) 0x1df8d80 con 0x1e38f20
-69> 2014-06-19 22:07:34.500319 7f356dd88700 1 -- 192.168.1.15:6800/2943 <== mon.0 192.168.1.16:6789/0 9 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (1074470259 0 0) 0x1e981c0 con 0x1e38f20
-68> 2014-06-19 22:07:34.500329 7f356dd88700 10 monclient: handle_subscribe_ack sent 2014-06-19 22:07:34.499521 renew after 2014-06-19 22:10:04.499521
-67> 2014-06-19 22:07:34.500336 7f356dd88700 1 -- 192.168.1.15:6800/2943 <== mon.0 192.168.1.16:6789/0 10 ==== mdsmap(e 56) v1 ==== 623+0+0 (1640617471 0 0) 0x1e21800 con 0x1e38f20
-66> 2014-06-19 22:07:34.500360 7f3572f6e7c0 10 monclient: _send_mon_message to mon.0 at 192.168.1.16:6789/0
-65> 2014-06-19 22:07:34.500365 7f3572f6e7c0 1 -- 192.168.1.15:6800/2943 --> 192.168.1.16:6789/0 -- mdsbeacon(11715/0 up:boot seq 1 v0) v2 -- ?+0 0x1eb02c0 con 0x1e38f20
-64> 2014-06-19 22:07:34.500532 7f356dd88700 5 mds.-1.0 handle_mds_map epoch 56 from mon.0
-63> 2014-06-19 22:07:34.500582 7f356dd88700 1 -- 192.168.1.15:6800/2943 <== mon.0 192.168.1.16:6789/0 11 ==== osd_map(346..346 src has 1..346) v3 ==== 6466+0+0 (1629858625 0 0) 0x1df9200 con 0x1e38f20
-62> 2014-06-19 22:07:34.500604 7f356dd88700 1 -- 192.168.1.15:6800/2943 <== mon.0 192.168.1.16:6789/0 12 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (1074470259 0 0) 0x1e98380 con 0x1e38f20
-61> 2014-06-19 22:07:34.500617 7f356dd88700 10 monclient: handle_subscribe_ack sent 0.000000, ignoring
-60> 2014-06-19 22:07:35.793201 7f356dd88700 1 -- 192.168.1.15:6800/2943 <== mon.0 192.168.1.16:6789/0 13 ==== mdsmap(e 57) v1 ==== 607+0+0 (2841522137 0 0) 0x1e21400 con 0x1e38f20
-59> 2014-06-19 22:07:35.793248 7f356dd88700 5 mds.-1.-1 handle_mds_map epoch 57 from mon.0
-58> 2014-06-19 22:07:35.793288 7f356dd88700 1 -- 192.168.1.15:6800/2943 mark_down 192.168.1.15:6800/2782 -- pipe dne
-57> 2014-06-19 22:07:35.793309 7f356dd88700 1 mds.-1.0 handle_mds_map standby
-56> 2014-06-19 22:07:35.876659 7f356dd88700 1 -- 192.168.1.15:6800/2943 <== mon.0 192.168.1.16:6789/0 14 ==== mdsmap(e 58) v1 ==== 623+0+0 (2343310359 0 0) 0x1e21c00 con 0x1e38f20
-55> 2014-06-19 22:07:35.876671 7f356dd88700 5 mds.-1.0 handle_mds_map epoch 58 from mon.0
-54> 2014-06-19 22:07:35.876689 7f356dd88700 1 mds.0.15 handle_mds_map i am now mds.0.15
-53> 2014-06-19 22:07:35.876695 7f356dd88700 1 mds.0.15 handle_mds_map state change up:standby --> up:creating
-52> 2014-06-19 22:07:35.876704 7f356dd88700 3 mds.0.15 boot_create
-51> 2014-06-19 22:07:35.876726 7f356dd88700 5 mds.0.log create empty log
-50> 2014-06-19 22:07:35.876754 7f356dd88700 1 mds.0.journaler(ro) set_writeable
-49> 2014-06-19 22:07:35.876760 7f356dd88700 1 mds.0.journaler(rw) create blank journal
-48> 2014-06-19 22:07:35.876891 7f356dd88700 1 -- 192.168.1.15:6800/2943 --> 192.168.1.16:6805/13028 -- osd_op(mds.0.15:1 200.00000000 [writefull 0~84] 7.844f3494 ondisk+write e346) v4 -- ?+0 0x1df9200 con 0x1e39e40
-47> 2014-06-19 22:07:35.876926 7f356dd88700 3 mds.0.15 boot_create creating fresh hierarchy
-46> 2014-06-19 22:07:35.876931 7f356dd88700 0 mds.0.cache creating system inode with ino:1
-45> 2014-06-19 22:07:35.877134 7f356dd88700 1 -- 192.168.1.15:6800/2943 --> 192.168.1.16:6810/13173 -- osd_op(mds.0.15:2 1.00000000 [??? 1~0,omap-set-header 0~222] 7.6b2cdaff ondisk+write e346) v4 -- ?+0 0x1df8480 con 0x1e39fa0
-44> 2014-06-19 22:07:35.877189 7f356dd88700 1 -- 192.168.1.15:6800/2943 --> 192.168.1.16:6805/13028 -- osd_op(mds.0.15:3 1.00000000.inode [writefull 0~452] 7.232c0e14 ondisk+write e346) v4 -- ?+0 0x1df8000 con 0x1e39e40
-43> 2014-06-19 22:07:35.877198 7f356dd88700 3 mds.0.15 boot_create creating mydir hierarchy
-42> 2014-06-19 22:07:35.877204 7f356dd88700 0 mds.0.cache creating system inode with ino:100
-41> 2014-06-19 22:07:35.877209 7f356dd88700 0 mds.0.cache creating system inode with ino:600
-40> 2014-06-19 22:07:35.877337 7f356dd88700 1 -- 192.168.1.15:6800/2943 --> 192.168.1.16:6800/12881 -- osd_op(mds.0.15:4 600.00000000 [??? 1~0,omap-set-header 0~222] 7.c2e541b0 ondisk+write e346) v4 -- ?+0 0x1df9b00 con 0x1e3a3c0
-39> 2014-06-19 22:07:35.877369 7f356dd88700 0 mds.0.cache creating system inode with ino:601
-38> 2014-06-19 22:07:35.877443 7f356dd88700 1 -- 192.168.1.15:6800/2943 --> 192.168.1.15:6801/1251 -- osd_op(mds.0.15:5 601.00000000 [??? 1~0,omap-set-header 0~222] 7.81580290 ondisk+write e346) v4 -- ?+0 0x1df98c0 con 0x1e3aaa0
-37> 2014-06-19 22:07:35.877455 7f356dd88700 0 mds.0.cache creating system inode with ino:602
-36> 2014-06-19 22:07:35.877507 7f356dd88700 1 -- 192.168.1.15:6800/2943 --> 192.168.1.16:6800/12881 -- osd_op(mds.0.15:6 602.00000000 [??? 1~0,omap-set-header 0~222] 7.2a8b6bd0 ondisk+write e346) v4 -- ?+0 0x1df9680 con 0x1e3a3c0
-35> 2014-06-19 22:07:35.877519 7f356dd88700 0 mds.0.cache creating system inode with ino:603
-34> 2014-06-19 22:07:35.877556 7f356dd88700 1 -- 192.168.1.15:6800/2943 --> 192.168.1.16:6800/12881 -- osd_op(mds.0.15:7 603.00000000 [??? 1~0,omap-set-header 0~222] 7.7a4d91b0 ondisk+write e346) v4 -- ?+0 0x1df9440 con 0x1e3a3c0
-33> 2014-06-19 22:07:35.877566 7f356dd88700 0 mds.0.cache creating system inode with ino:604
-32> 2014-06-19 22:07:35.877595 7f356dd88700 1 -- 192.168.1.15:6800/2943 --> 192.168.1.16:6810/13173 -- osd_op(mds.0.15:8 604.00000000 [??? 1~0,omap-set-header 0~222] 7.43e85c95 ondisk+write e346) v4 -- ?+0 0x1dfaf40 con 0x1e39fa0
-31> 2014-06-19 22:07:35.877606 7f356dd88700 0 mds.0.cache creating system inode with ino:605
-30> 2014-06-19 22:07:35.877672 7f356dd88700 1 -- 192.168.1.15:6800/2943 --> 192.168.1.15:6811/1540 -- osd_op(mds.0.15:9 605.00000000 [??? 1~0,omap-set-header 0~222] 7.d0c18e1d ondisk+write e346) v4 -- ?+0 0x1dfad00 con 0x1e3a7e0
-29> 2014-06-19 22:07:35.877683 7f356dd88700 0 mds.0.cache creating system inode with ino:606
-28> 2014-06-19 22:07:35.877714 7f356dd88700 1 -- 192.168.1.15:6800/2943 --> 192.168.1.16:6800/12881 -- osd_op(mds.0.15:10 606.00000000 [??? 1~0,omap-set-header 0~222] 7.f89eaaf4 ondisk+write e346) v4 -- ?+0 0x1dfaac0 con 0x1e3a3c0
-27> 2014-06-19 22:07:35.877723 7f356dd88700 0 mds.0.cache creating system inode with ino:607
-26> 2014-06-19 22:07:35.877769 7f356dd88700 1 -- 192.168.1.15:6800/2943 --> 192.168.1.16:6810/13173 -- osd_op(mds.0.15:11 607.00000000 [??? 1~0,omap-set-header 0~222] 7.bb590b7c ondisk+write e346) v4 -- ?+0 0x1dfa880 con 0x1e39fa0
-25> 2014-06-19 22:07:35.877780 7f356dd88700 0 mds.0.cache creating system inode with ino:608
-24> 2014-06-19 22:07:35.877810 7f356dd88700 1 -- 192.168.1.15:6800/2943 --> 192.168.1.16:6810/13173 -- osd_op(mds.0.15:12 608.00000000 [??? 1~0,omap-set-header 0~222] 7.c4405e8e ondisk+write e346) v4 -- ?+0 0x1dfa640 con 0x1e39fa0
-23> 2014-06-19 22:07:35.877819 7f356dd88700 0 mds.0.cache creating system inode with ino:609
-22> 2014-06-19 22:07:35.877849 7f356dd88700 1 -- 192.168.1.15:6800/2943 --> 192.168.1.16:6805/13028 -- osd_op(mds.0.15:13 609.00000000 [??? 1~0,omap-set-header 0~222] 7.60b82d07 ondisk+write e346) v4 -- ?+0 0x1dfa400 con 0x1e39e40
-21> 2014-06-19 22:07:35.877858 7f356dd88700 0 mds.0.cache creating system inode with ino:200
-20> 2014-06-19 22:07:35.878036 7f356dd88700 1 -- 192.168.1.15:6800/2943 --> 192.168.1.16:6810/13173 -- osd_op(mds.0.15:14 100.00000000 [??? 1~0,omap-set-header 0~222,omap-set-vals 0~4372] 7.c5265ab3 ondisk+write e346) v4 -- ?+0 0x1df9d40 con 0x1e39fa0
-19> 2014-06-19 22:07:35.878076 7f356dd88700 1 -- 192.168.1.15:6800/2943 --> 192.168.1.16:6810/13173 -- osd_op(mds.0.15:15 100.00000000.inode [writefull 0~452] 7.85dde07f ondisk+write e346) v4 -- ?+0 0x1dfa1c0 con 0x1e39fa0
-18> 2014-06-19 22:07:35.878119 7f356dd88700 1 -- 192.168.1.15:6800/2943 --> 192.168.1.16:6800/12881 -- osd_op(mds.0.15:16 mds0_inotable [writefull 0~34] 7.b852b893 ondisk+write e346) v4 -- ?+0 0x1dfc380 con 0x1e3a3c0
-17> 2014-06-19 22:07:35.878148 7f356dd88700 1 -- 192.168.1.15:6800/2943 --> 192.168.1.15:6801/1251 -- osd_op(mds.0.15:17 mds0_sessionmap [writefull 0~22] 7.3270c60b ondisk+write e346) v4 -- ?+0 0x1dfc140 con 0x1e3aaa0
-16> 2014-06-19 22:07:35.878179 7f356dd88700 1 -- 192.168.1.15:6800/2943 --> 192.168.1.15:6811/1540 -- osd_op(mds.0.15:18 mds_anchortable [writefull 0~34] 7.a977f6a7 ondisk+write e346) v4 -- ?+0 0x1dfbf00 con 0x1e3a7e0
-15> 2014-06-19 22:07:35.878230 7f356dd88700 1 -- 192.168.1.15:6800/2943 --> 192.168.1.16:6800/12881 -- osd_op(mds.0.15:19 mds_snaptable [writefull 0~46] 7.d90270ad ondisk+write e346) v4 -- ?+0 0x1dfbcc0 con 0x1e3a3c0
-14> 2014-06-19 22:07:35.878452 7f356dd88700 5 mds.0.log submit_entry 4194304~756 : ESubtreeMap 2 subtrees , 0 ambiguous [metablob 1, 2 dirs]
-13> 2014-06-19 22:07:35.878505 7f356dd88700 1 -- 192.168.1.15:6800/2943 --> 192.168.1.16:6800/12881 -- osd_op(mds.0.15:20 200.00000001 [delete] 7.6e5f474 ondisk+write e346) v4 -- ?+0 0x1dfba80 con 0x1e3a3c0
-12> 2014-06-19 22:07:35.878533 7f356dd88700 1 -- 192.168.1.15:6800/2943 --> 192.168.1.16:6800/12881 -- osd_op(mds.0.15:21 200.00000002 [delete] 7.eb272dbb ondisk+write e346) v4 -- ?+0 0x1dfb840 con 0x1e3a3c0
-11> 2014-06-19 22:07:35.878557 7f356dd88700 1 -- 192.168.1.15:6800/2943 --> 192.168.1.15:6811/1540 -- osd_op(mds.0.15:22 200.00000003 [delete] 7.95e3ab ondisk+write e346) v4 -- ?+0 0x1dfb600 con 0x1e3a7e0
-10> 2014-06-19 22:07:35.878581 7f356dd88700 1 -- 192.168.1.15:6800/2943 --> 192.168.1.16:6800/12881 -- osd_op(mds.0.15:23 200.00000004 [delete] 7.1ba6bc1f ondisk+write e346) v4 -- ?+0 0x1dfb3c0 con 0x1e3a3c0
-9> 2014-06-19 22:07:35.878605 7f356dd88700 1 -- 192.168.1.15:6800/2943 --> 192.168.1.16:6805/13028 -- osd_op(mds.0.15:24 200.00000005 [delete] 7.96d5827a ondisk+write e346) v4 -- ?+0 0x1dfd340 con 0x1e39e40
-8> 2014-06-19 22:07:35.878629 7f356dd88700 1 -- 192.168.1.15:6800/2943 --> 192.168.1.16:6805/13028 -- osd_op(mds.0.15:25 200.00000006 [delete] 7.ade7efde ondisk+write e346) v4 -- ?+0 0x1dfd100 con 0x1e39e40
-7> 2014-06-19 22:07:35.878673 7f356dd88700 5 mds.0.bal rebalance done
-6> 2014-06-19 22:07:35.878690 7f356dd88700 5 mds.0.15 ms_handle_connect on 192.168.1.15:6801/1251
-5> 2014-06-19 22:07:35.879367 7f356dd88700 5 mds.0.15 ms_handle_connect on 192.168.1.15:6811/1540
-4> 2014-06-19 22:07:35.879617 7f356dd88700 5 mds.0.15 ms_handle_connect on 192.168.1.16:6805/13028
-3> 2014-06-19 22:07:35.879898 7f356dd88700 5 mds.0.15 ms_handle_connect on 192.168.1.16:6810/13173
-2> 2014-06-19 22:07:35.879965 7f356dd88700 5 mds.0.15 ms_handle_connect on 192.168.1.16:6800/12881
-1> 2014-06-19 22:07:35.881174 7f356dd88700 1 -- 192.168.1.15:6800/2943 <== osd.3 192.168.1.15:6801/1251 1 ==== osd_op_reply(5 601.00000000 [??? 1~0,omap-set-header 0~222] v0'0 uv0 ondisk = -95 ((95) Operation not supported)) v6 ==== 221+0+0 (685458269 0 0) 0x1eda300 con 0x1e3aaa0
0> 2014-06-19 22:07:35.883239 7f356dd88700 -1 mds/CDir.cc: In function 'virtual void C_Dir_Committed::finish(int)' thread 7f356dd88700 time 2014-06-19 22:07:35.881337
mds/CDir.cc: 1809: FAILED assert(r == 0)
ceph version 0.80.1 (a38fe1169b6d2ac98b427334c12d7cf81f809b74)
1: ceph-mds() [0x75c6f1]
2: (Context::complete(int)+0x9) [0x56cff9]
3: (C_Gather::sub_finish(Context*, int)+0x1f7) [0x56e9a7]
4: (C_Gather::C_GatherSub::finish(int)+0x12) [0x56eab2]
5: (Context::complete(int)+0x9) [0x56cff9]
6: (Objecter::handle_osd_op_reply(MOSDOpReply*)+0xf4e) [0x7d26ee]
7: (MDS::handle_core_message(Message*)+0xb1f) [0x58e5ef]
8: (MDS::_dispatch(Message*)+0x32) [0x58e7f2]
9: (MDS::ms_dispatch(Message*)+0xa3) [0x5901d3]
10: (DispatchQueue::entry()+0x57a) [0x99d9da]
11: (DispatchQueue::DispatchThread::entry()+0xd) [0x8be63d]
12: (()+0x7c53) [0x7f3572366c53]
13: (clone()+0x6d) [0x7f3571257dbd]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 keyvaluestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
-2/-2 (syslog threshold)
99/99 (stderr threshold)
max_recent 10000
max_new 1000
log_file
--- end dump of recent events ---
terminate called after throwing an instance of 'ceph::FailedAssertion'
*** Caught signal (Aborted) **
in thread 7f356dd88700
ceph version 0.80.1 (a38fe1169b6d2ac98b427334c12d7cf81f809b74)
1: ceph-mds() [0x7f71df]
2: (()+0xef90) [0x7f357236df90]
3: (gsignal()+0x39) [0x7f35711979e9]
4: (abort()+0x148) [0x7f35711990f8]
5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f3571a9b565]
6: (()+0x5e6c6) [0x7f3571a996c6]
7: (()+0x5e6f3) [0x7f3571a996f3]
8: (()+0x5e91f) [0x7f3571a9991f]
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1f2) [0x8d6922]
10: ceph-mds() [0x75c6f1]
11: (Context::complete(int)+0x9) [0x56cff9]
12: (C_Gather::sub_finish(Context*, int)+0x1f7) [0x56e9a7]
13: (C_Gather::C_GatherSub::finish(int)+0x12) [0x56eab2]
14: (Context::complete(int)+0x9) [0x56cff9]
15: (Objecter::handle_osd_op_reply(MOSDOpReply*)+0xf4e) [0x7d26ee]
16: (MDS::handle_core_message(Message*)+0xb1f) [0x58e5ef]
17: (MDS::_dispatch(Message*)+0x32) [0x58e7f2]
18: (MDS::ms_dispatch(Message*)+0xa3) [0x5901d3]
19: (DispatchQueue::entry()+0x57a) [0x99d9da]
20: (DispatchQueue::DispatchThread::entry()+0xd) [0x8be63d]
21: (()+0x7c53) [0x7f3572366c53]
22: (clone()+0x6d) [0x7f3571257dbd]
2014-06-19 22:07:35.893814 7f356dd88700 -1 *** Caught signal (Aborted) **
in thread 7f356dd88700
ceph version 0.80.1 (a38fe1169b6d2ac98b427334c12d7cf81f809b74)
1: ceph-mds() [0x7f71df]
2: (()+0xef90) [0x7f357236df90]
3: (gsignal()+0x39) [0x7f35711979e9]
4: (abort()+0x148) [0x7f35711990f8]
5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f3571a9b565]
6: (()+0x5e6c6) [0x7f3571a996c6]
7: (()+0x5e6f3) [0x7f3571a996f3]
8: (()+0x5e91f) [0x7f3571a9991f]
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1f2) [0x8d6922]
10: ceph-mds() [0x75c6f1]
11: (Context::complete(int)+0x9) [0x56cff9]
12: (C_Gather::sub_finish(Context*, int)+0x1f7) [0x56e9a7]
13: (C_Gather::C_GatherSub::finish(int)+0x12) [0x56eab2]
14: (Context::complete(int)+0x9) [0x56cff9]
15: (Objecter::handle_osd_op_reply(MOSDOpReply*)+0xf4e) [0x7d26ee]
16: (MDS::handle_core_message(Message*)+0xb1f) [0x58e5ef]
17: (MDS::_dispatch(Message*)+0x32) [0x58e7f2]
18: (MDS::ms_dispatch(Message*)+0xa3) [0x5901d3]
19: (DispatchQueue::entry()+0x57a) [0x99d9da]
20: (DispatchQueue::DispatchThread::entry()+0xd) [0x8be63d]
21: (()+0x7c53) [0x7f3572366c53]
22: (clone()+0x6d) [0x7f3571257dbd]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- begin dump of recent events ---
0> 2014-06-19 22:07:35.893814 7f356dd88700 -1 *** Caught signal (Aborted) **
in thread 7f356dd88700
ceph version 0.80.1 (a38fe1169b6d2ac98b427334c12d7cf81f809b74)
1: ceph-mds() [0x7f71df]
2: (()+0xef90) [0x7f357236df90]
3: (gsignal()+0x39) [0x7f35711979e9]
4: (abort()+0x148) [0x7f35711990f8]
5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f3571a9b565]
6: (()+0x5e6c6) [0x7f3571a996c6]
7: (()+0x5e6f3) [0x7f3571a996f3]
8: (()+0x5e91f) [0x7f3571a9991f]
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1f2) [0x8d6922]
10: ceph-mds() [0x75c6f1]
11: (Context::complete(int)+0x9) [0x56cff9]
12: (C_Gather::sub_finish(Context*, int)+0x1f7) [0x56e9a7]
13: (C_Gather::C_GatherSub::finish(int)+0x12) [0x56eab2]
14: (Context::complete(int)+0x9) [0x56cff9]
15: (Objecter::handle_osd_op_reply(MOSDOpReply*)+0xf4e) [0x7d26ee]
16: (MDS::handle_core_message(Message*)+0xb1f) [0x58e5ef]
17: (MDS::_dispatch(Message*)+0x32) [0x58e7f2]
18: (MDS::ms_dispatch(Message*)+0xa3) [0x5901d3]
19: (DispatchQueue::entry()+0x57a) [0x99d9da]
20: (DispatchQueue::DispatchThread::entry()+0xd) [0x8be63d]
21: (()+0x7c53) [0x7f3572366c53]
22: (clone()+0x6d) [0x7f3571257dbd]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 keyvaluestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
-2/-2 (syslog threshold)
99/99 (stderr threshold)
max_recent 10000
max_new 1000
log_file
--- end dump of recent events ---
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
