Re: [ceph-users] [Problem] I cannot start the OSD daemon

Mon, 07 Sep 2015 02:47:05 -0700 


Supplement[On OSD-0 Node]:

[root@node1 ceph]# *ls /var/lib/ceph/osd/ceph-0/current/*
0.0_head 0.18_head 0.20_head 0.29_head 0.31_head 0.3a_head 0.6_head 0.f_head 2.0_head commit_op_seq 0.10_head 0.19_head 0.21_head 0.2a_head 0.32_head 0.3b_head 0.7_head 1.0_head 2.1_head meta 0.11_head 0.1a_head 0.22_head 0.2b_head 0.33_head 0.3c_head 0.8_head 1.1_head 2.2_head nosnap 0.12_head 0.1b_head 0.23_head 0.2c_head 0.34_head 0.3d_head 0.9_head 1.2_head 2.3_head omap 0.13_head 0.1c_head 0.24_head 0.2d_head 0.35_head 0.3e_head 0.a_head 1.3_head 2.4_head 0.14_head 0.1d_head 0.25_head 0.2e_head 0.36_head 0.3f_head 0.b_head 1.4_head 2.5_head 0.15_head 0.1e_head 0.26_head 0.2f_head 0.37_head 0.3_head 0.c_head 1.5_head 2.6_head 0.16_head 0.1f_head 0.27_head 0.2_head 0.38_head 0.4_head 0.d_head 1.6_head 2.7_head 0.17_head 0.1_head 0.28_head 0.30_head 0.39_head 0.5_head 0.e_head 1.7_head 3.5_head 
[root@node1 ceph]#
[root@node1 ceph]# *ls /var/lib/ceph/osd/ceph-0/current/3.5_head/*
__head_BD49D10D__3
[root@node1 ceph]#
[root@node1 ceph]# *ceph osd stat*
     osdmap e20: 2 osds: 0 up, 1 in
[root@node1 ceph]#
[root@node1 ceph]# *ceph osd tree*
ID WEIGHT  TYPE NAME      UP/DOWN REWEIGHT PRIMARY-AFFINITY
-1 0.21997 root default
-2 0.10999     host node1
 0 0.10999         osd.0     down        0          1.00000
-3 0.10999     host node2
 1 0.10999         osd.1     down  1.00000          1.00000
[root@node1 ceph]#
[root@node1 ceph]# *ceph osd dump*
epoch 20
fsid 86130593-7041-4066-aa9b-963f4c5a6832
created 2015-08-05 15:42:28.938948
modified 2015-09-07 10:48:45.182566
flags
pool 0 'rbd' replicated size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 flags hashpspool stripe_width 0 
max_osd 2
osd.0 down out weight 0 up_from 4 up_thru 14 down_at 17 last_clean_interval [0,0) 10.241.226.116:6800/1084 10.241.226.116:6801/1084 10.241.226.116:6802/1084 10.241.226.116:6803/1084 autoout,exists c5f917c4-fd48-4bbf-954a-ab1ee1b524ad osd.1 down in weight 1 up_from 8 up_thru 17 down_at 20 last_clean_interval [0,0) 10.241.226.117:6800/1949 10.241.226.117:6801/1949 10.241.226.117:6802/1949 10.241.226.117:6803/1949 exists e1aaeab0-627c-4f22-b506-4856f2c8befb 
[root@node1 ceph-0]# ls current/0.11_head/
__head_00000011__0
[root@node1 ceph]#


在 2015/9/7 16:16, Aaron 写道:

Hi All,
I cannot start the OSD daemon, and need your helps. Any advice is appreciated.
When I deployed the ceph cluster through ceph-deploy, it worked. But after some days, all the OSD daemons were down, and I could not start them. Then I redeployed it some times, and it was still this case. 

*The environment:*
OS: CentOS 6.6
Ceph: hammer(0.92.2)
*When the OSD daemon was down, I started the OSD daemon on the OSD node, and the console outputted:* 
[root@node1 ~]# service ceph start
=== osd.0 ===
libust[2285/2285]: Warning: HOME environment variable not set. Disabling LTTng-UST per-user tracing. (in setup_local_apps() at lttng-ust-comm.c:305) create-or-move updated item name 'osd.0' weight 0.11 at location {host=node1,root=default} to crush map 
Starting Ceph osd.0 on node1...
libust[2329/2329]: Warning: HOME environment variable not set. Disabling LTTng-UST per-user tracing. (in setup_local_apps() at lttng-ust-comm.c:305) starting osd.0 at :/0 osd_data /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal 
[root@node1 ~]#

*And the log file of ceph, /var/log/ceph/ceph-osd.0.log, recorded:*
2015-09-07 15:13:02.433546 7f74cea0d800 0 ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3), process ceph-osd, pid 1045 2015-09-07 15:13:02.508300 7f74cea0d800 0 filestore(/var/lib/ceph/osd/ceph-0) backend generic (magic 0xef53) 2015-09-07 15:13:02.511270 7f74cea0d800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: FIEMAP ioctl is supported and appears to work 2015-09-07 15:13:02.511293 7f74cea0d800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
2015-09-07 15:13:02.520826 7f74cea0d800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: syscall(SYS_syncfs, fd) fully supported 2015-09-07 15:13:02.523890 7f74cea0d800 0 filestore(/var/lib/ceph/osd/ceph-0) limited size xattrs 2015-09-07 15:13:02.527387 7f74cea0d800 0 filestore(/var/lib/ceph/osd/ceph-0) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled 2015-09-07 15:13:02.528870 7f74cea0d800 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2015-09-07 15:13:02.528890 7f74cea0d800 1 2015-09-07 15:13:02.528890 7f74cea0d800 1 journal _open /var/lib/ceph/osd/ceph-0/journal fd 19: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
2015-09-07 15:13:02.529881 7f74cea0d800 -1 filestore(/var/lib/ceph/osd/ceph-0) could not find 3//head//3 in index: (2) No such file or directory 2015-09-07 15:13:02.530039 7f74cea0d800 -1 filestore(/var/lib/ceph/osd/ceph-0) could not find 2//head//3 in index: (2) No such file or directory 2015-09-07 15:13:02.530273 7f74cea0d800 -1 filestore(/var/lib/ceph/osd/ceph-0) could not find 1//head//3 in index: (2) No such file or directory 2015-09-07 15:13:02.530349 7f74cea0d800 -1 filestore(/var/lib/ceph/osd/ceph-0) could not find 0//head//3 in index: (2) No such file or directory
2015-09-07 15:13:02.530487 7f74cea0d800 -1 filestore(/var/lib/ceph/osd/ceph-0) could not find 7//head//3 in index: (2) No such file or directory 2015-09-07 15:13:02.530569 7f74cea0d800 -1 filestore(/var/lib/ceph/osd/ceph-0) could not find 6//head//3 in index: (2) No such file or directory 2015-09-07 15:13:02.530770 7f74cea0d800 0 filestore(/var/lib/ceph/osd/ceph-0) error (39) Directory not empty not handled on operation 0x55d8ef6 (4434.0.1, or op 1, counting from 0) 2015-09-07 15:13:02.530798 7f74cea0d800 0 filestore(/var/lib/ceph/osd/ceph-0) ENOTEMPTY suggests garbage data in osd data dir
2015-09-07 15:13:02.530801 7f74cea0d800 0 filestore(/var/lib/ceph/osd/ceph-0) transaction dump: 
{
    "ops": [
        {
            "op_num": 0,
            "op_name": "remove",
            "collection": "3.5_head",
            "oid": "5\/\/head\/\/3"
        },
        {
            "op_num": 1,
            "op_name": "rmcoll",
            "collection": "3.5_head"
        }
    ]
}

          "op_name": "remove",
            "collection": "3.5_head",
            "oid": "5\/\/head\/\/3"
        },
        {
            "op_num": 1,
            "op_name": "rmcoll",
            "collection": "3.5_head"
        }
    ]
}
2015-09-07 15:13:02.533771 7f74cea0d800 -1 os/FileStore.cc: In function 'unsigned int FileStore::_do_transaction(ObjectStore::Transaction&, uint64_t, int, ThreadPool::TPHandle*)' thread 7f74cea0d800 time 2015-09-07 15:13:02.530848 
os/FileStore.cc: 2757: FAILED assert(0 == "unexpected error")

ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3)
1: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int, ThreadPool::TPHandle*)+0xc16) [0x975a06] 2: (FileStore::_do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long, ThreadPool::TPHandle*)+0x64) [0x97d794] 
3: (JournalingObjectStore::journal_replay(unsigned long)+0x61e) [0xa2927e]
4: (FileStore::mount()+0x2d0e) [0x963d5e]
5: (OSD::init()+0x273) [0x6891f3]
6: (main()+0x384f) [0x62e10f]
7: (__libc_start_main()+0xfd) [0x3c70c1ed5d]
8: /usr/bin/ceph-osd() [0x629819]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. 

--- begin dump of recent events ---
-126> 2015-09-07 15:13:02.431001 7f74cea0d800 5 asok(0x5630000) register_command perfcounters_dump hook 0x55a8050 -125> 2015-09-07 15:13:02.431054 7f74cea0d800 5 asok(0x5630000) register_command 1 hook 0x55a8050 -126> 2015-09-07 15:13:02.431001 7f74cea0d800 5 asok(0x5630000) register_command perfcounters_dump hook 0x55a8050 -125> 2015-09-07 15:13:02.431054 7f74cea0d800 5 asok(0x5630000) register_command 1 hook 0x55a8050 -124> 2015-09-07 15:13:02.431059 7f74cea0d800 5 asok(0x5630000) register_command perf dump hook 0x55a8050 -124> 2015-09-07 15:13:02.431059 7f74cea0d800 5 asok(0x5630000) register_command perf dump hook 0x55a8050 -123> 2015-09-07 15:13:02.431070 7f74cea0d800 5 asok(0x5630000) register_command perfcounters_schema hook 0x55a8050
-122> 2015-09-07 15:13:02.431075 7f74cea0d800 5 -122> 2015-09-07 15:13:02.431075 7f74cea0d800 5 asok(0x5630000) register_command 2 hook 0x55a8050
-121> 2015-09-07 15:13:02.431085 7f74cea0d800 5 asok(0x5630000) register_command perf schema hook 0x55a8050 -121> 2015-09-07 15:13:02.431085 7f74cea0d800 5 asok(0x5630000) register_command perf schema hook 0x55a8050 -120> 2015-09-07 15:13:02.431089 7f74cea0d800 5 asok(0x5630000) register_command perf reset hook 0x55a8050 -119> 2015-09-07 15:13:02.431097 7f74cea0d800 5 asok(0x5630000) register_command config show hook 0x55a8050 -119> 2015-09-07 15:13:02.431097 7f74cea0d800 5 asok(0x5630000) register_command config show hook 0x55a8050 -118> 2015-09-07 15:13:02.431101 7f74cea0d800 5 asok(0x5630000) register_command config set hook 0x55a8050 -118> 2015-09-07 15:13:02.431101 7f74cea0d800 5 asok(0x5630000) register_command config set hook 0x55a8050 -117> 2015-09-07 15:13:02.431109 7f74cea0d800 5 asok(0x5630000) register_command config get hook 0x55a8050 -116> 2015-09-07 15:13:02.431112 7f74cea0d800 5 asok(0x5630000) register_command config diff hook 0x55a8050 -115> 2015-09-07 15:13:02.431121 7f74cea0d800 5 asok(0x5630000) register_command log flush hook 0x55a8050 -117> 2015-09-07 15:13:02.431109 7f74cea0d800 5 asok(0x5630000) register_command config get hook 0x55a8050 -116> 2015-09-07 15:13:02.431112 7f74cea0d800 5 asok(0x5630000) register_command config diff hook 0x55a8050 -115> 2015-09-07 15:13:02.431121 7f74cea0d800 5 asok(0x5630000) register_command log flush hook 0x55a8050 -114> 2015-09-07 15:13:02.431124 7f74cea0d800 5 asok(0x5630000) register_command log dump hook 0x55a8050 
asok(0x5630000) register_command log dump hook 0x55a8050
-113> 2015-09-07 15:13:02.431133 7f74cea0d800 5 asok(0x5630000) register_command log reopen hook 0x55a8050 
asok(0x5630000) register_command log reopen hook 0x55a8050
-112> 2015-09-07 15:13:02.433546 7f74cea0d800 0 -112> 2015-09-07 15:13:02.433546 7f74cea0d800 0 ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3), process ceph-osd, pid 1045 -111> 2015-09-07 15:13:02.434302 7f74cea0d800 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6800/1045 need_addr=1accepter.accepter.bind my_inst.addr is 0.0.0.0:6800/1045 need_addr=1 -110> 2015-09-07 15:13:02.434355 7f74cea0d800 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6801/1045 need_addr=1 -109> 2015-09-07 15:13:02.434380 7f74cea0d800 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6802/1045 need_addr=1accepter.accepter.bind my_inst.addr is 0.0.0.0:6802/1045 need_addr=1 -108> 2015-09-07 15:13:02.434403 7f74cea0d800 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6803/1045 need_addr=1 
accepter.accepter.bind my_inst.addr is 0.0.0.0:6803/1045 need_addr=1
-107> 2015-09-07 15:13:02.435349 7f74cea0d800 1 finished global_init_daemonize -107> 2015-09-07 15:13:02.435349 7f74cea0d800 1 finished global_init_daemonize -106> 2015-09-07 15:13:02.491120 7f74cea0d800 5 asok(0x5630000) init /var/run/ceph/ceph-osd.0.asok -106> 2015-09-07 15:13:02.491120 7f74cea0d800 5 asok(0x5630000) init /var/run/ceph/ceph-osd.0.asok -105> 2015-09-07 15:13:02.491164 7f74cea0d800 5 asok(0x5630000) bind_and_listen /var/run/ceph/ceph-osd.0.asok -105> 2015-09-07 15:13:02.491164 7f74cea0d800 5 asok(0x5630000) bind_and_listen /var/run/ceph/ceph-osd.0.asok -104> 2015-09-07 15:13:02.491404 7f74cea0d800 5 asok(0x5630000) register_command 0 hook 0x55a00b0 -104> 2015-09-07 15:13:02.491404 7f74cea0d800 5 asok(0x5630000) register_command 0 hook 0x55a00b0 -103> 2015-09-07 15:13:02.491429 7f74cea0d800 5 asok(0x5630000) register_command version hook 0x55a00b0 -103> 2015-09-07 15:13:02.491429 7f74cea0d800 5 asok(0x5630000) register_command version hook 0x55a00b0 -102> 2015-09-07 15:13:02.491435 7f74cea0d800 5 asok(0x5630000) register_command git_version hook 0x55a00b0 -102> 2015-09-07 15:13:02.491435 7f74cea0d800 5 asok(0x5630000) register_command git_version hook 0x55a00b0 -101> 2015-09-07 15:13:02.491441 7f74cea0d800 5 f74cea0d800 5 asok(0x5630000) register_command help hook 0x55a8150 -100> 2015-09-07 15:13:02.491445 7f74cea0d800 5 asok(0x5630000) register_command get_command_descriptions hook 0x55a8140 -99> 2015-09-07 15:13:02.491545 7f74cea0d800 10 monclient(hunting): build_initial_monmap -98> 2015-09-07 15:13:02.491563 7f74cd141700 5 asok(0x5630000) entry start -97> 2015-09-07 15:13:02.506754 7f74cea0d800 5 adding auth protocol: cephx -96> 2015-09-07 15:13:02.506788 7f74cea0d800 5 adding auth protocol: cephx -95> 2015-09-07 15:13:02.507056 7f74cea0d800 5 asok(0x5630000) register_command objecter_requests hook 0x55a8180 -94> 2015-09-07 15:13:02.507286 7f74cea0d800 1 -- 0.0.0.0:6800/1045 messenger.start 
   -93> 2015-09-07 15:13:02.507367 7f74cea0d800  1 -- :/0 messenger.start
-92> 2015-09-07 15:13:02.507504 7f74cea0d800 1 -- 0.0.0.0:6803/1045 messenger.start -91> 2015-09-07 15:13:02.507637 7f74cea0d800 1 -- 0.0.0.0:6802/1045 messenger.start -90> 2015-09-07 15:13:02.507744 7f74cea0d800 1 -- 0.0.0.0:6801/1045 messenger.start 
   -89> 2015-09-07 15:13:02.507889 7f74cea0d800  1 -- :/0 messenger.start
-88> 2015-09-07 15:13:02.508178 7f74cea0d800 2 osd.0 0 mounting /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal -87> 2015-09-07 15:13:02.508300 7f74cea0d800 0 filestore(/var/lib/ceph/osd/ceph-0) backend generic (magic 0xef53) -86> 2015-09-07 15:13:02.511270 7f74cea0d800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: FIEMAP ioctl is supported and appears to work -85> 2015-09-07 15:13:02.511293 7f74cea0d800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: FIEMAP ioctl is supported and appears to work -85> 2015-09-07 15:13:02.511293 7f74cea0d800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option -84> 2015-09-07 15:13:02.520826 7f74cea0d800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: syscall(SYS_syncfs, fd) fully supported
-83> 2015-09-07 15:13:02.523890 7f74cea0d800 0 filestore(/var/lib/ceph/osd/ceph-0) limited size xattrsfilestore(/var/lib/ceph/osd/ceph-0) limited size xattrs -82> 2015-09-07 15:13:02.527387 7f74cea0d800 0 filestore(/var/lib/ceph/osd/ceph-0) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled filestore(/var/lib/ceph/osd/ceph-0) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled -81> 2015-09-07 15:13:02.528819 7f74cea0d800 2 -81> 2015-09-07 15:13:02.528819 7f74cea0d800 2 journal open /var/lib/ceph/osd/ceph-0/journal fsid c5f917c4-fd48-4bbf-954a-ab1ee1b524ad fs_op_seq 4411 -80> 2015-09-07 15:13:02.528870 7f74cea0d800 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
-79> 2015-09-07 15:13:02.528890 7f74cea0d800 1 -79> 2015-09-07 15:13:02.528890 7f74cea0d800 1 journal _open /var/lib/ceph/osd/ceph-0/journal fd 19: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0 -78> 2015-09-07 15:13:02.528954 7f74cea0d800 2 journal read_entry 85811200 : seq 4412 3703 bytes 
journal read_entry 85811200 : seq 4412 3703 bytes
-77> 2015-09-07 15:13:02.528966 7f74cea0d800 2 journal read_entry 85811200 : seq 4412 3703 bytes -77> 2015-09-07 15:13:02.528966 7f74cea0d800 2 journal read_entry 85811200 : seq 4412 3703 bytes -76> 2015-09-07 15:13:02.528971 7f74cea0d800 3 -76> 2015-09-07 15:13:02.528971 7f74cea0d800 3 journal journal_replay: applying op seq 4412 -75> 2015-09-07 15:13:02.529399 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4412journal journal_replay: r = 0, op_seq now 4412 -74> 2015-09-07 15:13:02.529432 7f74cea0d800 2 journal read_entry 85815296 : seq 4413 959 bytes 
journal read_entry 85815296 : seq 4413 959 bytes
-73> 2015-09-07 15:13:02.529436 7f74cea0d800 3 journal journal_replay: applying op seq 4413 -73> 2015-09-07 15:13:02.529436 7f74cea0d800 3 journal journal_replay: applying op seq 4413 -72> 2015-09-07 15:13:02.529476 7f74cea0d800 3 -72> 2015-09-07 15:13:02.529476 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4413 -71> 2015-09-07 15:13:02.529501 7f74cea0d800 2 journal read_entry 85819392 : seq 4414 46 bytes -70> 2015-09-07 15:13:02.529573 7f74cea0d800 3 -70> 2015-09-07 15:13:02.529573 7f74cea0d800 3 journal journal_replay: applying op seq 4414
-69> 2015-09-07 15:13:02.529579 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4414 -69> 2015-09-07 15:13:02.529579 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4414 -68> 2015-09-07 15:13:02.529590 7f74cea0d800 2 journal read_entry 85823488 : seq 4415 959 bytes 
journal read_entry 85823488 : seq 4415 959 bytes
-67> 2015-09-07 15:13:02.529592 7f74cea0d800 3 -67> 2015-09-07 15:13:02.529592 7f74cea0d800 3 journal journal_replay: applying op seq 4415 -66> 2015-09-07 15:13:02.529620 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4415 -65> 2015-09-07 15:13:02.529645 7f74cea0d800 2 -65> 2015-09-07 15:13:02.529645 7f74cea0d800 2 journal read_entry 85827584 : seq 4416 959 bytes
-64> 2015-09-07 15:13:02.529648 7f74cea0d800 3 journal journal_replay: applying op -64> 2015-09-07 15:13:02.529648 7f74cea0d800 3 journal journal_replay: applying op seq 4416 seq 4416 -63> 2015-09-07 15:13:02.529672 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4416 -62> 2015-09-07 15:13:02.529687 7f74cea0d800 2 journal read_entry 85831680 : seq 4417 959 bytes -61> 2015-09-07 15:13:02.529690 7f74cea0d800 3 journal journal_replay: applying op seq 4417 -61> 2015-09-07 15:13:02.529690 7f74cea0d800 3 journal journal_replay: applying op seq 4417 -60> 2015-09-07 15:13:02.529711 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4417 
journal journal_replay: r = 0, op_seq now 4417
-59> 2015-09-07 15:13:02.529776 7f74cea0d800 2 -59> 2015-09-07 15:13:02.529776 7f74cea0d800 2 journal read_entry 85835776 : seq 4418 959 bytes -58> 2015-09-07 15:13:02.529779 7f74cea0d800 3 journal journal_replay: applying op seq 4418 
journal journal_replay: applying op seq 4418
-57> 2015-09-07 15:13:02.529803 7f74cea0d800 3 -57> 2015-09-07 15:13:02.529803 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4418 -56> 2015-09-07 15:13:02.529819 7f74cea0d800 2 journal read_entry 85839872 : seq 4419 959 bytesjournal read_entry 85839872 : seq 4419 959 bytes -55> 2015-09-07 15:13:02.529822 7f74cea0d800 3 journal journal_replay: applying op seq 4419 
journal journal_replay: applying op seq 4419
-54> 2015-09-07 15:13:02.529842 7f74cea0d800 3 -54> 2015-09-07 15:13:02.529842 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4419 -53> 2015-09-07 15:13:02.529857 7f74cea0d800 2 journal read_entry 85843968 : seq 4420 259 bytes -52> 2015-09-07 15:13:02.529860 7f74cea0d800 3 journal journal_replay: applying op seq 4420 -51> 2015-09-07 15:13:02.529881 7f74cea0d800 -1 filestore(/var/lib/ceph/osd/ceph-0) could not find 3//head//3 in index: (2) No such file or directory -50> 2015-09-07 15:13:02.529946 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4420 -49> 2015-09-07 15:13:02.529958 7f74cea0d800 2 journal read_entry 85848064 : seq 4421 46 bytes -48> 2015-09-07 15:13:02.529962 7f74cea0d800 3 journal journal_replay: applying op seq 4421 -47> 2015-09-07 15:13:02.529967 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4421 -46> 2015-09-07 15:13:02.529976 7f74cea0d800 2 journal read_entry 85852160 : seq 4422 259 bytes -45> 2015-09-07 15:13:02.529979 7f74cea0d800 3 journal journal_replay: applying op seq 4422 -44> 2015-09-07 15:13:02.530039 7f74cea0d800 -1 filestore(/var/lib/ceph/osd/ceph-0) could not find 2//head//3 in index: (2) No such file or directory -43> 2015-09-07 15:13:02.530070 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4422 -42> 2015-09-07 15:13:02.530082 7f74cea0d800 2 journal read_entry 85856256 : seq 4423 46 bytes -41> 2015-09-07 15:13:02.530085 7f74cea0d800 3 journal journal_replay: applying op seq 4423 -40> 2015-09-07 15:13:02.530088 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4423 -39> 2015-09-07 15:13:02.530097 7f74cea0d800 2 -53> 2015-09-07 15:13:02.529857 7f74cea0d800 2 journal read_entry 85843968 : seq 4420 259 bytes -52> 2015-09-07 15:13:02.529860 7f74cea0d800 3 journal journal_replay: applying op seq 4420 -51> 2015-09-07 15:13:02.529881 7f74cea0d800 -1 filestore(/var/lib/ceph/osd/ceph-0) could not find 3//head//3 in index: (2) No such file or directory -50> 2015-09-07 15:13:02.529946 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4420 -49> 2015-09-07 15:13:02.529958 7f74cea0d800 2 journal read_entry 85848064 : seq 4421 46 bytes -48> 2015-09-07 15:13:02.529962 7f74cea0d800 3 journal journal_replay: applying op seq 4421 -47> 2015-09-07 15:13:02.529967 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4421 -46> 2015-09-07 15:13:02.529976 7f74cea0d800 2 journal read_entry 85852160 : seq 4422 259 bytes -45> 2015-09-07 15:13:02.529979 7f74cea0d800 3 journal journal_replay: applying op seq 4422 -44> 2015-09-07 15:13:02.530039 7f74cea0d800 -1 filestore(/var/lib/ceph/osd/ceph-0) could not find 2//head//3 in index: (2) No such file or directory -43> 2015-09-07 15:13:02.530070 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4422 -42> 2015-09-07 15:13:02.530082 7f74cea0d800 2 journal read_entry 85856256 : seq 4423 46 bytes -41> 2015-09-07 15:13:02.530085 7f74cea0d800 3 journal journal_replay: applying op seq 4423 -40> 2015-09-07 15:13:02.530088 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4423 -39> 2015-09-07 15:13:02.530097 7f74cea0d800 2 journal read_entry 85860352 : seq 4424 959 bytes -38> 2015-09-07 15:13:02.530100 7f74cea0d800 3 journal journal_replay: applying op seq 4424 -37> 2015-09-07 15:13:02.530135 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4424 -38> 2015-09-07 15:13:02.530100 7f74cea0d800 3 journal journal_replay: applying op seq 4424 -37> 2015-09-07 15:13:02.530135 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4424 -36> 2015-09-07 15:13:02.530152 7f74cea0d800 2 journal read_entry 85864448 : seq 4425 959 bytes -36> 2015-09-07 15:13:02.530152 7f74cea0d800 2 journal read_entry 85864448 : seq 4425 959 bytes -35> 2015-09-07 15:13:02.530155 7f74cea0d800 3 -35> 2015-09-07 15:13:02.530155 7f74cea0d800 3 journal journal_replay: applying op seq 4425 -34> 2015-09-07 15:13:02.530232 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4425journal journal_replay: r = 0, op_seq now 4425 -33> 2015-09-07 15:13:02.530245 7f74cea0d800 2 journal read_entry 85868544 : seq 4426 259 bytes 
journal read_entry 85868544 : seq 4426 259 bytes
-32> 2015-09-07 15:13:02.530251 7f74cea0d800 3 -32> 2015-09-07 15:13:02.530251 7f74cea0d800 3 journal journal_replay: applying op seq 4426
-31> 2015-09-07 15:13:02.530273 7f74cea0d800 -1 filestore(/var/lib/ceph/osd/ceph-0) could not find 1//head//3 in index: (2) No such file or directoryfilestore(/var/lib/ceph/osd/ceph-0) could not find 1//head//3 in index: (2) No such file or directory -30> 2015-09-07 15:13:02.530299 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4426 
journal journal_replay: r = 0, op_seq now 4426
-29> 2015-09-07 15:13:02.530310 7f74cea0d800 2 -29> 2015-09-07 15:13:02.530310 7f74cea0d800 2 journal read_entry 85872640 : seq 4427 46 bytes -28> 2015-09-07 15:13:02.530312 7f74cea0d800 3 journal journal_replay: applying op seq 4427 -27> 2015-09-07 15:13:02.530316 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4427 -27> 2015-09-07 15:13:02.530316 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4427 -26> 2015-09-07 15:13:02.530325 7f74cea0d800 2 journal read_entry 85876736 : seq 4428 259 bytes -26> 2015-09-07 15:13:02.530325 7f74cea0d800 2 journal read_entry 85876736 : seq 4428 259 bytes -25> 2015-09-07 15:13:02.530328 7f74cea0d800 3 journal journal_replay: applying op seq 4428 -25> 2015-09-07 15:13:02.530328 7f74cea0d800 3 journal journal_replay: applying op seq 4428 -24> 2015-09-07 15:13:02.530349 7f74cea0d800 -1 -24> 2015-09-07 15:13:02.530349 7f74cea0d800 -1 filestore(/var/lib/ceph/osd/ceph-0) could not find 0//head//3 in index: (2) No such file or directory 
0//head//3 in index: (2) No such file or directory
-23> 2015-09-07 15:13:02.530389 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4428 -22> 2015-09-07 15:13:02.530401 7f74cea0d800 2 journal read_entry 85880832 : seq 4429 46 bytes
-21> 2015-09-07 15:13:02.530404 7f74cea0d800 3 -21> 2015-09-07 15:13:02.530404 7f74cea0d800 3 journal journal_replay: applying op seq 4429
-20> 2015-09-07 15:13:02.530448 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4429journal journal_replay: r = 0, op_seq now 4429 -19> 2015-09-07 15:13:02.530458 7f74cea0d800 2 journal read_entry 85884928 : seq 4430 259 bytes 
journal read_entry 85884928 : seq 4430 259 bytes
-18> 2015-09-07 15:13:02.530465 7f74cea0d800 3 journal journal_replay: applying op seq 4430 -18> 2015-09-07 15:13:02.530465 7f74cea0d800 3 journal journal_replay: applying op seq 4430 -17> 2015-09-07 15:13:02.530487 7f74cea0d800 -1 filestore(/var/lib/ceph/osd/ceph-0) could not find 7//head//3 in index: (2) No such file or directory filestore(/var/lib/ceph/osd/ceph-0) could not find 7//head//3 in index: (2) No such file or directory -16> 2015-09-07 15:13:02.530514 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4430 -15> 2015-09-07 15:13:02.530525 7f74cea0d800 2 journal read_entry 85889024 : seq 4431 46 bytes -14> 2015-09-07 15:13:02.530528 7f74cea0d800 3 journal journal_replay: applying op seq 4431 -13> 2015-09-07 15:13:02.530532 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4431 -12> 2015-09-07 15:13:02.530541 7f74cea0d800 2 journal read_entry 85893120 : seq 4432 259 bytes -11> 2015-09-07 15:13:02.530549 7f74cea0d800 3 journal journal_replay: applying op seq 4432 -10> 2015-09-07 15:13:02.530569 7f74cea0d800 -1 filestore(/var/lib/ceph/osd/ceph-0) could not find 6//head//3 in index: (2) No such file or directory -9> 2015-09-07 15:13:02.530593 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4432 -8> 2015-09-07 15:13:02.530604 7f74cea0d800 2 journal read_entry 85897216 : seq 4433 46 bytes -7> 2015-09-07 15:13:02.530606 7f74cea0d800 3 journal journal_replay: applying op seq 4433 -6> 2015-09-07 15:13:02.530610 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4433 -5> 2015-09-07 15:13:02.530621 7f74cea0d800 2 journal read_entry 85901312 : seq 4434 259 bytes -4> 2015-09-07 15:13:02.530624 7f74cea0d800 3 journal journal_replay: applying op seq 4434 -3> 2015-09-07 15:13:02.530770 7f74cea0d800 0 filestore(/var/lib/ceph/osd/ceph-0) error (39) Directory not empty not handled on operation 0x55d8ef6 (4434.0.1, or op 1, counting from 0) -2> 2015-09-07 15:13:02.530798 7f74cea0d800 0 filestore(/var/lib/ceph/osd/ceph-0) ENOTEMPTY suggests garbage data in osd data dir -1> 2015-09-07 15:13:02.530801 7f74cea0d800 0 filestore(/var/lib/ceph/osd/ceph-0) transaction dump: 
{
    "ops": [
        {
            "op_num": 0,
            "op_name": "remove",
            "collection": "3.5_head",
            "oid": "5\/\/head\/\/3"
        },
        {
            "op_num": 1,
            "op_name": "rmcoll",
            "collection": "3.5_head"
        }
    ]
}
0> 2015-09-07 15:13:02.533771 7f74cea0d800 -1 os/FileStore.cc: In function 'unsigned int FileStore::_do_transaction(ObjectStore::Transaction&, uint64_t, int, ThreadPool::TPHandle*)' thread 7f74cea0d800 time 2015-09-07 15:13:02.530848 
os/FileStore.cc: 2757: FAILED assert(0 == "unexpected error")

ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3)
1: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int, ThreadPool::TPHandle*)+0xc16) [0x975a06] 2: (FileStore::_do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long, ThreadPool::TPHandle*)+0x64) [0x97d794] 
3: (JournalingObjectStore::journal_replay(unsigned long)+0x61e) [0xa2927e]
4: (FileStore::mount()+0x2d0e) [0x963d5e]
5: (OSD::init()+0x273) [0x6891f3]
6: (main()+0x384f) [0x62e10f]
7: (__libc_start_main()+0xfd) [0x3c70c1ed5d]
8: /usr/bin/ceph-osd() [0x629819]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. 

--- logging levels ---
   0/ 5 none
   0/ 1 lockdep
   0/ 1 context
   1/ 1 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 1 buffer
   0/ 1 timer
   0/ 1 filer
   0/ 1 striper
   0/ 1 objecter
   0/ 5 rados
   0/ 5 rbd
   0/ 5 rbd_replay
   0/ 5 journaler
   0/ 5 objectcacher
   0/ 5 client
0/ 5 osd -16> 2015-09-07 15:13:02.530514 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4430 -15> 2015-09-07 15:13:02.530525 7f74cea0d800 2 journal read_entry 85889024 : seq 4431 46 bytes -14> 2015-09-07 15:13:02.530528 7f74cea0d800 3 journal journal_replay: applying op seq 4431 -13> 2015-09-07 15:13:02.530532 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4431 -12> 2015-09-07 15:13:02.530541 7f74cea0d800 2 journal read_entry 85893120 : seq 4432 259 bytes -11> 2015-09-07 15:13:02.530549 7f74cea0d800 3 journal journal_replay: applying op seq 4432 -10> 2015-09-07 15:13:02.530569 7f74cea0d800 -1 filestore(/var/lib/ceph/osd/ceph-0) could not find 6//head//3 in index: (2) No such file or directory -9> 2015-09-07 15:13:02.530593 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4432 -8> 2015-09-07 15:13:02.530604 7f74cea0d800 2 journal read_entry 85897216 : seq 4433 46 bytes -7> 2015-09-07 15:13:02.530606 7f74cea0d800 3 journal journal_replay: applying op seq 4433 -6> 2015-09-07 15:13:02.530610 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4433 -5> 2015-09-07 15:13:02.530621 7f74cea0d800 2 journal read_entry 85901312 : seq 4434 259 bytes -4> 2015-09-07 15:13:02.530624 7f74cea0d800 3 journal journal_replay: applying op seq 4434 -3> 2015-09-07 15:13:02.530770 7f74cea0d800 0 filestore(/var/lib/ceph/osd/ceph-0) error (39) Directory not empty not handled on operation 0x55d8ef6 (4434.0.1, or op 1, counting from 0) -2> 2015-09-07 15:13:02.530798 7f74cea0d800 0 filestore(/var/lib/ceph/osd/ceph-0) ENOTEMPTY suggests garbage data in osd data dir -1> 2015-09-07 15:13:02.530801 7f74cea0d800 0 filestore(/var/lib/ceph/osd/ceph-0) transaction dump: 
{
    "ops": [
        {
            "op_num": 0,
            "op_name": "remove",
            "collection": "3.5_head",
            "oid": "5\/\/head\/\/3"
        },
        {
            "op_num": 1,
            "op_name": "rmcoll",
            "collection": "3.5_head"
        }
    ]
}
0> 2015-09-07 15:13:02.533771 7f74cea0d800 -1 os/FileStore.cc: In function 'unsigned int FileStore::_do_transaction(ObjectStore::Transaction&, uint64_t, int, ThreadPool::TPHandle*)' thread 7f74cea0d800 time 2015-09-07 15:13:02.530848 
os/FileStore.cc: 2757: FAILED assert(0 == "unexpected error")

ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3)
1: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int, ThreadPool::TPHandle*)+0xc16) [0x975a06] 2: (FileStore::_do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long, ThreadPool::TPHandle*)+0x64) [0x97d794] 
3: (JournalingObjectStore::journal_replay(unsigned long)+0x61e) [0xa2927e]
4: (FileStore::mount()+0x2d0e) [0x963d5e]
5: (OSD::init()+0x273) [0x6891f3]
6: (main()+0x384f) [0x62e10f]
7: (__libc_start_main()+0xfd) [0x3c70c1ed5d]
8: /usr/bin/ceph-osd() [0x629819]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. 

--- logging levels ---
   0/ 5 none
   0/ 1 lockdep
   0/ 1 context
   1/ 1 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 1 buffer
   0/ 1 timer
   0/ 1 filer
   0/ 1 striper
   0/ 1 objecter
   0/ 5 rados
   0/ 5 rbd
   0/ 5 rbd_replay
   0/ 5 journaler
   0/ 5 objectcacher
   0/ 5 client
   0/ 5 osd
   0/ 5 optracker
   0/ 5 objclass
   1/ 3 filestore
   1/ 3 keyvaluestore
   1/ 3 journal
   0/ 5 ms
   1/ 5 mon
   0/10 monc
   1/ 5 paxos
   0/ 5 tp
   1/ 5 auth
   1/ 5 crypto
   1/ 1 finisher
   1/ 5 heartbeatmap
   1/ 5 perfcounter
   1/ 5 rgw
   1/10 civetweb
   1/ 5 javaclient
   1/ 5 asok
   1/ 1 throttle
   0/ 0 refs
   1/ 5 xio
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
  max_recent     10000
  max_new         1000
  log_file /var/log/ceph/ceph-osd.0.log
--- end dump of recent events ---
   0/ 5 objclass
   1/ 3 filestore
   1/ 3 keyvaluestore
   1/ 3 journal
   0/ 5 ms
   1/ 5 mon
   0/10 monc
   1/ 5 paxos
   0/ 5 tp
   1/ 5 auth
   1/ 5 crypto
   1/ 1 finisher
   1/ 5 heartbeatmap
   1/ 5 perfcounter
   1/ 5 rgw
   1/10 civetweb
   1/ 5 javaclient
   1/ 5 asok
   1/ 1 throttle
   0/ 0 refs
   1/ 5 xio
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
  max_recent     10000
  max_new         1000
  log_file /var/log/ceph/ceph-osd.0.log
--- end dump of recent events ---
2015-09-07 15:13:02.540432 7f74cea0d800 -1 *** Caught signal (Aborted) **
in thread 7f74cea0d800

ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3)
1: /usr/bin/ceph-osd() [0xa476a5]
2: /lib64/libpthread.so.0() [0x3c7100f710]
3: (gsignal()+0x35) [0x3c70c32625]
4: (abort()+0x175) [0x3c70c33e05]
5: (__gnu_cxx::__verbose_terminate_handler()+0x12d) [0x39930bea7d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. 

--- begin dump of recent events ---
0> 2015-09-07 15:13:02.540432 7f74cea0d800 -1 *** Caught signal (Aborted) ** 
in thread 7f74cea0d800

ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3)
1: /usr/bin/ceph-osd() [0xa476a5]
2: /lib64/libpthread.so.0() [0x3c7100f710]
3: (gsignal()+0x35) [0x3c70c32625]
4: (abort()+0x175) [0x3c70c33e05]
5: (__gnu_cxx::__verbose_terminate_handler()+0x12d) [0x39930bea7d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. 

--- logging levels ---
   0/ 5 none
   0/ 1 lockdep
   0/ 1 context
   1/ 1 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 1 buffer
   0/ 1 timer
   0/ 1 filer
   0/ 1 striper
   0/ 1 objecter
   0/ 5 rados
   0/ 5 rbd
   0/ 5 rbd_replay
   0/ 5 journaler
   0/ 5 objectcacher
   0/ 5 client
   0/ 5 osd
   0/ 5 optracker
   0/ 5 objclass
   1/ 3 filestore
   1/ 3 keyvaluestore
   1/ 3 journal
   0/ 5 ms
   1/ 5 mon
   0/10 monc
   1/ 5 paxos
   0/ 5 tp
   1/ 5 auth
   1/ 5 crypto
   1/ 1 finisher
   1/ 5 heartbeatmap
   1/ 5 perfcounter
   1/ 5 rgw
   1/10 civetweb
   1/ 5 javaclient
   1/ 5 asok
   1/ 1 throttle
   0/ 0 refs
   1/ 5 xio
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
  max_recent     10000
  max_new         1000
  log_file /var/log/ceph/ceph-osd.0.log
--- end dump of recent events ---


I hope you could help me. Any advice is appreciated.

Thanks,
Aaron





_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Reply via email to