Re: [ceph-users] New deployment: errors starting OSDs: invalid (someone else's?) journal
On Wed, Mar 25, 2015 at 6:37 PM, Robert LeBlanc rob...@leblancnet.us wrote: As far as the foreign journal, I would run dd over the journal partition and try it again. It sounds like something didn't get cleaned up from a previous run. I wrote zeros on the journal device re-created the journal with ceph-osd --makejournal, and it seems that the latter command didn't write anything on the partition: starting osd.196 at :/0 osd_data /var/lib/ceph/osd/ceph-196 /var/lib/ceph/osd/ceph-196/journal SG_IO: bad/missing sense data, sb[]: 70 00 05 00 00 00 00 28 00 00 00 00 20 00 00 00 00 00 00 85 55 ac 01 00 00 00 00 00 00 00 20 00 SG_IO: bad/missing sense data, sb[]: 70 00 05 00 00 00 00 28 00 00 00 00 20 00 00 00 00 00 00 85 55 ac 01 00 00 00 00 00 00 00 20 00 HDIO_DRIVE_CMD(identify) failed: Input/output error 2015-04-07 14:07:53.703247 7f0e32433900 -1 journal FileJournal::open: ondisk fsid ---- doesn't match expected 35cd523e-4a74-41ae-908a-f25267a94dac, invalid (someone else's?) journal 2015-04-07 14:07:53.703569 7f0e32433900 -1 filestore(/var/lib/ceph/osd/ceph-196) mount failed to open journal /var/lib/ceph/osd/ceph-196/journal: (22) Invalid argument 2015-04-07 14:07:53.703956 7f0e32433900 -1 ** ERROR: error converting store /var/lib/ceph/osd/ceph-196: (22) Invalid argument Note the fsid 0... line. I've also tried to zap and re-create with ceph-deploy, same results. Maybe it's not writing to the journal file because of the bad missing data error? It's strange thought, that error seems to come from hdparm -W not working on those disks, but I have the same error on *all* of my disks... Any other idea? .a. -- antonio.s.mess...@gmail.com antonio.mess...@uzh.ch +41 (0)44 635 42 22 S3IT: Service and Support for Science IT http://www.s3it.uzh.ch/ University of Zurich Winterthurerstrasse 190 CH-8057 Zurich Switzerland ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] New deployment: errors starting OSDs: invalid (someone else's?) journal
I don't know much about ceph-deploy, but I know that ceph-disk has problems automatically adding an SSD OSD when there are journals of other disks already on it. I've had to partition the disk ahead of time and pass in the partitions to make ceph-disk work. Also, unless you are sure that the dev devices will be deterministicly named the same each time, I'd recommend you not use /dev/sd* for pointing to your journals. Instead use something that will always be the same, since Ceph with partition the disks with GPT, you can use the partuuid to point to the journal partition and it will always be right. A while back I used this to fix my journal links when I did it wrong. You will want to double check that it will work right for you. no warranty and all that jazz... #convert the /dev/sd* links for journals into UUIDs for lnk in $(ls /var/lib/ceph/osd/); do OSD=/var/lib/ceph/osd/$lnk; DEV=$(readlink $OSD/journal | cut -d'/' -f3); echo $DEV; PUUID=$(ls -lh /dev/disk/by-partuuid/ | grep $DEV | cut -d' ' -f 9); ln -sf /dev/disk/by-partuuid/$PUUID $OSD/journal; done On Wed, Mar 25, 2015 at 10:46 AM, Antonio Messina antonio.s.mess...@gmail.com wrote: Hi all, I'm trying to install ceph on a 7-nodes preproduction cluster. Each node has 24x 4TB SAS disks (2x dell md1400 enclosures) and 6x 800GB SSDs (for cache tiering, not journals). I'm using Ubuntu 14.04 and ceph-deploy to install the cluster, I've been trying both Firefly and Giant and getting the same error. However, the logs I'm reporting are relative to the Firefly installation. The installation seems to go fine until I try to install the last 2 OSDs (they are SSD disks) of each host. All the OSDs from 0 to 195 are UP and IN, but when I try to deploy the next OSD (no matter what host) ceph-osd daemon won't start. The error I get is: 2015-03-25 17:00:17.130937 7fe231312800 0 ceph version 0.80.9 (b5a67f0e1d15385bc0d60a6da6e7fc810bde6047), process ceph-osd, pid 20280 2015-03-25 17:00:17.133601 7fe231312800 10 filestore(/var/lib/ceph/osd/ceph-196) dump_stop 2015-03-25 17:00:17.133694 7fe231312800 5 filestore(/var/lib/ceph/osd/ceph-196) basedir /var/lib/ceph/osd/ceph-196 journal /var/lib/ceph/osd/ceph-196/journal 2015-03-25 17:00:17.133725 7fe231312800 10 filestore(/var/lib/ceph/osd/ceph-196) mount fsid is 8c2fa707-750a-4773-8918-a368367d9cf5 2015-03-25 17:00:17.133789 7fe231312800 0 filestore(/var/lib/ceph/osd/ceph-196) mount detected xfs (libxfs) 2015-03-25 17:00:17.133810 7fe231312800 1 filestore(/var/lib/ceph/osd/ceph-196) disabling 'filestore replica fadvise' due to known issues with fadvise(DONTNEED) on xfs 2015-03-25 17:00:17.135882 7fe231312800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-196) detect_features: FIEMAP ioctl is supported and appears to work 2015-03-25 17:00:17.135892 7fe231312800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-196) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-03-25 17:00:17.136318 7fe231312800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-196) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-03-25 17:00:17.136373 7fe231312800 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-196) detect_feature: extsize is disabled by conf 2015-03-25 17:00:17.136640 7fe231312800 5 filestore(/var/lib/ceph/osd/ceph-196) mount op_seq is 1 2015-03-25 17:00:17.137547 7fe231312800 20 filestore (init)dbobjectmap: seq is 1 2015-03-25 17:00:17.137560 7fe231312800 10 filestore(/var/lib/ceph/osd/ceph-196) open_journal at /var/lib/ceph/osd/ceph-196/journal 2015-03-25 17:00:17.137575 7fe231312800 0 filestore(/var/lib/ceph/osd/ceph-196) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled 2015-03-25 17:00:17.137580 7fe231312800 10 filestore(/var/lib/ceph/osd/ceph-196) list_collections 2015-03-25 17:00:17.137661 7fe231312800 10 journal journal_replay fs op_seq 1 2015-03-25 17:00:17.137668 7fe231312800 2 journal open /var/lib/ceph/osd/ceph-196/journal fsid 8c2fa707-750a-4773-8918-a368367d9cf5 fs_op_seq 1 2015-03-25 17:00:17.137670 7fe22b8b1700 20 filestore(/var/lib/ceph/osd/ceph-196) sync_entry waiting for max_interval 5.00 2015-03-25 17:00:17.137690 7fe231312800 10 journal _open_block_device: ignoring osd journal size. We'll use the entire block device (size: 5367661056) 2015-03-25 17:00:17.162489 7fe231312800 1 journal _open /var/lib/ceph/osd/ceph-196/journal fd 20: 5367660544 bytes, block size 4096 bytes, directio = 1, aio = 1 2015-03-25 17:00:17.162502 7fe231312800 10 journal read_header 2015-03-25 17:00:17.172249 7fe231312800 10 journal header: block_size 4096 alignment 4096 max_size 5367660544 2015-03-25 17:00:17.172256 7fe231312800 10 journal header: start 50987008 2015-03-25 17:00:17.172257 7fe231312800 10 journal write_pos 4096 2015-03-25 17:00:17.172259 7fe231312800 10 journal open header.fsid = 942f2d62-dd99-42a8-878a-feea443aaa61 2015-03-25 17:00:17.172264
Re: [ceph-users] New deployment: errors starting OSDs: invalid (someone else's?) journal
On Wed, Mar 25, 2015 at 6:06 PM, Robert LeBlanc rob...@leblancnet.us wrote: I don't know much about ceph-deploy, but I know that ceph-disk has problems automatically adding an SSD OSD when there are journals of other disks already on it. I've had to partition the disk ahead of time and pass in the partitions to make ceph-disk work. This is not my case: the journal is created automatically by ceph-deploy on the same disk, so that for each disk, /dev/sdX1 is the data partition and /dev/sdX2 is the journal partition. This is also what I want: I know there is a performance drop, but I expect it to be mitigated by the cache tier. (and I plan to test both configuration anyway) Also, unless you are sure that the dev devices will be deterministicly named the same each time, I'd recommend you not use /dev/sd* for pointing to your journals. Instead use something that will always be the same, since Ceph with partition the disks with GPT, you can use the partuuid to point to the journal partition and it will always be right. A while back I used this to fix my journal links when I did it wrong. You will want to double check that it will work right for you. no warranty and all that jazz... Thank you for pointing this out, it's an important point. However, the links are actually created using the partuuid. The command I posted in my previous email included the output of a pair of nested readlink in order to get the /dev/sd* names, because in this way it's easier to see if there are duplicates and where :) The output of ls -l /var/lib/ceph/osd/ceph-*/journal is actually: lrwxrwxrwx 1 root root 58 Mar 25 11:38 /var/lib/ceph/osd/ceph-0/journal - /dev/disk/by-partuuid/18305316-96b0-4654-aaad-7aeb891429f6 lrwxrwxrwx 1 root root 58 Mar 25 11:49 /var/lib/ceph/osd/ceph-7/journal - /dev/disk/by-partuuid/a263b19a-cb0d-4b4c-bd81-314619d5755d lrwxrwxrwx 1 root root 58 Mar 25 12:21 /var/lib/ceph/osd/ceph-14/journal - /dev/disk/by-partuuid/79734e0e-87dd-40c7-ba83-0d49695a75fb lrwxrwxrwx 1 root root 58 Mar 25 12:31 /var/lib/ceph/osd/ceph-21/journal - /dev/disk/by-partuuid/73a504bc-3179-43fd-942c-13c6bd8633c5 lrwxrwxrwx 1 root root 58 Mar 25 12:42 /var/lib/ceph/osd/ceph-28/journal - /dev/disk/by-partuuid/ecff10df-d757-4b1f-bef4-88dd84d84ef1 lrwxrwxrwx 1 root root 58 Mar 25 12:52 /var/lib/ceph/osd/ceph-35/journal - /dev/disk/by-partuuid/5be30238-3f07-4950-b39f-f5e4c7305e4c lrwxrwxrwx 1 root root 58 Mar 25 13:02 /var/lib/ceph/osd/ceph-42/journal - /dev/disk/by-partuuid/3cdb65f2-474c-47fb-8d07-83e7518418ff lrwxrwxrwx 1 root root 58 Mar 25 13:12 /var/lib/ceph/osd/ceph-49/journal - /dev/disk/by-partuuid/a47fe2b7-e375-4eea-b7a9-0354a24548dc lrwxrwxrwx 1 root root 58 Mar 25 13:22 /var/lib/ceph/osd/ceph-56/journal - /dev/disk/by-partuuid/fb42b7d6-bc6c-4063-8b73-29beb1f65107 lrwxrwxrwx 1 root root 58 Mar 25 13:33 /var/lib/ceph/osd/ceph-63/journal - /dev/disk/by-partuuid/72aff32b-ca56-4c25-b8ea-ff3aba8db507 lrwxrwxrwx 1 root root 58 Mar 25 13:43 /var/lib/ceph/osd/ceph-70/journal - /dev/disk/by-partuuid/b7c17a75-47cd-401e-b963-afe910612bd6 lrwxrwxrwx 1 root root 58 Mar 25 13:53 /var/lib/ceph/osd/ceph-77/journal - /dev/disk/by-partuuid/2c1c2501-fa82-4fc9-a586-03cc4d68faef lrwxrwxrwx 1 root root 58 Mar 25 14:03 /var/lib/ceph/osd/ceph-84/journal - /dev/disk/by-partuuid/46f619a5-3edf-44e9-99a6-24d98bcd174a lrwxrwxrwx 1 root root 58 Mar 25 14:13 /var/lib/ceph/osd/ceph-91/journal - /dev/disk/by-partuuid/5feef832-dd82-4aa0-9264-dc9496a3f93a lrwxrwxrwx 1 root root 58 Mar 25 14:24 /var/lib/ceph/osd/ceph-98/journal - /dev/disk/by-partuuid/055793a0-99d4-49c4-9698-bd8880c21d9c lrwxrwxrwx 1 root root 58 Mar 25 14:34 /var/lib/ceph/osd/ceph-105/journal - /dev/disk/by-partuuid/20547f26-6ef3-422b-9732-ad8b0b5b5379 lrwxrwxrwx 1 root root 58 Mar 25 14:44 /var/lib/ceph/osd/ceph-112/journal - /dev/disk/by-partuuid/2abea809-59c4-41da-bb52-28ef1911ec43 lrwxrwxrwx 1 root root 58 Mar 25 14:54 /var/lib/ceph/osd/ceph-119/journal - /dev/disk/by-partuuid/d8d15bb8-4b3d-4375-b6e1-62794971df7e lrwxrwxrwx 1 root root 58 Mar 25 15:05 /var/lib/ceph/osd/ceph-126/journal - /dev/disk/by-partuuid/ff6ee2b2-9c33-4902-a5e3-f6e9db5714e9 lrwxrwxrwx 1 root root 58 Mar 25 15:15 /var/lib/ceph/osd/ceph-133/journal - /dev/disk/by-partuuid/9faccb6e-ada9-4742-aa31-eb1308769205 lrwxrwxrwx 1 root root 58 Mar 25 15:25 /var/lib/ceph/osd/ceph-140/journal - /dev/disk/by-partuuid/2df13c88-ee58-4881-a373-a36a09fb6366 lrwxrwxrwx 1 root root 58 Mar 25 15:36 /var/lib/ceph/osd/ceph-147/journal - /dev/disk/by-partuuid/13cda9d1-0fec-40cc-a6fc-7cc56f7ffb78 lrwxrwxrwx 1 root root 58 Mar 25 15:46 /var/lib/ceph/osd/ceph-154/journal - /dev/disk/by-partuuid/5d37bfe9-c0f9-49e0-a951-b0ed04c5de51 lrwxrwxrwx 1 root root 58 Mar 25 15:57 /var/lib/ceph/osd/ceph-161/journal - /dev/disk/by-partuuid/d34f3abb-3fb7-4875-90d3-d2d3836f6e4d lrwxrwxrwx 1 root root 58 Mar 25 16:07 /var/lib/ceph/osd/ceph-168/journal - /dev/disk/by-partuuid/02c3db3e-159c-47d9-8a63-0389ea89fad1 lrwxrwxrwx 1 root root 58 Mar 25 16:16
Re: [ceph-users] New deployment: errors starting OSDs: invalid (someone else's?) journal
Probably a case of trying to read too fast. Sorry about that. As far as your theory on the cache pool, I haven't tried that, but my gut feeling is that it won't help as much as having the journal on the SSD. The Cache tier isn't trying to collate writes, not like the journal is doing. Then on the spindle you are having to write to two very different parts of the drive for every piece of data, although this is somewhat reduced by the journal, I feel it will still be significant. When I see writes coming off my SSD journals to the spindles, I'm still getting a lot of merged IO (at least during a backfill/recovery). I'm interested in your results. As far as the foreign journal, I would run dd over the journal partition and try it again. It sounds like something didn't get cleaned up from a previous run. On Wed, Mar 25, 2015 at 11:14 AM, Antonio Messina antonio.s.mess...@gmail.com wrote: On Wed, Mar 25, 2015 at 6:06 PM, Robert LeBlanc rob...@leblancnet.us wrote: I don't know much about ceph-deploy, but I know that ceph-disk has problems automatically adding an SSD OSD when there are journals of other disks already on it. I've had to partition the disk ahead of time and pass in the partitions to make ceph-disk work. This is not my case: the journal is created automatically by ceph-deploy on the same disk, so that for each disk, /dev/sdX1 is the data partition and /dev/sdX2 is the journal partition. This is also what I want: I know there is a performance drop, but I expect it to be mitigated by the cache tier. (and I plan to test both configuration anyway) Also, unless you are sure that the dev devices will be deterministicly named the same each time, I'd recommend you not use /dev/sd* for pointing to your journals. Instead use something that will always be the same, since Ceph with partition the disks with GPT, you can use the partuuid to point to the journal partition and it will always be right. A while back I used this to fix my journal links when I did it wrong. You will want to double check that it will work right for you. no warranty and all that jazz... Thank you for pointing this out, it's an important point. However, the links are actually created using the partuuid. The command I posted in my previous email included the output of a pair of nested readlink in order to get the /dev/sd* names, because in this way it's easier to see if there are duplicates and where :) The output of ls -l /var/lib/ceph/osd/ceph-*/journal is actually: lrwxrwxrwx 1 root root 58 Mar 25 11:38 /var/lib/ceph/osd/ceph-0/journal - /dev/disk/by-partuuid/18305316-96b0-4654-aaad-7aeb891429f6 lrwxrwxrwx 1 root root 58 Mar 25 11:49 /var/lib/ceph/osd/ceph-7/journal - /dev/disk/by-partuuid/a263b19a-cb0d-4b4c-bd81-314619d5755d lrwxrwxrwx 1 root root 58 Mar 25 12:21 /var/lib/ceph/osd/ceph-14/journal - /dev/disk/by-partuuid/79734e0e-87dd-40c7-ba83-0d49695a75fb lrwxrwxrwx 1 root root 58 Mar 25 12:31 /var/lib/ceph/osd/ceph-21/journal - /dev/disk/by-partuuid/73a504bc-3179-43fd-942c-13c6bd8633c5 lrwxrwxrwx 1 root root 58 Mar 25 12:42 /var/lib/ceph/osd/ceph-28/journal - /dev/disk/by-partuuid/ecff10df-d757-4b1f-bef4-88dd84d84ef1 lrwxrwxrwx 1 root root 58 Mar 25 12:52 /var/lib/ceph/osd/ceph-35/journal - /dev/disk/by-partuuid/5be30238-3f07-4950-b39f-f5e4c7305e4c lrwxrwxrwx 1 root root 58 Mar 25 13:02 /var/lib/ceph/osd/ceph-42/journal - /dev/disk/by-partuuid/3cdb65f2-474c-47fb-8d07-83e7518418ff lrwxrwxrwx 1 root root 58 Mar 25 13:12 /var/lib/ceph/osd/ceph-49/journal - /dev/disk/by-partuuid/a47fe2b7-e375-4eea-b7a9-0354a24548dc lrwxrwxrwx 1 root root 58 Mar 25 13:22 /var/lib/ceph/osd/ceph-56/journal - /dev/disk/by-partuuid/fb42b7d6-bc6c-4063-8b73-29beb1f65107 lrwxrwxrwx 1 root root 58 Mar 25 13:33 /var/lib/ceph/osd/ceph-63/journal - /dev/disk/by-partuuid/72aff32b-ca56-4c25-b8ea-ff3aba8db507 lrwxrwxrwx 1 root root 58 Mar 25 13:43 /var/lib/ceph/osd/ceph-70/journal - /dev/disk/by-partuuid/b7c17a75-47cd-401e-b963-afe910612bd6 lrwxrwxrwx 1 root root 58 Mar 25 13:53 /var/lib/ceph/osd/ceph-77/journal - /dev/disk/by-partuuid/2c1c2501-fa82-4fc9-a586-03cc4d68faef lrwxrwxrwx 1 root root 58 Mar 25 14:03 /var/lib/ceph/osd/ceph-84/journal - /dev/disk/by-partuuid/46f619a5-3edf-44e9-99a6-24d98bcd174a lrwxrwxrwx 1 root root 58 Mar 25 14:13 /var/lib/ceph/osd/ceph-91/journal - /dev/disk/by-partuuid/5feef832-dd82-4aa0-9264-dc9496a3f93a lrwxrwxrwx 1 root root 58 Mar 25 14:24 /var/lib/ceph/osd/ceph-98/journal - /dev/disk/by-partuuid/055793a0-99d4-49c4-9698-bd8880c21d9c lrwxrwxrwx 1 root root 58 Mar 25 14:34 /var/lib/ceph/osd/ceph-105/journal - /dev/disk/by-partuuid/20547f26-6ef3-422b-9732-ad8b0b5b5379 lrwxrwxrwx 1 root root 58 Mar 25 14:44 /var/lib/ceph/osd/ceph-112/journal - /dev/disk/by-partuuid/2abea809-59c4-41da-bb52-28ef1911ec43 lrwxrwxrwx 1 root root 58 Mar 25 14:54 /var/lib/ceph/osd/ceph-119/journal - /dev/disk/by-partuuid/d8d15bb8-4b3d-4375-b6e1-62794971df7e lrwxrwxrwx 1
[ceph-users] New deployment: errors starting OSDs: invalid (someone else's?) journal
Hi all, I'm trying to install ceph on a 7-nodes preproduction cluster. Each node has 24x 4TB SAS disks (2x dell md1400 enclosures) and 6x 800GB SSDs (for cache tiering, not journals). I'm using Ubuntu 14.04 and ceph-deploy to install the cluster, I've been trying both Firefly and Giant and getting the same error. However, the logs I'm reporting are relative to the Firefly installation. The installation seems to go fine until I try to install the last 2 OSDs (they are SSD disks) of each host. All the OSDs from 0 to 195 are UP and IN, but when I try to deploy the next OSD (no matter what host) ceph-osd daemon won't start. The error I get is: 2015-03-25 17:00:17.130937 7fe231312800 0 ceph version 0.80.9 (b5a67f0e1d15385bc0d60a6da6e7fc810bde6047), process ceph-osd, pid 20280 2015-03-25 17:00:17.133601 7fe231312800 10 filestore(/var/lib/ceph/osd/ceph-196) dump_stop 2015-03-25 17:00:17.133694 7fe231312800 5 filestore(/var/lib/ceph/osd/ceph-196) basedir /var/lib/ceph/osd/ceph-196 journal /var/lib/ceph/osd/ceph-196/journal 2015-03-25 17:00:17.133725 7fe231312800 10 filestore(/var/lib/ceph/osd/ceph-196) mount fsid is 8c2fa707-750a-4773-8918-a368367d9cf5 2015-03-25 17:00:17.133789 7fe231312800 0 filestore(/var/lib/ceph/osd/ceph-196) mount detected xfs (libxfs) 2015-03-25 17:00:17.133810 7fe231312800 1 filestore(/var/lib/ceph/osd/ceph-196) disabling 'filestore replica fadvise' due to known issues with fadvise(DONTNEED) on xfs 2015-03-25 17:00:17.135882 7fe231312800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-196) detect_features: FIEMAP ioctl is supported and appears to work 2015-03-25 17:00:17.135892 7fe231312800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-196) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-03-25 17:00:17.136318 7fe231312800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-196) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-03-25 17:00:17.136373 7fe231312800 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-196) detect_feature: extsize is disabled by conf 2015-03-25 17:00:17.136640 7fe231312800 5 filestore(/var/lib/ceph/osd/ceph-196) mount op_seq is 1 2015-03-25 17:00:17.137547 7fe231312800 20 filestore (init)dbobjectmap: seq is 1 2015-03-25 17:00:17.137560 7fe231312800 10 filestore(/var/lib/ceph/osd/ceph-196) open_journal at /var/lib/ceph/osd/ceph-196/journal 2015-03-25 17:00:17.137575 7fe231312800 0 filestore(/var/lib/ceph/osd/ceph-196) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled 2015-03-25 17:00:17.137580 7fe231312800 10 filestore(/var/lib/ceph/osd/ceph-196) list_collections 2015-03-25 17:00:17.137661 7fe231312800 10 journal journal_replay fs op_seq 1 2015-03-25 17:00:17.137668 7fe231312800 2 journal open /var/lib/ceph/osd/ceph-196/journal fsid 8c2fa707-750a-4773-8918-a368367d9cf5 fs_op_seq 1 2015-03-25 17:00:17.137670 7fe22b8b1700 20 filestore(/var/lib/ceph/osd/ceph-196) sync_entry waiting for max_interval 5.00 2015-03-25 17:00:17.137690 7fe231312800 10 journal _open_block_device: ignoring osd journal size. We'll use the entire block device (size: 5367661056) 2015-03-25 17:00:17.162489 7fe231312800 1 journal _open /var/lib/ceph/osd/ceph-196/journal fd 20: 5367660544 bytes, block size 4096 bytes, directio = 1, aio = 1 2015-03-25 17:00:17.162502 7fe231312800 10 journal read_header 2015-03-25 17:00:17.172249 7fe231312800 10 journal header: block_size 4096 alignment 4096 max_size 5367660544 2015-03-25 17:00:17.172256 7fe231312800 10 journal header: start 50987008 2015-03-25 17:00:17.172257 7fe231312800 10 journal write_pos 4096 2015-03-25 17:00:17.172259 7fe231312800 10 journal open header.fsid = 942f2d62-dd99-42a8-878a-feea443aaa61 2015-03-25 17:00:17.172264 7fe231312800 -1 journal FileJournal::open: ondisk fsid 942f2d62-dd99-42a8-878a-feea443aaa61 doesn't match expected 8c2fa707-750a-4773-8918-a368367d9cf5, invalid (someone else's?) journal 2015-03-25 17:00:17.172268 7fe231312800 3 journal journal_replay open failed with (22) Invalid argument 2015-03-25 17:00:17.172284 7fe231312800 -1 filestore(/var/lib/ceph/osd/ceph-196) mount failed to open journal /var/lib/ceph/osd/ceph-196/journal: (22) Invalid argument 2015-03-25 17:00:17.172304 7fe22b8b1700 20 filestore(/var/lib/ceph/osd/ceph-196) sync_entry woke after 0.034632 2015-03-25 17:00:17.172330 7fe22b8b1700 10 journal commit_start max_applied_seq 1, open_ops 0 2015-03-25 17:00:17.172333 7fe22b8b1700 10 journal commit_start blocked, all open_ops have completed 2015-03-25 17:00:17.172334 7fe22b8b1700 10 journal commit_start nothing to do 2015-03-25 17:00:17.172465 7fe231312800 -1 ** ERROR: error converting store /var/lib/ceph/osd/ceph-196: (22) Invalid argument I'm attaching the full log of ceph-deploy osd create osd-l2-05:sde and the /var/log/ceph/ceph-osd.196.log, after trying to re-start the osd with increased verbosing, as long as the ceph.conf I'm using. I've also checked if the journal symlinks were correct, and they all
