Hi Sam, In the 70.459 logs from the deep-scrub, there is an error:
$ zgrep "= \-2$" ceph-osd.307.log.1.gz 2016-03-07 16:11:41.828332 7ff30cdad700 10 filestore(/var/lib/ceph/osd/ceph-307) remove 70.459s0_head/79ced459/default.724733.17__shadow_prostate/rnaseq/8e5da6e8-8881-4813-a4e3-327df57fd1b7/UNCID_2409283.304a95c1-2180-4a81-a85a-880427e97d67.140304_UNC14-SN744_0400_AC3LWGACXX_7_GAGTGG.tar.gz.2~_1r_FGidmpEP8GRsJkNLfAh9CokxkLf.4_156/head//70/202909/0 = -2 2016-03-07 21:44:02.197676 7fe96b56f700 10 filestore(/var/lib/ceph/osd/ceph-307) remove 70.459s0_head/79ced459/default.724733.17__shadow_prostate/rnaseq/8e5da6e8-8881-4813-a4e3-327df57fd1b7/UNCID_2409283.304a95c1-2180-4a81-a85a-880427e97d67.140304_UNC14-SN744_0400_AC3LWGACXX_7_GAGTGG.tar.gz.2~_1r_FGidmpEP8GRsJkNLfAh9CokxkLf.4_156/head//70/202909/0 = -2 I'm taking this as an indication of the error you mentioned. It looks to me as if this bug leaves two files with "issues" based upon what I see on the filesystem. First, I have a size-0 file in a directory where I expect only to have directories: root@ceph03:/var/lib/ceph/osd/ceph-307/current/70.459s0_head/DIR_9/DIR_5/DIR_4/DIR_D# ls -ltr total 320 -rw-r--r-- 1 root root 0 Jan 23 21:49 default.724733.17\u\ushadow\uprostate\srnaseq\s8e5da6e8-8881-4813-a4e3-327df57fd1b7\sUNCID\u2409283.304a95c1-2180-4a81-a85a-880427e97d67.140304\uUNC14-SN744\u0400\uAC3LWGACXX\u7\uGAGTGG.tar.gz.2~\u1r\uFGidmpEP8GRsJkNLfAh9CokxkL_fa202ec9b4b3b217275a_0_long drwxr-xr-x 2 root root 16384 Feb 5 15:13 DIR_6 drwxr-xr-x 2 root root 16384 Feb 5 17:26 DIR_3 drwxr-xr-x 2 root root 16384 Feb 10 00:01 DIR_C drwxr-xr-x 2 root root 16384 Mar 4 10:50 DIR_7 drwxr-xr-x 2 root root 16384 Mar 4 16:46 DIR_A drwxr-xr-x 2 root root 16384 Mar 5 02:37 DIR_2 drwxr-xr-x 2 root root 16384 Mar 5 17:39 DIR_4 drwxr-xr-x 2 root root 16384 Mar 8 16:50 DIR_F drwxr-xr-x 2 root root 16384 Mar 15 15:51 DIR_8 drwxr-xr-x 2 root root 16384 Mar 15 21:18 DIR_D drwxr-xr-x 2 root root 16384 Mar 15 22:25 DIR_0 drwxr-xr-x 2 root root 16384 Mar 15 22:35 DIR_9 drwxr-xr-x 2 root root 16384 Mar 15 22:56 DIR_E drwxr-xr-x 2 root root 16384 Mar 15 23:21 DIR_1 drwxr-xr-x 2 root root 12288 Mar 16 00:07 DIR_B drwxr-xr-x 2 root root 16384 Mar 16 00:34 DIR_5 I assume that this file is an issue as well......and needs to be removed. then, in the directory where the file should be, I have the same file: root@ceph03:/var/lib/ceph/osd/ceph-307/current/70.459s0_head/DIR_9/DIR_5/DIR_4/DIR_D/DIR_E# ls -ltr | grep -v __head_ total 64840 -rw-r--r-- 1 root root 1048576 Jan 23 21:49 default.724733.17\u\ushadow\uprostate\srnaseq\s8e5da6e8-8881-4813-a4e3-327df57fd1b7\sUNCID\u2409283.304a95c1-2180-4a81-a85a-880427e97d67.140304\uUNC14-SN744\u0400\uAC3LWGACXX\u7\uGAGTGG.tar.gz.2~\u1r\uFGidmpEP8GRsJkNLfAh9CokxkL_fa202ec9b4b3b217275a_0_long In the directory DIR_E here (from above), there is only one file without a __head_ in the pathname -- the file above....Should I be deleting both these _long files without the __head_ in DIR_E and in one above .../DIR_E? Since there is no directory structure HASH in these files, is that the indication that it is an orphan? Thanks, Jeff On Tue, Mar 15, 2016 at 8:38 PM, Samuel Just <[email protected]> wrote: > Ah, actually, I think there will be duplicates only around half the > time -- either the old link or the new link could be orphaned > depending on which xfs decides to list first. Only if the old link is > orphaned will it match the name of the object once it's recreated. I > should be able to find time to put together a branch in the next week > or two if you want to wait. It's still probably worth trying removing > that object in 70.459. > -Sam > > On Tue, Mar 15, 2016 at 6:03 PM, Samuel Just <[email protected]> wrote: > > The bug is entirely independent of hardware issues -- entirely a ceph > > bug. xfs doesn't let us specify an ordering when reading a directory, > > so we have to keep directory sizes small. That means that when one of > > those pg collection subfolders has 320 files in it, we split it into > > up to 16 smaller directories. Overwriting or removing an ec object > > requires us to rename the old version out of the way in case we need > > to roll back (that's the generation number I mentioned above). For > > crash safety, this involves first creating a link to the new name, > > then removing the old one. Both the old and new link will be in the > > same subdirectory. If creating the new link pushes the directory to > > 320 files then we do a split while both links are present. If the > > file in question is using the special long filename handling, then a > > bug in the resulting link juggling causes us to orphan the old version > > of the file. Your cluster seems to have an unusual number of objects > > with very long names, which is why it is so visible on your cluster. > > > > There are critical pool sizes where the PGs will all be close to one > > of those limits. It's possible you are not close to one of those > > limits. It's also possible you are nearing one now. In any case, the > > remapping gave the orphaned files an opportunity to cause trouble, but > > they don't appear due to remapping. > > -Sam > > > > On Tue, Mar 15, 2016 at 5:41 PM, Jeffrey McDonald <[email protected]> > wrote: > >> One more question.....did we hit the bug because we had hardware issues > >> during the remapping or would it have happened regardless of the > hardware > >> issues? e.g. I'm not planning to add any additional hardware soon, but > >> would the bug pop again on an (unpatched) system not subject to any > >> remapping? > >> > >> thanks, > >> jeff > >> > >> On Tue, Mar 15, 2016 at 7:27 PM, Samuel Just <[email protected]> wrote: > >>> > >>> [back on list] > >>> > >>> ceph-objectstore-tool has a whole bunch of machinery for modifying an > >>> offline objectstore. It would be the easiest place to put it -- you > >>> could add a > >>> > >>> ceph-objectstore-tool --op filestore-repair-orphan-links [--dry-run] > ... > >>> > >>> command which would mount the filestore in a special mode and iterate > >>> over all collections and repair them. If you want to go that route, > >>> we'd be happy to help you get it written. Once it fixes your cluster, > >>> we'd then be able to merge and backport it in case anyone else hits > >>> it. > >>> > >>> You'd probably be fine doing it while the OSD is live...but as a rule > >>> I usually prefer to do my osd surgery offline. Journal doesn't matter > >>> here, the orphaned files are basically invisible to the filestore > >>> (except when doing a collection scan for scrub) since they are in the > >>> wrong directory. > >>> > >>> I don't think the orphans are necessarily going to be 0 size. There > >>> might be quirk of how radosgw creates these objects that always causes > >>> them to be created 0 size than then overwritten with a writefull -- if > >>> that's true it might be the case that you would only see 0 size ones. > >>> -Sam > >>> > >>> On Tue, Mar 15, 2016 at 4:02 PM, Jeffrey McDonald <[email protected]> > >>> wrote: > >>> > Thanks, I can try to write a tool to do this. Does > >>> > ceph-objectstore-tool > >>> > provide a framework? > >>> > > >>> > Can I safely delete the files while the OSD is alive or should I > take it > >>> > offline? Any concerns about the journal? > >>> > > >>> > Are there any other properties of the orphans, e.g. will the orphans > >>> > always > >>> > be size 0? > >>> > > >>> > Thanks! > >>> > Jeff > >>> > > >>> > On Tue, Mar 15, 2016 at 5:35 PM, Samuel Just <[email protected]> > wrote: > >>> >> > >>> >> Ok, a branch merged to master which should fix this > >>> >> (https://github.com/ceph/ceph/pull/8136). It'll be backported in > due > >>> >> course. The problem is that that patch won't clean orphaned files > >>> >> that already exist. > >>> >> > >>> >> Let me explain a bit about what the orphaned files look like. The > >>> >> problem is files with object names that result in escaped filenames > >>> >> longer than the max filename ceph will create (~250 iirc). > Normally, > >>> >> the name of the file is an escaped and sanitized version of the > object > >>> >> name: > >>> >> > >>> >> > >>> >> > >>> >> > ./DIR_9/DIR_5/DIR_4/DIR_D/DIR_C/default.325674.107\u\ushadow\u.KePEE8heghHVnlb1\uEIupG0I5eROwRn\u77__head_C1DCD459__46_ffffffffffffffff_0 > >>> >> > >>> >> corresponds to an object like > >>> >> > >>> >> > >>> >> > >>> >> > c1dcd459/default.325674.107__shadow_.KePEE8heghHVnlb1_EIupG0I5eROwRn_77/head//70 > >>> >> > >>> >> the DIR_9/DIR_5/DIR_4/DIR_D/DIR_C/ path is derived from the hash > >>> >> starting with the last value: cd459 -> > DIR_9/DIR_5/DIR_4/DIR_D/DIR_C/ > >>> >> > >>> >> It ends up in DIR_9/DIR_5/DIR_4/DIR_D/DIR_C/ because that's the > >>> >> longest path that exists (DIR_9/DIR_5/DIR_4/DIR_D/DIR_C/DIR_D does > not > >>> >> exist -- if DIR_9/DIR_5/DIR_4/DIR_D/DIR_C/ ever gets too full, > >>> >> DIR_9/DIR_5/DIR_4/DIR_D/DIR_C/DIR_D would be created and this file > >>> >> would be moved into it). > >>> >> > >>> >> When the escaped filename gets too long, we truncate the filename, > and > >>> >> then append a hash and a number yielding a name like: > >>> >> > >>> >> > >>> >> > >>> >> > ./DIR_9/DIR_5/DIR_4/DIR_D/DIR_E/default.724733.17\u\ushadow\uprostate\srnaseq\s8e5da6e8-8881-4813-a4e3-327df57fd1b7\sUNCID\u2409283.304a95c1-2180-4a81-a85a-880427e97d67.140304\uUNC14-SN744\u0400\uAC3LWGACXX\u7\uGAGTGG.tar.gz.2~\u1r\uFGidmpEP8GRsJkNLfAh9CokxkL_fa202ec9b4b3b217275a_0_long > >>> >> > >>> >> The _long at the end is always present with files like this. > >>> >> fa202ec9b4b3b217275a is the hash of the filename. The 0 indicates > >>> >> that it's the 0th file with this prefix and this hash -- if there > are > >>> >> hash collisions with the same prefix, you'll see _1_ and _2_ and so > on > >>> >> to distinguish them (very very unlikely). When the filename has > been > >>> >> truncated as with this one, you will find the full file name in the > >>> >> attrs (attr user.cephosd.lfn3): > >>> >> > >>> >> > >>> >> > >>> >> > ./DIR_9/DIR_5/DIR_4/DIR_D/DIR_E/default.724733.17\u\ushadow\uprostate\srnaseq\s8e5da6e8-8881-4813-a4e3-327df57fd1b7\sUNCID\u2409283.304a95c1-2180-4a81-a85a-880427e97d67.140304\uUNC14-SN744\u0400\uAC3LWGACXX\u7\uGAGTGG.tar.gz.2~\u1r\uFGidmpEP8GRsJkNLfAh9CokxkL_fa202ec9b4b3b217275a_0_long: > >>> >> user.cephos.lfn3: > >>> >> > >>> >> > >>> >> > default.724733.17\u\ushadow\uprostate\srnaseq\s8e5da6e8-8881-4813-a4e3-327df57fd1b7\sUNCID\u2409283.304a95c1-2180-4a81-a85a-880427e97d67.140304\uUNC14-SN744\u0400\uAC3LWGACXX\u7\uGAGTGG.tar.gz.2~\u1r\uFGidmpEP8GRsJkNLfAh9CokxkLf.4\u156__head_79CED459__46_ffffffffffffffff_0 > >>> >> > >>> >> Let's look at one of the orphaned files (the one with the same > >>> >> file-name as the previous one, actually): > >>> >> > >>> >> > >>> >> > >>> >> > ./DIR_9/DIR_5/DIR_4/DIR_D/default.724733.17\u\ushadow\uprostate\srnaseq\s8e5da6e8-8881-4813-a4e3-327df57fd1b7\sUNCID\u2409283.304a95c1-2180-4a81-a85a-880427e97d67.140304\uUNC14-SN744\u0400\uAC3LWGACXX\u7\uGAGTGG.tar.gz.2~\u1r\uFGidmpEP8GRsJkNLfAh9CokxkL_fa202ec9b4b3b217275a_0_long: > >>> >> user.cephos.lfn3: > >>> >> > >>> >> > >>> >> > default.724733.17\u\ushadow\uprostate\srnaseq\s8e5da6e8-8881-4813-a4e3-327df57fd1b7\sUNCID\u2409283.304a95c1-2180-4a81-a85a-880427e97d67.140304\uUNC14-SN744\u0400\uAC3LWGACXX\u7\uGAGTGG.tar.gz.2~\u1r\uFGidmpEP8GRsJkNLfAh9CokxkLf.4\u156__head_79CED459__46_3189d_0 > >>> >> > >>> >> This one has the same filename as the previous object, but is an > >>> >> orphan. What makes it an orphan is that it has hash 79CED459, but > is > >>> >> in ./DIR_9/DIR_5/DIR_4/DIR_D even though > >>> >> ./DIR_9/DIR_5/DIR_4/DIR_D/DIR_E exists (objects-files are always at > >>> >> the farthest directory from the root matching their hash). All of > the > >>> >> orphans will be long-file-name objects (but most long-file-name > >>> >> objects are fine and are neither orphans nor have duplicates -- > it's a > >>> >> fairly low occurrence bug). In your case, I think *all* of the > >>> >> orphans will probably happen to have files with duplicate names in > the > >>> >> correct directory -- though might not if the object had actually > been > >>> >> deleted since the bug happened. When there are duplicates, the full > >>> >> object names will either be the same or differ by the generation > >>> >> number at the end (ffffffffffffffff_0 vs 3189d_0) in this case. > >>> >> > >>> >> Once the orphaned files are cleaned up, your cluster should be back > to > >>> >> normal. If you want to wait, someone might get time to build a > patch > >>> >> for ceph-objectstore-tool to automate this. You can try removing > the > >>> >> orphan we identified in pg 70.459 and re-scrubbing to confirm that > >>> >> that fixes the pg. > >>> >> -Sam > >>> >> > >>> >> On Wed, Mar 9, 2016 at 6:58 AM, Jeffrey McDonald <[email protected]> > >>> >> wrote: > >>> >> > Hi, I went back to the mon logs to see if I could illicit any > >>> >> > additional > >>> >> > information about this PG. > >>> >> > Prior to 1/27/16, the deep-scrub on this OSD passes(then I see > >>> >> > obsolete > >>> >> > rollback objects found): > >>> >> > > >>> >> > ceph.log.4.gz:2016-01-20 09:43:36.195640 osd.307 > >>> >> > 10.31.0.67:6848/127170 > >>> >> > 538 > >>> >> > : cluster [INF] 70.459 deep-scrub ok > >>> >> > ceph.log.4.gz:2016-01-27 09:51:49.952459 osd.307 > >>> >> > 10.31.0.67:6848/127170 > >>> >> > 583 > >>> >> > : cluster [INF] 70.459 deep-scrub starts > >>> >> > ceph.log.4.gz:2016-01-27 10:10:57.196311 osd.108 > 10.31.0.69:6816/4283 > >>> >> > 335 : > >>> >> > cluster [ERR] osd.108 pg 70.459s5 found obsolete rollback obj > >>> >> > > >>> >> > > >>> >> > > 79ced459/default.724733.17__shadow_prostate/rnaseq/8e5da6e8-8881-4813-a4e3-327df57fd1b7/UNCID_2409283.304a95c1-2180-4a81-a85a-880427e97d67.140304_UNC14-SN744_0400_AC3LWGACXX_7_GAGTGG.tar.gz.2~_1r_FGidmpEP8GRsJkNLfAh9CokxkLf.4_156/head//70/202909/5 > >>> >> > generation < trimmed_to 130605'206504...repaired > >>> >> > ceph.log.4.gz:2016-01-27 10:10:57.043942 osd.307 > >>> >> > 10.31.0.67:6848/127170 > >>> >> > 584 > >>> >> > : cluster [ERR] osd.307 pg 70.459s0 found obsolete rollback obj > >>> >> > > >>> >> > > >>> >> > > 79ced459/default.724733.17__shadow_prostate/rnaseq/8e5da6e8-8881-4813-a4e3-327df57fd1b7/UNCID_2409283.304a95c1-2180-4a81-a85a-880427e97d67.140304_UNC14-SN744_0400_AC3LWGACXX_7_GAGTGG.tar.gz.2~_1r_FGidmpEP8GRsJkNLfAh9CokxkLf.4_156/head//70/202909/0 > >>> >> > generation < trimmed_to 130605'206504...repaired > >>> >> > ceph.log.4.gz:2016-01-27 10:10:58.225017 osd.307 > >>> >> > 10.31.0.67:6848/127170 > >>> >> > 585 > >>> >> > : cluster [ERR] 70.459s0 shard 4(3) missing > >>> >> > > >>> >> > > >>> >> > > cffed459/default.325671.93__shadow_wrfout_d01_2005-04-18_00_00_00.2~DyHqLoH7FFV_6fz8MOzmPEVO3Td4bZx.10_82/head//70 > >>> >> > ceph.log.4.gz:2016-01-27 10:10:58.225068 osd.307 > >>> >> > 10.31.0.67:6848/127170 > >>> >> > 586 > >>> >> > : cluster [ERR] 70.459s0 shard 10(2) missing > >>> >> > > >>> >> > > >>> >> > > cffed459/default.325671.93__shadow_wrfout_d01_2005-04-18_00_00_00.2~DyHqLoH7FFV_6fz8MOzmPEVO3Td4bZx.10_82/head//70 > >>> >> > ceph.log.4.gz:2016-01-27 10:10:58.225088 osd.307 > >>> >> > 10.31.0.67:6848/127170 > >>> >> > 587 > >>> >> > : cluster [ERR] 70.459s0 shard 26(1) missing > >>> >> > > >>> >> > > >>> >> > > cffed459/default.325671.93__shadow_wrfout_d01_2005-04-18_00_00_00.2~DyHqLoH7FFV_6fz8MOzmPEVO3Td4bZx.10_82/head//70 > >>> >> > ceph.log.4.gz:2016-01-27 10:10:58.225127 osd.307 > >>> >> > 10.31.0.67:6848/127170 > >>> >> > 588 > >>> >> > : cluster [ERR] 70.459s0 shard 132(4) missing > >>> >> > > >>> >> > > >>> >> > > cffed459/default.325671.93__shadow_wrfout_d01_2005-04-18_00_00_00.2~DyHqLoH7FFV_6fz8MOzmPEVO3Td4bZx.10_82/head//70 > >>> >> > ceph.log.4.gz:2016-01-27 10:13:52.926032 osd.307 > >>> >> > 10.31.0.67:6848/127170 > >>> >> > 589 > >>> >> > : cluster [ERR] 70.459s0 deep-scrub stat mismatch, got > 21324/21323 > >>> >> > objects, > >>> >> > 0/0 clones, 21324/21323 dirty, 0/0 omap, 0/0 hit_set_archive, 0/0 > >>> >> > whiteouts, > >>> >> > 64313094166/64308899862 bytes,0/0 hit_set_archive bytes. > >>> >> > ceph.log.4.gz:2016-01-27 10:13:52.927589 osd.307 > >>> >> > 10.31.0.67:6848/127170 > >>> >> > 590 > >>> >> > : cluster [ERR] 70.459s0 deep-scrub 1 missing, 0 inconsistent > objects > >>> >> > ceph.log.4.gz:2016-01-27 10:13:52.931250 osd.307 > >>> >> > 10.31.0.67:6848/127170 > >>> >> > 591 > >>> >> > : cluster [ERR] 70.459 deep-scrub 5 errors > >>> >> > ceph.log.4.gz:2016-01-28 10:32:37.083809 osd.307 > >>> >> > 10.31.0.67:6848/127170 > >>> >> > 592 > >>> >> > : cluster [INF] 70.459 repair starts > >>> >> > ceph.log.4.gz:2016-01-28 10:51:44.608297 osd.307 > >>> >> > 10.31.0.67:6848/127170 > >>> >> > 593 > >>> >> > : cluster [ERR] osd.307 pg 70.459s0 found obsolete rollback obj > >>> >> > > >>> >> > > >>> >> > > 79ced459/default.724733.17__shadow_prostate/rnaseq/8e5da6e8-8881-4813-a4e3-327df57fd1b7/UNCID_2409283.304a95c1-2180-4a81-a85a-880427e97d67.140304_UNC14-SN744_0400_AC3LWGACXX_7_GAGTGG.tar.gz.2~_1r_FGidmpEP8GRsJkNLfAh9CokxkLf.4_156/head//70/202909/0 > >>> >> > generation < trimmed_to 130605'206504...repaired > >>> >> > ceph.log.4.gz:2016-01-28 10:51:45.802549 osd.307 > >>> >> > 10.31.0.67:6848/127170 > >>> >> > 594 > >>> >> > : cluster [ERR] 70.459s0 shard 4(3) missing > >>> >> > > >>> >> > > >>> >> > > cffed459/default.325671.93__shadow_wrfout_d01_2005-04-18_00_00_00.2~DyHqLoH7FFV_6fz8MOzmPEVO3Td4bZx.10_82/head//70 > >>> >> > ceph.log.4.gz:2016-01-28 10:51:45.802933 osd.307 > >>> >> > 10.31.0.67:6848/127170 > >>> >> > 595 > >>> >> > : cluster [ERR] 70.459s0 shard 10(2) missing > >>> >> > > >>> >> > > >>> >> > > cffed459/default.325671.93__shadow_wrfout_d01_2005-04-18_00_00_00.2~DyHqLoH7FFV_6fz8MOzmPEVO3Td4bZx.10_82/head//70 > >>> >> > ceph.log.4.gz:2016-01-28 10:51:45.802978 osd.307 > >>> >> > 10.31.0.67:6848/127170 > >>> >> > 596 > >>> >> > : cluster [ERR] 70.459s0 shard 26(1) missing > >>> >> > > >>> >> > > >>> >> > > cffed459/default.325671.93__shadow_wrfout_d01_2005-04-18_00_00_00.2~DyHqLoH7FFV_6fz8MOzmPEVO3Td4bZx.10_82/head//70 > >>> >> > ceph.log.4.gz:2016-01-28 10:51:45.803039 osd.307 > >>> >> > 10.31.0.67:6848/127170 > >>> >> > 597 > >>> >> > : cluster [ERR] 70.459s0 shard 132(4) missing > >>> >> > > >>> >> > > >>> >> > > cffed459/default.325671.93__shadow_wrfout_d01_2005-04-18_00_00_00.2~DyHqLoH7FFV_6fz8MOzmPEVO3Td4bZx.10_82/head//70 > >>> >> > ceph.log.4.gz:2016-01-28 10:51:44.781639 osd.108 > 10.31.0.69:6816/4283 > >>> >> > 338 : > >>> >> > cluster [ERR] osd.108 pg 70.459s5 found obsolete rollback obj > >>> >> > > >>> >> > > >>> >> > > 79ced459/default.724733.17__shadow_prostate/rnaseq/8e5da6e8-8881-4813-a4e3-327df57fd1b7/UNCID_2409283.304a95c1-2180-4a81-a85a-880427e97d67.140304_UNC14-SN744_0400_AC3LWGACXX_7_GAGTGG.tar.gz.2~_1r_FGidmpEP8GRsJkNLfAh9CokxkLf.4_156/head//70/202909/5 > >>> >> > generation < trimmed_to 130605'206504...repaired > >>> >> > ceph.log.4.gz:2016-01-28 11:01:18.119350 osd.26 > >>> >> > 10.31.0.103:6812/77378 > >>> >> > 2312 > >>> >> > : cluster [INF] 70.459s1 restarting backfill on osd.305(0) from > >>> >> > (0'0,0'0] > >>> >> > MAX to 130605'206506 > >>> >> > ceph.log.4.gz:2016-02-01 13:40:55.096030 osd.307 > >>> >> > 10.31.0.67:6848/13421 > >>> >> > 16 : > >>> >> > cluster [INF] 70.459s0 restarting backfill on osd.210(1) from > >>> >> > (0'0,0'0] > >>> >> > MAX > >>> >> > to 135195'206996 > >>> >> > ceph.log.4.gz:2016-02-01 13:41:10.623892 osd.307 > >>> >> > 10.31.0.67:6848/13421 > >>> >> > 27 : > >>> >> > cluster [INF] 70.459s0 restarting backfill on osd.25(1) from > >>> >> > (0'0,0'0] > >>> >> > MAX > >>> >> > to 135195'206996 > >>> >> > ... > >>> >> > > >>> >> > > >>> >> > Regards, > >>> >> > Jeff > >>> >> > > >>> >> > > >>> >> > -- > >>> >> > > >>> >> > Jeffrey McDonald, PhD > >>> >> > Assistant Director for HPC Operations > >>> >> > Minnesota Supercomputing Institute > >>> >> > University of Minnesota Twin Cities > >>> >> > 599 Walter Library email: [email protected] > >>> >> > 117 Pleasant St SE phone: +1 612 625-6905 > >>> >> > Minneapolis, MN 55455 fax: +1 612 624-8861 > >>> >> > > >>> >> > > >>> > > >>> > > >>> > > >>> > > >>> > -- > >>> > > >>> > Jeffrey McDonald, PhD > >>> > Assistant Director for HPC Operations > >>> > Minnesota Supercomputing Institute > >>> > University of Minnesota Twin Cities > >>> > 599 Walter Library email: [email protected] > >>> > 117 Pleasant St SE phone: +1 612 625-6905 > >>> > Minneapolis, MN 55455 fax: +1 612 624-8861 > >>> > > >>> > > >> > >> > >> > >> > >> -- > >> > >> Jeffrey McDonald, PhD > >> Assistant Director for HPC Operations > >> Minnesota Supercomputing Institute > >> University of Minnesota Twin Cities > >> 599 Walter Library email: [email protected] > >> 117 Pleasant St SE phone: +1 612 625-6905 > >> Minneapolis, MN 55455 fax: +1 612 624-8861 > >> > >> > -- Jeffrey McDonald, PhD Assistant Director for HPC Operations Minnesota Supercomputing Institute University of Minnesota Twin Cities 599 Walter Library email: [email protected] 117 Pleasant St SE phone: +1 612 625-6905 Minneapolis, MN 55455 fax: +1 612 624-8861
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
