@lists.ceph.com] *On Behalf
> Of *Samuel Just
> *Sent:* 09 February 2017 19:22
> *To:* Nick Fisk
> *Cc:* ceph-users@lists.ceph.com
>
> *Subject:* Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during
> sleep?
>
>
>
> Ok, https://github.com/athanatos/c
Ok, https://github.com/athanatos/ceph/tree/wip-snap-trim-sleep (based on
master) passed a rados suite. It adds a configurable limit to the number
of pgs which can be trimming on any OSD (default: 2). PGs trimming will be
in snaptrim state, PGs waiting to trim will be in snaptrim_wait state. I
su
intended recipient of this message or received it
> erroneously, please notify the sender and delete it, together with any
> attachments, and be advised that any dissemination or copying of this
> message is prohibited.
>
> --
>
> ---
ete it, together with any
> attachments, and be advised that any dissemination or copying of this
> message is prohibited.
>
> --
>
> --
> *From:* Samuel Just [sj...@redhat.com]
> *Sent:* Thursday, January 26, 2017 1:14 P
am
On Fri, Jan 20, 2017 at 2:05 PM, Nick Fisk wrote:
> Hi Sam,
>
>
>
> I have a test cluster, albeit small. I’m happy to run tests + graph
> results with a wip branch and work out reasonable settings…etc
>
>
>
> *From:* Samuel Just [mailto:sj...@redhat.com]
> *S
> --
>
> --
> *From:* ceph-users [ceph-users-boun...@lists.ceph.com] on behalf of David
> Turner [david.tur...@storagecraft.com]
> *Sent:* Thursday, January 19, 2017 3:25 PM
> *To:* Samuel Just; Nick Fisk
>
> *Cc:* cep
application can priority tag individual
> IO's for CFQ to handle, instead of the current limitation of priority per
> thread, I realise this is probably very very hard or impossible. But it would
> allow Ceph to control IO queue's right down to the disk.
>
>> -Original M
fixing a bug with that.
>
> Nick
>
>> -Original Message-
>> From: Samuel Just [mailto:sj...@redhat.com]
>> Sent: 19 January 2017 15:47
>> To: Dan van der Ster
>> Cc: Nick Fisk ; ceph-users
>> Subject: Re: [ceph-users] osd_snap_trim_sleep k
Snaptrimming is now in the main op threadpool along with scrub,
recovery, and client IO. I don't think it's a good idea to use any of
the _sleep configs anymore -- the intention is that by setting the
priority low, they won't actually be scheduled much.
-Sam
On Thu, Jan 19, 2017 at 5:40 AM, Dan v
That would work.
-Sam
On Thu, Jan 12, 2017 at 1:40 PM, Gregory Farnum wrote:
> On Thu, Jan 12, 2017 at 1:37 PM, Samuel Just wrote:
>> Oh, this is basically working as intended. What happened is that the
>> mon died before the pending map was actually committed. The OSD has a
r. I will look tomorrow
> at the output again and create an ticket.
>
>
> Thanks
>
>
> Udo
>
>
> On 12.01.2017 20:02, Samuel Just wrote:
>> How long did it take for the cluster to recover?
>> -Sam
>>
>> On Thu, Jan 12, 2017 at 10:54 AM, Gregor
How long did it take for the cluster to recover?
-Sam
On Thu, Jan 12, 2017 at 10:54 AM, Gregory Farnum wrote:
> On Thu, Jan 12, 2017 at 2:03 AM, wrote:
>> Hi all,
>> I had just reboot all 3 nodes (one after one) of an small Proxmox-VE
>> ceph-cluster. All nodes are mons and have two OSDs.
>> Du
Jason: librbd itself uses the librados C++ api though, right?
-Sam
On Wed, Jan 11, 2017 at 9:37 AM, Jason Dillaman wrote:
> +1
>
> I'd be happy to tweak the internals of librbd to support pass-through
> of C buffers all the way to librados. librbd clients like QEMU use the
> C API and this curren
c/ceph/
>
>
> Bryan
>
> On 1/10/17, 10:51 AM, "Samuel Just" wrote:
>
>>Can you push that branch somewhere? I don't have a v11.1.1 or that sha1.
>>-Sam
>>
>>On Tue, Jan 10, 2017 at 9:41 AM, Stillwell, Bryan J
>> wrote:
>>> This is f
Can you push that branch somewhere? I don't have a v11.1.1 or that sha1.
-Sam
On Tue, Jan 10, 2017 at 9:41 AM, Stillwell, Bryan J
wrote:
> This is from:
>
> ceph version 11.1.1 (87597971b371d7f497d7eabad3545d72d18dd755)
>
> On 1/10/17, 10:23 AM, "Samuel Just" wro
What ceph sha1 is that? Does it include
6c3d015c6854a12cda40673848813d968ff6afae which fixed the messenger
spin?
-Sam
On Tue, Jan 10, 2017 at 9:00 AM, Stillwell, Bryan J
wrote:
> On 1/10/17, 5:35 AM, "John Spray" wrote:
>
>>On Mon, Jan 9, 2017 at 11:46 PM, Stillwell, Bryan J
>> wrote:
>>> Last
Shinobu isn't correct, you have 9/9 osds up and running. up does not
equal acting because crush is having trouble fulfilling the weights in
your crushmap and the acting set is being padded out with an extra osd
which happens to have the data to keep you up to the right number of
replicas. Please
{
"name": "Started\/Primary\/Peering",
"enter_time": "2017-01-10 13:43:34.933074",
"past_intervals": [
{
"first": 75858,
"last": 75860,
"maybe_went_rw": 1,
"up
Is there a particular reason you are sticking to the versions with
shorter support periods?
-Sam
On Fri, Dec 9, 2016 at 11:38 AM, Ben Hines wrote:
> Anyone have any good / bad experiences with Kraken? I haven't seen much
> discussion of it. Particularly from the RGW front.
>
> I'm still on Infern
Actually, Greg and Sage are working up other branches, nvm.
-Sam
On Wed, Dec 7, 2016 at 2:52 PM, Samuel Just wrote:
> I just pushed a branch wip-14120-10.2.4 with a possible fix.
>
> https://github.com/ceph/ceph/pull/12349/ is a fix for a known bug
> which didn't quite make it
I just pushed a branch wip-14120-10.2.4 with a possible fix.
https://github.com/ceph/ceph/pull/12349/ is a fix for a known bug
which didn't quite make it into 10.2.4, it's possible that
165e5abdbf6311974d4001e43982b83d06f9e0cc which did made the bug much
more likely to happen. wip-14120-10.2.4 ha
On Wed, Nov 23, 2016 at 4:11 PM, Chris Taylor wrote:
> Kevin,
>
> After changing the pool size to 3, make sure the min_size is set to 1 to
> allow 2 of the 3 hosts to be offline.
If you do this, the flip side is that while in that configuration
losing that single
host will render your data unreco
I don't mind trying to get this done if its felt to be worthwhile.
>
> Nick
>
>> -Original Message-
>> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of
>> Samuel Just
>> Sent: 19 November 2016 00:31
>> To: Nick Fisk
>
t;
>> This is like your mother telling not to cross the road when you were 4
>> years of age but not telling you it was because you could be flattened
>> by a car :)
>>
>> Can you expand on your answer? If you are in a DC with AB power,
>> redundant UPS, dual feed
Never *ever* use nobarrier with ceph under *any* circumstances. I
cannot stress this enough.
-Sam
On Fri, Nov 18, 2016 at 10:39 AM, Craig Chi wrote:
> Hi Nick and other Cephers,
>
> Thanks for your reply.
>
>> 2) Config Errors
>> This can be an easy one to say you are safe from. But I would say
Puzzling, added a question to the ticket.
-Sam
On Thu, Nov 17, 2016 at 4:32 AM, Nick Fisk wrote:
> Hi Sam,
>
> I've updated the ticket with logs from the wip run.
>
> Nick
>
>> -Original Message-
>> From: Samuel Just [mailto:sj...@redhat.com]
>>
http://tracker.ceph.com/issues/17916
I just pushed a branch wip-17916-jewel based on v10.2.3 with some
additional debugging. Once it builds, would you be able to start the
afflicted osds with that version of ceph-osd and
debug osd = 20
debug ms = 1
debug filestore = 20
and get me the log?
-Sam
I seriously doubt that it's ever going to be a winning strategy to let
rgw index objects go to a cold tier. Some practical problems:
1) We don't track omap size (the leveldb entries for an object)
because it would turn writes into rmw's -- so they always show up as 0
size. Thus, the target_max_by
Ah, same question then. If we can get logging on the primary for one
of those pgs, it should be fairly obvious.
-Sam
On Wed, Sep 21, 2016 at 4:08 AM, Pavan Rallabhandi
wrote:
> We find this as well in our fresh built Jewel clusters, and seems to happen
> only with a handful of PGs from couple o
Looks like the OSD didn't like an error return it got from the
underlying fs. Can you reproduce with
debug filestore = 20
debug osd = 20
debug ms = 1
on the osd and post the whole log?
-Sam
On Wed, Sep 21, 2016 at 12:10 AM, Peter Maloney
wrote:
> Hi,
>
> I created a one disk osd with data and
Can you reproduce with logging on the primary for that pg?
debug osd = 20
debug filestore = 20
debug ms = 1
Since restarting the osd may be a workaround, can you inject the debug
values without restarting the daemon?
-Sam
On Wed, Sep 21, 2016 at 2:44 AM, Tobias Böhm wrote:
> Hi,
>
> there is an
Probably an EIO. You can reproduce with debug filestore = 20 to confirm.
-Sam
On Fri, Sep 2, 2016 at 10:18 AM, Reed Dier wrote:
> OSD has randomly stopped for some reason. Lots of recovery processes
> currently running on the ceph cluster. OSD log with assert below:
>
> -14> 2016-09-02 11:32:38.
If it's bluestore, this is pretty likely to be a bluestore bug. If
you are interested in experimenting with bluestore, you probably want
to watch developements on the master branch, it's undergoing a bunch
of changes right now.
-Sam
On Thu, Sep 1, 2016 at 1:54 PM, Виталий Филиппов wrote:
> Hi! I
use peering and
>> backfill to restart)
>> 5) link all logs to the bug
>>
>> Thanks!
>> -Sam
>>
>> On Tue, Jul 26, 2016 at 9:11 AM, Samuel Just wrote:
>> > Hmm, nvm, it's not an lfn object anyway.
>> > -Sam
>> >
>> &g
, 2016 at 9:11 AM, Samuel Just wrote:
> Hmm, nvm, it's not an lfn object anyway.
> -Sam
>
> On Tue, Jul 26, 2016 at 7:07 AM, Brian Felton wrote:
>> If I search on osd.580, I find
>> default.421929.15\uTEPP\s84316222-6ddd-4ac9-8283-6fa1cdcf9b88\sbackups\s2016063009
osd min down reports = 2
Set that to 1?
-Sam
On Wed, Jul 27, 2016 at 10:24 AM, Patrick McGarry wrote:
> Moving this to ceph-user.
>
>
> On Wed, Jul 27, 2016 at 8:36 AM, Kostya Velychkovsky
> wrote:
>> Hello. I have test CEPH cluster with 5 nodes: 3 MON and 2 OSD
>>
>> This is my ceph.conf
>>
>
Well, it's kind of deliberately obfuscated because PGs aren't a
librados-level abstraction. Why do you want to list the objects in a
PG?
-Sam
On Wed, Jul 27, 2016 at 8:10 AM, David Blundell
wrote:
> Hi,
>
>
>
> I wasn’t sure if this is a ceph-users or ceph-devel question as it’s about
> the API
e plugin for RDP (or RAID-DP) kind of
> code.
>
> Thanks,
> Syed
>
> On Wed, Jul 27, 2016 at 4:12 AM, Samuel Just wrote:
>>
>> Why do you want them in serial increasing order?
>> -Sam
>>
>> On Tue, Jul 26, 2016 at 2:43 PM, Samuel Just wrote:
Why do you want them in serial increasing order?
-Sam
On Tue, Jul 26, 2016 at 2:43 PM, Samuel Just wrote:
> How would such a code work if there were more than 24 osds?
> -Sam
>
> On Tue, Jul 26, 2016 at 2:37 PM, Syed Hussain wrote:
>
>> Hi,
>>
>> I'm wor
How would such a code work if there were more than 24 osds?
-Sam
On Tue, Jul 26, 2016 at 2:37 PM, Syed Hussain wrote:
> Hi,
>
> I'm working to develop an Erasure Code plugin (variation of ISA) that have
> typical requirement that the active set of the Erasure Coded pool in serial
> order.
> For
about a dangling link, please
> point me in the right direction.
>
> Brian
>
> On Tue, Jul 26, 2016 at 8:59 AM, Samuel Just wrote:
>>
>> Did you also confirm that the backfill target does not have any of
>> those dangling links? I'd be looking for
e reviewing them with my team tomorrow morning to see if we can find
> anything. Thanks for your assistance.
>
> Brian
>
> On Mon, Jul 25, 2016 at 3:33 PM, Samuel Just wrote:
>>
>> The next thing I'd want is for you to reproduce with
>>
>> debug osd = 20
the special ceph long filename handling.
>
> Also, we are not using XFS on our OSDs; we are using ZFS instead.
>
> If I'm misunderstanding the issue linked above and the corresponding thread,
> please let me know.
>
> Brian
>
>
> On Mon, Jul 25, 2016 at 1:32 PM, Samu
The next thing I'd want is for you to reproduce with
debug osd = 20
debug filestore = 20
debug ms = 1
and post the file somewhere.
-Sam
On Mon, Jul 25, 2016 at 1:33 PM, Samuel Just wrote:
> If you don't have the orphaned file link, it's not the same bug.
> -Sam
>
>
You may have hit http://tracker.ceph.com/issues/14766. There was a
thread on the list a while back about diagnosing and fixing it.
-Sam
On Mon, Jul 25, 2016 at 10:45 AM, Brian Felton wrote:
> Greetings,
>
> Problem: After removing (out + crush remove + auth del + osd rm) three osds
> on a single
Try strace.
-Sam
On Wed, Jul 6, 2016 at 7:53 AM, RJ Nowling wrote:
> Hi all,
>
> I'm trying to use the ceph-ansible playbooks to deploy onto baremetal. I'm
> currently testing with the approach that uses an existing filesystem rather
> than giving raw access to the disks.
>
> The OSDs are failin
.rgw.bucket.index.pool is the pool with rgw's index objects, right?
The actual on-disk directory for one of those pgs would contain only
empty files -- the actual index data is stored in the osd's leveldb
instance. I suspect your index objects are very large (because the
buckets contain many objec
I think you hit the os process fd limit. You need to adjust it.
-Sam
On Wed, Jun 15, 2016 at 2:07 PM, Mansour Shafaei Moghaddam
wrote:
> It fails at "FileStore.cc: 2761". Here is a more complete log:
>
> -9> 2016-06-15 10:55:13.205014 7fa2dcd85700 -1 dump_open_fds unable to
> open /proc/self
I'd have to look more closely, but these days promotion is
probabilistic and throttled. During each read of those objects, it
will tend to promote a few more of them depending on how many
promotions are in progress and how hot it thinks a particular object
is. The lack of a speed up is a bummer,
r
> root as they were created while the older ceph processes were still running.
>
> Changing that after the fact was still a necessity and I will make sure that
> those are also properly changed.
>
> On Mon, Jun 6, 2016 at 12:12 PM Samuel Just wrote:
>>
>> Oh, what wa
Oh, what was the problem (for posterity)?
-Sam
On Mon, Jun 6, 2016 at 12:11 PM, Tu Holmes wrote:
> It totally did and I see what the problem is.
>
> Thanks for your input. I truly appreciate it.
>
>
> On Mon, Jun 6, 2016 at 12:01 PM Samuel Just wrote:
>>
>> If you
If you reproduce with
debug osd = 20
debug filestore = 20
debug ms = 1
that might make it clearer what is going on.
-Sam
On Mon, Jun 6, 2016 at 11:53 AM, Tu Holmes wrote:
> Hey cephers. I have been following the upgrade documents and I have done
> everything regarding upgrading the client to th
Sorry, I should have been more clear. The bug actually is due to a
difference in an on disk encoding from hammer. An infernalis cluster would
never had had such encodings and is fine.
-Sam
On Jun 3, 2016 6:53 AM, "Francois Lafont" wrote:
> Hi,
>
> On 03/06/2016 05:39
Due to http://tracker.ceph.com/issues/16113, it would be best to avoid
setting the sortbitwise flag on jewel clusters upgraded from previous
versions until we get a point release out with a fix.
The symptom is that setting the sortbitwise flag on a jewel cluster
upgraded from a previous version ca
I suspect the problem is that ReplicatedBackend::build_push_op assumes
that osd_recovery_max_chunk (defaults to 8MB) of omap entries is about
the same amount of work to get as 8MB of normal object data. The fix
would be to add another config osd_recovery_max_omap_entries_per_chunk
with a sane defa
as you noted in the issue.
>
> The cluster runs jewel 10.2.1 and was created a long time ago, I think it was
> giant.
>
> Thanks again!
>
> Uwe
>
>> Am 02.06.2016 um 00:19 schrieb Samuel Just :
>>
>> http://tracker.ceph.com/issues/16113
>>
>> I t
the upgrade (despite the well know systemd),
> and after that the cluster didn't show any suspicious behavior.
>
>
> ---
> Diego Castro / The CloudFather
> GetupCloud.com - Eliminamos a Gravidade
>
> 2016-06-01 18:57 GMT-03:00 Samuel Just :
>>
>> Was this cluster upg
t unset sortbitwise flag.
>
>
> ---
> Diego Castro / The CloudFather
> GetupCloud.com - Eliminamos a Gravidade
>
> 2016-06-01 13:39 GMT-03:00 Samuel Just :
>>
>> Can either of you reproduce with logs? That would make it a lot
>> easier to track down if it's a bug. I
Can either of you reproduce with logs? That would make it a lot
easier to track down if it's a bug. I'd want
debug osd = 20
debug ms = 1
debug filestore = 20
On all of the osds for a particular pg from when it is clean until it
develops an unfound object.
-Sam
On Wed, Jun 1, 2016 at 5:36 AM, D
Well, it's not supposed to do that if the backing storage is working
properly. If the filesystem/disk controller/disk combination is not
respecting barriers (or otherwise can lose committed data in a power
failure) in your configuration, a power failure could cause a node to
go backwards in time -
. Suggestions?
>
>
> Kevan
>
>
>
> On 5/24/16, 4:11 PM, "Kevan Rehm" wrote:
>
>>Okay, will do. If the problem goes away with filestore, I'll switch back
>>to bluestore again and re-duplicate the problem. In that case, are there
>>p
My money is on bluestore. If you can try to reproduce on filestore,
that would rapidly narrow it down.
-Sam
On Tue, May 24, 2016 at 1:53 PM, Kevan Rehm wrote:
> Nope, not using tiering.
>
> Also, this is my second attempt, this is repeatable for me, I'm trying to
> duplicate a previous occurrenc
Restart osd.1 with debugging enabled
debug osd = 20
debug filestore = 20
debug ms = 1
Then, run list_unfound once the pg is back in active+recovering. If
it still hangs, post osd.1's log to the list along with the output of
ceph osd dump and ceph pg dump.
-Sam
On Wed, May 18, 2016 at 6:20 PM, D
Try restarting the primary osd for that pg with
osd_find_best_info_ignore_history_les set to true (don't leave it set
long term).
-Sam
On Tue, May 17, 2016 at 7:50 AM, Hein-Pieter van Braam wrote:
> Hello,
>
> Today we had a power failure in a rack housing our OSD servers. We had
> 7 of our 30 to
Can you reproduce with
debug ms = 1
debug objecter = 20
on the radosgw side?
-Sam
On Thu, May 5, 2016 at 8:28 AM, Brian Felton wrote:
> Greetings,
>
> We are running a number of Ceph clusters in production to provide object
> storage services. We have stumbled upon an issue where objects of ce
egards
> Somnath
>
> -Original Message-
> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of
> Garg, Pankaj
> Sent: Friday, April 29, 2016 8:59 AM
> To: Samuel Just
> Cc: ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] OSD Crashes
>
Your fs is throwing an EIO on open.
-Sam
On Fri, Apr 29, 2016 at 8:54 AM, Garg, Pankaj
wrote:
> Hi,
>
> I had a fully functional Ceph cluster with 3 x86 Nodes and 3 ARM64 nodes,
> each with 12 HDD Drives and 2SSD Drives. All these were initially running
> Hammer, and then were successfully update
I'd guess that to make any progress we'll need debug ms = 20 on both
sides of the connection when a message is lost.
-Sam
On Thu, Apr 28, 2016 at 2:38 PM, Mike Lovell wrote:
> there was a problem on one of the clusters i manage a couple weeks ago where
> pairs of OSDs would wait indefinitely on s
I think? Probably worth reproducing on a vstart cluster to validate
the fix. Didn't we introduce something in the mon to validate new
crushmaps? Hammer maybe?
-Sam
On Tue, Apr 26, 2016 at 8:09 AM, Wido den Hollander wrote:
>
>> Op 26 april 2016 om 16:58 schreef Samuel Just :
&g
Can you attach the OSDMap (ceph osd getmap -o )?
-Sam
On Tue, Apr 26, 2016 at 2:07 AM, Henrik Svensson wrote:
> Hi!
>
> We got a three node CEPH cluster with 10 OSD each.
>
> We bought 3 new machines with additional 30 disks that should reside in
> another location.
> Before adding these machine
It doesn't seem like it would be wise to run such systems on top of rbd.
-Sam
On Thu, Apr 14, 2016 at 11:05 AM, Jianjian Huo wrote:
> On Wed, Apr 13, 2016 at 6:06 AM, Sage Weil wrote:
>> On Tue, 12 Apr 2016, Jan Schermer wrote:
>>> Who needs to have exactly the same data in two separate objects
Well, that bug is a completely different backtrace, so it's probably
safe to say that it's not related. It looks like you are missing
maps, which suggests you have inconsistent osd filesystems. Did you
powercycle this node with barriers disabled in xfs? Posting a log
from startup to crash would
f304ebf8bf1166aba6)
>
> If I run the command without root or sudo the command failed with a
> Permission denied
>
> German
>
> 2016-03-29 17:24 GMT-03:00 Samuel Just :
>>
>> Or you needed to run it as root?
>> -Sam
>>
>> On Tue, Mar 29, 2016 at 1:24
Or you needed to run it as root?
-Sam
On Tue, Mar 29, 2016 at 1:24 PM, Samuel Just wrote:
> Sounds like a version/compatibility thing. Are your rbd clients really old?
> -Sam
>
> On Tue, Mar 29, 2016 at 1:19 PM, German Anders wrote:
>> I've just upgrade to jewel, and
"2",
>> "filestore_queue_max_ops": "50",
>> "filestore_split_multiple": "2",
>> "fsid": "----0000",
>> "internal_safe_to_sta
That seems to be scrubbing pretty often. Can you attach a config diff
from osd.4 (ceph daemon osd.4 config diff)?
-Sam
On Tue, Mar 29, 2016 at 9:30 AM, German Anders wrote:
> Hi All,
>
> I've maybe a simple question, I've setup a new cluster with Infernalis
> release, there's no IO going on at t
Ok, like I said, most files with _long at the end are *not orphaned*.
The generation number also is *not* an indication of whether the file
is orphaned -- some of the orphaned files will have
as the generation number and others won't. For each long filename
object in a pg you woul
seen and only about 230 have the user.cephos.lfn3
> file attribute. The files will have other attributes, just not
> user.cephos.lfn3.
>
> Regards,
> Jeff
>
>
> On Wed, Mar 16, 2016 at 3:53 PM, Samuel Just wrote:
>>
>> Ok, like I said, most files with _long a
X\u7\uGAGTGG.tar.gz.2~\u1r\uFGidmpEP8GRsJkNLfAh9CokxkL_fa202ec9b4b3b217275a_0_long
>
> In the directory DIR_E here (from above), there is only one file without a
> __head_ in the pathname -- the file aboveShould I be deleting both these
> _long files without the __head_ in DIR_E
p.
> Regards,
> Jeff
>
> On Thu, Mar 17, 2016 at 11:58 AM, Samuel Just wrote:
>>
>> Oh, it's getting a stat mismatch. I think what happened is that on
>> one of the earlier repairs it reset the stats to the wrong value (the
>> orphan was causing the primary
ly have this one example so far.is the 79C of the hash
> constant? Would the hash pick up another hex character if the pg splits
> again?
>
> Thanks,
> Jeff
>
> On Wed, Mar 16, 2016 at 10:24 AM, Samuel Just wrote:
>>
>> There is a directory structure hash, it&#x
time to put together a branch in the next week
or two if you want to wait. It's still probably worth trying removing
that object in 70.459.
-Sam
On Tue, Mar 15, 2016 at 6:03 PM, Samuel Just wrote:
> The bug is entirely independent of hardware issues -- entirely a ceph
> bug. xf
had hardware issues
> during the remapping or would it have happened regardless of the hardware
> issues? e.g. I'm not planning to add any additional hardware soon, but
> would the bug pop again on an (unpatched) system not subject to any
> remapping?
>
> thanks,
> jeff
&
.g. will the orphans always
> be size 0?
>
> Thanks!
> Jeff
>
> On Tue, Mar 15, 2016 at 5:35 PM, Samuel Just wrote:
>>
>> Ok, a branch merged to master which should fix this
>> (https://github.com/ceph/ceph/pull/8136). It'll be backported in due
>> cour
Ok, a branch merged to master which should fix this
(https://github.com/ceph/ceph/pull/8136). It'll be backported in due
course. The problem is that that patch won't clean orphaned files
that already exist.
Let me explain a bit about what the orphaned files look like. The
problem is files with
s?The PGs there seem to now be remapped elsewhere.
>
> Regards,
> Jeff
>
>
> On Tue, Mar 8, 2016 at 2:09 PM, Samuel Just wrote:
>>
>> The pgs are not actually inconsistent (that is, I think that all of
>> the real objects are present and healthy). I thin
Yep
On Tue, Mar 8, 2016 at 12:08 PM, Jeffrey McDonald wrote:
> Will do, I'm in the midst of the xfs filesystem check so it will be a bit
> before I have the filesystem back.
> jeff
>
> On Tue, Mar 8, 2016 at 2:05 PM, Samuel Just wrote:
>>
>> There are 3 other
ctive+clean
> 43 active+clean+inconsistent
>7 active+clean+scrubbing+deep
> 7 active+clean+scrubbing
>
> Jeff
>
> On Tue, Mar 8, 2016 at 2:00 PM, Samuel Just wrote:
>>
>> Yeah, that procedure should have i
\u0386\uBC2E4WACXX\u2\uACTTGA.tar.gz.2~TWfBUlEkk4EPH4u\uNkMjmz65CRSkJA3._215ce1442b16dc173b77_0_long
On Tue, Mar 8, 2016 at 12:00 PM, Samuel Just wrote:
> Yeah, that procedure should have isolated any filesystem issues. Are
> there still unfound objects?
> -sam
>
> On Tue, Mar 8, 2
lowing fewer backfill
> threads.
>
> Jeff
>
> On Tue, Mar 8, 2016 at 1:56 PM, Samuel Just wrote:
>>
>> By "migrated", you mean you marked them out one at a time and let ceph
>> recover them over to new nodes?
>> -Sam
>>
>> On Tue, M
hardware.
> jeff
>
> On Tue, Mar 8, 2016 at 1:52 PM, Samuel Just wrote:
>>
>> ceph3 is not the same host as ceph03?
>> -Sam
>>
>> On Tue, Mar 8, 2016 at 11:48 AM, Jeffrey McDonald
>> wrote:
>> > Hi Sam,
>> >
>> > 1) Are t
.
>
>
> 4) Has anything unusual ever happened to the host which osd.307 is on?
> Particularly a power loss?
>
> I don't recall anything. A couple of times the data center overheated (air
> ) but these nodes are in a water-cooled enclosure and were OK. What I did
> hav
e in the PGs and are labeled as unfound.Its important for us to
>> maintain user confidence in the system so I need a fix as soon as
>> possible..
>>
>> The log files from the OSDs are here:
>>
>> https://drive.google.com/open?id=0Bzz8TrxFvfembkt2XzlCZFVJZFU
&
Nevermind on http://tracker.ceph.com/issues/14766 , OSD::remove_dir
uses the right collection_list_partial.
-Sam
On Mon, Mar 7, 2016 at 5:26 PM, Samuel Just wrote:
> Yep, just as before. Actually, do it twice (wait for 'scrubbing' to
> go away each time).
> -Sam
>
> On
70.459
> 3- stop once the 70.459 pg goes inconsistent?
>
> Thanks,
> Jeff
>
>
> On Mon, Mar 7, 2016 at 6:52 PM, Samuel Just wrote:
>>
>> Hmm, I'll look into this a bit more tomorrow. Can you get the tree
>> structure of the 70.459 pg directory on osd.30
On the plus side, I think I figured out http://tracker.ceph.com/issues/14766.
-Sam
On Mon, Mar 7, 2016 at 4:52 PM, Samuel Just wrote:
> Hmm, I'll look into this a bit more tomorrow. Can you get the tree
> structure of the 70.459 pg directory on osd.307 (find . will do fine).
> -
Hmm, I'll look into this a bit more tomorrow. Can you get the tree
structure of the 70.459 pg directory on osd.307 (find . will do fine).
-Sam
On Mon, Mar 7, 2016 at 4:50 PM, Jeffrey McDonald wrote:
> 307 is on ceph03.
> Jeff
>
> On Mon, Mar 7, 2016 at 6:48 PM, Samuel Just w
Which node is osd.307 on?
-Sam
On Mon, Mar 7, 2016 at 4:43 PM, Samuel Just wrote:
> ' I didn't see the errors in the tracker on the new nodes, but they
> were only receiving new data, not migrating it.' -- What do you mean
> by that?
> -Sam
>
> On Mon, Mar 7,
2 22:08:27
> UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
> ceph09: Linux ceph09 3.19.0-51-generic #58~14.04.1-Ubuntu SMP Fri Feb 26
> 22:02:58 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
>
>
> On Mon, Mar 7, 2016 at 6:39 PM, Samuel Just wrote:
>>
>> What filesystem and kernel
I'd rather you just scrubbed the same pg with the same osds and the
same debugging.
-Sam
On Mon, Mar 7, 2016 at 4:40 PM, Samuel Just wrote:
> Yes, the unfound objects are due to a bug in the repair command. I
> suggest you don't repair anything, actually. I don't think
1 - 100 of 305 matches
Mail list logo
