Re: [ceph-users] Ceph luminous - troubleshooting performance issues overall DSK 100%, busy 1%

[Konstantin Shalygin]

ceph osd df tree
ID CLASS WEIGHT  REWEIGHT SIZE  USEAVAIL %USE  VAR  PGS TYPE NAME
-1   3.44714-  588G 80693M  509G 00   - root default
-9   0.57458-  588G 80693M  509G 13.39 1.13   - host osd01
  5   hdd 0.57458  1.0  588G 80693M  509G 13.39 1.13  64 osd.5
-7   1.14899- 1176G   130G 1046G 11.06 0.94   - host osd02
  0   hdd 0.57500  1.0  588G 70061M  519G 11.63 0.98  50 osd.0
  1   hdd 0.57500  1.0  588G 63200M  526G 10.49 0.89  41 osd.1
-3   1.14899- 1176G   138G 1038G 11.76 1.00   - host osd03
  2   hdd 0.57500  1.0  588G 68581M  521G 11.38 0.96  48 osd.2
  3   hdd 0.57500  1.0  588G 73185M  516G 12.15 1.03  53 osd.3
-4   0.57458- 0  0 0 00   - host osd04
  4   hdd 0.574580 0  0 0 00   0 osd.4



By adding new hosts with half of osds of present hosts you a imbalance 
your crush.


osd.4 and osd.5 do double work in compare with present hosts if your 
failure domain is host.




k

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] cephfs snapshot format upgrade

[Yan, Zheng]

On Wed, Apr 11, 2018 at 3:34 AM, Gregory Farnum  wrote:
> On Tue, Apr 10, 2018 at 5:54 AM, John Spray  wrote:
>> On Tue, Apr 10, 2018 at 1:44 PM, Yan, Zheng  wrote:
>>> Hello
>>>
>>> To simplify snapshot handling in multiple active mds setup, we changed
>>> format of snaprealm in mimic dev.
>>> https://github.com/ceph/ceph/pull/16779.
>>>
>>> The new version mds can handle old format snaprealm in single active
>>> setup. It also can convert old format snaprealm to the new format when
>>> snaprealm is modified. The problem is that new version mds can not
>>> properly handle old format snaprealm in multiple active setup. It may
>>> crash when it encounter old format snaprealm. For existing filesystem
>>> with snapshots, upgrading mds to mimic seems to be no problem at first
>>> glance. But if user later enables multiple active mds,  mds may
>>> crashes continuously. No easy way to switch back to single acitve mds.
>>>
>>> I don't have clear idea how to handle this situation. I can think of a
>>> few options.
>>>
>>> 1. Forbid multiple active before all old snapshots are deleted or
>>> before all snaprealms are converted to new format. Format conversion
>>> requires traversing while whole filesystem tree.  Not easy to
>>> implement.
>>
>> This has been a general problem with metadata format changes: we can
>> never know if all the metadata in a filesystem has been brought up to
>> a particular version.  Scrubbing (where scrub does the updates) should
>> be the answer, but we don't have the mechanism for recording/ensuring
>> the scrub has really happened.
>>
>> Maybe we need the MDS to be able to report a complete whole-filesystem
>> scrub to the monitor, and record a field like
>> "latest_scrubbed_version" in FSMap, so that we can be sure that all
>> the filesystem metadata has been brought up to a certain version
>> before enabling certain features?  So we'd then have a
>> "latest_scrubbed_version >= mimic" test before enabling multiple
>> active daemons.
>
> Don't we have a (recursive!) last_scrub_[stamp|version] on all
> directories? There's not (yet) a mechanism for associating that with
> specific data versions like you describe here, but for a one-time
> upgrade with unsupported features I don't think we need anything too
> sophisticated.
> -Greg
>
No, we don't.  Besides, normal recursive stats (record last update) does not
work for this case. We need a recursive stat that tracks the oldest
update on all
directories..

Regards
Yan, Zheng

>>
>> For this particular situation, we'd also need to protect against
>> people who had enabled multimds and snapshots experimentally, with an
>> MDS startup check like:
>>  ((ever_allowed_features & CEPH_MDSMAP_ALLOW_SNAPS) &&
>> (allows_multimds() || in.size() >1)) && latest_scrubbed_version <
>> mimic
>>
>> John
>>
>>> 2. Ask user to delete all old snapshots before upgrading to mimic,
>>> make mds just ignore old format snaprealms.
>>>
>>>
>>> Regards
>>> Yan, Zheng
>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] cephfs snapshot format upgrade

[Yan, Zheng]

On Wed, Apr 11, 2018 at 10:10 AM, Sage Weil  wrote:
> On Tue, 10 Apr 2018, Patrick Donnelly wrote:
>> On Tue, Apr 10, 2018 at 5:54 AM, John Spray  wrote:
>> > On Tue, Apr 10, 2018 at 1:44 PM, Yan, Zheng  wrote:
>> >> Hello
>> >>
>> >> To simplify snapshot handling in multiple active mds setup, we changed
>> >> format of snaprealm in mimic dev.
>> >> https://github.com/ceph/ceph/pull/16779.
>> >>
>> >> The new version mds can handle old format snaprealm in single active
>> >> setup. It also can convert old format snaprealm to the new format when
>> >> snaprealm is modified. The problem is that new version mds can not
>> >> properly handle old format snaprealm in multiple active setup. It may
>> >> crash when it encounter old format snaprealm. For existing filesystem
>> >> with snapshots, upgrading mds to mimic seems to be no problem at first
>> >> glance. But if user later enables multiple active mds,  mds may
>> >> crashes continuously. No easy way to switch back to single acitve mds.
>> >>
>> >> I don't have clear idea how to handle this situation. I can think of a
>> >> few options.
>> >>
>> >> 1. Forbid multiple active before all old snapshots are deleted or
>> >> before all snaprealms are converted to new format. Format conversion
>> >> requires traversing while whole filesystem tree.  Not easy to
>> >> implement.
>> >
>> > This has been a general problem with metadata format changes: we can
>> > never know if all the metadata in a filesystem has been brought up to
>> > a particular version.  Scrubbing (where scrub does the updates) should
>> > be the answer, but we don't have the mechanism for recording/ensuring
>> > the scrub has really happened.
>> >
>> > Maybe we need the MDS to be able to report a complete whole-filesystem
>> > scrub to the monitor, and record a field like
>> > "latest_scrubbed_version" in FSMap, so that we can be sure that all
>> > the filesystem metadata has been brought up to a certain version
>> > before enabling certain features?  So we'd then have a
>> > "latest_scrubbed_version >= mimic" test before enabling multiple
>> > active daemons.
>> >
>> > For this particular situation, we'd also need to protect against
>> > people who had enabled multimds and snapshots experimentally, with an
>> > MDS startup check like:
>> >  ((ever_allowed_features & CEPH_MDSMAP_ALLOW_SNAPS) &&
>> > (allows_multimds() || in.size() >1)) && latest_scrubbed_version <
>> > mimic
>>
>> This sounds like the right approach to me. The mons should also be
>> capable of performing the same test and raising a health error that
>> pre-Mimic MDSs must be started and the number of actives be reduced to
>> 1.
>
> Does scrub actually do the conversion already, though, or does that need
> to be implemented?
>

need to be implemented

Regards
Yan, Zheng

> sage
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] rbd-nbd not resizing even after kernel tweaks

[Alex Gorbachev]

On Sun, Mar 11, 2018 at 3:50 PM, Alex Gorbachev  
wrote:
> On Sun, Mar 11, 2018 at 4:23 AM, Mykola Golub  wrote:
>> On Sat, Mar 10, 2018 at 08:25:15PM -0500, Alex Gorbachev wrote:
>>> I am running into the problem described in
>>> https://lkml.org/lkml/2018/2/19/565 and
>>> https://tracker.ceph.com/issues/23137
>>>
>>> I went ahead and built a custom kernel reverting the change
>>> https://github.com/torvalds/linux/commit/639812a1ed9bf49ae2c026086fbf975339cd1eef
>>>
>>> After that a resize shows in lsblk and /sys/block/nbdX/size, but not
>>> in parted for a mounted filesystem.
>>>
>>> Unmapping and remapping the NBD device shows the size in parted.
>>
>> Note 639812a is only a part of changes. The more invasive changes are
>> in 29eaadc [1]. For me the most suspicious looks removing
>> bd_set_size() in nbd_size_update(), but this is just a wild guess.
>>
>> I would recommend to contact the authors of the change. This would
>> also be a gentle remainder for Josef that he promised to fix this.
>>
>> [1] 
>> https://github.com/torvalds/linux/commit/29eaadc0364943b6352e8994158febcb699c9f9b
>


So Josef fixed the one issue that enables e.g. lsblk and sysfs size to
reflect the correct siz on change.  However, partptobe and parted
still do not detect the change, complete unmap and remap of rbd-nbd
device and remount of the filesystem is required.

In limbo with this issue at this point.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Purged a pool, buckets remain

[Robert Stanford]

 I used this command to purge my rgw data:

 rados purge default.rgw.buckets.data --yes-i-really-really-mean-it

 Now, when I list the buckets with s3cmd, I still see the buckets (s3cmd ls
shows a listing of them.)  When I try to delete one (s3cmd rb) I get this:

  ERROR: S3 error: 404 (NoSuchKey)

 I thought maybe the names were sticking around in
default.rgw.buckets.index, so I purged that too.  But no luck, the phantom
buckets are still there.

 My questions are:

 1) How can I take care of these phantom buckets?

 2) How can I purge / delete all data in the pool, without remnants such as
these?

 Thank you
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] cephfs snapshot format upgrade

[Sage Weil]

On Tue, 10 Apr 2018, Patrick Donnelly wrote:
> On Tue, Apr 10, 2018 at 5:54 AM, John Spray  wrote:
> > On Tue, Apr 10, 2018 at 1:44 PM, Yan, Zheng  wrote:
> >> Hello
> >>
> >> To simplify snapshot handling in multiple active mds setup, we changed
> >> format of snaprealm in mimic dev.
> >> https://github.com/ceph/ceph/pull/16779.
> >>
> >> The new version mds can handle old format snaprealm in single active
> >> setup. It also can convert old format snaprealm to the new format when
> >> snaprealm is modified. The problem is that new version mds can not
> >> properly handle old format snaprealm in multiple active setup. It may
> >> crash when it encounter old format snaprealm. For existing filesystem
> >> with snapshots, upgrading mds to mimic seems to be no problem at first
> >> glance. But if user later enables multiple active mds,  mds may
> >> crashes continuously. No easy way to switch back to single acitve mds.
> >>
> >> I don't have clear idea how to handle this situation. I can think of a
> >> few options.
> >>
> >> 1. Forbid multiple active before all old snapshots are deleted or
> >> before all snaprealms are converted to new format. Format conversion
> >> requires traversing while whole filesystem tree.  Not easy to
> >> implement.
> >
> > This has been a general problem with metadata format changes: we can
> > never know if all the metadata in a filesystem has been brought up to
> > a particular version.  Scrubbing (where scrub does the updates) should
> > be the answer, but we don't have the mechanism for recording/ensuring
> > the scrub has really happened.
> >
> > Maybe we need the MDS to be able to report a complete whole-filesystem
> > scrub to the monitor, and record a field like
> > "latest_scrubbed_version" in FSMap, so that we can be sure that all
> > the filesystem metadata has been brought up to a certain version
> > before enabling certain features?  So we'd then have a
> > "latest_scrubbed_version >= mimic" test before enabling multiple
> > active daemons.
> >
> > For this particular situation, we'd also need to protect against
> > people who had enabled multimds and snapshots experimentally, with an
> > MDS startup check like:
> >  ((ever_allowed_features & CEPH_MDSMAP_ALLOW_SNAPS) &&
> > (allows_multimds() || in.size() >1)) && latest_scrubbed_version <
> > mimic
> 
> This sounds like the right approach to me. The mons should also be
> capable of performing the same test and raising a health error that
> pre-Mimic MDSs must be started and the number of actives be reduced to
> 1.

Does scrub actually do the conversion already, though, or does that need 
to be implemented?

sage
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] cephfs snapshot format upgrade

[Patrick Donnelly]

On Tue, Apr 10, 2018 at 5:54 AM, John Spray  wrote:
> On Tue, Apr 10, 2018 at 1:44 PM, Yan, Zheng  wrote:
>> Hello
>>
>> To simplify snapshot handling in multiple active mds setup, we changed
>> format of snaprealm in mimic dev.
>> https://github.com/ceph/ceph/pull/16779.
>>
>> The new version mds can handle old format snaprealm in single active
>> setup. It also can convert old format snaprealm to the new format when
>> snaprealm is modified. The problem is that new version mds can not
>> properly handle old format snaprealm in multiple active setup. It may
>> crash when it encounter old format snaprealm. For existing filesystem
>> with snapshots, upgrading mds to mimic seems to be no problem at first
>> glance. But if user later enables multiple active mds,  mds may
>> crashes continuously. No easy way to switch back to single acitve mds.
>>
>> I don't have clear idea how to handle this situation. I can think of a
>> few options.
>>
>> 1. Forbid multiple active before all old snapshots are deleted or
>> before all snaprealms are converted to new format. Format conversion
>> requires traversing while whole filesystem tree.  Not easy to
>> implement.
>
> This has been a general problem with metadata format changes: we can
> never know if all the metadata in a filesystem has been brought up to
> a particular version.  Scrubbing (where scrub does the updates) should
> be the answer, but we don't have the mechanism for recording/ensuring
> the scrub has really happened.
>
> Maybe we need the MDS to be able to report a complete whole-filesystem
> scrub to the monitor, and record a field like
> "latest_scrubbed_version" in FSMap, so that we can be sure that all
> the filesystem metadata has been brought up to a certain version
> before enabling certain features?  So we'd then have a
> "latest_scrubbed_version >= mimic" test before enabling multiple
> active daemons.
>
> For this particular situation, we'd also need to protect against
> people who had enabled multimds and snapshots experimentally, with an
> MDS startup check like:
>  ((ever_allowed_features & CEPH_MDSMAP_ALLOW_SNAPS) &&
> (allows_multimds() || in.size() >1)) && latest_scrubbed_version <
> mimic

This sounds like the right approach to me. The mons should also be
capable of performing the same test and raising a health error that
pre-Mimic MDSs must be started and the number of actives be reduced to
1.

-- 
Patrick Donnelly
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph luminous - troubleshooting performance issues overall DSK 100%, busy 1%

[Steven Vacaroaia]

Hi

With osd_debug increased to 5/5 I am seeing lots of these in the
ceph-osd.5.log
( newly added OSD)

Anyone know what it means ?

2018-04-10 16:05:33.317451 7f33610be700  5 osd.5 300 heartbeat:
osd_stat(43897 MB used, 545 GB avail, 588 GB total, peers [0,1,2,3,4] op
hist [0,0,0,0,0,1,0,3,0,0,2])
2018-04-10 16:05:33.455358 7f33630c2700  5 write_log_and_missing with:
dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from:
300'42, trimmed: , trimmed_dups: , clear_divergent_priors: 0
2018-04-10 16:05:33.463539 7f33638c3700  5 write_log_and_missing with:
dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from:
300'46, trimmed: , trimmed_dups: , clear_divergent_priors: 0
2018-04-10 16:05:33.470797 7f33628c1700  5 write_log_and_missing with:
dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from:
300'40, trimmed: , trimmed_dups: , clear_divergent_priors: 0
2018-04-10 16:05:33.507682 7f33628c1700  5 write_log_and_missing with:
dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from:
300'38, trimmed: , trimmed_dups: , clear_divergent_priors: 0
2018-04-10 16:05:36.995387 7f33640c4700  5 write_log_and_missing with:
dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from:
300'36, trimmed: , trimmed_dups: , clear_divergent_priors: 0
2018-04-10 16:05:37.000745 7f33648c5700  5 write_log_and_missing with:
dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from:
300'42, trimmed: , trimmed_dups: , clear_divergent_priors: 0
2018-04-10 16:05:37.005926 7f33638c3700  5 write_log_and_missing with:
dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from:
300'44, trimmed: , trimmed_dups: , clear_divergent_priors: 0
2018-04-10 16:05:37.011209 7f33640c4700  5 write_log_and_missing with:
dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from:
300'44, trimmed: , trimmed_dups: , clear_divergent_priors: 0
2018-04-10 16:05:37.016410 7f33640c4700  5 write_log_and_missing with:
dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from:
300'49, trimmed: , trimmed_dups: , clear_divergent_priors: 0
2018-04-10 16:05:37.021478 7f33640c4700  5 write_log_and_missing with:
dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from:
300'50, trimmed: , trimmed_dups: , clear_divergent_priors: 0
2018-04-10 16:05:37.038838 7f33640c4700  5 write_log_and_missing with:
dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from:
300'49, trimmed: , trimmed_dups: , clear_divergent_priors: 0
2018-04-10 16:05:37.057197 7f33628c1700  5 write_log_and_missing with:
dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from:
300'43, trimmed: , trimmed_dups: , clear_divergent_priors: 0
2018-04-10 16:05:37.084913 7f33630c2700  5 write_log_and_missing with:
dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from:
300'44, trimmed: , trimmed_dups: , clear_divergent_priors: 0
2018-04-10 16:05:37.094249 7f33638c3700  5 write_log_and_missing with:
dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from:
300'45, trimmed: , trimmed_dups: , clear_divergent_priors: 0
2018-04-10 16:05:37.095475 7f33628c1700  5 write_log_and_missing with:
dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from:
300'44, trimmed: , trimmed_dups: , clear_divergent_priors: 0
2018-04-10 16:05:37.129598 7f33628c1700  5 write_log_and_missing with:
dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from:
300'45, trimmed: , trimmed_dups: , clear_divergent_priors: 0
2018-04-10 16:05:37.148653 7f33638c3700  5 write_log_and_missing with:
dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from:
300'45, trimmed: , trimmed_dups: , clear_divergent_priors: 0
2018-04-10 16:05:38.320251 7f33630c2700  5 write_log_and_missing with:
dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from:
300'45, trimmed: , trimmed_dups: , clear_divergent_priors: 0
2018-04-10 16:05:38.327982 7f33640c4700  5 write_log_and_missing with:
dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from:
300'50, trimmed: , trimmed_dups: , clear_divergent_priors: 0
2018-04-10 16:05:38.334373 7f33638c3700  5 write_log_and_missing with:
dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from:
300'46, trimmed: , trimmed_dups: , clear_divergent_priors: 0
2018-04-10 16:05:38.344398 7f33638c3700  5 write_log_and_missing with:
dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from:
300'45, trimmed: , trimmed_dups: , clear_divergent_priors: 0
2018-04-10 16:05:38.357411 7f33638c3700  5 write_log_and_missing with:
dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from:
300'46, trimmed: , trimmed_dups: , clear_divergent_priors: 0
2018-04-10 16:05:38.374542 7f33640c4700  5 write_log_and_missing with:
dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, 

Re: [ceph-users] Dying OSDs

[Jan Marquardt]

Am 10.04.18 um 20:22 schrieb Paul Emmerich:
> Hi,
> 
> I encountered the same crash a few months ago, see
> https://tracker.ceph.com/issues/23030
> 
> Can you post the output of
> 
>    ceph osd pool ls detail -f json-pretty
> 
> 
> Paul

Yes, of course.

# ceph osd pool ls detail -f json-pretty

[
{
"pool_name": "rbd",
"flags": 1,
"flags_names": "hashpspool",
"type": 1,
"size": 3,
"min_size": 2,
"crush_rule": 0,
"object_hash": 2,
"pg_num": 768,
"pg_placement_num": 768,
"crash_replay_interval": 0,
"last_change": "91256",
"last_force_op_resend": "0",
"last_force_op_resend_preluminous": "0",
"auid": 0,
"snap_mode": "selfmanaged",
"snap_seq": 35020,
"snap_epoch": 91219,
"pool_snaps": [],
"removed_snaps":
"[1~4562,47f1~58,484a~9,4854~70,48c5~36,48fc~48,4945~d,4953~1,4957~1,495a~3,4960~1,496e~3,497a~1,4980~2,4983~3,498b~1,4997~1,49a8~1,49ae~1,49b1~2,49b4~1,49b7~1,49b9~3,49bd~5,49c3~6,49ca~5,49d1~4,49d6~1,49d8~2,49df~2,49e2~1,49e4~2,49e7~5,49ef~2,49f2~2,49f5~6,49fc~1,49fe~3,4a05~9,4a0f~4,4a14~4,4a1a~6,4a21~6,4a29~2,4a2c~3,4a30~1,4a33~5,4a39~3,4a3e~b,4a4a~1,4a4c~2,4a50~1,4a52~7,4a5a~1,4a5c~2,4a5f~4,4a64~1,4a66~2,4a69~2,4a6c~4,4a72~1,4a74~2,4a78~3,4a7c~6,4a84~2,4a87~b,4a93~4,4a99~1,4a9c~4,4aa1~7,4aa9~1,4aab~6,4ab2~2,4ab5~5,4abb~2,4abe~9,4ac8~a,4ad3~4,4ad8~13,4aec~16,4b03~6,4b0a~c,4b17~2,4b1a~3,4b1f~4,4b24~c,4b31~d,4b3f~13,4b53~1,4bfc~13ed,61e1~4a,622c~8,6235~a0,62d6~ac,63a6~2,63b2~2,63d0~2,63f7~2,6427~2,6434~10f]",
"quota_max_bytes": 0,
"quota_max_objects": 0,
"tiers": [],
"tier_of": -1,
"read_tier": -1,
"write_tier": -1,
"cache_mode": "none",
"target_max_bytes": 0,
"target_max_objects": 0,
"cache_target_dirty_ratio_micro": 0,
"cache_target_dirty_high_ratio_micro": 0,
"cache_target_full_ratio_micro": 0,
"cache_min_flush_age": 0,
"cache_min_evict_age": 0,
"erasure_code_profile": "",
"hit_set_params": {
"type": "none"
},
"hit_set_period": 0,
"hit_set_count": 0,
"use_gmt_hitset": true,
"min_read_recency_for_promote": 0,
"min_write_recency_for_promote": 0,
"hit_set_grade_decay_rate": 0,
"hit_set_search_last_n": 0,
"grade_table": [],
"stripe_width": 0,
"expected_num_objects": 0,
"fast_read": false,
"options": {},
"application_metadata": {
"rbd": {}
}
}
]

"Unfortunately" I started the crashed OSDs again in the meantime,
because the first pgs have been down before. So currently all OSDs are
running.

Regards,

Jan


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph-fuse CPU and Memory usage vs CephFS kclient

[Gregory Farnum]

On Tue, Apr 10, 2018 at 12:36 PM, Wido den Hollander  wrote:
>
>
> On 04/10/2018 09:22 PM, Gregory Farnum wrote:
>> On Tue, Apr 10, 2018 at 6:32 AM Wido den Hollander > > wrote:
>>
>> Hi,
>>
>> There have been numerous threads about this in the past, but I wanted to
>> bring this up again in a new situation.
>>
>> Running with Luminous v12.2.4 I'm seeing some odd Memory and CPU usage
>> when using the ceph-fuse client to mount a multi-MDS CephFS filesystem.
>>
>> health: HEALTH_OK
>>
>>   services:
>> mon: 3 daemons, quorum luvil,sanomat,tide
>> mgr: luvil(active), standbys: tide, sanomat
>> mds: svw-2/2/2 up  {0=luvil=up:active,1=tide=up:active}, 1
>> up:standby
>> osd: 112 osds: 111 up, 111 in
>>
>>   data:
>> pools:   2 pools, 4352 pgs
>> objects: 85549k objects, 4415 GB
>> usage:   50348 GB used, 772 TB / 821 TB avail
>> pgs: 4352 active+clean
>>
>> After running a rsync with millions of files (and some directories
>> having 1M files) a ceph-fuse process was using 44GB RSS and using
>> between 100% and 200% CPU usage.
>>
>> Looking at this FUSE client through the admin socket the objecter was
>> one of my first suspects, but it claimed to only use ~300M of data in
>> it's case spread out over tens of thousands of files.
>>
>> After unmounting and mounting again the Memory usage was gone and we
>> tried the rsync again, but it wasn't reproducible.
>>
>> The CPU usage however is, a "simple" rsync would cause ceph-fuse to use
>> up to 100% CPU.
>>
>> Switching to the kernel client (4.16 kernel) seems to solve this, but
>> the reason for using ceph-fuse in this would be the lack of a recent
>> kernel in Debian 9 in this case and the easiness to upgrade the FUSE
>> client.
>>
>> I've tried to disable all logging inside the FUSE client, but that
>> didn't help.
>>
>> When checking on the FUSE client's socket I saw that rename() operations
>> were hanging and that's something which rsync does a lot.
>>
>> At the same time I saw a getfattr() being done to the same inode by the
>> FUSE client, but to a different MDS:
>>
>> rename(): mds rank 0
>> getfattr: mds rank 1
>>
>> Although the kernel client seems to perform better it has the same
>> behavior when looking at the mdsc file in /sys
>>
>> 216729  mds0create  (unsafe)
>> #100021abbd9/.ddd.010236269.mpeg21.a0065.folia.xml.gz.AuxBQj
>> (reddata2/.ddd.010236269.mpeg21.a0065.folia.xml.gz.AuxBQj)
>>
>> 216731  mds1rename
>>  #100021abbd9/ddd.010236269.mpeg21.a0065.folia.xml.gz
>> (reddata2/ddd.010236269.mpeg21.a0065.folia.xml.gz)
>> #100021abbd9/.ddd.010236269.mpeg21.a0065.folia.xml.gz.AuxBQj
>> (reddata2/.ddd.010236269.mpeg21.a0065.folia.xml.gz.AuxBQj)
>>
>> So this is rsync talking to two MDS, one for a create and one for a
>> rename.
>>
>> Is this normal? Is this expected behavior?
>>
>>
>> If the directory got large enough to be sharded across MDSes, yes, it's
>> expected behavior. There are filesystems that attempt to recognize rsync
>> and change their normal behavior specifically to deal with this case,
>> but CephFS isn't one of them (yet, anyway).
>>
>
> Yes, that directory is rather large.
>
> I've set max_mds to 1 for now and suddenly both FUSE and the kclient are
> a lot after, not 10% but something like 80 to 100% faster.
>
> It seems like that directory was being balanced between two MDS and that
> caused a 'massive' slow down.
>
> This can probably be influenced by tuning the MDS balancer settings, but
> I am not sure yet where to start, any suggestions?

Well, you can disable directory fragmentation, but if it's happening
automatically that means it's probably necessary. You can also pin the
directory to a specific MDS, which will prevent the balancer from
moving it or its descendants around. I'd try that; it should have the
same impact on the client.
-Greg

>
> Wido
>
>> Not sure about the specifics of the client memory or CPU usage; I think
>> you'd have to profile. rsync is a pretty pessimal CephFS workload though
>> and I think I've heard about this before...
>> -Greg
>>
>>
>>
>> To me it seems like that possibly the Subtree Partitioning might be
>> interfering here, but it wanted to double check.
>>
>> Apart from that the CPU and Memory usage of the FUSE client seems very
>> high and that might be related to this.
>>
>> Any ideas?
>>
>> Thanks,
>>
>> Wido
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com 
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
___
ceph-users mailing list
ceph-users@lists.ceph.com

Re: [ceph-users] ceph-fuse CPU and Memory usage vs CephFS kclient

[Wido den Hollander]

On 04/10/2018 09:22 PM, Gregory Farnum wrote:
> On Tue, Apr 10, 2018 at 6:32 AM Wido den Hollander  > wrote:
> 
> Hi,
> 
> There have been numerous threads about this in the past, but I wanted to
> bring this up again in a new situation.
> 
> Running with Luminous v12.2.4 I'm seeing some odd Memory and CPU usage
> when using the ceph-fuse client to mount a multi-MDS CephFS filesystem.
> 
>     health: HEALTH_OK
> 
>   services:
>     mon: 3 daemons, quorum luvil,sanomat,tide
>     mgr: luvil(active), standbys: tide, sanomat
>     mds: svw-2/2/2 up  {0=luvil=up:active,1=tide=up:active}, 1
> up:standby
>     osd: 112 osds: 111 up, 111 in
> 
>   data:
>     pools:   2 pools, 4352 pgs
>     objects: 85549k objects, 4415 GB
>     usage:   50348 GB used, 772 TB / 821 TB avail
>     pgs:     4352 active+clean
> 
> After running a rsync with millions of files (and some directories
> having 1M files) a ceph-fuse process was using 44GB RSS and using
> between 100% and 200% CPU usage.
> 
> Looking at this FUSE client through the admin socket the objecter was
> one of my first suspects, but it claimed to only use ~300M of data in
> it's case spread out over tens of thousands of files.
> 
> After unmounting and mounting again the Memory usage was gone and we
> tried the rsync again, but it wasn't reproducible.
> 
> The CPU usage however is, a "simple" rsync would cause ceph-fuse to use
> up to 100% CPU.
> 
> Switching to the kernel client (4.16 kernel) seems to solve this, but
> the reason for using ceph-fuse in this would be the lack of a recent
> kernel in Debian 9 in this case and the easiness to upgrade the FUSE
> client.
> 
> I've tried to disable all logging inside the FUSE client, but that
> didn't help.
> 
> When checking on the FUSE client's socket I saw that rename() operations
> were hanging and that's something which rsync does a lot.
> 
> At the same time I saw a getfattr() being done to the same inode by the
> FUSE client, but to a different MDS:
> 
> rename(): mds rank 0
> getfattr: mds rank 1
> 
> Although the kernel client seems to perform better it has the same
> behavior when looking at the mdsc file in /sys
> 
> 216729  mds0    create  (unsafe)
> #100021abbd9/.ddd.010236269.mpeg21.a0065.folia.xml.gz.AuxBQj
> (reddata2/.ddd.010236269.mpeg21.a0065.folia.xml.gz.AuxBQj)
> 
> 216731  mds1    rename 
>  #100021abbd9/ddd.010236269.mpeg21.a0065.folia.xml.gz
> (reddata2/ddd.010236269.mpeg21.a0065.folia.xml.gz)
> #100021abbd9/.ddd.010236269.mpeg21.a0065.folia.xml.gz.AuxBQj
> (reddata2/.ddd.010236269.mpeg21.a0065.folia.xml.gz.AuxBQj)
> 
> So this is rsync talking to two MDS, one for a create and one for a
> rename.
> 
> Is this normal? Is this expected behavior?
> 
> 
> If the directory got large enough to be sharded across MDSes, yes, it's
> expected behavior. There are filesystems that attempt to recognize rsync
> and change their normal behavior specifically to deal with this case,
> but CephFS isn't one of them (yet, anyway).
> 

Yes, that directory is rather large.

I've set max_mds to 1 for now and suddenly both FUSE and the kclient are
a lot after, not 10% but something like 80 to 100% faster.

It seems like that directory was being balanced between two MDS and that
caused a 'massive' slow down.

This can probably be influenced by tuning the MDS balancer settings, but
I am not sure yet where to start, any suggestions?

Wido

> Not sure about the specifics of the client memory or CPU usage; I think
> you'd have to profile. rsync is a pretty pessimal CephFS workload though
> and I think I've heard about this before...
> -Greg
>  
> 
> 
> To me it seems like that possibly the Subtree Partitioning might be
> interfering here, but it wanted to double check.
> 
> Apart from that the CPU and Memory usage of the FUSE client seems very
> high and that might be related to this.
> 
> Any ideas?
> 
> Thanks,
> 
> Wido
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com 
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] cephfs snapshot format upgrade

[Gregory Farnum]

On Tue, Apr 10, 2018 at 5:54 AM, John Spray  wrote:
> On Tue, Apr 10, 2018 at 1:44 PM, Yan, Zheng  wrote:
>> Hello
>>
>> To simplify snapshot handling in multiple active mds setup, we changed
>> format of snaprealm in mimic dev.
>> https://github.com/ceph/ceph/pull/16779.
>>
>> The new version mds can handle old format snaprealm in single active
>> setup. It also can convert old format snaprealm to the new format when
>> snaprealm is modified. The problem is that new version mds can not
>> properly handle old format snaprealm in multiple active setup. It may
>> crash when it encounter old format snaprealm. For existing filesystem
>> with snapshots, upgrading mds to mimic seems to be no problem at first
>> glance. But if user later enables multiple active mds,  mds may
>> crashes continuously. No easy way to switch back to single acitve mds.
>>
>> I don't have clear idea how to handle this situation. I can think of a
>> few options.
>>
>> 1. Forbid multiple active before all old snapshots are deleted or
>> before all snaprealms are converted to new format. Format conversion
>> requires traversing while whole filesystem tree.  Not easy to
>> implement.
>
> This has been a general problem with metadata format changes: we can
> never know if all the metadata in a filesystem has been brought up to
> a particular version.  Scrubbing (where scrub does the updates) should
> be the answer, but we don't have the mechanism for recording/ensuring
> the scrub has really happened.
>
> Maybe we need the MDS to be able to report a complete whole-filesystem
> scrub to the monitor, and record a field like
> "latest_scrubbed_version" in FSMap, so that we can be sure that all
> the filesystem metadata has been brought up to a certain version
> before enabling certain features?  So we'd then have a
> "latest_scrubbed_version >= mimic" test before enabling multiple
> active daemons.

Don't we have a (recursive!) last_scrub_[stamp|version] on all
directories? There's not (yet) a mechanism for associating that with
specific data versions like you describe here, but for a one-time
upgrade with unsupported features I don't think we need anything too
sophisticated.
-Greg

>
> For this particular situation, we'd also need to protect against
> people who had enabled multimds and snapshots experimentally, with an
> MDS startup check like:
>  ((ever_allowed_features & CEPH_MDSMAP_ALLOW_SNAPS) &&
> (allows_multimds() || in.size() >1)) && latest_scrubbed_version <
> mimic
>
> John
>
>> 2. Ask user to delete all old snapshots before upgrading to mimic,
>> make mds just ignore old format snaprealms.
>>
>>
>> Regards
>> Yan, Zheng
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph luminous - troubleshooting performance issues overall DSK 100%, busy 1%

[Steven Vacaroaia]

I've just added another server ( same specs) with one osd and the behavior
is the same - bad performance ..cur MB/s 0
Check network with iperf3 ..no issues

So it is not a server issue since I am getting same behavior with 2
different servers

... but I checked network with iperf3 ..no issues

What can it be  ?

ceph osd df tree
ID CLASS WEIGHT  REWEIGHT SIZE  USEAVAIL %USE  VAR  PGS TYPE NAME
-1   3.44714-  588G 80693M  509G 00   - root default
-9   0.57458-  588G 80693M  509G 13.39 1.13   - host osd01
 5   hdd 0.57458  1.0  588G 80693M  509G 13.39 1.13  64 osd.5
-7   1.14899- 1176G   130G 1046G 11.06 0.94   - host osd02
 0   hdd 0.57500  1.0  588G 70061M  519G 11.63 0.98  50 osd.0
 1   hdd 0.57500  1.0  588G 63200M  526G 10.49 0.89  41 osd.1
-3   1.14899- 1176G   138G 1038G 11.76 1.00   - host osd03
 2   hdd 0.57500  1.0  588G 68581M  521G 11.38 0.96  48 osd.2
 3   hdd 0.57500  1.0  588G 73185M  516G 12.15 1.03  53 osd.3
-4   0.57458- 0  0 0 00   - host osd04
 4   hdd 0.574580 0  0 0 00   0 osd.4

2018-04-10 15:11:58.542507 min lat: 0.0201432 max lat: 13.9308 avg lat:
0.466235
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg
lat(s)
   40  16  1294  1278   127.785 0   -
0.466235
   41  16  1294  1278   124.668 0   -
0.466235
   42  16  1294  1278 121.7 0   -
0.466235
   43  16  1294  1278118.87 0   -
0.466235
   44  16  1302  1286   116.896   6.4   0.0302793
0.469203
   45  16  1395  1379   122.564   3720.312525
 0.51994
   46  16  1458  1442   125.377   252   0.0387492
0.501892
   47  16  1458  1442   122.709 0   -
0.501892
   48  16  1458  1442   120.153 0   -
0.501892
   49  16  1458  1442   117.701 0   -
0.501892
   50  16  1522  1506   120.466640.137913
0.516969
   51  16  1522  1506   118.104 0   -
0.516969
   52  16  1522  1506   115.833 0   -
0.516969
   53  16  1522  1506   113.648 0   -
0.516969
   54  16  1522  1506   111.543 0   -
0.516969
   55  16  1522  1506   109.515 0   -
0.516969
   56  16  1522  1506   107.559 0   -
0.516969
   57  16  1522  1506   105.672 0   -
0.516969
   58  16  1522  1506   103.851 0   -
0.516969
Total time run:   58.927431
Total reads made: 1522
Read size:4194304
Object size:  4194304
Bandwidth (MB/sec):   103.314
Average IOPS: 25
Stddev IOPS:  35
Max IOPS: 106
Min IOPS: 0
Average Latency(s):   0.618812
Max latency(s):   13.9308
Min latency(s):   0.0201432


iperf3 -c 192.168.0.181 -i1 -t 10
Connecting to host 192.168.0.181, port 5201
[  4] local 192.168.0.182 port 57448 connected to 192.168.0.181 port 5201
[ ID] Interval   Transfer Bandwidth   Retr  Cwnd
[  4]   0.00-1.00   sec  1.15 GBytes  9.92 Gbits/sec0830 KBytes
[  4]   1.00-2.00   sec  1.15 GBytes  9.90 Gbits/sec0830 KBytes
[  4]   2.00-3.00   sec  1.15 GBytes  9.91 Gbits/sec0918 KBytes
[  4]   3.00-4.00   sec  1.15 GBytes  9.90 Gbits/sec0918 KBytes
[  4]   4.00-5.00   sec  1.15 GBytes  9.90 Gbits/sec0918 KBytes
[  4]   5.00-6.00   sec  1.15 GBytes  9.90 Gbits/sec0918 KBytes
[  4]   6.00-7.00   sec  1.15 GBytes  9.90 Gbits/sec0918 KBytes
[  4]   7.00-8.00   sec  1.15 GBytes  9.90 Gbits/sec0918 KBytes
[  4]   8.00-9.00   sec  1.15 GBytes  9.90 Gbits/sec0918 KBytes
[  4]   9.00-10.00  sec  1.15 GBytes  9.91 Gbits/sec0918 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval   Transfer Bandwidth   Retr
[  4]   0.00-10.00  sec  11.5 GBytes  9.90 Gbits/sec0 sender
[  4]   0.00-10.00  sec  11.5 GBytes  9.90 Gbits/sec
receiver




On Tue, 10 Apr 2018 at 08:49, Steven Vacaroaia  wrote:

> Hi,
> Thanks for providing guidance
>
> VD0 is the SSD drive
> Many people suggested to not enable WB for SSD so that cache can be used
> for HDD where is needed more
>
> Setup is 3 identical DELL R620 server OSD01, OSD02, OSD04
> 10 GB  separate networks, 600 GB Entreprise HDD , 320 GB Entreprise SSD
> Blustore, separate WAL / DB on SSD ( 1 GB partition for WAL, 30GB for DB)
>
> With 2 OSD per servers and only OSD01, OSD02 , performance is as expected
> ( no gaps CUR MB/s )
>
> Adding one OSD from OSD04, tanks performance ( lots of gaps CUR MB/s 0 )
>
> See below
>
> ceph -s
>   

Re: [ceph-users] Ceph scrub logs: _scan_snaps no head for $object?

[Marc Roos]

 
Hi Paul,

This is a small test cluster, and the rbd pool is replicated. I am 
hardly using any clients on the cluster. Furthermore I have been the 
only one creating the snapshots and I know for sure that I was not 
trying to delete them. If so I have been doing this on one of the ceph 
nodes.

I have these issues on images with
create_timestamp: Tue Jul 18 20:51:40 2017
create_timestamp: Fri Sep  1 13:55:25 2017
create_timestamp: Fri Sep  1 13:59:10 2017
create_timestamp: Wed Jan  3 16:38:57 2018

Updates have been done in February, so theoretically I should not be 
seeing these than any more?
Feb 21 15:13:35 Updated: 2:ceph-osd-12.2.3-0.el7.x86_64
Feb 28 13:33:27 Updated: 2:ceph-osd-12.2.4-0.el7.x86_64

How can I determine what snapshot is bad of this image? 
Should this snapshot be considered lost?
And is deleting this snapshot the only way to fix this? 


-Original Message-
From: Paul Emmerich [mailto:paul.emmer...@croit.io] 
Sent: dinsdag 10 april 2018 20:14
To: Marc Roos
Cc: ceph-users
Subject: Re: [ceph-users] Ceph scrub logs: _scan_snaps no head for 
$object?

Hi,


you'll usually see this if there are "orphaned" snapshot objects. One 
common cause for this are
pre-12.2.2 clients trying to delete RBD snapshots with a data pool 
(i.e., erasure coded pools) They send the snapshot requests to the wrong 
pool and you end up with lots of problems.



Paul


2018-04-09 16:55 GMT+02:00 Marc Roos :



I have this on a rbd pool with images/snapshots that have been 
created
in Luminous

> Hi Stefan, Mehmet,
>
> Are these clusters that were upgraded from prior versions, or 
fresh
> luminous installs?
>
>
> This message indicates that there is a stray clone object with no
> associated head or snapdir object.  That normally should never
> happen--it's presumably the result of a (hopefully old) bug.  The
scrub
> process doesn't even clean them up, which maybe says something 
about
how
> common it is/was...
>
> sage

>




___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 
 





-- 

--
Paul Emmerich

croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph-fuse CPU and Memory usage vs CephFS kclient

[Gregory Farnum]

On Tue, Apr 10, 2018 at 6:32 AM Wido den Hollander  wrote:

> Hi,
>
> There have been numerous threads about this in the past, but I wanted to
> bring this up again in a new situation.
>
> Running with Luminous v12.2.4 I'm seeing some odd Memory and CPU usage
> when using the ceph-fuse client to mount a multi-MDS CephFS filesystem.
>
> health: HEALTH_OK
>
>   services:
> mon: 3 daemons, quorum luvil,sanomat,tide
> mgr: luvil(active), standbys: tide, sanomat
> mds: svw-2/2/2 up  {0=luvil=up:active,1=tide=up:active}, 1 up:standby
> osd: 112 osds: 111 up, 111 in
>
>   data:
> pools:   2 pools, 4352 pgs
> objects: 85549k objects, 4415 GB
> usage:   50348 GB used, 772 TB / 821 TB avail
> pgs: 4352 active+clean
>
> After running a rsync with millions of files (and some directories
> having 1M files) a ceph-fuse process was using 44GB RSS and using
> between 100% and 200% CPU usage.
>
> Looking at this FUSE client through the admin socket the objecter was
> one of my first suspects, but it claimed to only use ~300M of data in
> it's case spread out over tens of thousands of files.
>
> After unmounting and mounting again the Memory usage was gone and we
> tried the rsync again, but it wasn't reproducible.
>
> The CPU usage however is, a "simple" rsync would cause ceph-fuse to use
> up to 100% CPU.
>
> Switching to the kernel client (4.16 kernel) seems to solve this, but
> the reason for using ceph-fuse in this would be the lack of a recent
> kernel in Debian 9 in this case and the easiness to upgrade the FUSE
> client.
>
> I've tried to disable all logging inside the FUSE client, but that
> didn't help.
>
> When checking on the FUSE client's socket I saw that rename() operations
> were hanging and that's something which rsync does a lot.
>
> At the same time I saw a getfattr() being done to the same inode by the
> FUSE client, but to a different MDS:
>
> rename(): mds rank 0
> getfattr: mds rank 1
>
> Although the kernel client seems to perform better it has the same
> behavior when looking at the mdsc file in /sys
>
> 216729  mds0create  (unsafe)
> #100021abbd9/.ddd.010236269.mpeg21.a0065.folia.xml.gz.AuxBQj
> (reddata2/.ddd.010236269.mpeg21.a0065.folia.xml.gz.AuxBQj)
>
> 216731  mds1rename
>  #100021abbd9/ddd.010236269.mpeg21.a0065.folia.xml.gz
> (reddata2/ddd.010236269.mpeg21.a0065.folia.xml.gz)
> #100021abbd9/.ddd.010236269.mpeg21.a0065.folia.xml.gz.AuxBQj
> (reddata2/.ddd.010236269.mpeg21.a0065.folia.xml.gz.AuxBQj)
>
> So this is rsync talking to two MDS, one for a create and one for a rename.
>
> Is this normal? Is this expected behavior?
>

If the directory got large enough to be sharded across MDSes, yes, it's
expected behavior. There are filesystems that attempt to recognize rsync
and change their normal behavior specifically to deal with this case, but
CephFS isn't one of them (yet, anyway).

Not sure about the specifics of the client memory or CPU usage; I think
you'd have to profile. rsync is a pretty pessimal CephFS workload though
and I think I've heard about this before...
-Greg


>
> To me it seems like that possibly the Subtree Partitioning might be
> interfering here, but it wanted to double check.
>
> Apart from that the CPU and Memory usage of the FUSE client seems very
> high and that might be related to this.
>
> Any ideas?
>
> Thanks,
>
> Wido
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Move ceph admin node to new other server

[Nghia Than]

I appreciate for your kind Paul.

On Wed, Apr 11, 2018 at 1:47 AM, Paul Emmerich 
wrote:

> http://docs.ceph.com/ceph-deploy/docs/gatherkeys.html
>
> 2018-04-10 20:39 GMT+02:00 Nghia Than :
>
>> Hi Paul,
>>
>> Thanks for your information.
>>
>> May i know if i destroy this node so how they can gatherkeys as this node
>> is already terminated and no data available. As you said it will get from
>> cluster so will they get them all or i have to manual backup them (surely i
>> will do this task before terminate any server) and copy to new node.
>>
>> Thanks,
>>
>>
>> On Wed, Apr 11, 2018 at 1:25 AM, Paul Emmerich 
>> wrote:
>>
>>> Hi,
>>>
>>> yes, that folder contains everything you need. You can also use
>>> ceph-deploy gatherkeys to get them from your cluster.
>>>
>>>
>>> Paul
>>>
>>>
>>> 2018-04-09 10:04 GMT+02:00 Nghia Than :
>>>
 Hello,

 We have use 1 server for deploy (called ceph-admin-node) for 3 mon and
 4 OSD node.

 We have created a folder called *ceph-deploy* to deploy all node
 members. May we move this folder to other server?

 This folder contains all following files:

 total 1408
 -rw--- 1 root root 113 Oct 26 16:48 ceph.bootstrap-mds.keyring
 -rw--- 1 root root  71 Oct 26 16:48 ceph.bootstrap-mgr.keyring
 -rw--- 1 root root 113 Oct 26 16:48 ceph.bootstrap-osd.keyring
 -rw--- 1 root root 113 Oct 26 16:48 ceph.bootstrap-rgw.keyring
 -rw--- 1 root root 129 Oct 26 16:48 ceph.client.admin.keyring
 -rw-r--r-- 1 root root 818 Oct 26 12:20 ceph.conf
 -rw-r--r-- 1 root root 1405043 Apr  2 14:16 ceph-deploy-ceph.log
 -rw--- 1 root root  73 Oct 26 12:19 ceph.mon.keyring

 Any guide for this task?

 Thanks,

 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


>>>
>>>
>>> --
>>> --
>>> Paul Emmerich
>>>
>>> croit GmbH
>>> Freseniusstr. 31h
>>> 
>>> 81247 München
>>> 
>>> www.croit.io
>>> Tel: +49 89 1896585 90
>>>
>>
>>
>>
>> --
>> ==
>> Nghia Than
>>
>
>
>
> --
> --
> Paul Emmerich
>
> croit GmbH
> Freseniusstr. 31h
> 
> 81247 München
> 
> www.croit.io
> Tel: +49 89 1896585 90
>



-- 
==
Nghia Than
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Move ceph admin node to new other server

[Paul Emmerich]

http://docs.ceph.com/ceph-deploy/docs/gatherkeys.html

2018-04-10 20:39 GMT+02:00 Nghia Than :

> Hi Paul,
>
> Thanks for your information.
>
> May i know if i destroy this node so how they can gatherkeys as this node
> is already terminated and no data available. As you said it will get from
> cluster so will they get them all or i have to manual backup them (surely i
> will do this task before terminate any server) and copy to new node.
>
> Thanks,
>
>
> On Wed, Apr 11, 2018 at 1:25 AM, Paul Emmerich 
> wrote:
>
>> Hi,
>>
>> yes, that folder contains everything you need. You can also use
>> ceph-deploy gatherkeys to get them from your cluster.
>>
>>
>> Paul
>>
>>
>> 2018-04-09 10:04 GMT+02:00 Nghia Than :
>>
>>> Hello,
>>>
>>> We have use 1 server for deploy (called ceph-admin-node) for 3 mon and 4
>>> OSD node.
>>>
>>> We have created a folder called *ceph-deploy* to deploy all node
>>> members. May we move this folder to other server?
>>>
>>> This folder contains all following files:
>>>
>>> total 1408
>>> -rw--- 1 root root 113 Oct 26 16:48 ceph.bootstrap-mds.keyring
>>> -rw--- 1 root root  71 Oct 26 16:48 ceph.bootstrap-mgr.keyring
>>> -rw--- 1 root root 113 Oct 26 16:48 ceph.bootstrap-osd.keyring
>>> -rw--- 1 root root 113 Oct 26 16:48 ceph.bootstrap-rgw.keyring
>>> -rw--- 1 root root 129 Oct 26 16:48 ceph.client.admin.keyring
>>> -rw-r--r-- 1 root root 818 Oct 26 12:20 ceph.conf
>>> -rw-r--r-- 1 root root 1405043 Apr  2 14:16 ceph-deploy-ceph.log
>>> -rw--- 1 root root  73 Oct 26 12:19 ceph.mon.keyring
>>>
>>> Any guide for this task?
>>>
>>> Thanks,
>>>
>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>
>>>
>>
>>
>> --
>> --
>> Paul Emmerich
>>
>> croit GmbH
>> Freseniusstr. 31h
>> 
>> 81247 München
>> 
>> www.croit.io
>> Tel: +49 89 1896585 90
>>
>
>
>
> --
> ==
> Nghia Than
>



-- 
-- 
Paul Emmerich

croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Move ceph admin node to new other server

[Nghia Than]

Hi Paul,

Thanks for your information.

May i know if i destroy this node so how they can gatherkeys as this node
is already terminated and no data available. As you said it will get from
cluster so will they get them all or i have to manual backup them (surely i
will do this task before terminate any server) and copy to new node.

Thanks,


On Wed, Apr 11, 2018 at 1:25 AM, Paul Emmerich 
wrote:

> Hi,
>
> yes, that folder contains everything you need. You can also use
> ceph-deploy gatherkeys to get them from your cluster.
>
>
> Paul
>
>
> 2018-04-09 10:04 GMT+02:00 Nghia Than :
>
>> Hello,
>>
>> We have use 1 server for deploy (called ceph-admin-node) for 3 mon and 4
>> OSD node.
>>
>> We have created a folder called *ceph-deploy* to deploy all node
>> members. May we move this folder to other server?
>>
>> This folder contains all following files:
>>
>> total 1408
>> -rw--- 1 root root 113 Oct 26 16:48 ceph.bootstrap-mds.keyring
>> -rw--- 1 root root  71 Oct 26 16:48 ceph.bootstrap-mgr.keyring
>> -rw--- 1 root root 113 Oct 26 16:48 ceph.bootstrap-osd.keyring
>> -rw--- 1 root root 113 Oct 26 16:48 ceph.bootstrap-rgw.keyring
>> -rw--- 1 root root 129 Oct 26 16:48 ceph.client.admin.keyring
>> -rw-r--r-- 1 root root 818 Oct 26 12:20 ceph.conf
>> -rw-r--r-- 1 root root 1405043 Apr  2 14:16 ceph-deploy-ceph.log
>> -rw--- 1 root root  73 Oct 26 12:19 ceph.mon.keyring
>>
>> Any guide for this task?
>>
>> Thanks,
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
>
>
> --
> --
> Paul Emmerich
>
> croit GmbH
> Freseniusstr. 31h
> 
> 81247 München
> 
> www.croit.io
> Tel: +49 89 1896585 90
>



-- 
==
Nghia Than
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Move ceph admin node to new other server

[Paul Emmerich]

Hi,

yes, that folder contains everything you need. You can also use ceph-deploy
gatherkeys to get them from your cluster.


Paul


2018-04-09 10:04 GMT+02:00 Nghia Than :

> Hello,
>
> We have use 1 server for deploy (called ceph-admin-node) for 3 mon and 4
> OSD node.
>
> We have created a folder called *ceph-deploy* to deploy all node members.
> May we move this folder to other server?
>
> This folder contains all following files:
>
> total 1408
> -rw--- 1 root root 113 Oct 26 16:48 ceph.bootstrap-mds.keyring
> -rw--- 1 root root  71 Oct 26 16:48 ceph.bootstrap-mgr.keyring
> -rw--- 1 root root 113 Oct 26 16:48 ceph.bootstrap-osd.keyring
> -rw--- 1 root root 113 Oct 26 16:48 ceph.bootstrap-rgw.keyring
> -rw--- 1 root root 129 Oct 26 16:48 ceph.client.admin.keyring
> -rw-r--r-- 1 root root 818 Oct 26 12:20 ceph.conf
> -rw-r--r-- 1 root root 1405043 Apr  2 14:16 ceph-deploy-ceph.log
> -rw--- 1 root root  73 Oct 26 12:19 ceph.mon.keyring
>
> Any guide for this task?
>
> Thanks,
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>


-- 
-- 
Paul Emmerich

croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Dying OSDs

[Paul Emmerich]

Hi,

I encountered the same crash a few months ago, see
https://tracker.ceph.com/issues/23030

Can you post the output of

   ceph osd pool ls detail -f json-pretty


Paul


2018-04-10 18:09 GMT+02:00 Jan Marquardt :

> Am 10.04.18 um 15:29 schrieb Brady Deetz:
> > What distribution and kernel are you running?
> >
> > I recently found my cluster running the 3.10 centos kernel when I
> > thought it was running the elrepo kernel. After forcing it to boot
> > correctly, my flapping osd issue went away.
>
> We are running on Ubuntu 16.04 and its current linux-hwe in version
> 4.13.0-32-generic.
>
> Regards
>
> Jan
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>



-- 
-- 
Paul Emmerich

croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph scrub logs: _scan_snaps no head for $object?

[Paul Emmerich]

Hi,

you'll usually see this if there are "orphaned" snapshot objects. One
common cause for this are
pre-12.2.2 clients trying to delete RBD snapshots with a data pool (i.e.,
erasure coded pools)
They send the snapshot requests to the wrong pool and you end up with lots
of problems.


Paul

2018-04-09 16:55 GMT+02:00 Marc Roos :

>
> I have this on a rbd pool with images/snapshots that have been created
> in Luminous
>
> > Hi Stefan, Mehmet,
> >
> > Are these clusters that were upgraded from prior versions, or fresh
> > luminous installs?
> >
> >
> > This message indicates that there is a stray clone object with no
> > associated head or snapdir object.  That normally should never
> > happen--it's presumably the result of a (hopefully old) bug.  The
> scrub
> > process doesn't even clean them up, which maybe says something about
> how
> > common it is/was...
> >
> > sage
> >
>
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>



-- 
-- 
Paul Emmerich

croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Dying OSDs

[Jan Marquardt]

Am 10.04.18 um 15:29 schrieb Brady Deetz:
> What distribution and kernel are you running?
> 
> I recently found my cluster running the 3.10 centos kernel when I
> thought it was running the elrepo kernel. After forcing it to boot
> correctly, my flapping osd issue went away. 

We are running on Ubuntu 16.04 and its current linux-hwe in version
4.13.0-32-generic.

Regards

Jan
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] ceph-fuse CPU and Memory usage vs CephFS kclient

[Wido den Hollander]

Hi,

There have been numerous threads about this in the past, but I wanted to
bring this up again in a new situation.

Running with Luminous v12.2.4 I'm seeing some odd Memory and CPU usage
when using the ceph-fuse client to mount a multi-MDS CephFS filesystem.

health: HEALTH_OK

  services:
mon: 3 daemons, quorum luvil,sanomat,tide
mgr: luvil(active), standbys: tide, sanomat
mds: svw-2/2/2 up  {0=luvil=up:active,1=tide=up:active}, 1 up:standby
osd: 112 osds: 111 up, 111 in

  data:
pools:   2 pools, 4352 pgs
objects: 85549k objects, 4415 GB
usage:   50348 GB used, 772 TB / 821 TB avail
pgs: 4352 active+clean

After running a rsync with millions of files (and some directories
having 1M files) a ceph-fuse process was using 44GB RSS and using
between 100% and 200% CPU usage.

Looking at this FUSE client through the admin socket the objecter was
one of my first suspects, but it claimed to only use ~300M of data in
it's case spread out over tens of thousands of files.

After unmounting and mounting again the Memory usage was gone and we
tried the rsync again, but it wasn't reproducible.

The CPU usage however is, a "simple" rsync would cause ceph-fuse to use
up to 100% CPU.

Switching to the kernel client (4.16 kernel) seems to solve this, but
the reason for using ceph-fuse in this would be the lack of a recent
kernel in Debian 9 in this case and the easiness to upgrade the FUSE client.

I've tried to disable all logging inside the FUSE client, but that
didn't help.

When checking on the FUSE client's socket I saw that rename() operations
were hanging and that's something which rsync does a lot.

At the same time I saw a getfattr() being done to the same inode by the
FUSE client, but to a different MDS:

rename(): mds rank 0
getfattr: mds rank 1

Although the kernel client seems to perform better it has the same
behavior when looking at the mdsc file in /sys

216729  mds0create  (unsafe)
#100021abbd9/.ddd.010236269.mpeg21.a0065.folia.xml.gz.AuxBQj
(reddata2/.ddd.010236269.mpeg21.a0065.folia.xml.gz.AuxBQj)

216731  mds1rename   #100021abbd9/ddd.010236269.mpeg21.a0065.folia.xml.gz
(reddata2/ddd.010236269.mpeg21.a0065.folia.xml.gz)
#100021abbd9/.ddd.010236269.mpeg21.a0065.folia.xml.gz.AuxBQj
(reddata2/.ddd.010236269.mpeg21.a0065.folia.xml.gz.AuxBQj)

So this is rsync talking to two MDS, one for a create and one for a rename.

Is this normal? Is this expected behavior?

To me it seems like that possibly the Subtree Partitioning might be
interfering here, but it wanted to double check.

Apart from that the CPU and Memory usage of the FUSE client seems very
high and that might be related to this.

Any ideas?

Thanks,

Wido
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Dying OSDs

[Brady Deetz]

What distribution and kernel are you running?

I recently found my cluster running the 3.10 centos kernel when I thought
it was running the elrepo kernel. After forcing it to boot correctly, my
flapping osd issue went away.

On Tue, Apr 10, 2018, 2:18 AM Jan Marquardt  wrote:

> Hi,
>
> we are experiencing massive problems with our Ceph setup. After starting
> a "repair pg" because of scrub errors OSDs started to crash, which we
> could not stop so far. We are running Ceph 12.2.4. Crashed OSDs are both
> bluestore and filestore.
>
> Our cluster currently looks like this:
>
> # ceph -s
>   cluster:
> id: c59e56df-2043-4c92-9492-25f05f268d9f
> health: HEALTH_ERR
> 1 osds down
> 73005/17149710 objects misplaced (0.426%)
> 5 scrub errors
> Reduced data availability: 2 pgs inactive, 2 pgs down
> Possible data damage: 1 pg inconsistent
> Degraded data redundancy: 611518/17149710 objects degraded
> (3.566%), 86 pgs degraded, 86 pgs undersized
>
>   services:
> mon: 3 daemons, quorum head1,head2,head3
> mgr: head3(active), standbys: head2, head1
> osd: 34 osds: 24 up, 25 in; 18 remapped pgs
>
>   data:
> pools:   1 pools, 768 pgs
> objects: 5582k objects, 19500 GB
> usage:   62030 GB used, 31426 GB / 93456 GB avail
> pgs: 0.260% pgs not active
>  611518/17149710 objects degraded (3.566%)
>  73005/17149710 objects misplaced (0.426%)
>  670 active+clean
>  75  active+undersized+degraded
>  8   active+undersized+degraded+remapped+backfill_wait
>  8   active+clean+remapped
>  2   down
>  2   active+undersized+degraded+remapped+backfilling
>  2   active+clean+scrubbing+deep
>  1   active+undersized+degraded+inconsistent
>
>   io:
> client:   10911 B/s rd, 118 kB/s wr, 0 op/s rd, 54 op/s wr
> recovery: 31575 kB/s, 8 objects/s
>
> # ceph osd tree
> ID  CLASS WEIGHTTYPE NAME  STATUS REWEIGHT PRI-AFF
>  -1   124.07297 root default
>  -229.08960 host ceph1
>   0   hdd   3.63620 osd.0  up  1.0 1.0
>   1   hdd   3.63620 osd.1down0 1.0
>   2   hdd   3.63620 osd.2  up  1.0 1.0
>   3   hdd   3.63620 osd.3  up  1.0 1.0
>   4   hdd   3.63620 osd.4down0 1.0
>   5   hdd   3.63620 osd.5down0 1.0
>   6   hdd   3.63620 osd.6  up  1.0 1.0
>   7   hdd   3.63620 osd.7  up  1.0 1.0
>  -3 7.27240 host ceph2
>  14   hdd   3.63620 osd.14 up  1.0 1.0
>  15   hdd   3.63620 osd.15 up  1.0 1.0
>  -429.11258 host ceph3
>  16   hdd   3.63620 osd.16 up  1.0 1.0
>  18   hdd   3.63620 osd.18   down0 1.0
>  19   hdd   3.63620 osd.19   down0 1.0
>  20   hdd   3.65749 osd.20 up  1.0 1.0
>  21   hdd   3.63620 osd.21 up  1.0 1.0
>  22   hdd   3.63620 osd.22 up  1.0 1.0
>  23   hdd   3.63620 osd.23 up  1.0 1.0
>  24   hdd   3.63789 osd.24   down0 1.0
>  -929.29919 host ceph4
>  17   hdd   3.66240 osd.17 up  1.0 1.0
>  25   hdd   3.66240 osd.25 up  1.0 1.0
>  26   hdd   3.66240 osd.26   down0 1.0
>  27   hdd   3.66240 osd.27 up  1.0 1.0
>  28   hdd   3.66240 osd.28   down0 1.0
>  29   hdd   3.66240 osd.29 up  1.0 1.0
>  30   hdd   3.66240 osd.30 up  1.0 1.0
>  31   hdd   3.66240 osd.31   down0 1.0
> -1129.29919 host ceph5
>  32   hdd   3.66240 osd.32 up  1.0 1.0
>  33   hdd   3.66240 osd.33 up  1.0 1.0
>  34   hdd   3.66240 osd.34 up  1.0 1.0
>  35   hdd   3.66240 osd.35 up  1.0 1.0
>  36   hdd   3.66240 osd.36   down  1.0 1.0
>  37   hdd   3.66240 osd.37 up  1.0 1.0
>  38   hdd   3.66240 osd.38 up  1.0 1.0
>  39   hdd   3.66240 osd.39 up  1.0 1.0
>
> The last OSDs that crashed are #28 and #36. Please find the
> corresponding log files here:
>
> http://af.janno.io/ceph/ceph-osd.28.log.1.gz
> http://af.janno.io/ceph/ceph-osd.36.log.1.gz
>
> The backtraces look almost the same for all crashed OSDs.
>
> Any help, hint or advice would really be appreciated. Please let me know
> if you need any further information.
>
> Best Regards
>
> Jan
>
> --
> Artfiles New Media GmbH | Zirkusweg 1 | 20359 Hamburg
> Tel: 040 - 32 02 72 90 | Fax: 040 - 32 02 72 95
> E-Mail: supp...@artfiles.de | Web: http://www.artfiles.de
> Geschäftsführer: Harald Oltmanns | Tim Evers
> 

Re: [ceph-users] cephfs snapshot format upgrade

[John Spray]

On Tue, Apr 10, 2018 at 1:44 PM, Yan, Zheng  wrote:
> Hello
>
> To simplify snapshot handling in multiple active mds setup, we changed
> format of snaprealm in mimic dev.
> https://github.com/ceph/ceph/pull/16779.
>
> The new version mds can handle old format snaprealm in single active
> setup. It also can convert old format snaprealm to the new format when
> snaprealm is modified. The problem is that new version mds can not
> properly handle old format snaprealm in multiple active setup. It may
> crash when it encounter old format snaprealm. For existing filesystem
> with snapshots, upgrading mds to mimic seems to be no problem at first
> glance. But if user later enables multiple active mds,  mds may
> crashes continuously. No easy way to switch back to single acitve mds.
>
> I don't have clear idea how to handle this situation. I can think of a
> few options.
>
> 1. Forbid multiple active before all old snapshots are deleted or
> before all snaprealms are converted to new format. Format conversion
> requires traversing while whole filesystem tree.  Not easy to
> implement.

This has been a general problem with metadata format changes: we can
never know if all the metadata in a filesystem has been brought up to
a particular version.  Scrubbing (where scrub does the updates) should
be the answer, but we don't have the mechanism for recording/ensuring
the scrub has really happened.

Maybe we need the MDS to be able to report a complete whole-filesystem
scrub to the monitor, and record a field like
"latest_scrubbed_version" in FSMap, so that we can be sure that all
the filesystem metadata has been brought up to a certain version
before enabling certain features?  So we'd then have a
"latest_scrubbed_version >= mimic" test before enabling multiple
active daemons.

For this particular situation, we'd also need to protect against
people who had enabled multimds and snapshots experimentally, with an
MDS startup check like:
 ((ever_allowed_features & CEPH_MDSMAP_ALLOW_SNAPS) &&
(allows_multimds() || in.size() >1)) && latest_scrubbed_version <
mimic

John

> 2. Ask user to delete all old snapshots before upgrading to mimic,
> make mds just ignore old format snaprealms.
>
>
> Regards
> Yan, Zheng
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph luminous - troubleshooting performance issues overall DSK 100%, busy 1%

[Steven Vacaroaia]

Hi,
Thanks for providing guidance

VD0 is the SSD drive
Many people suggested to not enable WB for SSD so that cache can be used
for HDD where is needed more

Setup is 3 identical DELL R620 server OSD01, OSD02, OSD04
10 GB  separate networks, 600 GB Entreprise HDD , 320 GB Entreprise SSD
Blustore, separate WAL / DB on SSD ( 1 GB partition for WAL, 30GB for DB)

With 2 OSD per servers and only OSD01, OSD02 , performance is as expected (
no gaps CUR MB/s )

Adding one OSD from OSD04, tanks performance ( lots of gaps CUR MB/s 0 )

See below

ceph -s
  cluster:
id: 1e98e57a-ef41-4327-b88a-dd2531912632
health: HEALTH_WARN
noscrub,nodeep-scrub flag(s) set




WITH OSD04

ceph osd tree
ID CLASS WEIGHT  TYPE NAME  STATUS REWEIGHT PRI-AFF
-1   2.87256 root default
-7   1.14899 host osd02
 0   hdd 0.57500 osd.0  up  1.0 1.0
 1   hdd 0.57500 osd.1  up  1.0 1.0
-3   1.14899 host osd03
 2   hdd 0.57500 osd.2  up  1.0 1.0
 3   hdd 0.57500 osd.3  up  1.0 1.0
-4   0.57458 host osd04
 4   hdd 0.57458 osd.4  up  1.0 1.0


2018-04-10 08:37:08.111037 min lat: 0.0128562 max lat: 13.1623 avg lat:
0.528273
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg
lat(s)
  100  16  3001  2985   119.38890   0.0169507
0.528273
  101  16  3029  3013   119.315   112   0.0410565
0.524325
  102  16  3029  3013   118.145 0   -
0.524325
  103  16  3029  3013   116.998 0   -
0.524325
  104  16  3029  3013   115.873 0   -
0.524325
  105  16  3071  3055116.3742   0.0888923
 0.54832
  106  16  3156  3140   118.479   340   0.0162464
0.535244
  107  16  3156  3140   117.372 0   -
0.535244
  108  16  3156  3140   116.285 0   -
0.535244
  109  16  3156  3140   115.218 0   -
0.535244
  110  16  3156  3140   114.171 0   -
0.535244
  111  16  3156  3140   113.142 0   -
0.535244
  112  16  3156  3140   112.132 0   -
0.535244
  113  16  3156  3140111.14 0   -
0.535244
  114  16  3156  3140   110.165 0   -
0.535244
  115  16  3156  3140   109.207 0   -
0.535244
  116  16  3230  3214   110.817  29.6   0.0169969
0.574856
  117  16  3311  3295   112.639   324   0.0704851
0.565529
  118  16  3311  3295   111.684 0   -
0.565529
  119  16  3311  3295   110.746 0   -
0.565529
2018-04-10 08:37:28.112886 min lat: 0.0128562 max lat: 14.7293 avg lat:
0.565529
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg
lat(s)
  120  16  3311  3295   109.823 0   -
0.565529
  121  16  3311  3295   108.915 0   -
0.565529
  122  16  3311  3295   108.022 0   -
0.565529
Total time run: 122.568983
Total writes made:  3312
Write size: 4194304
Object size:4194304
Bandwidth (MB/sec): 108.086
Stddev Bandwidth:   121.191
Max bandwidth (MB/sec): 520
Min bandwidth (MB/sec): 0
Average IOPS:   27
Stddev IOPS:30
Max IOPS:   130
Min IOPS:   0
Average Latency(s): 0.591771
Stddev Latency(s):  1.74753
Max latency(s): 14.7293
Min latency(s): 0.0128562


AFTER ceph osd down osd.4; ceph osd out osd.4

ceph osd tree
ID CLASS WEIGHT  TYPE NAME  STATUS REWEIGHT PRI-AFF
-1   2.87256 root default
-7   1.14899 host osd02
 0   hdd 0.57500 osd.0  up  1.0 1.0
 1   hdd 0.57500 osd.1  up  1.0 1.0
-3   1.14899 host osd03
 2   hdd 0.57500 osd.2  up  1.0 1.0
 3   hdd 0.57500 osd.3  up  1.0 1.0
-4   0.57458 host osd04
 4   hdd 0.57458 osd.4  up0 1.0


2018-04-10 08:46:55.193642 min lat: 0.0156532 max lat: 2.5884 avg lat:
0.310681
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg
lat(s)
  100  16  5144  5128   205.097   220   0.037
0.310681
  101  16  5196  5180   205.126   2080.421245
0.310908
  102  16  5232  5216   204.526   1440.543723
0.311544
  103  16  5271  5255   204.055   1560.465998
0.312394
  104  16  5310  5294   203.593   1560.483188
0.313355
  105  16  5357  5341   203.444   188   0.0313209
0.313267
  106  16  5402  5386   203.223   1800.517098
0.313714
  107  16  5457  5441   203.379   220   0.0277359
0.313288
  108 

[ceph-users] cephfs snapshot format upgrade

[Yan, Zheng]

Hello

To simplify snapshot handling in multiple active mds setup, we changed
format of snaprealm in mimic dev.
https://github.com/ceph/ceph/pull/16779.

The new version mds can handle old format snaprealm in single active
setup. It also can convert old format snaprealm to the new format when
snaprealm is modified. The problem is that new version mds can not
properly handle old format snaprealm in multiple active setup. It may
crash when it encounter old format snaprealm. For existing filesystem
with snapshots, upgrading mds to mimic seems to be no problem at first
glance. But if user later enables multiple active mds,  mds may
crashes continuously. No easy way to switch back to single acitve mds.

I don't have clear idea how to handle this situation. I can think of a
few options.

1. Forbid multiple active before all old snapshots are deleted or
before all snaprealms are converted to new format. Format conversion
requires traversing while whole filesystem tree.  Not easy to
implement.

2. Ask user to delete all old snapshots before upgrading to mimic,
make mds just ignore old format snaprealms.


Regards
Yan, Zheng
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Moving bluestore WAL and DB after bluestore creation

[Eugen Block]

Hi,

I just wanted to make sure that our latest findings reach the OP of  
this thread. We posted it in a different thread [1] and hope this  
helps some of you.
It is possible to migrate a journal from one partition to another  
almost without downtime of the OSD. But it's *not* sufficient to dd  
the journal to the new partition and replace the symlink. The OSD will  
restart successfully only if the old partition still exists, and  
you'll find references to it in /proc/fd/. Removing the old  
partition will prevent the OSD from starting. You can find details in  
the provided link [1].


We managed to replace the journals of six 1 TB OSDs residing on the  
same host within 25 minutes in our production environment.


Note: this only applies if the wal/db already reside on a separate partition.

Currently, I'm looking for a way to extract the journal of an  
all-in-one OSD (bluestore) into a separate partition, I thought maybe  
"ceph-objectstore-tool --op dump-journal" could do the trick, but this  
command doesn't work for me. Has anyone any insights on this?


Regards,
Eugen

[1] http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-April/025930.html



 -
 Datum: Fri, 17 Nov 2017 17:04:36 +0100
   Von: Ronny Aasen 
Betreff: Re: [ceph-users] Moving bluestore WAL and DB after  
bluestore creation

An: ceph-users@lists.ceph.com

On 16.11.2017 09:45, Loris Cuoghi wrote:

Le Wed, 15 Nov 2017 19:46:48 +,
Shawn Edwards  a écrit :


On Wed, Nov 15, 2017, 11:07 David Turner 
wrote:


I'm not going to lie.  This makes me dislike Bluestore quite a
bit.  Using multiple OSDs to an SSD journal allowed for you to
monitor the write durability of the SSD and replace it without
having to out and re-add all of the OSDs on the device.  Having to
now out and backfill back onto the HDDs is awful and would have
made a time when I realized that 20 journal SSDs all ran low on
writes at the same time nearly impossible to recover from.

Flushing journals, replacing SSDs, and bringing it all back online
was a slick process.  Formatting the HDDs and backfilling back onto
the same disks sounds like a big regression.  A process to migrate
the WAL and DB onto the HDD and then back off to a new device would
be very helpful.

On Wed, Nov 15, 2017 at 10:51 AM Mario Giammarco
 wrote:


It seems it is not possible. I recreated the OSD

2017-11-12 17:44 GMT+01:00 Shawn Edwards :


I've created some Bluestore OSD with all data (wal, db, and data)
all on the same rotating disk.  I would like to now move the wal
and db onto an nvme disk.  Is that possible without re-creating
the OSD?

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



This.  Exactly this.  Not being able to move the .db and .wal data on
and off the main storage disk on Bluestore is a regression.


Hello,

What stops you from dd'ing the DB/WAL's partitions on another disk and
updating the symlinks in the OSD's mount point under /var/lib/ceph/osd?



this probably works when you deployed bluestore with partitions, but  
if you did not create partitions for block.db on orginal bluestore  
creation there is no block.db symlink, db and wal are mixed into the  
block partition and not easy to extract.  also just dd the block  
device may not help if you want to change the size of the db  
partition. this needs more testing.  probably tools can be created  
in the future for resizing  db and wal partitions, and for  
extracting db data from block into a separate block.db partition.


dd block.db would probably work when you need to replace a worn out  
ssd drive. but not so much if you want to deploy separate block.db  
from a bluestore made without block.db



kind regards
Ronny Aasen





___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



--
Eugen Block voice   : +49-40-559 51 75
NDE Netzdesign und -entwicklung AG  fax : +49-40-559 51 77
Postfach 61 03 15
D-22423 Hamburg e-mail  : ebl...@nde.ag

 Vorsitzende des Aufsichtsrates: Angelika Mozdzen
   Sitz und Registergericht: Hamburg, HRB 90934
   Vorstand: Jens-U. Mozdzen
USt-IdNr. DE 814 013 983


--
Eugen Block voice   : +49-40-559 51 75
NDE Netzdesign und -entwicklung AG  fax : +49-40-559 51 77
Postfach 61 03 15
D-22423 Hamburg e-mail  : ebl...@nde.ag

Vorsitzende des Aufsichtsrates: Angelika Mozdzen
  Sitz und 

Re: [ceph-users] Ceph luminous - troubleshooting performance issues overall DSK 100%, busy 1%

[Kai Wagner]

Is this just from one server or from all servers? Just wondering why VD
0 is using WriteThrough compared to the others. If that's the setup for
the OSD's you already have a cache setup problem.


On 10.04.2018 13:44, Mohamad Gebai wrote:
> megacli -LDGetProp -cache -Lall -a0
>
> Adapter 0-VD 0(target id: 0): Cache Policy:WriteThrough,
> ReadAheadNone, Direct, Write Cache OK if bad BBU
> Adapter 0-VD 1(target id: 1): Cache Policy:WriteBack, ReadAdaptive,
> Cached, No Write Cache if bad BBU
> Adapter 0-VD 2(target id: 2): Cache Policy:WriteBack, ReadAdaptive,
> Cached, No Write Cache if bad BBU
> Adapter 0-VD 3(target id: 3): Cache Policy:WriteBack, ReadAdaptive,
> Cached, No Write Cache if bad BBU

-- 
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 
(AG Nürnberg)




signature.asc
Description: OpenPGP digital signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph luminous - troubleshooting performance issues overall DSK 100%, busy 1%

[Mohamad Gebai]

Just to be clear about the issue:

You have a 3 servers setup, performance is good. You add a server (with
1 OSD?) and performance goes down, is that right?

Can you give us more details? What's your complete setup? How many OSDs
per node, bluestore/filestore, WAL/DB setup, etc. You're talking about
sdb, sde, etc.. are those supposed to be OSD disks? What performance do
you see before adding the last server? And how does it compare to the
performance after? Are your OSD weights set correctly after the move
(and after data settles)?

Mohamad


On 04/05/2018 11:23 AM, Steven Vacaroaia wrote:
> Hi,
>
> I have a strange issue - OSDs from a specific server are introducing
> huge performance issue
>
> This is a brand new installation on 3 identical servers -
>  DELL R620 with PERC H710 , bluestore  DB and WAL on SSD, 10GB
> dedicated private/public networks 
>
>
> When I add the OSD I see gaps like below and huge latency  
>
> atop provides no  clear culprit EXCEPT very low network and specific
> disk utilization BUT 100% DSK for ceph-osd process  which stay like
> that ( 100%) for the duration of the test
> ( see below)
>
> Not sure why ceph-osd process  DSK stays at 100% while all the
> specific DSK ( for sdb, sde ..etc) are 1% busy ?
>
> Any help/ instructions for how to troubleshooting this will be
> appreciated 
>
> (apologies if the format is not being kept)
>
>
> CPU | sys       4%  | user      1%  |               | irq       1%  | 
>              | idle    794%  | wait      0%  |              |         
>      |  steal     0% |  guest     0% |  curf 2.20GHz |             
>  |  curscal   ?% |
> CPL | avg1    0.00  |               | avg5    0.00  | avg15   0.00  | 
>              |               |               | csw    547/s |         
>      |  intr   832/s |               |               |  numcpu     8
> |               |
> MEM | tot    62.9G  | free   61.4G  | cache 520.6M  | dirty   0.0M  |
> buff    7.5M  | slab   98.9M  | slrec  64.8M  | shmem   8.8M |  shrss 
>  0.0M |  shswp   0.0M |  vmbal   0.0M |               |  hptot   0.0M
> |  hpuse   0.0M |
> SWP | tot     6.0G  | free    6.0G  |               |               | 
>              |               |               |              |         
>      |               |               |  vmcom   1.5G |             
>  |  vmlim  37.4G |
> LVM |         dm-0  | busy      1%  |               | read     0/s  |
> write   54/s  |               | KiB/r      0  | KiB/w    455 |  MBr/s 
>   0.0 |               |  MBw/s   24.0 |  avq     3.69 |             
>  |  avio 0.14 ms |
> DSK |          sdb  | busy      1%  |               | read     0/s  |
> write  102/s  |               | KiB/r      0  | KiB/w    240 |  MBr/s 
>   0.0 |               |  MBw/s   24.0 |  avq     6.69 |             
>  |  avio 0.08 ms |
> DSK |          sda  | busy      0%  |               | read     0/s  |
> write   12/s  |               | KiB/r      0  | KiB/w      4 |  MBr/s 
>   0.0 |               |  MBw/s    0.1 |  avq     1.00 |             
>  |  avio 0.05 ms |
> DSK |          sde  | busy      0%  |               | read     0/s  |
> write    0/s  |               | KiB/r      0  | KiB/w      0 |  MBr/s 
>   0.0 |               |  MBw/s    0.0 |  avq     1.00 |             
>  |  avio 2.50 ms |
> NET | transport     | tcpi   718/s  | tcpo   972/s  | udpi     0/s  | 
>              | udpo     0/s  | tcpao    0/s  | tcppo    0/s |  tcprs 
>  21/s |  tcpie    0/s |  tcpor    0/s |               |  udpnp    0/s
> |  udpie    0/s |
> NET | network       | ipi    719/s  |               | ipo    399/s  |
> ipfrw    0/s  |               | deliv  719/s  |              |       
>        |               |               |  icmpi    0/s |             
>  |  icmpo    0/s |
> NET | eth5      1%  | pcki  2214/s  | pcko   939/s  |               |
> sp   10 Gbps  | si  154 Mbps  | so   52 Mbps  |              |  coll 
>    0/s |  mlti     0/s |  erri     0/s |  erro     0/s |  drpi     0/s
> |  drpo     0/s |
> NET | eth4      0%  | pcki   712/s  | pcko    54/s  |               |
> sp   10 Gbps  | si   50 Mbps  | so   90 Kbps  |              |  coll 
>    0/s |  mlti     0/s |  erri     0/s |  erro     0/s |  drpi     0/s
> |  drpo     0/s |
>
>     PID                                 TID                           
>    RDDSK                               WRDSK                         
>    WCANCL                               DSK                           
>   CMD       1/21
>    2067                                   -                           
>     0K/s                              0.0G/s                         
>      0K/s                              100%                           
>   ceph-osd
>
>
>
>   
>
> 2018-04-05 10:55:24.316549 min lat: 0.0203278 max lat: 10.7501 avg
> lat: 0.496822
>   sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg
> lat(s)
>    40      16      1096      1080   107.988         0           -   
> 0.496822
>    41      16      1096      1080   

Re: [ceph-users] Fwd: Separate --block.wal --block.db bluestore not working as expected.

[Richard Hesketh]

No, you shouldn't invoke it that way, you should just not specify a WAL device 
at all if you want it to be stored with the DB - if not otherwise specified the 
WAL is automatically stored with the other metadata on the DB device. You 
should do something like:

ceph-volume lvm prepare --bluestore --data /dev/sdc --block.db /dev/sda1

Rich

On 09/04/18 09:19, Hervé Ballans wrote:
> Hi,
> 
> Just a little question regarding this operation :
> 
> [root@osdhost osd]# ceph-volume lvm prepare --bluestore --data /dev/sdc 
> --block.wal /dev/sda2 --block.db /dev/sda1
> 
> On a previous post, I understood that if both wal and db are stored on the 
> same separate device, then we could use a single partition for both...which 
> means we could do :
> 
> # ceph-volume lvm prepare --bluestore --data /dev/sdc --block.wal /dev/sda1 
> --block.db /dev/sda1
> 
> and so on with other uniq wal/db partition for other OSD...
> 
> Did I get that correctly ?
> 
> Thanks,
> 
> Hervé
> 
> 
> Le 07/04/2018 à 17:59, Gary Verhulp a écrit :
>>
>> I’m trying to create bluestore osds with separate --block.wal --block.db 
>> devices on a write intensive SSD
>>
>> I’ve split the SSD (/dev/sda) into two partditions sda1 and sda2 for db and 
>> wal
>>
>>  
>>
>> I seems to me the osd uuid is getting changed and I’m only able to start the 
>> last OSD
>>
>> Do I need to create a new partition or logical volume on the SSD for each 
>> OSD?
>>
>> I’m sure this is a simple fail in my understanding of how it is supposed to 
>> be provisioned.
>>
>> Any advice would be appreciated.
>>
>> Thanks,
>>
>> Gary



signature.asc
Description: OpenPGP digital signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Dying OSDs

[Jan Marquardt]

Hi,

we are experiencing massive problems with our Ceph setup. After starting
a "repair pg" because of scrub errors OSDs started to crash, which we
could not stop so far. We are running Ceph 12.2.4. Crashed OSDs are both
bluestore and filestore.

Our cluster currently looks like this:

# ceph -s
  cluster:
id: c59e56df-2043-4c92-9492-25f05f268d9f
health: HEALTH_ERR
1 osds down
73005/17149710 objects misplaced (0.426%)
5 scrub errors
Reduced data availability: 2 pgs inactive, 2 pgs down
Possible data damage: 1 pg inconsistent
Degraded data redundancy: 611518/17149710 objects degraded
(3.566%), 86 pgs degraded, 86 pgs undersized

  services:
mon: 3 daemons, quorum head1,head2,head3
mgr: head3(active), standbys: head2, head1
osd: 34 osds: 24 up, 25 in; 18 remapped pgs

  data:
pools:   1 pools, 768 pgs
objects: 5582k objects, 19500 GB
usage:   62030 GB used, 31426 GB / 93456 GB avail
pgs: 0.260% pgs not active
 611518/17149710 objects degraded (3.566%)
 73005/17149710 objects misplaced (0.426%)
 670 active+clean
 75  active+undersized+degraded
 8   active+undersized+degraded+remapped+backfill_wait
 8   active+clean+remapped
 2   down
 2   active+undersized+degraded+remapped+backfilling
 2   active+clean+scrubbing+deep
 1   active+undersized+degraded+inconsistent

  io:
client:   10911 B/s rd, 118 kB/s wr, 0 op/s rd, 54 op/s wr
recovery: 31575 kB/s, 8 objects/s

# ceph osd tree
ID  CLASS WEIGHTTYPE NAME  STATUS REWEIGHT PRI-AFF
 -1   124.07297 root default
 -229.08960 host ceph1
  0   hdd   3.63620 osd.0  up  1.0 1.0
  1   hdd   3.63620 osd.1down0 1.0
  2   hdd   3.63620 osd.2  up  1.0 1.0
  3   hdd   3.63620 osd.3  up  1.0 1.0
  4   hdd   3.63620 osd.4down0 1.0
  5   hdd   3.63620 osd.5down0 1.0
  6   hdd   3.63620 osd.6  up  1.0 1.0
  7   hdd   3.63620 osd.7  up  1.0 1.0
 -3 7.27240 host ceph2
 14   hdd   3.63620 osd.14 up  1.0 1.0
 15   hdd   3.63620 osd.15 up  1.0 1.0
 -429.11258 host ceph3
 16   hdd   3.63620 osd.16 up  1.0 1.0
 18   hdd   3.63620 osd.18   down0 1.0
 19   hdd   3.63620 osd.19   down0 1.0
 20   hdd   3.65749 osd.20 up  1.0 1.0
 21   hdd   3.63620 osd.21 up  1.0 1.0
 22   hdd   3.63620 osd.22 up  1.0 1.0
 23   hdd   3.63620 osd.23 up  1.0 1.0
 24   hdd   3.63789 osd.24   down0 1.0
 -929.29919 host ceph4
 17   hdd   3.66240 osd.17 up  1.0 1.0
 25   hdd   3.66240 osd.25 up  1.0 1.0
 26   hdd   3.66240 osd.26   down0 1.0
 27   hdd   3.66240 osd.27 up  1.0 1.0
 28   hdd   3.66240 osd.28   down0 1.0
 29   hdd   3.66240 osd.29 up  1.0 1.0
 30   hdd   3.66240 osd.30 up  1.0 1.0
 31   hdd   3.66240 osd.31   down0 1.0
-1129.29919 host ceph5
 32   hdd   3.66240 osd.32 up  1.0 1.0
 33   hdd   3.66240 osd.33 up  1.0 1.0
 34   hdd   3.66240 osd.34 up  1.0 1.0
 35   hdd   3.66240 osd.35 up  1.0 1.0
 36   hdd   3.66240 osd.36   down  1.0 1.0
 37   hdd   3.66240 osd.37 up  1.0 1.0
 38   hdd   3.66240 osd.38 up  1.0 1.0
 39   hdd   3.66240 osd.39 up  1.0 1.0

The last OSDs that crashed are #28 and #36. Please find the
corresponding log files here:

http://af.janno.io/ceph/ceph-osd.28.log.1.gz
http://af.janno.io/ceph/ceph-osd.36.log.1.gz

The backtraces look almost the same for all crashed OSDs.

Any help, hint or advice would really be appreciated. Please let me know
if you need any further information.

Best Regards

Jan

-- 
Artfiles New Media GmbH | Zirkusweg 1 | 20359 Hamburg
Tel: 040 - 32 02 72 90 | Fax: 040 - 32 02 72 95
E-Mail: supp...@artfiles.de | Web: http://www.artfiles.de
Geschäftsführer: Harald Oltmanns | Tim Evers
Eingetragen im Handelsregister Hamburg - HRB 81478
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com