Re: [ceph-users] [External Email] RE: Beginner questions

[DHilsbos]

Paul;

So is the 3/30/300GB a limit of RocksDB, or of Bluestore?

The percentages you list, are they used DB / used data?  If so... Where do you 
get the used DB data from?

Thank you,

Dominic L. Hilsbos, MBA 
Director – Information Technology 
Perform Air International Inc.
dhils...@performair.com 
www.PerformAir.com


From: Paul Emmerich [mailto:paul.emmer...@croit.io] 
Sent: Thursday, January 16, 2020 3:23 PM
To: Bastiaan Visser
Cc: Dominic Hilsbos; Ceph Users
Subject: Re: [ceph-users] [External Email] RE: Beginner questions

Discussing DB size requirements without knowing the exact cluster requirements 
doesn't work.

Here are some real-world examples:

cluster1: CephFS, mostly large files, replicated x3
0.2% used for metadata

cluster2: radosgw, mix between replicated and erasure, mixed file sizes (lots 
of tiny files, though)
1.3% used for metadata

The 4%-10% quoted in the docs are *not based on any actual usage data*, they 
are just an absolute worst case estimate.


A 30 GB DB partition for a 12 TiB disk is 0.25% if the disk is completely full 
(which it won't be) is sufficient for many use cases.
I think cluster2 with 1.3% is one of the highest metadata usages that I've seen 
on an actual production cluster.
I can think of a setup that probably has more but I haven't ever explicitly 
checked it.

The restriction to 3/30/300 is temporary and might be fixed in a future 
release, so I'd just partition that disk into X DB devices.

Paul

-- 
Paul Emmerich

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90


On Thu, Jan 16, 2020 at 10:28 PM Bastiaan Visser  wrote:
Dave made a good point WAL + DB might end up a little over 60G, I would 
probably go with ~70Gig partitions /LV's per OSD in your case. (if the nvme 
drive is smart enough to spread the writes over all available capacity, mort 
recent nvme's are). I have not yet seen a WAL larger or even close to than a 
gigabyte.

We don't even think about EC-coded pools on clusters with less than 6 nodes 
(spindles, full SSD is another story).
EC pools neer more processing resources  We usually settle with 1 gig per TB of 
storage on replicated only sluters, but whet EC polls are involved, we add at 
least 50% to that. Also make sure your processors are up for it.

Do not base your calculations on a healthy cluster -> build to fail. 
How long are you willing to be in a degraded state on node failure. Especially 
when using many larger spindles. recovery time might be way longer than you 
think. 12 * 12TB is 144TB storage, on a 4+2 EC pool you might end up with over 
200 TB of traffic, on a 10Gig network that's roughly 2 and a half days to 
recover. IF your processors are not bottleneck due to EC parity calculations 
and all capacity is available for recovery (which is usually not the case, 
there is still production traffic that will eat up resources).

Op do 16 jan. 2020 om 21:30 schreef :
Dave;

I don't like reading inline responses, so...

I have zero experience with EC pools, so I won't pretend to give advice in that 
area.

I would think that small NVMe for DB would be better than nothing, but I don't 
know.

Once I got the hang of building clusters, it was relatively easy to wipe a 
cluster out and rebuild it.  Perhaps you could take some time, and benchmark 
different configurations?

Thank you,

Dominic L. Hilsbos, MBA 
Director – Information Technology 
Perform Air International Inc.
dhils...@performair.com 
www.PerformAir.com


-Original Message-
From: Dave Hall [mailto:kdh...@binghamton.edu] 
Sent: Thursday, January 16, 2020 1:04 PM
To: Dominic Hilsbos; ceph-users@lists.ceph.com
Subject: Re: [External Email] RE: [ceph-users] Beginner questions

Dominic,

We ended up with a 1.6TB PCIe NVMe in each node.  For 8 drives this 
worked out to a DB size of something like 163GB per OSD. Allowing for 
expansion to 12 drives brings it down to 124GB. So maybe just put the 
WALs on NVMe and leave the DBs on the platters?

Understood that we will want to move to more nodes rather than more 
drives per node, but our funding is grant and donation based, so we may 
end up adding drives in the short term.  The long term plan is to get to 
separate MON/MGR/MDS nodes and 10s of OSD nodes.

Due to our current low node count, we are considering erasure-coded PGs 
rather than replicated in order to maximize usable space.  Any 
guidelines or suggestions on this?

Also, sorry for not replying inline.  I haven't done this much in a 
while - I'll figure it out.

Thanks.

-Dave

On 1/16/2020 2:48 PM, dhils...@performair.com wrote:
> Dave;
>
> I'd like to expand on this answer, briefly...
>
> The information in the docs is wrong.  There have been many discussions about 
> changing it, but no good alternative has been suggested, thus it hasn't been 
> changed.
>
> The 3rd party project that Ceph's BlueStore uses for its database (RocksDB), 
> apparently only uses 

Re: [ceph-users] [External Email] RE: Beginner questions

[DHilsbos]

Dave;

I don't like reading inline responses, so...

I have zero experience with EC pools, so I won't pretend to give advice in that 
area.

I would think that small NVMe for DB would be better than nothing, but I don't 
know.

Once I got the hang of building clusters, it was relatively easy to wipe a 
cluster out and rebuild it.  Perhaps you could take some time, and benchmark 
different configurations?

Thank you,

Dominic L. Hilsbos, MBA 
Director – Information Technology 
Perform Air International Inc.
dhils...@performair.com 
www.PerformAir.com


-Original Message-
From: Dave Hall [mailto:kdh...@binghamton.edu] 
Sent: Thursday, January 16, 2020 1:04 PM
To: Dominic Hilsbos; ceph-users@lists.ceph.com
Subject: Re: [External Email] RE: [ceph-users] Beginner questions

Dominic,

We ended up with a 1.6TB PCIe NVMe in each node.  For 8 drives this 
worked out to a DB size of something like 163GB per OSD. Allowing for 
expansion to 12 drives brings it down to 124GB. So maybe just put the 
WALs on NVMe and leave the DBs on the platters?

Understood that we will want to move to more nodes rather than more 
drives per node, but our funding is grant and donation based, so we may 
end up adding drives in the short term.  The long term plan is to get to 
separate MON/MGR/MDS nodes and 10s of OSD nodes.

Due to our current low node count, we are considering erasure-coded PGs 
rather than replicated in order to maximize usable space.  Any 
guidelines or suggestions on this?

Also, sorry for not replying inline.  I haven't done this much in a 
while - I'll figure it out.

Thanks.

-Dave

On 1/16/2020 2:48 PM, dhils...@performair.com wrote:
> Dave;
>
> I'd like to expand on this answer, briefly...
>
> The information in the docs is wrong.  There have been many discussions about 
> changing it, but no good alternative has been suggested, thus it hasn't been 
> changed.
>
> The 3rd party project that Ceph's BlueStore uses for its database (RocksDB), 
> apparently only uses DB sizes of 3GB, 30GB, and 300GB.  As Dave mentions 
> below, when RocksDB executes a compact operation, it creates a new blob of 
> the same target size, and writes the compacted data into it.  This doubles 
> the necessary space.  In addition, BlueStore places its Write Ahead Log (WAL) 
> into the fastest storage that is available to OSD daemon,  i.e. NVMe if 
> available.  Since this is done before the first compaction is requested, the 
> WAL can force compaction onto slower storage.
>
> Thus, the numbers I've had floating around in my head for our next cluster 
> are: 7GB, 66GB, and 630GB.  From all the discussion I've seen around RocksDB, 
> those seem like good, common sense targets.  Pick the largest one that works 
> for your setup.
>
> All that said... You would really want to pair a 600GB+ NVMe with 12TB 
> drives, otherwise your DB is almost guaranteed to overflow onto the spinning 
> drive, and affect performance.
>
> I became aware of most of this after we planned our clusters, so I haven't 
> tried it, YMMV.
>
> One final note: more hosts, and more spindles usually translates into better 
> cluster-wide performance.  I can't predict what the relatively low client 
> counts you're suggesting would impact that.
>
> Thank you,
>
> Dominic L. Hilsbos, MBA
> Director – Information Technology
> Perform Air International Inc.
> dhils...@performair.com
> www.PerformAir.com
>
>
> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of 
> Bastiaan Visser
> Sent: Thursday, January 16, 2020 10:55 AM
> To: Dave Hall
> Cc: ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] Beginner questions
>
> I would definitely go for Nautilus. there are quite some optimizations that 
> went in after mimic.
>
> Bluestore DB size usually ends up at either 30 or 60 GB.
> 30 GB is one of the sweet spots during normal operation. But during 
> compaction, ceph writes the new data before removing the old, hence the 60GB.
> Next sweetspot is 300/600GB. any size between 60 and 300 will never be unused.
>
> DB Usage is also dependent on ceph usage, object storage is known to use a 
> lot more db space than rbd images for example.
>
> Op do 16 jan. 2020 om 17:46 schreef Dave Hall :
> Hello all.
> Sorry for the beginner questions...
> I am in the process of setting up a small (3 nodes, 288TB) Ceph cluster to 
> store some research data.  It is expected that this cluster will grow 
> significantly in the next year, possibly to multiple petabytes and 10s of 
> nodes.  At this time I'm expected a relatively small number of clients, with 
> only one or two actively writing collected data - albeit at a high volume per 
> day.
> Currently I'm deploying on Debian 9 via ceph-ansible.
> Before I put this cluster into production I have a couple questions based on 
> my experience to date:
> Luminous, Mimic, or Nautilus?  I need stability for this deployment, so I am 
> sticking with Debian 9 since Debian 10 is fairly new, and I have been 
> hesitant to go 

Re: [ceph-users] Beginner questions

[DHilsbos]

Dave;

I'd like to expand on this answer, briefly...

The information in the docs is wrong.  There have been many discussions about 
changing it, but no good alternative has been suggested, thus it hasn't been 
changed.

The 3rd party project that Ceph's BlueStore uses for its database (RocksDB), 
apparently only uses DB sizes of 3GB, 30GB, and 300GB.  As Dave mentions below, 
when RocksDB executes a compact operation, it creates a new blob of the same 
target size, and writes the compacted data into it.  This doubles the necessary 
space.  In addition, BlueStore places its Write Ahead Log (WAL) into the 
fastest storage that is available to OSD daemon,  i.e. NVMe if available.  
Since this is done before the first compaction is requested, the WAL can force 
compaction onto slower storage.

Thus, the numbers I've had floating around in my head for our next cluster are: 
7GB, 66GB, and 630GB.  From all the discussion I've seen around RocksDB, those 
seem like good, common sense targets.  Pick the largest one that works for your 
setup.

All that said... You would really want to pair a 600GB+ NVMe with 12TB drives, 
otherwise your DB is almost guaranteed to overflow onto the spinning drive, and 
affect performance.

I became aware of most of this after we planned our clusters, so I haven't 
tried it, YMMV.

One final note: more hosts, and more spindles usually translates into better 
cluster-wide performance.  I can't predict what the relatively low client 
counts you're suggesting would impact that.

Thank you,

Dominic L. Hilsbos, MBA 
Director – Information Technology 
Perform Air International Inc.
dhils...@performair.com 
www.PerformAir.com


From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of 
Bastiaan Visser
Sent: Thursday, January 16, 2020 10:55 AM
To: Dave Hall
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Beginner questions

I would definitely go for Nautilus. there are quite some optimizations that 
went in after mimic.

Bluestore DB size usually ends up at either 30 or 60 GB.
30 GB is one of the sweet spots during normal operation. But during compaction, 
ceph writes the new data before removing the old, hence the 60GB.
Next sweetspot is 300/600GB. any size between 60 and 300 will never be unused.

DB Usage is also dependent on ceph usage, object storage is known to use a lot 
more db space than rbd images for example.

Op do 16 jan. 2020 om 17:46 schreef Dave Hall :
Hello all.
Sorry for the beginner questions...
I am in the process of setting up a small (3 nodes, 288TB) Ceph cluster to 
store some research data.  It is expected that this cluster will grow 
significantly in the next year, possibly to multiple petabytes and 10s of 
nodes.  At this time I'm expected a relatively small number of clients, with 
only one or two actively writing collected data - albeit at a high volume per 
day.
Currently I'm deploying on Debian 9 via ceph-ansible.  
Before I put this cluster into production I have a couple questions based on my 
experience to date:
Luminous, Mimic, or Nautilus?  I need stability for this deployment, so I am 
sticking with Debian 9 since Debian 10 is fairly new, and I have been hesitant 
to go with Nautilus.  Yet Mimic seems to have had a hard road on Debian but for 
the efforts at Croit.  
• Statements on the Releases page are now making more sense to me, but I would 
like to confirm that Nautilus is the right choice at this time?
Bluestore DB size:  My nodes currently have 8 x 12TB drives (plus 4 empty bays) 
and a PCIe NVMe drive.  If I understand the suggested calculation correctly, 
the DB size for a 12 TB Bluestore OSD would be 480GB.  If my NVMe isn't big 
enough to provide this size, should I skip provisioning the DBs on the NVMe, or 
should I give each OSD 1/12th of what I have available?  Also, should I try to 
shift budget a bit to get more NVMe as soon as I can, and redo the OSDs when 
sufficient NVMe is available?
Thanks.
-Dave
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Separate disk sets for high IO?

[DHilsbos]

Philip;

Ah, ok.  I suspect that isn't documented because the developers don't want 
average users doing it.

It's also possible that it won't work as expected, as there is discussion on 
the web of device classes being changed at startup of the OSD daemon.

That said...

"ceph osd crush class create " is the command to create a custom device 
class, at least in Nautilus 14.2.4.

Theoretically, a custom device class can then be used the same as the built in 
device classes.

Caveat: I'm a user, not a developer of Ceph.

Thank you,

Dominic L. Hilsbos, MBA 
Director - Information Technology 
Perform Air International Inc.
dhils...@performair.com 
www.PerformAir.com



-Original Message-
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Philip 
Brown
Sent: Monday, December 16, 2019 4:42 PM
To: ceph-users
Subject: Re: [ceph-users] Separate disk sets for high IO?

Yes I saw that thanks.

Unfortunately, that doesnt show use of "custom classes" as someone hinted at.



- Original Message -
From: dhils...@performair.com
To: "ceph-users" 
Cc: "Philip Brown" 
Sent: Monday, December 16, 2019 3:38:49 PM
Subject: RE: Separate disk sets for high IO?

Philip;

There's isn't any documentation that shows specifically how to do that, though 
the below comes close.

Here's the documentation, for Nautilus, on CRUSH operations:
https://docs.ceph.com/docs/nautilus/rados/operations/crush-map/

About a third of the way down the page is a discussion of "Device Classes."  In 
that sections it talks about creating CRUSH rules that target certain device 
classes (hdd, ssd, nvme, by default).

Once you have a rule, you can configure a pool to use the rule.

Thank you,

Dominic L. Hilsbos, MBA 
Director - Information Technology 
Perform Air International Inc.
dhils...@performair.com 
www.PerformAir.com


-Original Message-
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Philip 
Brown
Sent: Monday, December 16, 2019 3:43 PM
To: Nathan Fish
Cc: ceph-users
Subject: Re: [ceph-users] Separate disk sets for high IO?

Sounds very useful.

Any online example documentation for this?
havent found any so far?


- Original Message -
From: "Nathan Fish" 
To: "Marc Roos" 
Cc: "ceph-users" , "Philip Brown" 
Sent: Monday, December 16, 2019 2:07:44 PM
Subject: Re: [ceph-users] Separate disk sets for high IO?

Indeed, you can set device class to pretty much arbitrary strings and
specify them. By default, 'hdd', 'ssd', and I think 'nvme' are
autodetected - though my Optanes showed up as 'ssd'.

On Mon, Dec 16, 2019 at 4:58 PM Marc Roos  wrote:
>
>
>
> You can classify osd's, eg as ssd. And you can assign this class to a
> pool you create. This way you have have rbd's running on only ssd's. I
> think you have also a class for nvme and you can create custom classes.
>
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Separate disk sets for high IO?

[DHilsbos]

Philip;

There's isn't any documentation that shows specifically how to do that, though 
the below comes close.

Here's the documentation, for Nautilus, on CRUSH operations:
https://docs.ceph.com/docs/nautilus/rados/operations/crush-map/

About a third of the way down the page is a discussion of "Device Classes."  In 
that sections it talks about creating CRUSH rules that target certain device 
classes (hdd, ssd, nvme, by default).

Once you have a rule, you can configure a pool to use the rule.

Thank you,

Dominic L. Hilsbos, MBA 
Director - Information Technology 
Perform Air International Inc.
dhils...@performair.com 
www.PerformAir.com


-Original Message-
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Philip 
Brown
Sent: Monday, December 16, 2019 3:43 PM
To: Nathan Fish
Cc: ceph-users
Subject: Re: [ceph-users] Separate disk sets for high IO?

Sounds very useful.

Any online example documentation for this?
havent found any so far?


- Original Message -
From: "Nathan Fish" 
To: "Marc Roos" 
Cc: "ceph-users" , "Philip Brown" 
Sent: Monday, December 16, 2019 2:07:44 PM
Subject: Re: [ceph-users] Separate disk sets for high IO?

Indeed, you can set device class to pretty much arbitrary strings and
specify them. By default, 'hdd', 'ssd', and I think 'nvme' are
autodetected - though my Optanes showed up as 'ssd'.

On Mon, Dec 16, 2019 at 4:58 PM Marc Roos  wrote:
>
>
>
> You can classify osd's, eg as ssd. And you can assign this class to a
> pool you create. This way you have have rbd's running on only ssd's. I
> think you have also a class for nvme and you can create custom classes.
>
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] HEALTH_WARN 1 MDSs report oversized cache

[DHilsbos]

Patrick;

I agree with Ranjan, though not in the particulars.

The issue is that "oversized" is ambiguous, though undersized is also ambiguous.

I personally prefer unambiguous error messages which also suggest solutions, 
like:
"1 MDSs reporting cache exceeds 'mds cache memory limit,' of: ."

My 2 cents.

Thank you,

Dominic L. Hilsbos, MBA 
Director - Information Technology 
Perform Air International Inc.
dhils...@performair.com 
www.PerformAir.com



-Original Message-
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of 
Patrick Donnelly
Sent: Thursday, December 05, 2019 11:41 AM
To: Ranjan Ghosh
Cc: Ceph Users
Subject: Re: [ceph-users] HEALTH_WARN 1 MDSs report oversized cache

On Thu, Dec 5, 2019 at 9:45 AM Ranjan Ghosh  wrote:
> Ah, that seems to have fixed it. Hope it stays that way. I've raised it
> to 4 GB. Thanks to you both!

Just be aware the warning could come back. You just moved the goal posts.

The 1GB default is probably too low for most deployments, I have a PR
to increase this: https://github.com/ceph/ceph/pull/32042

> Although I have to say that the message is IMHO *very* misleading: "1
> MDSs report oversized cache" sounds to me like the cache is too large
> (i.e. wasting RAM unnecessarily). Shouldn't the message rather be "1
> MDSs report *undersized* cache"? Weird.

No. I means the MDS cache is larger than its target. This means the
MDS cannot trim its cache to go back under the limit. This could be
for many reasons but probably due to clients not releasing
capabilities, perhaps due to a bug.

-- 
Patrick Donnelly, Ph.D.
He / Him / His
Senior Software Engineer
Red Hat Sunnyvale, CA
GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Large OMAP Object

[DHilsbos]

All;

Since I haven't heard otherwise, I have to assume that the only way to get this 
to go away is to dump the contents of the RGW bucket(s), and  recreate it 
(them)?

How did this get past release approval?  A change which makes a valid cluster 
state in-valid, with no mitigation other than downtime, in a minor release.

Thank you,

Dominic L. Hilsbos, MBA 
Director – Information Technology 
Perform Air International Inc.
dhils...@performair.com 
www.PerformAir.com


-Original Message-
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of 
dhils...@performair.com
Sent: Friday, November 15, 2019 9:13 AM
To: ceph-users@lists.ceph.com
Cc: Stephen Self
Subject: Re: [ceph-users] Large OMAP Object

Wido;

Ok, yes, I have tracked it down to the index for one of our buckets.  I missed 
the ID in the ceph df output previously.  Next time I'll wait to read replies 
until I've finished my morning coffee.

How would I go about correcting this?

The content for this bucket is basically just junk, as we're still doing 
production qualification, and workflow planning.  Moving from Windows file 
shares to self-hosted cloud storage is a significant undertaking.

Thank you,

Dominic L. Hilsbos, MBA 
Director – Information Technology 
Perform Air International Inc.
dhils...@performair.com 
www.PerformAir.com



-Original Message-
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Wido 
den Hollander
Sent: Friday, November 15, 2019 8:40 AM
To: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Large OMAP Object



On 11/15/19 4:35 PM, dhils...@performair.com wrote:
> All;
> 
> Thank you for your help so far.  I have found the log entries from when the 
> object was found, but don't see a reference to the pool.
> 
> Here the logs:
> 2019-11-14 03:10:16.508601 osd.1 (osd.1) 21 : cluster [DBG] 56.7 deep-scrub 
> starts
> 2019-11-14 03:10:18.325881 osd.1 (osd.1) 22 : cluster [WRN] Large omap object 
> found. Object: 
> 56:f7d15b13:::.dir.f91aeff8-a365-47b4-a1c8-928cd66134e8.44130.1:head Key 
> count: 380425 Size (bytes): 82896978
> 

In this case it's in pool 56, check 'ceph df' to see which pool that is.

To me this seems like a RGW bucket which index grew too big.

Use:

$ radosgw-admin bucket list
$ radosgw-admin metadata get bucket:

And match that UUID back to the bucket.

Wido

> Thank you,
> 
> Dominic L. Hilsbos, MBA 
> Director – Information Technology 
> Perform Air International Inc.
> dhils...@performair.com 
> www.PerformAir.com
> 
> 
> 
> -Original Message-
> From: Wido den Hollander [mailto:w...@42on.com] 
> Sent: Friday, November 15, 2019 1:56 AM
> To: Dominic Hilsbos; ceph-users@lists.ceph.com
> Cc: Stephen Self
> Subject: Re: [ceph-users] Large OMAP Object
> 
> Did you check /var/log/ceph/ceph.log on one of the Monitors to see which
> pool and Object the large Object is in?
> 
> Wido
> 
> On 11/15/19 12:23 AM, dhils...@performair.com wrote:
>> All;
>>
>> We had a warning about a large OMAP object pop up in one of our clusters 
>> overnight.  The cluster is configured for CephFS, but nothing mounts a 
>> CephFS, at this time.
>>
>> The cluster mostly uses RGW.  I've checked the cluster log, the MON log, and 
>> the MGR log on one of the mons, with no useful references to the pool / pg 
>> where the large OMAP objects resides.
>>
>> Is my only option to find this large OMAP object to go through the OSD logs 
>> for the individual OSDs in the cluster?
>>
>> Thank you,
>>
>> Dominic L. Hilsbos, MBA 
>> Director - Information Technology 
>> Perform Air International Inc.
>> dhils...@performair.com 
>> www.PerformAir.com
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Large OMAP Object

[DHilsbos]

Wido;

Ok, yes, I have tracked it down to the index for one of our buckets.  I missed 
the ID in the ceph df output previously.  Next time I'll wait to read replies 
until I've finished my morning coffee.

How would I go about correcting this?

The content for this bucket is basically just junk, as we're still doing 
production qualification, and workflow planning.  Moving from Windows file 
shares to self-hosted cloud storage is a significant undertaking.

Thank you,

Dominic L. Hilsbos, MBA 
Director – Information Technology 
Perform Air International Inc.
dhils...@performair.com 
www.PerformAir.com



-Original Message-
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Wido 
den Hollander
Sent: Friday, November 15, 2019 8:40 AM
To: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Large OMAP Object



On 11/15/19 4:35 PM, dhils...@performair.com wrote:
> All;
> 
> Thank you for your help so far.  I have found the log entries from when the 
> object was found, but don't see a reference to the pool.
> 
> Here the logs:
> 2019-11-14 03:10:16.508601 osd.1 (osd.1) 21 : cluster [DBG] 56.7 deep-scrub 
> starts
> 2019-11-14 03:10:18.325881 osd.1 (osd.1) 22 : cluster [WRN] Large omap object 
> found. Object: 
> 56:f7d15b13:::.dir.f91aeff8-a365-47b4-a1c8-928cd66134e8.44130.1:head Key 
> count: 380425 Size (bytes): 82896978
> 

In this case it's in pool 56, check 'ceph df' to see which pool that is.

To me this seems like a RGW bucket which index grew too big.

Use:

$ radosgw-admin bucket list
$ radosgw-admin metadata get bucket:

And match that UUID back to the bucket.

Wido

> Thank you,
> 
> Dominic L. Hilsbos, MBA 
> Director – Information Technology 
> Perform Air International Inc.
> dhils...@performair.com 
> www.PerformAir.com
> 
> 
> 
> -Original Message-
> From: Wido den Hollander [mailto:w...@42on.com] 
> Sent: Friday, November 15, 2019 1:56 AM
> To: Dominic Hilsbos; ceph-users@lists.ceph.com
> Cc: Stephen Self
> Subject: Re: [ceph-users] Large OMAP Object
> 
> Did you check /var/log/ceph/ceph.log on one of the Monitors to see which
> pool and Object the large Object is in?
> 
> Wido
> 
> On 11/15/19 12:23 AM, dhils...@performair.com wrote:
>> All;
>>
>> We had a warning about a large OMAP object pop up in one of our clusters 
>> overnight.  The cluster is configured for CephFS, but nothing mounts a 
>> CephFS, at this time.
>>
>> The cluster mostly uses RGW.  I've checked the cluster log, the MON log, and 
>> the MGR log on one of the mons, with no useful references to the pool / pg 
>> where the large OMAP objects resides.
>>
>> Is my only option to find this large OMAP object to go through the OSD logs 
>> for the individual OSDs in the cluster?
>>
>> Thank you,
>>
>> Dominic L. Hilsbos, MBA 
>> Director - Information Technology 
>> Perform Air International Inc.
>> dhils...@performair.com 
>> www.PerformAir.com
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Large OMAP Object

[DHilsbos]

Paul;

I upgraded the cluster in question from 14.2.2 to 14.2.4 just before this came 
up, so that makes sense.

Thank you,

Dominic L. Hilsbos, MBA 
Director – Information Technology 
Perform Air International Inc.
dhils...@performair.com 
www.PerformAir.com



-Original Message-
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Paul 
Emmerich
Sent: Friday, November 15, 2019 8:48 AM
To: Wido den Hollander
Cc: Ceph Users
Subject: Re: [ceph-users] Large OMAP Object

Note that the size limit changed from 2M keys to 200k keys recently
(14.2.3 or 14.2.2 or something), so that object is probably older and
that's just the first deep scrub with the reduced limit that triggered
the warning.


Paul

-- 
Paul Emmerich

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90

On Fri, Nov 15, 2019 at 4:40 PM Wido den Hollander  wrote:
>
>
>
> On 11/15/19 4:35 PM, dhils...@performair.com wrote:
> > All;
> >
> > Thank you for your help so far.  I have found the log entries from when the 
> > object was found, but don't see a reference to the pool.
> >
> > Here the logs:
> > 2019-11-14 03:10:16.508601 osd.1 (osd.1) 21 : cluster [DBG] 56.7 deep-scrub 
> > starts
> > 2019-11-14 03:10:18.325881 osd.1 (osd.1) 22 : cluster [WRN] Large omap 
> > object found. Object: 
> > 56:f7d15b13:::.dir.f91aeff8-a365-47b4-a1c8-928cd66134e8.44130.1:head Key 
> > count: 380425 Size (bytes): 82896978
> >
>
> In this case it's in pool 56, check 'ceph df' to see which pool that is.
>
> To me this seems like a RGW bucket which index grew too big.
>
> Use:
>
> $ radosgw-admin bucket list
> $ radosgw-admin metadata get bucket:
>
> And match that UUID back to the bucket.
>
> Wido
>
> > Thank you,
> >
> > Dominic L. Hilsbos, MBA
> > Director – Information Technology
> > Perform Air International Inc.
> > dhils...@performair.com
> > www.PerformAir.com
> >
> >
> >
> > -Original Message-
> > From: Wido den Hollander [mailto:w...@42on.com]
> > Sent: Friday, November 15, 2019 1:56 AM
> > To: Dominic Hilsbos; ceph-users@lists.ceph.com
> > Cc: Stephen Self
> > Subject: Re: [ceph-users] Large OMAP Object
> >
> > Did you check /var/log/ceph/ceph.log on one of the Monitors to see which
> > pool and Object the large Object is in?
> >
> > Wido
> >
> > On 11/15/19 12:23 AM, dhils...@performair.com wrote:
> >> All;
> >>
> >> We had a warning about a large OMAP object pop up in one of our clusters 
> >> overnight.  The cluster is configured for CephFS, but nothing mounts a 
> >> CephFS, at this time.
> >>
> >> The cluster mostly uses RGW.  I've checked the cluster log, the MON log, 
> >> and the MGR log on one of the mons, with no useful references to the pool 
> >> / pg where the large OMAP objects resides.
> >>
> >> Is my only option to find this large OMAP object to go through the OSD 
> >> logs for the individual OSDs in the cluster?
> >>
> >> Thank you,
> >>
> >> Dominic L. Hilsbos, MBA
> >> Director - Information Technology
> >> Perform Air International Inc.
> >> dhils...@performair.com
> >> www.PerformAir.com
> >> ___
> >> ceph-users mailing list
> >> ceph-users@lists.ceph.com
> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >>
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Large OMAP Object

[DHilsbos]

All;

Thank you for your help so far.  I have found the log entries from when the 
object was found, but don't see a reference to the pool.

Here the logs:
2019-11-14 03:10:16.508601 osd.1 (osd.1) 21 : cluster [DBG] 56.7 deep-scrub 
starts
2019-11-14 03:10:18.325881 osd.1 (osd.1) 22 : cluster [WRN] Large omap object 
found. Object: 
56:f7d15b13:::.dir.f91aeff8-a365-47b4-a1c8-928cd66134e8.44130.1:head Key count: 
380425 Size (bytes): 82896978

Thank you,

Dominic L. Hilsbos, MBA 
Director – Information Technology 
Perform Air International Inc.
dhils...@performair.com 
www.PerformAir.com



-Original Message-
From: Wido den Hollander [mailto:w...@42on.com] 
Sent: Friday, November 15, 2019 1:56 AM
To: Dominic Hilsbos; ceph-users@lists.ceph.com
Cc: Stephen Self
Subject: Re: [ceph-users] Large OMAP Object

Did you check /var/log/ceph/ceph.log on one of the Monitors to see which
pool and Object the large Object is in?

Wido

On 11/15/19 12:23 AM, dhils...@performair.com wrote:
> All;
> 
> We had a warning about a large OMAP object pop up in one of our clusters 
> overnight.  The cluster is configured for CephFS, but nothing mounts a 
> CephFS, at this time.
> 
> The cluster mostly uses RGW.  I've checked the cluster log, the MON log, and 
> the MGR log on one of the mons, with no useful references to the pool / pg 
> where the large OMAP objects resides.
> 
> Is my only option to find this large OMAP object to go through the OSD logs 
> for the individual OSDs in the cluster?
> 
> Thank you,
> 
> Dominic L. Hilsbos, MBA 
> Director - Information Technology 
> Perform Air International Inc.
> dhils...@performair.com 
> www.PerformAir.com
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Large OMAP Object

[DHilsbos]

All;

We had a warning about a large OMAP object pop up in one of our clusters 
overnight.  The cluster is configured for CephFS, but nothing mounts a CephFS, 
at this time.

The cluster mostly uses RGW.  I've checked the cluster log, the MON log, and 
the MGR log on one of the mons, with no useful references to the pool / pg 
where the large OMAP objects resides.

Is my only option to find this large OMAP object to go through the OSD logs for 
the individual OSDs in the cluster?

Thank you,

Dominic L. Hilsbos, MBA 
Director - Information Technology 
Perform Air International Inc.
dhils...@performair.com 
www.PerformAir.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] rgw: multisite support

[DHilsbos]

Swami;

For 12.2.11 (Luminous), the previously linked document would be:
https://docs.ceph.com/docs/luminous/radosgw/multisite/#migrating-a-single-site-system-to-multi-site

Thank you,

Dominic L. Hilsbos, MBA 
Director – Information Technology 
Perform Air International Inc.
dhils...@performair.com 
www.PerformAir.com


From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of 
Joachim Kraftmayer
Sent: Friday, October 04, 2019 7:50 AM
To: M Ranga Swami Reddy
Cc: ceph-users; d...@ceph.io
Subject: Re: [ceph-users] rgw: multisite support

Maybe this will help you:
https://docs.ceph.com/docs/master/radosgw/multisite/#migrating-a-single-site-system-to-multi-site

___

Clyso GmbH


Am 03.10.2019 um 13:32 schrieb M Ranga Swami Reddy:
Thank you. Do we have a quick document to do this migration? 

Thanks
Swami

On Thu, Oct 3, 2019 at 4:38 PM Paul Emmerich  wrote:
On Thu, Oct 3, 2019 at 12:03 PM M Ranga Swami Reddy
 wrote:
>
> Below url says: "Switching from a standalone deployment to a multi-site 
> replicated deployment is not supported.
> https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/latest/app-rgw-multisite.html

this is wrong, might be a weird openstack-specific restriction.

Migrating single-site to multi-site is trivial, you just add the second site.


Paul

>
> Please advise.
>
>
> On Thu, Oct 3, 2019 at 3:28 PM M Ranga Swami Reddy  
> wrote:
>>
>> Hi,
>> Iam using the 2 ceph clusters in diff DCs (away by 500 KM) with ceph 12.2.11 
>> version.
>> Now, I want to setup rgw multisite using the above 2 ceph clusters.
>>
>> is it possible? if yes, please share good document to do the same.
>>
>> Thanks
>> Swami
>
> ___
> Dev mailing list -- d...@ceph.io
> To unsubscribe send an email to dev-le...@ceph.io


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Mutliple CephFS Filesystems Nautilus (14.2.2)

[DHilsbos]

All;

How experimental is the multiple CephFS filesystems per cluster feature?  We 
plan to use different sets of pools (meta / data) per filesystem.

Are there any known issues?

While we're on the subject, is it possible to assign a different active MDS to 
each filesystem?

Thank you,

Dominic L. Hilsbos, MBA 
Director - Information Technology 
Perform Air International Inc.
dhils...@performair.com 
www.PerformAir.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] New Cluster Failing to Start (Resolved)

[DHilsbos]

All;

We found the problem, we had the v2 ports incorrect in the monmap.

Thank you,

Dominic L. Hilsbos, MBA 
Director - Information Technology 
Perform Air International Inc.
dhils...@performair.com 
www.PerformAir.com



-Original Message-
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of 
dhils...@performair.com
Sent: Wednesday, August 14, 2019 10:13 AM
To: ceph-users@lists.ceph.com
Subject: [ceph-users] New Cluster Failing to Start

All;

We're working to deploy our first production Ceph cluster, and we've run into a 
snag.

The MONs start, but the "cluster" doesn't appear to come up.  Ceph -s never 
returns.

These are the last lines in the event log of one of the mons:
2019-08-13 16:20:03.706 7f668108f180  0 starting mon.s700034 rank 0 at public 
addrs [v2:10.0.80.10:3330/0,v1:10.0.80.10:6789/0] at bind addrs 
[v2:10.0.80.10:3330/0,v1:10.0.80.10:6789/0] mon_data 
/var/lib/ceph/mon/ceph-s700034 fsid effc5134-e0cc-4628-a079-d67b60071f90
2019-08-13 16:20:03.709 7f668108f180  1 mon.s700034@-1(???) e0 preinit fsid 
effc5134-e0cc-4628-a079-d67b60071f90
2019-08-13 16:20:03.709 7f668108f180  1 mon.s700034@-1(???) e0  initial_members 
s700034,s700035,s700036, filtering seed monmap
2019-08-13 16:20:03.713 7f668108f180  0 mon.s700034@-1(probing) e0  my rank is 
now 0 (was -1)

Aside from the address and hostname, the others logs end with the same 
statements.

I'm not seeing the log entries that I would expect as each MON joins the 
cluster, nor am I seeing the "cluster" log files being generated (i.e. I'm used 
to seeing ceph.log, and ceph-audit.log on one of the MONs).

Each machine can ping the others.  Firewall rules are in place for ports 330 & 
6789.

Any idea what I'm missing?

Thank you,

Dominic L. Hilsbos, MBA 
Director - Information Technology 
Perform Air International Inc.
dhils...@performair.com 
www.PerformAir.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] New Cluster Failing to Start

[DHilsbos]

All;

We're working to deploy our first production Ceph cluster, and we've run into a 
snag.

The MONs start, but the "cluster" doesn't appear to come up.  Ceph -s never 
returns.

These are the last lines in the event log of one of the mons:
2019-08-13 16:20:03.706 7f668108f180  0 starting mon.s700034 rank 0 at public 
addrs [v2:10.0.80.10:3330/0,v1:10.0.80.10:6789/0] at bind addrs 
[v2:10.0.80.10:3330/0,v1:10.0.80.10:6789/0] mon_data 
/var/lib/ceph/mon/ceph-s700034 fsid effc5134-e0cc-4628-a079-d67b60071f90
2019-08-13 16:20:03.709 7f668108f180  1 mon.s700034@-1(???) e0 preinit fsid 
effc5134-e0cc-4628-a079-d67b60071f90
2019-08-13 16:20:03.709 7f668108f180  1 mon.s700034@-1(???) e0  initial_members 
s700034,s700035,s700036, filtering seed monmap
2019-08-13 16:20:03.713 7f668108f180  0 mon.s700034@-1(probing) e0  my rank is 
now 0 (was -1)

Aside from the address and hostname, the others logs end with the same 
statements.

I'm not seeing the log entries that I would expect as each MON joins the 
cluster, nor am I seeing the "cluster" log files being generated (i.e. I'm used 
to seeing ceph.log, and ceph-audit.log on one of the MONs).

Each machine can ping the others.  Firewall rules are in place for ports 330 & 
6789.

Any idea what I'm missing?

Thank you,

Dominic L. Hilsbos, MBA 
Director - Information Technology 
Perform Air International Inc.
dhils...@performair.com 
www.PerformAir.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] WAL/DB size

[DHilsbos]

Wildo / Hemant;

Current recommendations (since at least luminous) say that a block.db device 
should be at least 4% of the block device.  For a 6 TB drive, this would be 240 
GB, not 60 GB.

Thank you,

Dominic L. Hilsbos, MBA 
Director – Information Technology 
Perform Air International Inc.
dhils...@performair.com 
www.PerformAir.com



-Original Message-
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Wido 
den Hollander
Sent: Tuesday, August 13, 2019 12:51 PM
To: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] WAL/DB size



On 8/13/19 5:54 PM, Hemant Sonawane wrote:
> Hi All,
> I have 4 6TB of HDD and 2 450GB SSD and I am going to partition each
> disk to 220GB for rock.db. So my question is does it make sense to use
> wal for my configuration? if yes then what could be the size of it? help
> will be really appreciated.

Yes, the WAL needs to be about 1GB in size. That should work in allmost
all configurations.

220GB is more then you need for the DB as well. It's doesn't hurt, but
it's not needed. For each 6TB drive you need about ~60GB of space for
the DB.

Wido

> -- 
> Thanks and Regards,
> 
> Hemant Sonawane
> 
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] More than 100% in a dashboard PG Status

[DHilsbos]

All;

I also noticed this behavior. It may have started after inducing a failure in 
the cluster in order to observe the self-healing behavior.

 In the "PG Status" section of the dashboard, I have "Clean (200%)."  This has 
not seemed to affect the functioning of the cluster.

Cluster is a new deployment, using nautilus (14.2.2).

Are there any commands I can run on the cluster to show what the numbers under 
this look like?

Thank you,

Dominic L. Hilsbos, MBA 
Director - Information Technology 
Perform Air International Inc.
dhils...@performair.com 
www.PerformAir.com


-Original Message-
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Fyodor 
Ustinov
Sent: Tuesday, August 13, 2019 3:48 AM
To: ceph-users
Subject: [ceph-users] More than 100% in a dashboard PG Status

Hi!

I have ceph nautilus (upgraded from mimic, if it is important) and in dashboard 
in "PG Status" section I see "Clean (2397%)"

It's a bug?

WBR,
Fyodor.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Error Mounting CephFS

[DHilsbos]

JC;

Excellent, thank you!

I apologize, normally I'm better about RTFM...

Thank you,

Dominic L. Hilsbos, MBA 
Director – Information Technology 
Perform Air International Inc.
dhils...@performair.com 
www.PerformAir.com


From: JC Lopez [mailto:jelo...@redhat.com] 
Sent: Wednesday, August 07, 2019 11:52 AM
To: Dominic Hilsbos
Cc: Lopez Jean-Charles; fr...@dtu.dk; ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Error Mounting CephFS

Hi,

See https://docs.ceph.com/docs/nautilus/cephfs/kernel/

-o mds_namespace={fsname}

Regards
JC

On Aug 7, 2019, at 10:24, dhils...@performair.com wrote:

All;

Thank you for your assistance, this led me to the fact that I hadn't set up the 
Ceph repo on this client server, and the ceph-common I had installed was 
version 10.

I got all of that squared away, and it all works.

I do have a couple follow up questions:
Can more than one system mount the same  CephFS, at the same time?
If your cluster has several CephFS filesystems defined, how do you select which 
gets mounted, as the fs name doesn't appear to be used in the mount command?

Thank you,

Dominic L. Hilsbos, MBA 
Director - Information Technology 
Perform Air International Inc.
dhils...@performair.com 
www.PerformAir.com



-Original Message-
From: Frank Schilder [mailto:fr...@dtu.dk] 
Sent: Wednesday, August 07, 2019 2:48 AM
To: Dominic Hilsbos
Cc: ceph-users
Subject: Re: [ceph-users] Error Mounting CephFS

On Centos7, the option "secretfile" requires installation of ceph-fuse.

Best regards,

=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14


From: ceph-users  on behalf of Yan, Zheng 

Sent: 07 August 2019 10:10:19
To: dhils...@performair.com
Cc: ceph-users
Subject: Re: [ceph-users] Error Mounting CephFS

On Wed, Aug 7, 2019 at 3:46 PM  wrote:


All;

I have a server running CentOS 7.6 (1810), that I want to set up with CephFS 
(full disclosure, I'm going to be running samba on the CephFS).  I can mount 
the CephFS fine when I use the option secret=, but when I switch to 
secretfile=, I get an error "No such process."  I installed ceph-common.

Is there a service that I'm not aware I should be starting?
Do I need to install another package?

mount.ceph is missing.  check if it exists and is located in $PATH


Thank you,

Dominic L. Hilsbos, MBA
Director - Information Technology
Perform Air International Inc.
dhils...@performair.com
www.PerformAir.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Error Mounting CephFS

[DHilsbos]

All;

Thank you for your assistance, this led me to the fact that I hadn't set up the 
Ceph repo on this client server, and the ceph-common I had installed was 
version 10.

I got all of that squared away, and it all works.

I do have a couple follow up questions:
Can more than one system mount the same  CephFS, at the same time?
If your cluster has several CephFS filesystems defined, how do you select which 
gets mounted, as the fs name doesn't appear to be used in the mount command?

Thank you,

Dominic L. Hilsbos, MBA 
Director - Information Technology 
Perform Air International Inc.
dhils...@performair.com 
www.PerformAir.com



-Original Message-
From: Frank Schilder [mailto:fr...@dtu.dk] 
Sent: Wednesday, August 07, 2019 2:48 AM
To: Dominic Hilsbos
Cc: ceph-users
Subject: Re: [ceph-users] Error Mounting CephFS

On Centos7, the option "secretfile" requires installation of ceph-fuse.

Best regards,

=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14


From: ceph-users  on behalf of Yan, Zheng 

Sent: 07 August 2019 10:10:19
To: dhils...@performair.com
Cc: ceph-users
Subject: Re: [ceph-users] Error Mounting CephFS

On Wed, Aug 7, 2019 at 3:46 PM  wrote:
>
> All;
>
> I have a server running CentOS 7.6 (1810), that I want to set up with CephFS 
> (full disclosure, I'm going to be running samba on the CephFS).  I can mount 
> the CephFS fine when I use the option secret=, but when I switch to 
> secretfile=, I get an error "No such process."  I installed ceph-common.
>
> Is there a service that I'm not aware I should be starting?
> Do I need to install another package?
>

mount.ceph is missing.  check if it exists and is located in $PATH

> Thank you,
>
> Dominic L. Hilsbos, MBA
> Director - Information Technology
> Perform Air International Inc.
> dhils...@performair.com
> www.PerformAir.com
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] RadosGW (Ceph Object Gateay) Pools

[DHilsbos]

All;

Based on the PG Calculator, on the Ceph website, I have this list of pools to 
pre-create for my Object Gateway:
.rgw.root
default.rgw.control
default.rgw.data.root
default.rgw.gc
default.rgw.log
default.rgw.intent-log
default.rgw.meta
default.rgw.usage
default.rgw.users.keys
default.rgw.users.email
default.rgw.users.uid
default.rgw.buckets.extra
default.rgw.buckets.index
default.rgw.buckets.data

I have a limited amount of SSDs, and I plan to create rules which limit pools 
to either HDD or SSD.  My HDDs have their block.db on NVMe devices.

I intend to use the SSDs primarily to back RBD for ISCSi, to support 
virtualization, but I'm not opposed to using some of the space to speed up RGW.

Which pool(s) would have the most impact on the performance of RGW to have on 
SSDs?

Thank you,

Dominic L. Hilsbos, MBA 
Director - Information Technology 
Perform Air International Inc.
dhils...@performair.com 
www.PerformAir.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Error Mounting CephFS

[DHilsbos]

All;

I have a server running CentOS 7.6 (1810), that I want to set up with CephFS 
(full disclosure, I'm going to be running samba on the CephFS).  I can mount 
the CephFS fine when I use the option secret=, but when I switch to 
secretfile=, I get an error "No such process."  I installed ceph-common.

Is there a service that I'm not aware I should be starting?
Do I need to install another package?

Thank you,

Dominic L. Hilsbos, MBA 
Director - Information Technology 
Perform Air International Inc.
dhils...@performair.com 
www.PerformAir.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] even number of monitors

[DHilsbos]

All;

While most discussion of MONs, and their failure modes revolves around the 
failure of the MONs themselves, the recommendation for od numbers of MONs has 
nothing to do with the loss of one or more MONs.  It's actually in response to 
the split brain problem.

Imagine you have the following (where 's" is a switch):
s1--mon1
|
s2--mon2
|
s3--mon3
|
s4--mon4

Now imagine what happens when the link between s2 and s3 breaks (imagine 
accidently pulling the wrong cable, port failure on switch, WAN fiber cut, etc.)
s1--mon1
|
s2--mon2
X
s3--mon3
|
s4--mon4

All 4 MONs are alive, but which has the official state of the cluster?  Which 
MON(s) can make decisions on behalf of the cluster?

Now imagine a similar situation for 3 MONs:
s1--mon1
X
s2--mon2
|
s3--mon3

or:
s1--mon1
|
s2--mon2
X
s3--mon3

The cluster can continue.

Similarly imagine 5 MONs:
s1--mon1
X
s2--mon2
|
s3--mon3
|
s4--mon4
|
s5--mon5
or:
s1--mon1
|
s2--mon2
X
s3--mon3
|
s4--mon4
|
s5--mon5
or:
s1--mon1
|
s2--mon2
|
s3--mon3
X
s4--mon4
|
s5--mon5
or:
s1--mon1
|
s2--mon2
|
s3--mon3
|
s4--mon4
X
s5--mon5

In each case, one side retains a quorum; enough MONs to definitively make 
decisions on behalf of the cluster. 

Note that it is just as important, in solving the split-brain problem, to 
recognize when you are NOT in the quorum (and thus should not make decisions), 
as to recognize when you are.

Within a single datacenter it is relatively easy to ensure that this kind of 
failure shouldn't occur (ring-style switch stacking for instance), but imagine 
that you cluster covers a good portion of the Eastern U.S., with MON(s) in 
Philadelphia, New York, and Baltimore.  Can you achieve redundant interconnects 
without going through the same fiber bundler?

Thank you,

Dominic L. Hilsbos, MBA 
Director - Information Technology 
Perform Air International Inc.
dhils...@performair.com 
www.PerformAir.com


-Original Message-
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of 
Alfredo Daniel Rezinovsky
Sent: Monday, August 05, 2019 3:28 AM
To: ceph-users
Subject: [ceph-users] even number of monitors

With 3 monitors, paxos needs at least 2 to reach consensus about the 
cluster status

With 4 monitors, more than half is 3. The only problem I can see here is 
that I will have only 1 spare monitor.

There's any other problem with and even number of monitors?

--
Alfrenovsky

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] [Disarmed] Re: ceph-ansible firewalld blocking ceph comms

[DHilsbos]

Nathan;

I'm not an expert on firewalld, but shouldn't you have a list of open ports?

 ports: ?

Here's the configuration on my test cluster:
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: bond0
  sources:
  services: ssh dhcpv6-client
  ports: 6789/tcp 3300/tcp 6800-7300/tcp 8443/tcp
  protocols:
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:
trusted (active)
  target: ACCEPT
  icmp-block-inversion: no
  interfaces: bond1
  sources:
  services:
  ports: 6789/tcp 3300/tcp 6800-7300/tcp 8443/tcp
  protocols:
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:

I use interfaces as selectors, but would think source selectors would work the 
same.

You might start by adding the MON ports to the firewall on the MONs:
firewall-cmd --zone=public --add-port=6789/tcp --permanent
firewall-cmd --zone=public --add-port=3300/tcp --permanent
firewall-cmd --reload

Thank you,

Dominic L. Hilsbos, MBA 
Director – Information Technology 
Perform Air International Inc.
dhils...@performair.com 
www.PerformAir.com


From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Nathan 
Harper
Sent: Thursday, July 25, 2019 2:08 PM
To: ceph-us...@ceph.com
Subject: [Disarmed] Re: [ceph-users] ceph-ansible firewalld blocking ceph comms

This is a new issue to us, and did not have the same problem running the same 
activity on our test system. 
Regards,
Nathan

On 25 Jul 2019, at 22:00, solarflow99  wrote:
I used ceph-ansible just fine, never had this problem.  

On Thu, Jul 25, 2019 at 1:31 PM Nathan Harper  wrote:
Hi all,

We've run into a strange issue with one of our clusters managed with 
ceph-ansible.   We're adding some RGW nodes to our cluster, and so re-ran 
site.yml against the cluster.  The new RGWs added successfully, but

When we did, we started to get slow requests, effectively across the whole 
cluster.   Quickly we realised that the firewall was now (apparently) blocking 
Ceph communications.   I say apparently, because the config looks correct:

[root@osdsrv05 ~]# firewall-cmd --list-all
public (active)
  target: default
  icmp-block-inversion: no
  interfaces:
  sources: MailScanner has detected a possible fraud attempt from "172.20.22.0" 
claiming to be 172.20.22.0/24 MailScanner has detected a possible fraud attempt 
from "172.20.23.0" claiming to be 172.20.23.0/24
  services: ssh dhcpv6-client ceph
  ports:
  protocols:
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:

If we drop the firewall everything goes back healthy.   All the clients 
(Openstack cinder) are on the 172.20.22.0 network (172.20.23.0 is the 
replication network).  Has anyone seen this?
-- 
Nathan Harper // IT Systems Lead

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] how to power off a cephfs cluster cleanly

[DHilsbos]

Dan;

I don't have  a lot of experience with Ceph, but I generally set all of the 
following before taking a cluster offline:
ceph osd set noout
ceph osd set nobackfill
ceph osd set norecover
ceph osd set norebalance
ceph osd set nodown
ceph osd set pause

I then unset them in the opposite order:

ceph osd unset pause
ceph osd unset nodown
ceph osd unset norebalance
ceph osd unset norecover
ceph osd unset nobackfill
ceph osd unset noout

This may be overkill though.

Will the MONs still have a quorum (i.e. will n / 2 + 1 still be running)?

Thank you,

Dominic L. Hilsbos, MBA 
Director - Information Technology 
Perform Air International Inc.
dhils...@performair.com 
www.PerformAir.com



-Original Message-
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Dan 
van der Ster
Sent: Thursday, July 25, 2019 7:48 AM
To: ceph-users
Subject: [ceph-users] how to power off a cephfs cluster cleanly

Hi all,

In September we'll need to power down a CephFS cluster (currently
mimic) for a several-hour electrical intervention.

Having never done this before, I thought I'd check with the list.
Here's our planned procedure:

1. umounts /cephfs from all hpc clients.
2. ceph osd set noout
3. wait until there is zero IO on the cluster
4. stop all mds's (active + standby)
5. stop all osds.
(6. we won't stop all mon's as they are not affected by that
electrical intervention)
7. power off the cluster.
...
8. power on the cluster, osd's first, then mds's. wait for health_ok.
9. ceph osd unset noout

Seems pretty simple... Are there any gotchas I'm missing? Maybe
there's some special procedure to stop the mds's cleanly?

Cheers, dan
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Kernel, Distro & Ceph

[DHilsbos]

All;

There's been a lot of discussion of various kernel versions on this list 
lately, so I thought I'd seek some clarification.

I prefer to run CentOS, and I prefer to keep the number of "extra" repositories 
to a minimum.  Ceph requires adding a Ceph repo, and the EPEL repo.  Updating 
the kernel requires (from the research I've done) adding EL-Repo.  I believe 
CentOS 7 uses the 3.10 kernel.

Under what circumstances would you recommend adding EL-Repo to CentOS 7.6, and 
installing kernel-ml?  Are there certain parts of Ceph which particularly 
benefit from kernels newer that 3.10?

Thank you,

Dominic L. Hilsbos, MBA 
Director - Information Technology 
Perform Air International Inc.
dhils...@performair.com 
www.PerformAir.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] MON / MDS Storage Location

[DHilsbos]

All;

Where, in the filesystem, do MONs and MDSs store their data?

Thank you,

Dominic L. Hilsbos, MBA 
Director - Information Technology 
Perform Air International Inc.
dhils...@performair.com 
www.PerformAir.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] MON DNS Lookup & Version 2 Protocol

[DHilsbos]

All;

I'm trying to firm up my understanding of how Ceph works, and ease of 
management tools and capabilities.

I stumbled upon this: 
http://docs.ceph.com/docs/nautilus/rados/configuration/mon-lookup-dns/

It got me wondering; how do you convey protocol version 2 capabilities in this 
format?

The examples all list port 6789, which is the port for protocol version 1.  
Would I add SRV records for port 3300?  How does the client distinguish v1 from 
v2 in this case?

Thank you,

Dominic L. Hilsbos, MBA 
Director - Information Technology 
Perform Air International, Inc.
dhils...@performair.com 
www.PerformAir.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Natlius, RBD-Mirroring & Cluster Names

[DHilsbos]

Paul;

If I understand you correctly:
I will have 2 clusters, each named "ceph" (internally).
As such, each will have a configuration file at: /etc/ceph/ceph.conf
I would copy the other clusters configuration file to something like: 
/etc/ceph/remote.conf
Then the commands (run on the local mirror) would look like this:
rbd mirror pool peer add image-pool [client-name]@ceph (uses default cluster 
name to reference local cluster)
rbd --cluster remote mirror pool add image-pool [client-name]@remote

Thank you,

Dominic L. Hilsbos, MBA 
Director – Information Technology 
Perform Air International, Inc.
dhils...@performair.com 
www.PerformAir.com


From: Paul Emmerich [mailto:paul.emmer...@croit.io] 
Sent: Monday, July 15, 2019 1:31 PM
To: Dominic Hilsbos
Cc: Ceph Users
Subject: Re: [ceph-users] Natlius, RBD-Mirroring & Cluster Names

No worries, that's just the names of the config files/keyrings on the mirror 
server which needs to access both clusters and hence two different ceph.conf 
files.

Paul

-- 
Paul Emmerich

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90


On Mon, Jul 15, 2019 at 10:25 PM  wrote:
All;

I'm digging deeper into the capabilities of Ceph, and I ran across this:
http://docs.ceph.com/docs/nautilus/rbd/rbd-mirroring/
Which seems really interesting, except...

This feature seems to require custom cluster naming to function, which is 
deprecated in Nautilus, and not all commands adhere to a passed cluster name 
parameter.

Does RBD-Mirroring still work in Nautilus?
Does RBD-Mirroring in Nautilus still depend on custom cluster names?
How does a custom cluster name get properly implemented?

Thank you,

Dominic L. Hilsbos, MBA 
Director - Information Technology 
Perform Air International, Inc.
dhils...@performair.com 
www.PerformAir.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Natlius, RBD-Mirroring & Cluster Names

[DHilsbos]

All;

I'm digging deeper into the capabilities of Ceph, and I ran across this:
http://docs.ceph.com/docs/nautilus/rbd/rbd-mirroring/
Which seems really interesting, except...

This feature seems to require custom cluster naming to function, which is 
deprecated in Nautilus, and not all commands adhere to a passed cluster name 
parameter.

Does RBD-Mirroring still work in Nautilus?
Does RBD-Mirroring in Nautilus still depend on custom cluster names?
How does a custom cluster name get properly implemented?

Thank you,

Dominic L. Hilsbos, MBA 
Director - Information Technology 
Perform Air International, Inc.
dhils...@performair.com 
www.PerformAir.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RADOSGW S3 - Continuation Token Ignored?

[DHilsbos]

Matt;

Yep, that would certainly explain it.

My apologies, I almost searched for that information before sending the email.

Thank you,

Dominic L. Hilsbos, MBA 
Director – Information Technology 
Perform Air International Inc.
dhils...@performair.com 
www.PerformAir.com



-Original Message-
From: Matt Benjamin [mailto:mbenj...@redhat.com] 
Sent: Friday, June 28, 2019 9:48 AM
To: Dominic Hilsbos
Cc: ceph-users
Subject: Re: [ceph-users] RADOSGW S3 - Continuation Token Ignored?

Hi Dominic,

The reason is likely that RGW doesn't yet support ListObjectsV2.

Support is nearly here though:  https://github.com/ceph/ceph/pull/28102

Matt


On Fri, Jun 28, 2019 at 12:43 PM  wrote:
>
> All;
>
> I've got a RADOSGW instance setup, backed by my demonstration Ceph cluster.  
> I'm using Amazon's S3 SDK, and I've run into an annoying little snag.
>
> My code looks like this:
> amazonS3 = builder.build();
>
> ListObjectsV2Request req = new 
> ListObjectsV2Request().withBucketName("WorkOrder").withMaxKeys(MAX_KEYS);
> ListObjectsV2Result result;
>
> do
> {
> result = amazonS3.listObjectsV2(req);
>
> for (S3ObjectSummary objectSummary : result.getObjectSummaries())
> {
> summaries.add(objectSummary);
> }
>
> String token = result.getNextContinuationToken();
> req.setContinuationToken(token);
> }
> while (result.isTruncated());
>
> The problem is, the ContinuationToken seems to be ignored, i.e. every call to 
> amazonS3.listObjectsV2(req) returns the same set, and the loop never ends 
> (until the summaries LinkedList overflows).
>
> Thoughts?
>
> Thank you,
>
> Dominic L. Hilsbos, MBA
> Director - Information Technology
> Perform Air International Inc.
> dhils...@performair.com
> www.PerformAir.com
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>


--

Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103

http://www.redhat.com/en/technologies/storage

tel.  734-821-5101
fax.  734-769-8938
cel.  734-216-5309
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] RADOSGW S3 - Continuation Token Ignored?

[DHilsbos]

All;

I've got a RADOSGW instance setup, backed by my demonstration Ceph cluster.  
I'm using Amazon's S3 SDK, and I've run into an annoying little snag.

My code looks like this:
amazonS3 = builder.build();

ListObjectsV2Request req = new 
ListObjectsV2Request().withBucketName("WorkOrder").withMaxKeys(MAX_KEYS);
ListObjectsV2Result result;

do
{
result = amazonS3.listObjectsV2(req);

for (S3ObjectSummary objectSummary : result.getObjectSummaries())
{
summaries.add(objectSummary);
}

String token = result.getNextContinuationToken();
req.setContinuationToken(token);
}
while (result.isTruncated());

The problem is, the ContinuationToken seems to be ignored, i.e. every call to 
amazonS3.listObjectsV2(req) returns the same set, and the loop never ends 
(until the summaries LinkedList overflows).

Thoughts?

Thank you,

Dominic L. Hilsbos, MBA 
Director - Information Technology 
Perform Air International Inc.
dhils...@performair.com 
www.PerformAir.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] MGR Logs after Failure Testing

[DHilsbos]

Eugen;

All services are running, yes, though they didn't all start when I brought the 
host up (configured not to start because the last thing I had done is 
physically relocate the entire cluster).

All services are running, and happy.

# ceph status
  cluster:
id: 1a8a1693-fa54-4cb3-89d2-7951d4cee6a3
health: HEALTH_OK

  services:
mon: 3 daemons, quorum S700028,S700029,S700030 (age 20h)
mgr: S700028(active, since 17h), standbys: S700029, S700030
mds: cifs:1 {0=S700029=up:active} 2 up:standby
osd: 6 osds: 6 up (since 21h), 6 in (since 21h)

  data:
pools:   16 pools, 192 pgs
objects: 449 objects, 761 MiB
usage:   724 GiB used, 65 TiB / 66 TiB avail
pgs: 192 active+clean

# ceph osd tree
ID CLASS WEIGHT   TYPE NAMESTATUS REWEIGHT PRI-AFF
-1   66.17697 root default
-5   22.05899 host S700029
 2   hdd 11.02950 osd.2up  1.0 1.0
 3   hdd 11.02950 osd.3up  1.0 1.0
-7   22.05899 host S700030
 4   hdd 11.02950 osd.4up  1.0 1.0
 5   hdd 11.02950 osd.5up  1.0 1.0
-3   22.05899 host s700028
 0   hdd 11.02950 osd.0up  1.0 1.0
 1   hdd 11.02950 osd.1up  1.0 1.0

The question about configuring the MDS as failover struck me as a potential, 
since I don't remember doing that, however it look like S700029 (10.0.200.111) 
took over from S700028 (10.0.200.110) as the active MDS.

Thank you,

Dominic L. Hilsbos, MBA 
Director - Information Technology 
Perform Air International Inc.
dhils...@performair.com 
www.PerformAir.com



-Original Message-
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Eugen 
Block
Sent: Thursday, June 27, 2019 8:23 AM
To: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] MGR Logs after Failure Testing

Hi,

some more information about the cluster status would be helpful, such as

ceph -s
ceph osd tree

service status of all MONs, MDSs, MGRs.
Are all services up? Did you configure the spare MDS as standby for  
rank 0 so that a failover can happen?

Regards,
Eugen


Zitat von dhils...@performair.com:

> All;
>
> I built a demonstration and testing cluster, just 3 hosts  
> (10.0.200.110, 111, 112).  Each host runs mon, mgr, osd, mds.
>
> During the demonstration yesterday, I pulled the power on one of the hosts.
>
> After bringing the host back up, I'm getting several error messages  
> every second or so:
> 2019-06-26 16:01:56.424 7fcbe0af9700  0 ms_deliver_dispatch:  
> unhandled message 0x55e80a728f00 mgrreport(mds.S700030 +0-0 packed  
> 6) v7 from mds.? v2:10.0.200.112:6808/980053124
> 2019-06-26 16:01:56.425 7fcbf4cd1700  1 mgr finish mon failed to  
> return metadata for mds.S700030: (2) No such file or directory
> 2019-06-26 16:01:56.429 7fcbe0af9700  0 ms_deliver_dispatch:  
> unhandled message 0x55e809f8e600 mgrreport(mds.S700029 +110-0 packed  
> 1366) v7 from mds.0 v2:10.0.200.111:6808/2726495738
> 2019-06-26 16:01:56.430 7fcbf4cd1700  1 mgr finish mon failed to  
> return metadata for mds.S700029: (2) No such file or directory
>
> Thoughts?
>
> Thank you,
>
> Dominic L. Hilsbos, MBA
> Director - Information Technology
> Perform Air International Inc.
> dhils...@performair.com
> www.PerformAir.com
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] MGR Logs after Failure Testing

[DHilsbos]

All;

I built a demonstration and testing cluster, just 3 hosts (10.0.200.110, 111, 
112).  Each host runs mon, mgr, osd, mds.

During the demonstration yesterday, I pulled the power on one of the hosts.

After bringing the host back up, I'm getting several error messages every 
second or so:
2019-06-26 16:01:56.424 7fcbe0af9700  0 ms_deliver_dispatch: unhandled message 
0x55e80a728f00 mgrreport(mds.S700030 +0-0 packed 6) v7 from mds.? 
v2:10.0.200.112:6808/980053124
2019-06-26 16:01:56.425 7fcbf4cd1700  1 mgr finish mon failed to return 
metadata for mds.S700030: (2) No such file or directory
2019-06-26 16:01:56.429 7fcbe0af9700  0 ms_deliver_dispatch: unhandled message 
0x55e809f8e600 mgrreport(mds.S700029 +110-0 packed 1366) v7 from mds.0 
v2:10.0.200.111:6808/2726495738
2019-06-26 16:01:56.430 7fcbf4cd1700  1 mgr finish mon failed to return 
metadata for mds.S700029: (2) No such file or directory

Thoughts?

Thank you,

Dominic L. Hilsbos, MBA 
Director - Information Technology 
Perform Air International Inc.
dhils...@performair.com 
www.PerformAir.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Nautilus HEALTH_WARN for msgr2 protocol

[DHilsbos]

Bob;

Have you verified that port 3300 is open for TCP on that host?

The extra host firewall rules for v2 protocol caused me all kinds of grief when 
I was setting up my MONs.

Thank you,

Dominic L. Hilsbos, MBA 
Director – Information Technology 
Perform Air International Inc.
dhils...@performair.com 
www.PerformAir.com


From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Paul 
Emmerich
Sent: Friday, June 14, 2019 10:23 AM
To: Brett Chancellor
Cc: ceph-users
Subject: Re: [ceph-users] Nautilus HEALTH_WARN for msgr2 protocol



On Fri, Jun 14, 2019 at 6:23 PM Brett Chancellor  
wrote:
If you don't figure out how to enable it on your monitor, you can always 
disable it to squash the warnings
ceph config set mon.node01 ms_bind_msgr2 false

No, that just disables msgr2 on that mon. 

Use this option if you want to disable the warning

mon_warn_on_msgr2_not_enabled false


But that's probably not a good idea since there's clearly something wrong with 
that mon.

Paul


-- 
Paul Emmerich

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
 


On Fri, Jun 14, 2019 at 12:11 PM Bob Farrell  wrote:
Hi. Firstly thanks to all involved in this great mailing list, I learn lots 
from it every day.

We are running Ceph with a huge amount of success to store website 
themes/templates across a large collection of websites. We are very pleased 
with the solution in every way.

The only issue we have, which we have had since day 1, is we always see 
HEALTH_WARN:

    health: HEALTH_WARN
            1 monitors have not enabled msgr2

And this is reflected in the monmap:

monmaptool: monmap file /tmp/monmap
epoch 7
fsid 7273720d-04d7-480f-a77c-f0207ae35852
last_changed 2019-04-02 17:21:56.935381
created 2019-04-02 17:21:09.925941
min_mon_release 14 (nautilus)
0: v1:172.30.0.144:6789/0 mon.node01.homeflow.co.uk
1: [v2:172.30.0.146:3300/0,v1:172.30.0.146:6789/0] mon.node03.homeflow.co.uk
2: [v2:172.30.0.147:3300/0,v1:172.30.0.147:6789/0] mon.node04.homeflow.co.uk
3: [v2:172.30.0.148:3300/0,v1:172.30.0.148:6789/0] mon.node05.homeflow.co.uk
4: [v2:172.30.0.145:3300/0,v1:172.30.0.145:6789/0] mon.node02.homeflow.co.uk
5: [v2:172.30.0.149:3300/0,v1:172.30.0.149:6789/0] mon.node06.homeflow.co.uk
6: [v2:172.30.0.150:3300/0,v1:172.30.0.150:6789/0] mon.node07.homeflow.co.uk

I never figured out the correct syntax to set up the first monitor to use both 
6789 and 3300. The other monitors that join the cluster set this config 
automatically but I couldn't work out how to apply it to the first monitor node.

The cluster has been operating in production for at least a month now with no 
issues at all, so it would be nice to remove this warning as, at the moment, 
it's not really very useful as a monitoring metric.

Could somebody advise me on the safest/most sensible way to update the monmap 
so that node01 listens on v2 and v1 ?

Thanks for any help !
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Ceph Cluster Replication / Disaster Recovery

[DHilsbos]

All;

I'm testing and evaluating Ceph for the next generation of storage architecture 
for our company, and so far I'm fairly impressed, but I've got a couple of 
questions around cluster replication and disaster recovery.

First; intended uses.
Ceph Object Gateway will be used to support new software projects presently in 
the works.
CephFS behind Samba will be used for Windows file shares both during 
transition, and to support long term needs.
The ISCSi gateway and RADOS Block Devices will be used to support 
virtualization.

My research suggests that native replication isn't available within the Ceph 
Cluster (i.e. have a cluster replicate all objects to a second cluster).  
RADOSgw support replicating objects into more than one Ceph cluster, but I 
can't find information on multi-site / replication for ISCSigw or CephFS.

So... How do you plan / manage major event disaster recovery with your Ceph 
Clusters (i.e. loss of the entire cluster)?
What backup solutions do you use / recommend with your Ceph clusters?  Are you 
doing any off-site backup?
Anyone backing up to the cloud?  What kind of bandwidth are you using for this?

Thank you,

Dominic L. Hilsbos, MBA 
Director - Information Technology 
Perform Air International Inc.
dhils...@performair.com 
www.PerformAir.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] radosgw dying

[DHilsbos]

All;

Thank you to all who assisted, this was the problem!

My default PG/pool was too high for my total OSD count, and it was unable to 
create all of these pools.

I remove the other pools I had created, and reduced the default PGs / pool, and 
radosgw was able to create all of its default pools, and is now running 
properly.

Tank you,

Dominic L. Hilsbos, MBA
Director - Information Technology
Perform Air International Inc.
dhils...@performair.com
www.PerformAir.com

From: Torben Hørup [tor...@t-hoerup.dk]
Sent: Sunday, June 09, 2019 11:12 AM
To: Paul Emmerich
Cc: Dominic Hilsbos; Ceph Users
Subject: Re: [ceph-users] radosgw dying

For just core rgw services it will need these 4
.rgw.root



  default.rgw.control



   default.rgw.meta
default.rgw.log

When creating buckets and uploading data RGW will need additional 3:

default.rgw.buckets.index
default.rgw.buckets.non-ec
default.rgw.buckets.data

/Torben


On 09.06.2019 19:34, Paul Emmerich wrote:

> rgw uses more than one pool. (5 or 6 IIRC)
>
> --
> Paul Emmerich
>
> Looking for help with your Ceph cluster? Contact us at https://croit.io
>
> croit GmbH
> Freseniusstr. 31h
> 81247 München
> www.croit.io
> Tel: +49 89 1896585 90
>
> On Sun, Jun 9, 2019 at 7:00 PM  wrote:
>
> Huan;
>
> I get that, but the pool already exists, why is radosgw trying to
> create one?
>
> Dominic Hilsbos
>
> Get Outlook for Android
>
> On Sat, Jun 8, 2019 at 2:55 AM -0700, "huang jun" 
> wrote:
>
> From the error message, i'm decline to that 'mon_max_pg_per_osd' was
> exceed,
> you can check the value of it, and its default value is 250, so you
> can at most have 1500pgs(250*6osds),
> and for replicated pools with size=3, you can have 500pgs for all
> pools,
> you already have 448pgs, so the next pool can create at most
> 500-448=52pgs.
>
> 于2019年6月8日周六 下午2:41写道：
>>
>> All;
>>
>> I have a test and demonstration cluster running (3 hosts, MON, MGR, 2x
>> OSD per host), and I'm trying to add a 4th host for gateway purposes.
>>
>> The radosgw process keeps dying with:
>> 2019-06-07 15:59:50.700 7fc4ef273780  0 ceph version 14.2.1
>> (d555a9489eb35f84f2e1ef49b77e19da9d113972) nautilus (stable), process
>> radosgw, pid 17588
>> 2019-06-07 15:59:51.358 7fc4ef273780  0 rgw_init_ioctx ERROR:
>> librados::Rados::pool_create returned (34) Numerical result out of
>> range (this can be due to a pool or placement group misconfiguration,
>> e.g. pg_num < pgp_num or mon_max_pg_per_osd exceeded)
>> 2019-06-07 15:59:51.396 7fc4ef273780 -1 Couldn't init storage provider
>> (RADOS)
>>
>> The .rgw.root pool already exists.
>>
>> ceph status returns:
>> cluster:
>> id: 1a8a1693-fa54-4cb3-89d2-7951d4cee6a3
>> health: HEALTH_OK
>>
>> services:
>> mon: 3 daemons, quorum S700028,S700029,S700030 (age 30m)
>> mgr: S700028(active, since 47h), standbys: S700030, S700029
>> osd: 6 osds: 6 up (since 2d), 6 in (since 3d)
>>
>> data:
>> pools:   5 pools, 448 pgs
>> objects: 12 objects, 1.2 KiB
>> usage:   722 GiB used, 65 TiB / 66 TiB avail
>> pgs: 448 active+clean
>>
>> and ceph osd tree returns:
>> ID CLASS WEIGHT   TYPE NAMESTATUS REWEIGHT PRI-AFF
>> -1   66.17697 root default
>> -5   22.05899 host S700029
>> 2   hdd 11.02950 osd.2up  1.0 1.0
>> 3   hdd 11.02950 osd.3up  1.0 1.0
>> -7   22.05899 host S700030
>> 4   hdd 11.02950 osd.4up  1.0 1.0
>> 5   hdd 11.02950 osd.5up  1.0 1.0
>> -3   22.05899 host s700028
>> 0   hdd 11.02950 osd.0up  1.0 1.0
>> 1   hdd 11.02950 osd.1up  1.0 1.0
>>
>> Any thoughts on what I'm missing?
>>
>> Thank you,
>>
>> Dominic L. Hilsbos, MBA
>> Director - Information Technology
>> Perform Air International Inc.
>> dhils...@performair.com
>> www.PerformAir.com
>>
>>
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
> --
> Thank you!
> HuangJun
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] radosgw dying

[DHilsbos]

Certainly.

Output of ceph osd df:
ID CLASS WEIGHT   REWEIGHT SIZE   RAW USE DATAOMAP META  AVAIL  %USE VAR  
PGS STATUS 
 2   hdd 11.02950  1.0 11 TiB 120 GiB  51 MiB  0 B 1 GiB 11 TiB 1.07 1.00 
227 up 
 3   hdd 11.02950  1.0 11 TiB 120 GiB  51 MiB  0 B 1 GiB 11 TiB 1.07 1.00 
221 up 
 4   hdd 11.02950  1.0 11 TiB 120 GiB  51 MiB  0 B 1 GiB 11 TiB 1.07 1.00 
226 up 
 5   hdd 11.02950  1.0 11 TiB 120 GiB  51 MiB  0 B 1 GiB 11 TiB 1.07 1.00 
222 up 
 0   hdd 11.02950  1.0 11 TiB 120 GiB  51 MiB  0 B 1 GiB 11 TiB 1.07 1.00 
217 up 
 1   hdd 11.02950  1.0 11 TiB 120 GiB  51 MiB  0 B 1 GiB 11 TiB 1.07 1.00 
231 up 
 TOTAL 66 TiB 722 GiB 306 MiB  0 B 6 GiB 65 TiB 1.07
 
MIN/MAX VAR: 1.00/1.00  STDDEV: 0

Thank you,

Dominic Hilsbos
Perform Air International Inc.

From: 
Sent: Saturday, June 08, 2019 3:35 AM
To: Dominic Hilsbos
Subject: Re: [ceph-users] radosgw dying

Can you post this?

ceph osd df

On Fri, Jun 7, 2019 at 7:31 PM 
mailto:dhils...@performair.com>> wrote:
All;

I have a test and demonstration cluster running (3 hosts, MON, MGR, 2x OSD per 
host), and I'm trying to add a 4th host for gateway purposes.

The radosgw process keeps dying with:
2019-06-07 15:59:50.700 7fc4ef273780  0 ceph version 14.2.1 
(d555a9489eb35f84f2e1ef49b77e19da9d113972) nautilus (stable), process radosgw, 
pid 17588
2019-06-07 15:59:51.358 7fc4ef273780  0 rgw_init_ioctx ERROR: 
librados::Rados::pool_create returned (34) Numerical result out of range (this 
can be due to a pool or placement group misconfiguration, e.g. pg_num < pgp_num 
or mon_max_pg_per_osd exceeded)
2019-06-07 15:59:51.396 7fc4ef273780 -1 Couldn't init storage provider (RADOS)

The .rgw.root pool already exists.

ceph status returns:
  cluster:
id: 1a8a1693-fa54-4cb3-89d2-7951d4cee6a3
health: HEALTH_OK

  services:
mon: 3 daemons, quorum S700028,S700029,S700030 (age 30m)
mgr: S700028(active, since 47h), standbys: S700030, S700029
osd: 6 osds: 6 up (since 2d), 6 in (since 3d)

  data:
pools:   5 pools, 448 pgs
objects: 12 objects, 1.2 KiB
usage:   722 GiB used, 65 TiB / 66 TiB avail
pgs: 448 active+clean

and ceph osd tree returns:
ID CLASS WEIGHT   TYPE NAMESTATUS REWEIGHT PRI-AFF
-1   66.17697 root default
-5   22.05899 host S700029
 2   hdd 11.02950 osd.2up  1.0 1.0
 3   hdd 11.02950 osd.3up  1.0 1.0
-7   22.05899 host S700030
 4   hdd 11.02950 osd.4up  1.0 1.0
 5   hdd 11.02950 osd.5up  1.0 1.0
-3   22.05899 host s700028
 0   hdd 11.02950 osd.0up  1.0 1.0
 1   hdd 11.02950 osd.1up  1.0 1.0

Any thoughts on what I'm missing?

Thank you,

Dominic L. Hilsbos, MBA
Director - Information Technology
Perform Air International Inc.
dhils...@performair.com
www.PerformAir.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


--
Shawn Iverson, CETL
Director of Technology
Rush County Schools
765-932-3901 option 7
ivers...@rushville.k12.in.us

[https://docs.google.com/uc?export=download=0Bw5iD0ToYvs_Zkh4eEs3R01yWXc=0Bw5iD0ToYvs_QWpBK2Y2ajJtYjhOMDRFekZwK2xOamk5Q3Y0PQ][https://docs.google.com/uc?export=download=1aBrlQou4gjB04FY-twHN_0Dn3GHVNxqa=0Bw5iD0ToYvs_RnQ0eDhHcm95WHBFdkNRbXhQRXpoYkR6SEEwPQ][Cybersecurity]
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] radosgw dying

[DHilsbos]

Huan;

I get that, but the pool already exists, why is radosgw trying to create one?

Dominic Hilsbos

Get Outlook for Android




On Sat, Jun 8, 2019 at 2:55 AM -0700, "huang jun" 
mailto:hjwsm1...@gmail.com>> wrote:


>From the error message, i'm decline to that 'mon_max_pg_per_osd' was exceed,
you can check the value of it, and its default value is 250, so you
can at most have 1500pgs(250*6osds),
and for replicated pools with size=3, you can have 500pgs for all pools,
you already have 448pgs, so the next pool can create at most 500-448=52pgs.

 于2019年6月8日周六 下午2:41写道：
>
> All;
>
> I have a test and demonstration cluster running (3 hosts, MON, MGR, 2x OSD 
> per host), and I'm trying to add a 4th host for gateway purposes.
>
> The radosgw process keeps dying with:
> 2019-06-07 15:59:50.700 7fc4ef273780  0 ceph version 14.2.1 
> (d555a9489eb35f84f2e1ef49b77e19da9d113972) nautilus (stable), process 
> radosgw, pid 17588
> 2019-06-07 15:59:51.358 7fc4ef273780  0 rgw_init_ioctx ERROR: 
> librados::Rados::pool_create returned (34) Numerical result out of range 
> (this can be due to a pool or placement group misconfiguration, e.g. pg_num < 
> pgp_num or mon_max_pg_per_osd exceeded)
> 2019-06-07 15:59:51.396 7fc4ef273780 -1 Couldn't init storage provider (RADOS)
>
> The .rgw.root pool already exists.
>
> ceph status returns:
>   cluster:
> id: 1a8a1693-fa54-4cb3-89d2-7951d4cee6a3
> health: HEALTH_OK
>
>   services:
> mon: 3 daemons, quorum S700028,S700029,S700030 (age 30m)
> mgr: S700028(active, since 47h), standbys: S700030, S700029
> osd: 6 osds: 6 up (since 2d), 6 in (since 3d)
>
>   data:
> pools:   5 pools, 448 pgs
> objects: 12 objects, 1.2 KiB
> usage:   722 GiB used, 65 TiB / 66 TiB avail
> pgs: 448 active+clean
>
> and ceph osd tree returns:
> ID CLASS WEIGHT   TYPE NAMESTATUS REWEIGHT PRI-AFF
> -1   66.17697 root default
> -5   22.05899 host S700029
>  2   hdd 11.02950 osd.2up  1.0 1.0
>  3   hdd 11.02950 osd.3up  1.0 1.0
> -7   22.05899 host S700030
>  4   hdd 11.02950 osd.4up  1.0 1.0
>  5   hdd 11.02950 osd.5up  1.0 1.0
> -3   22.05899 host s700028
>  0   hdd 11.02950 osd.0up  1.0 1.0
>  1   hdd 11.02950 osd.1up  1.0 1.0
>
> Any thoughts on what I'm missing?
>
> Thank you,
>
> Dominic L. Hilsbos, MBA
> Director - Information Technology
> Perform Air International Inc.
> dhils...@performair.com
> www.PerformAir.com
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



--
Thank you!
HuangJun

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] radosgw dying

[DHilsbos]

All;

I have a test and demonstration cluster running (3 hosts, MON, MGR, 2x OSD per 
host), and I'm trying to add a 4th host for gateway purposes.

The radosgw process keeps dying with:
2019-06-07 15:59:50.700 7fc4ef273780  0 ceph version 14.2.1 
(d555a9489eb35f84f2e1ef49b77e19da9d113972) nautilus (stable), process radosgw, 
pid 17588
2019-06-07 15:59:51.358 7fc4ef273780  0 rgw_init_ioctx ERROR: 
librados::Rados::pool_create returned (34) Numerical result out of range (this 
can be due to a pool or placement group misconfiguration, e.g. pg_num < pgp_num 
or mon_max_pg_per_osd exceeded)
2019-06-07 15:59:51.396 7fc4ef273780 -1 Couldn't init storage provider (RADOS)

The .rgw.root pool already exists.

ceph status returns:
  cluster:
id: 1a8a1693-fa54-4cb3-89d2-7951d4cee6a3
health: HEALTH_OK

  services:
mon: 3 daemons, quorum S700028,S700029,S700030 (age 30m)
mgr: S700028(active, since 47h), standbys: S700030, S700029
osd: 6 osds: 6 up (since 2d), 6 in (since 3d)

  data:
pools:   5 pools, 448 pgs
objects: 12 objects, 1.2 KiB
usage:   722 GiB used, 65 TiB / 66 TiB avail
pgs: 448 active+clean

and ceph osd tree returns:
ID CLASS WEIGHT   TYPE NAMESTATUS REWEIGHT PRI-AFF
-1   66.17697 root default
-5   22.05899 host S700029
 2   hdd 11.02950 osd.2up  1.0 1.0
 3   hdd 11.02950 osd.3up  1.0 1.0
-7   22.05899 host S700030
 4   hdd 11.02950 osd.4up  1.0 1.0
 5   hdd 11.02950 osd.5up  1.0 1.0
-3   22.05899 host s700028
 0   hdd 11.02950 osd.0up  1.0 1.0
 1   hdd 11.02950 osd.1up  1.0 1.0

Any thoughts on what I'm missing?

Thank you,

Dominic L. Hilsbos, MBA 
Director - Information Technology 
Perform Air International Inc.
dhils...@performair.com 
www.PerformAir.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] ceph-volume ignores cluster name?

[DHilsbos]

All;

I'm working on spinning up a demonstration cluster using ceph, and yes, I'm 
installing it manually, for the purpose of learning.

I can't seem to correctly create an OSD, as ceph-volume seems to only work if 
the cluster name is the default.  If I rename my configuration file (at 
/etc/ceph/) to ceph.conf, I can manage to create an OSD, but then it fails to 
start.

I've tried adding the --cluster argument to ceph-volume, but that doesn't seem 
to affect anything.

Any thoughts?

Thank you,

Dominic L. Hilsbos, MBA 
Director - Information Technology 
Perform Air International Inc.
dhils...@performair.com 
www.PerformAir.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Bluestore Hardwaresetup

[DHilsbos]

Peter;

I was just looking at this myself.
With regards to BlueStore, the Config Reference is useful:
http://docs.ceph.com/docs/master/rados/configuration/bluestore-config-ref/

As far as tiering goes, the OSD Config Reference talks about it:
http://docs.ceph.com/docs/master/rados/configuration/osd-config-ref/

The following talks extensively about tiering, and recommends not doing it:
http://docs.ceph.com/docs/master/rados/operations/cache-tiering/

Thank you,

Dominic L. Hilsbos, MBA 
Director - Information Technology 
Perform Air International Inc.


-Original Message-
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Jan 
Peters
Sent: Thursday, February 15, 2018 11:45 AM
To: ceph-users@lists.ceph.com
Subject: [ceph-users] Bluestore Hardwaresetup

Hi everybody,

I want to evaluate ceph with bluestore, so I need some hardware/configure 
advices from you. 

My Setup should be:

3 Nodes Cluster, on each with:

- Intel Gold Processor SP 5118, 12 core / 2.30Ghz
- 64GB RAM
- 6 x 7,2k, 4 TB SAS
- 2 x SSDs, 480GB

On the POSIX FS you have to set your journal on SSDs. What is the best way for 
bluestore? 

Should I configure separate SSDs for block.db and block.wal?

Is there a way to use CacheTiering or a cachepool? 

Thanks in advance

Peter


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Shutting down half / full cluster

[DHilsbos]

All;

This might be a noob type question, but this thread is interesting, and there's 
one thing I would like clarified.

David Turner mentions setting 3 flags on OSDs, Götz has mentioned 5 flags, do 
the commands need to be run on all OSD nodes, or just one?

Thank you,

Dominic L. Hilsbos, MBA 
Director – Information Technology 
Perform Air International Inc.
dhils...@performair.com 
300 S. Hamilton Pl. 
Gilbert, AZ 85233 
Phone: (480) 610-3500 
Fax: (480) 610-3501 
www.PerformAir.com


From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of David 
Turner
Sent: Wednesday, February 14, 2018 9:02 AM
To: Götz Reinicke
Cc: ceph-users
Subject: Re: [ceph-users] Shutting down half / full cluster

ceph osd set noout
ceph osd set nobackfill
ceph osd set norecover

Noout will prevent OSDs from being marked out during the maintenance and no PGs 
will be able to shift data around with the other 2 flags.  After everything is 
done, unset the 3 flags and you're good to go.

On Wed, Feb 14, 2018 at 5:25 AM Götz Reinicke  
wrote:
Thanks!

Götz



Am 14.02.2018 um 11:16 schrieb Kai Wagner :

Hi,
maybe it's worth looking at this:
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-April/017378.html
Kai

On 02/14/2018 11:06 AM, Götz Reinicke wrote:
Hi,

We have some work to do on our power lines for all building and we have to shut 
down all systems. So there is also no traffic on any ceph client.

Pitty, we have to shot down some ceph nodes too in an affected building.

To avoid rebalancing - as I see there is no need for it, as there is no traffic 
on clients - how can I safely set the remaining cluster nodes in a „keep calm 
and wait“ state?

Is that the noout option?

Thanks for feedback and suggestions! Regards . Götz



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


-- 
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 
(AG Nürnberg)
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


Götz Reinicke 
IT-Koordinator
IT-OfficeNet
+49 7141 969 82420 
goetz.reini...@filmakademie.de
Filmakademie Baden-Württemberg GmbH 
Akademiehof 10 71638 Ludwigsburg 
http://www.filmakademie.de 




Eintragung Amtsgericht Stuttgart HRB 205016
Vorsitzende des Aufsichtsrates:
Petra Olschowski
Staatssekretärin im Ministerium für Wissenschaft,
Forschung und Kunst Baden-Württemberg
Geschäftsführer:
Prof. Thomas Schadt 



Götz Reinicke 
IT-Koordinator
IT-OfficeNet
+49 7141 969 82420 
goetz.reini...@filmakademie.de
Filmakademie Baden-Württemberg GmbH 
Akademiehof 10 71638 Ludwigsburg 
http://www.filmakademie.de 




Eintragung Amtsgericht Stuttgart HRB 205016
Vorsitzende des Aufsichtsrates:
Petra Olschowski
Staatssekretärin im Ministerium für Wissenschaft,
Forschung und Kunst Baden-Württemberg
Geschäftsführer:
Prof. Thomas Schadt 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] BlueStore & Journal

[DHilsbos]

David;

Thank you for responding so quickly.  I believe I've been looking at Master.

I found the information on BlueStore five or ten minutes after I sent the 
email, but I appreciate the summary.

Thank you,

Dominic L. Hilsbos, MBA 
Director – Information Technology 
Perform Air International Inc.
dhils...@performair.com 
300 S. Hamilton Pl. 
Gilbert, AZ 85233 
Phone: (480) 610-3500 
Fax: (480) 610-3501 
www.PerformAir.com


From: David Turner [mailto:drakonst...@gmail.com] 
Sent: Tuesday, February 13, 2018 5:14 PM
To: Dominic Hilsbos
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] BlueStore & Journal

Bluestore uses BlocksDB and WAL. When you look through the Ceph documentation 
make sure you're looking at the proper branch for the version you are using, 
most likely luminous. All ceph documentation URLs are versioned with Jewel, 
Hammer, master, etc. There have been a lot of discussions here on the mailing 
list about proper sizes and types of disks for these components.

On Tue, Feb 13, 2018, 5:57 PM  wrote:
All;
 
I’m sorry if this question has been asked before.
 
I’m reading through Ceph’s documentation in preparation to build a cluster, and 
O keep coming across the recommendation to place journals on SSDs.  Does 
BlueStore uses journals, or was this a nod to improving XFS and BTRFS 
performance?
 
Thank you,
 
Dominic L. Hilsbos, MBA 
Director – Information Technology 

dhils...@performair.com 
300 S. Hamilton Pl. 
Gilbert, AZ 85233 
Phone: (480) 610-3500 
Fax: (480) 610-3501 
www.PerformAir.com
 
 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] BlueStore & Journal

[DHilsbos]

All;

I'm sorry if this question has been asked before.

I'm reading through Ceph's documentation in preparation to build a cluster, and 
O keep coming across the recommendation to place journals on SSDs.  Does 
BlueStore uses journals, or was this a nod to improving XFS and BTRFS 
performance?

Thank you,

Dominic L. Hilsbos, MBA

Director - Information Technology

[Perform Air International, Inc.]

dhils...@performair.com

300 S. Hamilton Pl.

Gilbert, AZ 85233

Phone: (480) 610-3500

Fax: (480) 610-3501

www.PerformAir.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com