Re: [ceph-users] balancer module makes OSD distribution worse

[Josh Haft]

PGs are not perfectly balanced per OSD, but I think that's expected/OK
due to setting crush_compat_metrics to bytes? Though realizing as I
type this that what I really want is equal percent-used, which may not
be possible given the slight variation in disk size (see below) in my
cluster?

# ceph osd df | awk '{print $3}'|sort -g|uniq -c
 84 8.90959
 11 9.03159
 96 9.03200
144 9.14240
131 10.69179
 60 10.92470

Josh

On Sat, Jun 8, 2019 at 8:08 AM Igor Podlesny  wrote:
>
> On Thu, 6 Jun 2019 at 03:01, Josh Haft  wrote:
> >
> > Hi everyone,
> >
> > On my 13.2.5 cluster, I recently enabled the ceph balancer module in
> > crush-compat mode.
>
> Why did you choose compat mode? Don't you want to try another one instead?
>
> --
> End of message. Next message?
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] balancer module makes OSD distribution worse

[Josh Haft]

95% of usage is CephFS. Remaining is split between RGW and RBD.

On Wed, Jun 5, 2019 at 3:05 PM Gregory Farnum  wrote:
>
> I think the mimic balancer doesn't include omap data when trying to
> balance the cluster. (Because it doesn't get usable omap stats from
> the cluster anyway; in Nautilus I think it does.) Are you using RGW or
> CephFS?
> -Greg
>
> On Wed, Jun 5, 2019 at 1:01 PM Josh Haft  wrote:
> >
> > Hi everyone,
> >
> > On my 13.2.5 cluster, I recently enabled the ceph balancer module in
> > crush-compat mode. A couple manual 'eval' and 'execute' runs showed
> > the score improving, so I set the following and enabled the auto
> > balancer.
> >
> > mgr/balancer/crush_compat_metrics:bytes # from
> > https://github.com/ceph/ceph/pull/20665
> > mgr/balancer/max_misplaced:0.01
> > mgr/balancer/mode:crush-compat
> >
> > Log messages from the mgr showed lower scores with each iteration, so
> > I thought things were moving in the right direction.
> >
> > Initially my highest-utilized OSD was at 79% and MAXVAR was 1.17. I
> > let the balancer do its thing for 5 days, at which point my highest
> > utilized OSD was just over 90% and MAXVAR was about 1.28.
> >
> > I do have pretty low PG-per-OSD counts (average of about 60 - that's
> > next on my list), but I explicitly asked the balancer to use the bytes
> > metric. Was I just being impatient? Is it expected that usage would go
> > up overall for a time before starting to trend downward? Is my low PG
> > count affecting this somehow? I would have expected things to move in
> > the opposite direction pretty quickly as they do with 'ceph osd
> > reweight-by-utilization'.
> >
> > Thoughts?
> >
> > Regards,
> > Josh
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] balancer module makes OSD distribution worse

[Josh Haft]

Hi everyone,

On my 13.2.5 cluster, I recently enabled the ceph balancer module in
crush-compat mode. A couple manual 'eval' and 'execute' runs showed
the score improving, so I set the following and enabled the auto
balancer.

mgr/balancer/crush_compat_metrics:bytes # from
https://github.com/ceph/ceph/pull/20665
mgr/balancer/max_misplaced:0.01
mgr/balancer/mode:crush-compat

Log messages from the mgr showed lower scores with each iteration, so
I thought things were moving in the right direction.

Initially my highest-utilized OSD was at 79% and MAXVAR was 1.17. I
let the balancer do its thing for 5 days, at which point my highest
utilized OSD was just over 90% and MAXVAR was about 1.28.

I do have pretty low PG-per-OSD counts (average of about 60 - that's
next on my list), but I explicitly asked the balancer to use the bytes
metric. Was I just being impatient? Is it expected that usage would go
up overall for a time before starting to trend downward? Is my low PG
count affecting this somehow? I would have expected things to move in
the opposite direction pretty quickly as they do with 'ceph osd
reweight-by-utilization'.

Thoughts?

Regards,
Josh
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Newly added OSDs will not stay up

[Josh Haft]

Turns out this was due to a switch misconfiguration on the cluster
network. I use jumbo frames and essentially the new server's
connections were not configured with the correct MTU on the switch. So
this caused some traffic to flow, but eventually the servers wanted to
send larger frame sizes than the switch allowed, and that prevented
OSDs from receiving osd_ping messages.

Sorry for the noise!
Josh


On Thu, Mar 14, 2019 at 3:45 PM Josh Haft  wrote:
>
> Hello fellow Cephers,
>
> My 12.2.2 cluster is pretty full so I've been adding new nodes/OSDs.
> Last week I added two new nodes with 12 OSDs each and they are still
> backfilling. I have max_backfills tuned quite low across the board to
> minimize client impact. Yesterday I brought two more nodes online each
> with 12 OSDs and added them to the crushmap under a staging root,
> planning to add those to root=default when the two from last week
> complete backfilling. When the OSDs processes came up they all did
> what I describe below and since it only takes two OSDs on different
> hosts... the mons started marking existing OSDs down. So I backed that
> out and am now just working with a single OSD on of the new nodes
> until I can figure this out.
>
> When the OSD process starts up it's listening on ports 6800 and 6801
> on both the cluster and public interfaces. It successfully gets the
> current osdmap from a monitor and chooses 10 OSDs to peer with, all of
> which fail.
>
> It doesn't appear to be a basic networking issue; I turned up debug
> osd and ms to 20 and based on the following it looks like a successful
> ping/reply with the OSD peer (osd.0), but after a while the log says
> it's never heard from this OSD.
>
> 2019-03-14 14:17:42.350902 7fe698776700 10 osd.403 103451
> _add_heartbeat_peer: new peer osd.0 10.8.78.23:6814/8498484
> 10.8.76.23:6805/8498484
> 2019-03-14 14:17:44.165460 7fe68df61700  1 -- 10.8.76.48:0/67279 -->
> 10.8.76.23:6805/8498484 -- osd_ping(ping e103451 stamp 2019-03-14
> 14:17:44.165415) v4 -- 0x55844222aa00 con 0
> 2019-03-14 14:17:44.165467 7fe68df61700 20 -- 10.8.76.48:0/67279 >>
> 10.8.76.23:6805/8498484 conn(0x558442368000 :-1 s=STATE_OPEN pgs=2349
> cs=1 l=1).prepare_send_message m osd_ping(ping e103451 stamp
> 2019-03-14 14:17:44.165415) v4
> 2019-03-14 14:17:44.165471 7fe68df61700 20 -- 10.8.76.48:0/67279 >>
> 10.8.76.23:6805/8498484 conn(0x558442368000 :-1 s=STATE_OPEN pgs=2349
> cs=1 l=1).prepare_send_message encoding features 2305244844532236283
> 0x55844222aa00 osd_ping(ping e103451 stamp 2019-03-14 14:17:44.165415)
> v4
> 2019-03-14 14:17:44.165691 7fe6a574e700  5 -- 10.8.76.48:0/67279 >>
> 10.8.76.23:6805/8498484 conn(0x558442368000 :-1
> s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=2349 cs=1 l=1). rx
> osd.0 seq 1 0x55844206ba00 osd_ping(ping_reply e103451 stamp
> 2019-03-14 14:17:44.165415) v4
> 2019-03-14 14:17:44.165697 7fe6a574e700  1 -- 10.8.76.48:0/67279 <==
> osd.0 10.8.76.23:6805/8498484 1  osd_ping(ping_reply e103451 stamp
> 2019-03-14 14:17:44.165415) v4  2004+0+0 (4204681659 0 0)
> 0x55844206ba00 con 0x558442368000
>
> ... seq 2-6...
>
> 2019-03-14 14:17:57.468338 7fe68df61700  1 -- 10.8.76.48:0/67279 -->
> 10.8.76.23:6805/8498484 -- osd_ping(ping e103451 stamp 2019-03-14
> 14:17:57.468301) v4 -- 0x5584422e2c00 con 0
> 2019-03-14 14:17:57.468343 7fe68df61700 20 -- 10.8.76.48:0/67279 >>
> 10.8.76.23:6805/8498484 conn(0x558442368000 :-1 s=STATE_OPEN pgs=2349
> cs=1 l=1).prepare_send_message m osd_ping(ping e103451 stamp
> 2019-03-14 14:17:57.468301) v4
> 2019-03-14 14:17:57.468348 7fe68df61700 20 -- 10.8.76.48:0/67279 >>
> 10.8.76.23:6805/8498484 conn(0x558442368000 :-1 s=STATE_OPEN pgs=2349
> cs=1 l=1).prepare_send_message encoding features 2305244844532236283
> 0x5584422e2c00 osd_ping(ping e103451 stamp 2019-03-14 14:17:57.468301)
> v4
> 2019-03-14 14:17:57.468554 7fe6a574e700  5 -- 10.8.76.48:0/67279 >>
> 10.8.76.23:6805/8498484 conn(0x558442368000 :-1
> s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=2349 cs=1 l=1). rx
> osd.0 seq 6 0x55844222a600 osd_ping(ping_reply e103451 stamp
> 2019-03-14 14:17:57.468301) v4
> 2019-03-14 14:17:57.468561 7fe6a574e700  1 -- 10.8.76.48:0/67279 <==
> osd.0 10.8.76.23:6805/8498484 6  osd_ping(ping_reply e103451 stamp
> 2019-03-14 14:17:57.468301) v4  2004+0+0 (306125004 0 0)
> 0x55844222a600 con 0x558442368000
> 2019-03-14 14:18:04.266809 7fe6a1f89700 -1 osd.403 103451
> heartbeat_check: no reply from 10.8.76.23:6805 osd.0 ever on either
> front or back, first ping sent 2019-03-14 14:17:44.165415 (cutoff
> 2019-03-14 14:17:44.266808)
> 2019-03-14 14:18:05.267163 7fe6a1f89700 -1 osd.403 103451
> heartbeat_check: no reply from 10.8.76.23:6805 osd.0 ever on

[ceph-users] Newly added OSDs will not stay up

[Josh Haft]

Hello fellow Cephers,

My 12.2.2 cluster is pretty full so I've been adding new nodes/OSDs.
Last week I added two new nodes with 12 OSDs each and they are still
backfilling. I have max_backfills tuned quite low across the board to
minimize client impact. Yesterday I brought two more nodes online each
with 12 OSDs and added them to the crushmap under a staging root,
planning to add those to root=default when the two from last week
complete backfilling. When the OSDs processes came up they all did
what I describe below and since it only takes two OSDs on different
hosts... the mons started marking existing OSDs down. So I backed that
out and am now just working with a single OSD on of the new nodes
until I can figure this out.

When the OSD process starts up it's listening on ports 6800 and 6801
on both the cluster and public interfaces. It successfully gets the
current osdmap from a monitor and chooses 10 OSDs to peer with, all of
which fail.

It doesn't appear to be a basic networking issue; I turned up debug
osd and ms to 20 and based on the following it looks like a successful
ping/reply with the OSD peer (osd.0), but after a while the log says
it's never heard from this OSD.

2019-03-14 14:17:42.350902 7fe698776700 10 osd.403 103451
_add_heartbeat_peer: new peer osd.0 10.8.78.23:6814/8498484
10.8.76.23:6805/8498484
2019-03-14 14:17:44.165460 7fe68df61700  1 -- 10.8.76.48:0/67279 -->
10.8.76.23:6805/8498484 -- osd_ping(ping e103451 stamp 2019-03-14
14:17:44.165415) v4 -- 0x55844222aa00 con 0
2019-03-14 14:17:44.165467 7fe68df61700 20 -- 10.8.76.48:0/67279 >>
10.8.76.23:6805/8498484 conn(0x558442368000 :-1 s=STATE_OPEN pgs=2349
cs=1 l=1).prepare_send_message m osd_ping(ping e103451 stamp
2019-03-14 14:17:44.165415) v4
2019-03-14 14:17:44.165471 7fe68df61700 20 -- 10.8.76.48:0/67279 >>
10.8.76.23:6805/8498484 conn(0x558442368000 :-1 s=STATE_OPEN pgs=2349
cs=1 l=1).prepare_send_message encoding features 2305244844532236283
0x55844222aa00 osd_ping(ping e103451 stamp 2019-03-14 14:17:44.165415)
v4
2019-03-14 14:17:44.165691 7fe6a574e700  5 -- 10.8.76.48:0/67279 >>
10.8.76.23:6805/8498484 conn(0x558442368000 :-1
s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=2349 cs=1 l=1). rx
osd.0 seq 1 0x55844206ba00 osd_ping(ping_reply e103451 stamp
2019-03-14 14:17:44.165415) v4
2019-03-14 14:17:44.165697 7fe6a574e700  1 -- 10.8.76.48:0/67279 <==
osd.0 10.8.76.23:6805/8498484 1  osd_ping(ping_reply e103451 stamp
2019-03-14 14:17:44.165415) v4  2004+0+0 (4204681659 0 0)
0x55844206ba00 con 0x558442368000

... seq 2-6...

2019-03-14 14:17:57.468338 7fe68df61700  1 -- 10.8.76.48:0/67279 -->
10.8.76.23:6805/8498484 -- osd_ping(ping e103451 stamp 2019-03-14
14:17:57.468301) v4 -- 0x5584422e2c00 con 0
2019-03-14 14:17:57.468343 7fe68df61700 20 -- 10.8.76.48:0/67279 >>
10.8.76.23:6805/8498484 conn(0x558442368000 :-1 s=STATE_OPEN pgs=2349
cs=1 l=1).prepare_send_message m osd_ping(ping e103451 stamp
2019-03-14 14:17:57.468301) v4
2019-03-14 14:17:57.468348 7fe68df61700 20 -- 10.8.76.48:0/67279 >>
10.8.76.23:6805/8498484 conn(0x558442368000 :-1 s=STATE_OPEN pgs=2349
cs=1 l=1).prepare_send_message encoding features 2305244844532236283
0x5584422e2c00 osd_ping(ping e103451 stamp 2019-03-14 14:17:57.468301)
v4
2019-03-14 14:17:57.468554 7fe6a574e700  5 -- 10.8.76.48:0/67279 >>
10.8.76.23:6805/8498484 conn(0x558442368000 :-1
s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=2349 cs=1 l=1). rx
osd.0 seq 6 0x55844222a600 osd_ping(ping_reply e103451 stamp
2019-03-14 14:17:57.468301) v4
2019-03-14 14:17:57.468561 7fe6a574e700  1 -- 10.8.76.48:0/67279 <==
osd.0 10.8.76.23:6805/8498484 6  osd_ping(ping_reply e103451 stamp
2019-03-14 14:17:57.468301) v4  2004+0+0 (306125004 0 0)
0x55844222a600 con 0x558442368000
2019-03-14 14:18:04.266809 7fe6a1f89700 -1 osd.403 103451
heartbeat_check: no reply from 10.8.76.23:6805 osd.0 ever on either
front or back, first ping sent 2019-03-14 14:17:44.165415 (cutoff
2019-03-14 14:17:44.266808)
2019-03-14 14:18:05.267163 7fe6a1f89700 -1 osd.403 103451
heartbeat_check: no reply from 10.8.76.23:6805 osd.0 ever on either
front or back, first ping sent 2019-03-14 14:17:44.165415 (cutoff
2019-03-14 14:17:45.267163)
2019-03-14 14:18:06.267296 7fe6a1f89700 -1 osd.403 103451
heartbeat_check: no reply from 10.8.76.23:6805 osd.0 ever on either
front or back, first ping sent 2019-03-14 14:17:44.165415 (cutoff
2019-03-14 14:17:46.267295)

This whole time other OSDs are marking 403 down so after the grace
period expires the monitor takes it down.
cluster [INF] osd.403 failed (root=staging,host=chhq-supcphsn34) (2
reporters from different host after 25.757343 >= grace 21.560865)

After about 45 seconds the OSD then stops listening on the
aforementioned ports and binds to different ones. Is this normal?
There's no obvious indication in the OSD log of why or when it does
this.
# date;ss -tlnp|grep 67279
Thu Mar 14 14:18:27 CDT 2019
LISTEN 0  12810.8.78.40:6800 *:*

Re: [ceph-users] OSDs crashing

[Josh Haft]

Created: https://tracker.ceph.com/issues/36250

On Tue, Sep 25, 2018 at 9:08 PM Brad Hubbard  wrote:
>
> On Tue, Sep 25, 2018 at 11:31 PM Josh Haft  wrote:
> >
> > Hi cephers,
> >
> > I have a cluster of 7 storage nodes with 12 drives each and the OSD
> > processes are regularly crashing. All 84 have crashed at least once in
> > the past two days. Cluster is Luminous 12.2.2 on CentOS 7.4.1708,
> > kernel version 3.10.0-693.el7.x86_64. I rebooted one of the OSD nodes
> > to see if that cleared up the issue, but it did not. This problem has
> > been going on for about a month now, but it was much less frequent
> > initially - I'd see a crash once every few days or so. I took a look
> > through the mailing list and bug reports, but wasn't able to find
> > anything resembling this problem.
> >
> > I am running a second cluster - also 12.2.2, CentOS 7.4.1708, and
> > kernel version 3.10.0-693.el7.x86_64 - but I do not see the issue
> > there.
> >
> > Log messages always look similar to the following, and I've pulled out
> > the back trace from a core dump as well. The aborting thread always
> > looks to be msgr-worker.
> >
>
> 
>
> > #7  0x7f9e731a3a36 in __cxxabiv1::__terminate (handler= > out>) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:38
> > #8  0x7f9e731a3a63 in std::terminate () at
> > ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48
> > #9  0x7f9e731fa345 in std::(anonymous
> > namespace)::execute_native_thread_routine (__p=) at
> > ../../../../../libstdc++-v3/src/c++11/thread.cc:92
>
> That is this code executing.
>
> https://gcc.gnu.org/git/?p=gcc.git;a=blob;f=libstdc%2B%2B-v3/src/c%2B%2B11/thread.cc;h=0351f19e042b0701ba3c2597ecec87144fd631d5;hb=cf82a597b0d189857acb34a08725762c4f5afb50#l76
>
> So the problem is we are generating an exception when our thread gets
> run, we should probably catch that before it gets to here but that's
> another story...
>
> The exception is "buffer::malformed_input: entity_addr_t marker != 1"
> and there is some precedent for this
> (https://tracker.ceph.com/issues/21660,
> https://tracker.ceph.com/issues/24819) but I don't think they are your
> issue.
>
> We generated that exception because we encountered an ill-formed
> entity_addr_t whilst decoding a message.
>
> Could you open a tracker for this issue and upload the entire log from
> a crash, preferably with "debug ms >= 5" but be careful as this will
> create very large log files. You can use ceph-post-file to upload
> large compressed files.
>
> Let me know the tracker ID here once you've created it.
>
> P.S. This is likely fixed in a later version of Luminous since you
> seem to be the only one hitting it. Either that or there is something
> unusual about your environment.
>
> >
> > Has anyone else seen this? Any suggestions on how to proceed? I do
> > intend to upgrade to Mimic but would prefer to do it when the cluster
> > is stable.
> >
> > Thanks for your help.
> > Josh
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
>
> --
> Cheers,
> Brad
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] OSDs crashing

[Josh Haft]

Hi cephers,

I have a cluster of 7 storage nodes with 12 drives each and the OSD
processes are regularly crashing. All 84 have crashed at least once in
the past two days. Cluster is Luminous 12.2.2 on CentOS 7.4.1708,
kernel version 3.10.0-693.el7.x86_64. I rebooted one of the OSD nodes
to see if that cleared up the issue, but it did not. This problem has
been going on for about a month now, but it was much less frequent
initially - I'd see a crash once every few days or so. I took a look
through the mailing list and bug reports, but wasn't able to find
anything resembling this problem.

I am running a second cluster - also 12.2.2, CentOS 7.4.1708, and
kernel version 3.10.0-693.el7.x86_64 - but I do not see the issue
there.

Log messages always look similar to the following, and I've pulled out
the back trace from a core dump as well. The aborting thread always
looks to be msgr-worker.

Sep 25 00:31:09 sn02 ceph-osd[26077]: terminate called after throwing
an instance of 'ceph::buffer::malformed_input'
Sep 25 00:31:09 sn02 ceph-osd[26077]: what():
buffer::malformed_input: entity_addr_t marker != 1
Sep 25 00:31:09 sn02 ceph-osd[26077]: *** Caught signal (Aborted) **
Sep 25 00:31:09 sn02 ceph-osd[26077]: in thread 7fd7d5200700
thread_name:msgr-worker-2
Sep 25 00:31:09 sn02 ceph-osd[26077]: ceph version 12.2.2
(cf0baba3b47f9427c6c97e2144b094b7e5ba) luminous (stable)
Sep 25 00:31:09 sn02 ceph-osd[26077]: 1: (()+0xa339e1) [0x56310fbe39e1]
Sep 25 00:31:09 sn02 ceph-osd[26077]: 2: (()+0xf5e0) [0x7fd7d8ae25e0]
Sep 25 00:31:09 sn02 ceph-osd[26077]: 3: (gsignal()+0x37) [0x7fd7d7b0b1f7]
Sep 25 00:31:09 sn02 ceph-osd[26077]: 4: (abort()+0x148) [0x7fd7d7b0c8e8]
Sep 25 00:31:09 sn02 ceph-osd[26077]: 5:
(__gnu_cxx::__verbose_terminate_handler()+0x165) [0x7fd7d8411ac5]
Sep 25 00:31:09 sn02 ceph-osd[26077]: 6: (()+0x5ea36) [0x7fd7d840fa36]
Sep 25 00:31:09 sn02 ceph-osd[26077]: 7: (()+0x5ea63) [0x7fd7d840fa63]
Sep 25 00:31:09 sn02 ceph-osd[26077]: 8: (()+0xb5345) [0x7fd7d8466345]
Sep 25 00:31:09 sn02 ceph-osd[26077]: 9: (()+0x7e25) [0x7fd7d8adae25]
Sep 25 00:31:09 sn02 ceph-osd[26077]: 10: (clone()+0x6d) [0x7fd7d7bce34d]
Sep 25 00:31:09 sn02 ceph-osd[26077]: 2018-09-25 00:31:09.849285
7fd7d5200700 -1 *** Caught signal (Aborted) **
  in thread
7fd7d5200700 thread_name:msgr-worker-2

  ceph version 12.2.2
(cf0baba3b47f9427c6c97e2144b094b7e5ba) luminous (stable)
  1: (()+0xa339e1)
[0x56310fbe39e1]
  2: (()+0xf5e0)
[0x7fd7d8ae25e0]
  3: (gsignal()+0x37)
[0x7fd7d7b0b1f7]
  4: (abort()+0x148)
[0x7fd7d7b0c8e8]
  5:
(__gnu_cxx::__verbose_terminate_handler()+0x165) [0x7fd7d8411ac5]
  6: (()+0x5ea36)
[0x7fd7d840fa36]
  7: (()+0x5ea63)
[0x7fd7d840fa63]
  8: (()+0xb5345)
[0x7fd7d8466345]
  9: (()+0x7e25)
[0x7fd7d8adae25]
  10: (clone()+0x6d)
[0x7fd7d7bce34d]
  NOTE: a copy of the
executable, or `objdump -rdS ` is needed to interpret
this.


#0  0x7f9e738764ab in raise (sig=6) at
../nptl/sysdeps/unix/sysv/linux/pt-raise.c:37
#1  0x55925e1edab6 in reraise_fatal (signum=6) at
/usr/src/debug/ceph-12.2.2/src/global/signal_handler.cc:74
#2  handle_fatal_signal (signum=6) at
/usr/src/debug/ceph-12.2.2/src/global/signal_handler.cc:138
#3  
#4  0x7f9e7289f1f7 in __GI_raise (sig=sig@entry=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:56
#5  0x7f9e728a08e8 in __GI_abort () at abort.c:90
#6  0x7f9e731a5ac5 in __gnu_cxx::__verbose_terminate_handler () at
../../../../libstdc++-v3/libsupc++/vterminate.cc:95
#7  0x7f9e731a3a36 in __cxxabiv1::__terminate (handler=) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:38
#8  0x7f9e731a3a63 in std::terminate () at
../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48
#9  0x7f9e731fa345 in std::(anonymous
namespace)::execute_native_thread_routine (__p=) at
../../../../../libstdc++-v3/src/c++11/thread.cc:92
#10 0x7f9e7386ee25 in start_thread (arg=0x7f9e6ff94700) at
pthread_create.c:308
#11 0x7f9e7296234d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Has anyone else seen this? Any suggestions on how to proceed? I do
intend to upgrade to Mimic but would prefer to do it when the cluster
is stable.

Thanks for your help.
Josh
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Group-based permissions issue when using ACLs on CephFS

[Josh Haft]

Here's what I'm seeing using basic owner/group permissions. Both
directories are mounted on my NFS client with the same options. Only
difference is underneath, from the NFS server, 'aclsupport' is mounted
via ceph-fuse with fuse_default_permissions=0 (acls enabled), and
'noaclsupport' is mounted via ceph-fuse with
fuse_default_permissions=1.

'user2' is part of 'group1' and should have r/w access to 'dir', but
does not when trying to access the filesystem mounted with ACL
support.

[user2@test01 ]$ groups
user2 group1

[user2@test01 ]$ stat -c "%i" /mnt/cephfs/aclsupport/dir/
1099511790134
[user2@test01 ]$ stat -c "%i" /mnt/cephfs/noaclsupport/dir/
1099511790134

[user2@test01 ]$ ls -lh /mnt/cephfs/aclsupport
total 1.5K
drwxrws--- 1 user1   group1  0 Mar 22 15:32 dir

[user2@test01 ]$ ls /mnt/cephfs/aclsupport/dir/
ls: reading directory /mnt/cephfs/aclsupport/dir/: Permission denied

[user2@test01 ]$ ls /mnt/cephfs/noaclsupport/dir/
foo

On Sat, Mar 24, 2018 at 3:26 AM, Yan, Zheng <uker...@gmail.com> wrote:
> On Sat, Mar 24, 2018 at 11:34 AM, Josh Haft <pacc...@gmail.com> wrote:
>>
>>
>> On Fri, Mar 23, 2018 at 8:49 PM, Yan, Zheng <uker...@gmail.com> wrote:
>>>
>>> On Fri, Mar 23, 2018 at 9:50 PM, Josh Haft <pacc...@gmail.com> wrote:
>>> > On Fri, Mar 23, 2018 at 12:14 AM, Yan, Zheng <uker...@gmail.com> wrote:
>>> >>
>>> >> On Fri, Mar 23, 2018 at 5:14 AM, Josh Haft <pacc...@gmail.com> wrote:
>>> >> > Hello!
>>> >> >
>>> >> > I'm running Ceph 12.2.2 with one primary and one standby MDS.
>>> >> > Mounting
>>> >> > CephFS via ceph-fuse (to leverage quotas), and enabled ACLs by adding
>>> >> > fuse_default_permissions=0 and client_acl_type=posix_acl to the mount
>>> >> > options. I then export this mount via NFS and the clients mount
>>> >> > NFS4.1.
>>> >> >
>>> >> does fuse_default_permissions=0 work?
>>> >
>>> > Yes, ACLs work as expected when I set fuse_default_permissions=0.
>>> >
>>> >> > After doing some in-depth testing it seems I'm unable to allow access
>>> >> > from
>>> >> > the NFS clients to a directory/file based on group membership when
>>> >> > the
>>> >> > underlying CephFS was mounted with ACL support. This issue appears
>>> >> > using
>>> >> > both filesystem permissions (e.g. chgrp) and NFSv4 ACLs. However,
>>> >> > ACLs do
>>> >> > work if the principal is a user instead of a group. If I disable ACL
>>> >> > support
>>> >> > on the ceph-fuse mount, things work as expected using fs permissions;
>>> >> > obviously I don't get ACL support.
>>> >> >
>>> >> > As an intermediate step I did check whether this works directly on
>>> >> > the
>>> >> > CephFS filesystem - on the NFS server - and it does. So it appears to
>>> >> > be an
>>> >> > issue re-exporting it via NFS.
>>> >> >
>>> >> > I do not see this issue when mounting CephFS via the kernel,
>>> >> > exporting via
>>> >> > NFS, and re-running these tests.
>>> >> >
>>> >> > I searched the ML and bug reports but only found this -
>>> >> > http://tracker.ceph.com/issues/12617 - which seems close to the issue
>>> >> > I'm
>>> >> > running into, but was closed as resolved 2+ years ago.
>>> >> >
>>> >> > Has anyone else run into this? Am I missing something obvious?
>>> >> >
>>> >>
>>> >> ceph-fuse does permission check according to localhost's config of
>>> >> supplement group. that's why you see this behavior.
>>> >
>>> > You're saying both the NFS client and server (where ceph-fuse is
>>> > running) need to use the same directory backend? (they are)
>>> > I should have mentioned I'm using LDAP/AD on client and server, so I
>>> > don't think that is the problem.
>>> >
>>> > Either way, I would not expect the behavior to change simply by
>>> > enabling ACLs, especially when I'm using filesystem permissions, and
>>> > ACLs aren't part of the equation.
>>>
>>> More specifically, ceph-fuse find which groups request initiator are
>>> in by function fuse_req_getgroups(). this function does tricks on
>>> "/proc/%lu/task/%lu/status".  It only works  when nfs client and
>>> ceph-fuse are running on the same machine.
>>>
>> So why does this work when I'm using ceph-fuse but ACLs are disabled?
>>>
>
> Really?
>
> Please check if supplement groups work for inodes without ACL (mount
> fuse with config option fuse_default_permissions=0)
>
>
>>>
>>> >> Yan, Zheng
>>> >>
>>> >> > Thanks!
>>> >> > Josh
>>> >> >
>>> >> >
>>> >> > ___
>>> >> > ceph-users mailing list
>>> >> > ceph-users@lists.ceph.com
>>> >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>> >> >
>>
>>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Group-based permissions issue when using ACLs on CephFS

[Josh Haft]

On Fri, Mar 23, 2018 at 8:49 PM, Yan, Zheng <uker...@gmail.com> wrote:

> On Fri, Mar 23, 2018 at 9:50 PM, Josh Haft <pacc...@gmail.com> wrote:
> > On Fri, Mar 23, 2018 at 12:14 AM, Yan, Zheng <uker...@gmail.com> wrote:
> >>
> >> On Fri, Mar 23, 2018 at 5:14 AM, Josh Haft <pacc...@gmail.com> wrote:
> >> > Hello!
> >> >
> >> > I'm running Ceph 12.2.2 with one primary and one standby MDS. Mounting
> >> > CephFS via ceph-fuse (to leverage quotas), and enabled ACLs by adding
> >> > fuse_default_permissions=0 and client_acl_type=posix_acl to the mount
> >> > options. I then export this mount via NFS and the clients mount
> NFS4.1.
> >> >
> >> does fuse_default_permissions=0 work?
> >
> > Yes, ACLs work as expected when I set fuse_default_permissions=0.
> >
> >> > After doing some in-depth testing it seems I'm unable to allow access
> from
> >> > the NFS clients to a directory/file based on group membership when the
> >> > underlying CephFS was mounted with ACL support. This issue appears
> using
> >> > both filesystem permissions (e.g. chgrp) and NFSv4 ACLs. However,
> ACLs do
> >> > work if the principal is a user instead of a group. If I disable ACL
> support
> >> > on the ceph-fuse mount, things work as expected using fs permissions;
> >> > obviously I don't get ACL support.
> >> >
> >> > As an intermediate step I did check whether this works directly on the
> >> > CephFS filesystem - on the NFS server - and it does. So it appears to
> be an
> >> > issue re-exporting it via NFS.
> >> >
> >> > I do not see this issue when mounting CephFS via the kernel,
> exporting via
> >> > NFS, and re-running these tests.
> >> >
> >> > I searched the ML and bug reports but only found this -
> >> > http://tracker.ceph.com/issues/12617 - which seems close to the
> issue I'm
> >> > running into, but was closed as resolved 2+ years ago.
> >> >
> >> > Has anyone else run into this? Am I missing something obvious?
> >> >
> >>
> >> ceph-fuse does permission check according to localhost's config of
> >> supplement group. that's why you see this behavior.
> >
> > You're saying both the NFS client and server (where ceph-fuse is
> > running) need to use the same directory backend? (they are)
> > I should have mentioned I'm using LDAP/AD on client and server, so I
> > don't think that is the problem.
> >
> > Either way, I would not expect the behavior to change simply by
> > enabling ACLs, especially when I'm using filesystem permissions, and
> > ACLs aren't part of the equation.
>
> More specifically, ceph-fuse find which groups request initiator are
> in by function fuse_req_getgroups(). this function does tricks on
> "/proc/%lu/task/%lu/status".  It only works  when nfs client and
> ceph-fuse are running on the same machine.
>
> So why does this work when I'm using ceph-fuse but ACLs are disabled?

>
> >> Yan, Zheng
> >>
> >> > Thanks!
> >> > Josh
> >> >
> >> >
> >> > ___
> >> > ceph-users mailing list
> >> > ceph-users@lists.ceph.com
> >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >> >
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Group-based permissions issue when using ACLs on CephFS

[Josh Haft]

On Fri, Mar 23, 2018 at 12:14 AM, Yan, Zheng <uker...@gmail.com> wrote:
>
> On Fri, Mar 23, 2018 at 5:14 AM, Josh Haft <pacc...@gmail.com> wrote:
> > Hello!
> >
> > I'm running Ceph 12.2.2 with one primary and one standby MDS. Mounting
> > CephFS via ceph-fuse (to leverage quotas), and enabled ACLs by adding
> > fuse_default_permissions=0 and client_acl_type=posix_acl to the mount
> > options. I then export this mount via NFS and the clients mount NFS4.1.
> >
> does fuse_default_permissions=0 work?

Yes, ACLs work as expected when I set fuse_default_permissions=0.

> > After doing some in-depth testing it seems I'm unable to allow access from
> > the NFS clients to a directory/file based on group membership when the
> > underlying CephFS was mounted with ACL support. This issue appears using
> > both filesystem permissions (e.g. chgrp) and NFSv4 ACLs. However, ACLs do
> > work if the principal is a user instead of a group. If I disable ACL support
> > on the ceph-fuse mount, things work as expected using fs permissions;
> > obviously I don't get ACL support.
> >
> > As an intermediate step I did check whether this works directly on the
> > CephFS filesystem - on the NFS server - and it does. So it appears to be an
> > issue re-exporting it via NFS.
> >
> > I do not see this issue when mounting CephFS via the kernel, exporting via
> > NFS, and re-running these tests.
> >
> > I searched the ML and bug reports but only found this -
> > http://tracker.ceph.com/issues/12617 - which seems close to the issue I'm
> > running into, but was closed as resolved 2+ years ago.
> >
> > Has anyone else run into this? Am I missing something obvious?
> >
>
> ceph-fuse does permission check according to localhost's config of
> supplement group. that's why you see this behavior.

You're saying both the NFS client and server (where ceph-fuse is
running) need to use the same directory backend? (they are)
I should have mentioned I'm using LDAP/AD on client and server, so I
don't think that is the problem.

Either way, I would not expect the behavior to change simply by
enabling ACLs, especially when I'm using filesystem permissions, and
ACLs aren't part of the equation.

> Regards
> Yan, Zheng
>
> > Thanks!
> > Josh
> >
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Group-based permissions issue when using ACLs on CephFS

[Josh Haft]

Hello!

I'm running Ceph 12.2.2 with one primary and one standby MDS. Mounting
CephFS via ceph-fuse (to leverage quotas), and enabled ACLs by adding
fuse_default_permissions=0 and client_acl_type=posix_acl to the mount
options. I then export this mount via NFS and the clients mount NFS4.1.

After doing some in-depth testing it seems I'm unable to allow access from
the NFS clients to a directory/file based on group membership when the
underlying CephFS was mounted with ACL support. This issue appears using
both filesystem permissions (e.g. chgrp) and NFSv4 ACLs. However, ACLs do
work if the principal is a user instead of a group. If I disable ACL
support on the ceph-fuse mount, things work as expected using fs
permissions; obviously I don't get ACL support.

As an intermediate step I did check whether this works directly on the
CephFS filesystem - on the NFS server - and it does. So it appears to be an
issue re-exporting it via NFS.

I do not see this issue when mounting CephFS via the kernel, exporting via
NFS, and re-running these tests.

I searched the ML and bug reports but only found this -
http://tracker.ceph.com/issues/12617 - which seems close to the issue I'm
running into, but was closed as resolved 2+ years ago.

Has anyone else run into this? Am I missing something obvious?

Thanks!
Josh
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Object gateway and LDAP Auth

[Josh Haft]

Finally got back around to working on this and wanted to provide a solution
in case anyone else runs into the same problem.

I was able to reproduce the problem using s3cmd, and noticed different
calls utilized different signature versions. Doing a GET operation on '/'
seemed to use v2 while a 'make bucket' command attempted to use v4. Since
the former succeeded and the latter failed, I called s3cmd with
'--signature-v2' and now all operations work. I'm still not able to use
boto3, but it's no longer an LDAP issue.

Josh



On Tue, Sep 5, 2017 at 10:26 AM, Josh Haft <pacc...@gmail.com> wrote:

> Thanks for your suggestions, Matt. ldapsearch functionality from the rados
> gw machines works fine using the same parameters specified in ceph.conf
> (uri, binddn, searchdn, ldap_secret). As expected I see network traffic
> to/from the ldap host when performing a search as well.
>
> The only configuration I have in /etc/openldap/ldap.conf is 'TLSREQCERT
> demand' and TLS_CACERTDIR pointing at the location of my certdb... is there
> something else required here for ceph-rgw or does it look elsewhere?
>
> Josh
>
>
>
>
> On Fri, Sep 1, 2017 at 11:15 PM, Matt Benjamin <mbenj...@redhat.com>
> wrote:
>
>> Hi Josh,
>>
>> I'm not certain, but you might try disabling the searchfilter to start
>> with.  If you're not seeing traffic, I would focus on verifying ldap
>> search connectivity using the same credentials, using the openldap
>> client, to rule out something low level.
>>
>> Matt
>>
>>
>> On Thu, Aug 31, 2017 at 3:33 PM, Josh <pacc...@gmail.com> wrote:
>> > Hello!
>> >
>> > I've setup LDAP authentication on an object gateway and am attempting to
>> > create a bucket via s3 using python's boto3. It works fine using the
>> access
>> > and secret key for a radosgw user, but access is denied using a token
>> > generated via radosgw-token with the LDAP user's credentials. The user
>> does
>> > exist in the directory (I'm using Active Directory), and I am able to
>> query
>> > for that user using the creds specified in rgw_ldap_binddn and
>> > rgw_ldap_secret.
>> >
>> > I've bumped the rgw logging to 20 and can see the request come in, but
>> it
>> > ultimately gets denied:
>> > 2017-08-30 15:44:55.754721 7f4878ff9700  2 req 1:0.76:s3:PUT
>> > /foobar:create_bucket:authorizing
>> > 2017-08-30 15:44:55.754738 7f4878ff9700 10 v4 signature format = 
>> > 2017-08-30 15:44:55.754746 7f4878ff9700 10 v4 credential format =
>> > /20170830/us-east-1/s3/aws4_request
>> > 2017-08-30 15:44:55.754750 7f4878ff9700 10 access key id = 
>> > 2017-08-30 15:44:55.754755 7f4878ff9700 10 credential scope =
>> > 20170830/us-east-1/s3/aws4_request
>> > 2017-08-30 15:44:55.754769 7f4878ff9700 20 get_system_obj_state:
>> > rctx=0x7f4878ff2060 obj=default.rgw.users.keys:
>> state=0x7f48f40131a8
>> > s->prefetch_data=0
>> > 2017-08-30 15:44:55.754778 7f4878ff9700 10 cache get:
>> > name=default.rgw.users.keys+ : miss
>> > 2017-08-30 15:44:55.755312 7f4878ff9700 10 cache put:
>> > name=default.rgw.users.keys+ info.flags=0
>> > 2017-08-30 15:44:55.755321 7f4878ff9700 10 adding
>> > default.rgw.users.keys+ to cache LRU end
>> > 2017-08-30 15:44:55.755328 7f4878ff9700 10 error reading user info,
>> uid=
>> > can't authenticate
>> > 2017-08-30 15:44:55.755330 7f4878ff9700 10 failed to authorize request
>> > 2017-08-30 15:44:55.755331 7f4878ff9700 20 handler->ERRORHANDLER:
>> > err_no=-2028 new_err_no=-2028
>> > 2017-08-30 15:44:55.755393 7f4878ff9700  2 req 1:0.000747:s3:PUT
>> > /foobar:create_bucket:op status=0
>> > 2017-08-30 15:44:55.755398 7f4878ff9700  2 req 1:0.000752:s3:PUT
>> > /foobar:create_bucket:http status=403
>> > 2017-08-30 15:44:55.755402 7f4878ff9700  1 == req done
>> > req=0x7f4878ff3710 op status=0 http_status=403 ==
>> > 2017-08-30 15:44:55.755409 7f4878ff9700 20 process_request() returned
>> -2028
>> >
>> > I am also running a tcpdump on the machine while I see these log
>> messages,
>> > but strangely I see no traffic destined for my configured LDAP server.
>> > Here's some info on my setup. It seems like I'm missing something very
>> > obvious; any help would be appreciated!
>> >
>> > # rpm -q ceph-radosgw
>> > ceph-radosgw-10.2.9-0.el7.x86_64
>> >
>> > # grep rgw /etc/ceph/ceph.conf
>> > [client.rgw.hostname]
>> > rgw_frontends = ci

Re: [ceph-users] Object gateway and LDAP Auth

[Josh Haft]

Thanks for your suggestions, Matt. ldapsearch functionality from the rados
gw machines works fine using the same parameters specified in ceph.conf
(uri, binddn, searchdn, ldap_secret). As expected I see network traffic
to/from the ldap host when performing a search as well.

The only configuration I have in /etc/openldap/ldap.conf is 'TLSREQCERT
demand' and TLS_CACERTDIR pointing at the location of my certdb... is there
something else required here for ceph-rgw or does it look elsewhere?

Josh




On Fri, Sep 1, 2017 at 11:15 PM, Matt Benjamin  wrote:

> Hi Josh,
>
> I'm not certain, but you might try disabling the searchfilter to start
> with.  If you're not seeing traffic, I would focus on verifying ldap
> search connectivity using the same credentials, using the openldap
> client, to rule out something low level.
>
> Matt
>
>
> On Thu, Aug 31, 2017 at 3:33 PM, Josh  wrote:
> > Hello!
> >
> > I've setup LDAP authentication on an object gateway and am attempting to
> > create a bucket via s3 using python's boto3. It works fine using the
> access
> > and secret key for a radosgw user, but access is denied using a token
> > generated via radosgw-token with the LDAP user's credentials. The user
> does
> > exist in the directory (I'm using Active Directory), and I am able to
> query
> > for that user using the creds specified in rgw_ldap_binddn and
> > rgw_ldap_secret.
> >
> > I've bumped the rgw logging to 20 and can see the request come in, but it
> > ultimately gets denied:
> > 2017-08-30 15:44:55.754721 7f4878ff9700  2 req 1:0.76:s3:PUT
> > /foobar:create_bucket:authorizing
> > 2017-08-30 15:44:55.754738 7f4878ff9700 10 v4 signature format = 
> > 2017-08-30 15:44:55.754746 7f4878ff9700 10 v4 credential format =
> > /20170830/us-east-1/s3/aws4_request
> > 2017-08-30 15:44:55.754750 7f4878ff9700 10 access key id = 
> > 2017-08-30 15:44:55.754755 7f4878ff9700 10 credential scope =
> > 20170830/us-east-1/s3/aws4_request
> > 2017-08-30 15:44:55.754769 7f4878ff9700 20 get_system_obj_state:
> > rctx=0x7f4878ff2060 obj=default.rgw.users.keys: state=0x7f48f40131a8
> > s->prefetch_data=0
> > 2017-08-30 15:44:55.754778 7f4878ff9700 10 cache get:
> > name=default.rgw.users.keys+ : miss
> > 2017-08-30 15:44:55.755312 7f4878ff9700 10 cache put:
> > name=default.rgw.users.keys+ info.flags=0
> > 2017-08-30 15:44:55.755321 7f4878ff9700 10 adding
> > default.rgw.users.keys+ to cache LRU end
> > 2017-08-30 15:44:55.755328 7f4878ff9700 10 error reading user info,
> uid=
> > can't authenticate
> > 2017-08-30 15:44:55.755330 7f4878ff9700 10 failed to authorize request
> > 2017-08-30 15:44:55.755331 7f4878ff9700 20 handler->ERRORHANDLER:
> > err_no=-2028 new_err_no=-2028
> > 2017-08-30 15:44:55.755393 7f4878ff9700  2 req 1:0.000747:s3:PUT
> > /foobar:create_bucket:op status=0
> > 2017-08-30 15:44:55.755398 7f4878ff9700  2 req 1:0.000752:s3:PUT
> > /foobar:create_bucket:http status=403
> > 2017-08-30 15:44:55.755402 7f4878ff9700  1 == req done
> > req=0x7f4878ff3710 op status=0 http_status=403 ==
> > 2017-08-30 15:44:55.755409 7f4878ff9700 20 process_request() returned
> -2028
> >
> > I am also running a tcpdump on the machine while I see these log
> messages,
> > but strangely I see no traffic destined for my configured LDAP server.
> > Here's some info on my setup. It seems like I'm missing something very
> > obvious; any help would be appreciated!
> >
> > # rpm -q ceph-radosgw
> > ceph-radosgw-10.2.9-0.el7.x86_64
> >
> > # grep rgw /etc/ceph/ceph.conf
> > [client.rgw.hostname]
> > rgw_frontends = civetweb port=8081s ssl_certificate=/path/to/
> private/key.pem
> > debug rgw = 20
> > rgw_s3_auth_use_ldap = true
> > rgw_ldap_secret = "/path/to/creds/file"
> > rgw_ldap_uri = "ldaps://hostname.domain.com:636"
> > rgw_ldap_binddn = "CN=valid_user,OU=Accounts,DC=domain,DC=com"
> > rgw_ldap_searchdn = "ou=Accounts,dc=domain,dc=com"
> > rgw_ldap_dnattr = "uid"
> > rgw_ldap_searchfilter = "objectclass=user"
> >
> >
> > Thanks,
> > Josh
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
>
>
>
> --
>
> Matt Benjamin
> Red Hat, Inc.
> 315 West Huron Street, Suite 140A
> Ann Arbor, Michigan 48103
>
> http://www.redhat.com/en/technologies/storage
>
> tel.  734-821-5101
> fax.  734-769-8938
> cel.  734-216-5309
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com