[ceph-users] default.rgw.log contains large omap object
Hi folks, Mimic cluster here, RGW pool with only default zone. I have a persistent error here LARGE_OMAP_OBJECTS 1 large omap objects 1 large objects found in pool 'default.rgw.log' Search the cluster log for 'Large omap object found' for more details. I think I've narrowed it down to this namespace: -[~:#]- rados -p default.rgw.log --namespace usage ls usage.17 usage.19 usage.24 -[~:#]- rados -p default.rgw.log --namespace usage listomapkeys usage.17 | wc -l 21284 -[~:#]- rados -p default.rgw.log --namespace usage listomapkeys usage.19 | wc -l 1355968 -[~:#]- rados -p default.rgw.log --namespace usage listomapkeys usage.24 | wc -l 0 Now, this last one take a long time to return -- minutes, even with a 0 response, and listomapvals indeed returns a very large amount of data. I'm doing `wc -c` on listomapvals but this hasn't returned at the time of writing this message. Is there anything I can do about this? Whenever PGs on the default.rgw.log are recovering or backfilling, my RGW cluster appears to block writes for almost two hours, and I think it points to this object, or at least this pool. I've been having trouble finding any documentation about how this log pool is used by RGW. I have a feeling updating this object happens on every write to the cluster. How would I remove this bottleneck? Can I? Thanks -Troy ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] default.rgw.log contains large omap object
Looks like the usage log (radosgw-admin usage show), how often do you trim it? -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Mon, Oct 14, 2019 at 11:55 PM Troy Ablan wrote: > > Hi folks, > > Mimic cluster here, RGW pool with only default zone. I have a > persistent error here > > LARGE_OMAP_OBJECTS 1 large omap objects > > 1 large objects found in pool 'default.rgw.log' > > Search the cluster log for 'Large omap object found' for more > details. > > I think I've narrowed it down to this namespace: > > -[~:#]- rados -p default.rgw.log --namespace usage ls > > > > > > usage.17 > usage.19 > usage.24 > > -[~:#]- rados -p default.rgw.log --namespace usage listomapkeys usage.17 > | wc -l > 21284 > > -[~:#]- rados -p default.rgw.log --namespace usage listomapkeys usage.19 > | wc -l > > > > > 1355968 > > -[~:#]- rados -p default.rgw.log --namespace usage listomapkeys usage.24 > | wc -l > > > > > 0 > > > Now, this last one take a long time to return -- minutes, even with a 0 > response, and listomapvals indeed returns a very large amount of data. > I'm doing `wc -c` on listomapvals but this hasn't returned at the time > of writing this message. Is there anything I can do about this? > > Whenever PGs on the default.rgw.log are recovering or backfilling, my > RGW cluster appears to block writes for almost two hours, and I think it > points to this object, or at least this pool. > > I've been having trouble finding any documentation about how this log > pool is used by RGW. I have a feeling updating this object happens on > every write to the cluster. How would I remove this bottleneck? Can I? > > Thanks > > -Troy > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] problem returning mon back to cluster
How big is the mon's DB? As in just the total size of the directory you copied FWIW I recently had to perform mon surgery on a 14.2.4 (or was it 14.2.2?) cluster with 8 GB mon size and I encountered no such problems while syncing a new mon which took 10 minutes or so. Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Mon, Oct 14, 2019 at 9:41 PM Nikola Ciprich wrote: > > On Mon, Oct 14, 2019 at 04:31:22PM +0200, Nikola Ciprich wrote: > > On Mon, Oct 14, 2019 at 01:40:19PM +0200, Harald Staub wrote: > > > Probably same problem here. When I try to add another MON, "ceph > > > health" becomes mostly unresponsive. One of the existing ceph-mon > > > processes uses 100% CPU for several minutes. Tried it on 2 test > > > clusters (14.2.4, 3 MONs, 5 storage nodes with around 2 hdd osds > > > each). To avoid errors like "lease timeout", I temporarily increase > > > "mon lease", from 5 to 50 seconds. > > > > > > Not sure how bad it is from a customer PoV. But it is a problem by > > > itself to be several minutes without "ceph health", when there is an > > > increased risk of losing the quorum ... > > > > Hi Harry, > > > > thanks a lot for your reply! not sure we're experiencing the same issue, > > i don't have it on any other cluster.. when this is happening to you, does > > only ceph health stop working, or it also blocks all clients IO? > > > > BR > > > > nik > > > > > > > > > > Harry > > > > > > On 13.10.19 20:26, Nikola Ciprich wrote: > > > >dear ceph users and developers, > > > > > > > >on one of our production clusters, we got into pretty unpleasant > > > >situation. > > > > > > > >After rebooting one of the nodes, when trying to start monitor, whole > > > >cluster > > > >seems to hang, including IO, ceph -s etc. When this mon is stopped again, > > > >everything seems to continue. Traying to spawn new monitor leads to the > > > >same problem > > > >(even on different node). > > > > > > > >I had to give up after minutes of outage, since it's unacceptable. I > > > >think we had this > > > >problem once in the past on this cluster, but after some (but much > > > >shorter) time, monitor > > > >joined and it worked fine since then. > > > > > > > >All cluster nodes are centos 7 machines, I have 3 monitors (so 2 are now > > > >running), I'm > > > >using ceph 13.2.6 > > > > > > > >Network connection seems to be fine. > > > > > > > >Anyone seen similar problem? I'd be very grateful for tips on how to > > > >debug and solve this.. > > > > > > > >for those interested, here's log of one of running monitors with > > > >debug_mon set to 10/10: > > > > > > > >https://storage.lbox.cz/public/d258d0 > > > > > > > >if I could provide more info, please let me know > > > > > > > >with best regards > > > > > > > >nikola ciprich > > just to add quick update, I was able to reproduce the issue by transferring > monitor > directories to test environmen with same IP adressing, so I can safely play > with that > now.. > > increasing lease timeout didn't help me to fix the problem, > but at least I seem to be able to use ceph -s now. > > few things I noticed in the meantime: > > - when I start problematic monitor, monitor slow ops start to appear for > quorum leader and the count is slowly increasing: > > 44 slow ops, oldest one blocked for 130 sec, mon.nodev1c has slow > ops > > - removing and recreating monitor didn't help > > - checking mon_status of problematic monitor shows it remains in the > "synchronizing" state > > I tried increasing debug_ms and debug_paxos but didn't see anything usefull > there.. > > will report further when I got something. I anyone has any idea in the > meantime, please > let me know. > > BR > > nik > > > > > -- > - > Ing. Nikola CIPRICH > LinuxBox.cz, s.r.o. > 28. rijna 168, 709 00 Ostrava > > tel.: +420 591 166 214 > fax:+420 596 621 273 > mobil: +420 777 093 799 > > www.linuxbox.cz > > mobil servis: +420 737 238 656 > email servis: ser...@linuxbox.cz > - > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] default.rgw.log contains large omap object
Yep, that's on me. I did enable it in the config originally, and I
think that I thought at the time that it might be useful, but I wasn't
aware of a sharding caveat owing to most of our traffic is happening on
one rgw user.
I think I know what I need to do to fix it now though.
Thanks again!
-Troy
On 10/14/19 3:23 PM, Paul Emmerich wrote:
Yeah, the number of shards is configurable ("rgw usage num shards"? or
something).
Are you sure you aren't using it? This feature is not enabled by
default, someone had to explicitly set "rgw enable usage log" for you
to run into this problem.
Paul
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] default.rgw.log contains large omap object
Paul, Apparently never. Appears to (potentially) have every request from the beginning of time (late last year, in my case). In our use case, we don't really need this data (not multi-tenant), so I might simply clear it. But in the case where this were an extremely high transaction cluster where we did care about historical data for longer, can this be spread or sharded across more than just this small handful of objects? Thanks for the insight. -Troy On 10/14/19 3:01 PM, Paul Emmerich wrote: Looks like the usage log (radosgw-admin usage show), how often do you trim it? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] default.rgw.log contains large omap object
Yeah, the number of shards is configurable ("rgw usage num shards"? or
something).
Are you sure you aren't using it? This feature is not enabled by
default, someone had to explicitly set "rgw enable usage log" for you
to run into this problem.
Paul
--
Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
On Tue, Oct 15, 2019 at 12:15 AM Troy Ablan wrote:
>
> Paul,
>
> Apparently never. Appears to (potentially) have every request from the
> beginning of time (late last year, in my case). In our use case, we
> don't really need this data (not multi-tenant), so I might simply clear it.
>
> But in the case where this were an extremely high transaction cluster
> where we did care about historical data for longer, can this be spread
> or sharded across more than just this small handful of objects?
>
> Thanks for the insight.
>
> -Troy
>
>
> On 10/14/19 3:01 PM, Paul Emmerich wrote:
> > Looks like the usage log (radosgw-admin usage show), how often do you trim
> > it?
> >
> >
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] problem returning mon back to cluster
On Tue, Oct 15, 2019 at 06:50:31AM +0200, Nikola Ciprich wrote: > > > On Mon, Oct 14, 2019 at 11:52:55PM +0200, Paul Emmerich wrote: > > How big is the mon's DB? As in just the total size of the directory you > > copied > > > > FWIW I recently had to perform mon surgery on a 14.2.4 (or was it > > 14.2.2?) cluster with 8 GB mon size and I encountered no such problems > > while syncing a new mon which took 10 minutes or so. > Hi Paul, > > yup I forgot to mention this.. It doesn't seem to be too big, just about > 100MB. I also noticed that while third monitor tries to join the cluster, > leader starts flapping between "leader" and "electing", so I suppose it's > quorum forming problem.. I tried bumping debug_ms and debug_paxos but > couldn't make head or tails of it.. can paste the logs somewhere if it > can help btw I just noticed, that on test cluster, third mon finally managed to join the cluster and forum got formed.. after more then 6 hours.. knowing that during it, the IO blocks for clients, it's pretty scary now I can stop/start monitors without problems on it.. so it somehow got "fixed" still dunno what to do with this production cluster though, so I'll just prepare test environment again and try digging more into it BR nik > > BR > > nik > > > > > > > Paul > > > > -- > > Paul Emmerich > > > > Looking for help with your Ceph cluster? Contact us at https://croit.io > > > > croit GmbH > > Freseniusstr. 31h > > 81247 München > > www.croit.io > > Tel: +49 89 1896585 90 > > > > On Mon, Oct 14, 2019 at 9:41 PM Nikola Ciprich > > wrote: > > > > > > On Mon, Oct 14, 2019 at 04:31:22PM +0200, Nikola Ciprich wrote: > > > > On Mon, Oct 14, 2019 at 01:40:19PM +0200, Harald Staub wrote: > > > > > Probably same problem here. When I try to add another MON, "ceph > > > > > health" becomes mostly unresponsive. One of the existing ceph-mon > > > > > processes uses 100% CPU for several minutes. Tried it on 2 test > > > > > clusters (14.2.4, 3 MONs, 5 storage nodes with around 2 hdd osds > > > > > each). To avoid errors like "lease timeout", I temporarily increase > > > > > "mon lease", from 5 to 50 seconds. > > > > > > > > > > Not sure how bad it is from a customer PoV. But it is a problem by > > > > > itself to be several minutes without "ceph health", when there is an > > > > > increased risk of losing the quorum ... > > > > > > > > Hi Harry, > > > > > > > > thanks a lot for your reply! not sure we're experiencing the same issue, > > > > i don't have it on any other cluster.. when this is happening to you, > > > > does > > > > only ceph health stop working, or it also blocks all clients IO? > > > > > > > > BR > > > > > > > > nik > > > > > > > > > > > > > > > > > > Harry > > > > > > > > > > On 13.10.19 20:26, Nikola Ciprich wrote: > > > > > >dear ceph users and developers, > > > > > > > > > > > >on one of our production clusters, we got into pretty unpleasant > > > > > >situation. > > > > > > > > > > > >After rebooting one of the nodes, when trying to start monitor, > > > > > >whole cluster > > > > > >seems to hang, including IO, ceph -s etc. When this mon is stopped > > > > > >again, > > > > > >everything seems to continue. Traying to spawn new monitor leads to > > > > > >the same problem > > > > > >(even on different node). > > > > > > > > > > > >I had to give up after minutes of outage, since it's unacceptable. I > > > > > >think we had this > > > > > >problem once in the past on this cluster, but after some (but much > > > > > >shorter) time, monitor > > > > > >joined and it worked fine since then. > > > > > > > > > > > >All cluster nodes are centos 7 machines, I have 3 monitors (so 2 are > > > > > >now running), I'm > > > > > >using ceph 13.2.6 > > > > > > > > > > > >Network connection seems to be fine. > > > > > > > > > > > >Anyone seen similar problem? I'd be very grateful for tips on how to > > > > > >debug and solve this.. > > > > > > > > > > > >for those interested, here's log of one of running monitors with > > > > > >debug_mon set to 10/10: > > > > > > > > > > > >https://storage.lbox.cz/public/d258d0 > > > > > > > > > > > >if I could provide more info, please let me know > > > > > > > > > > > >with best regards > > > > > > > > > > > >nikola ciprich > > > > > > just to add quick update, I was able to reproduce the issue by > > > transferring monitor > > > directories to test environmen with same IP adressing, so I can safely > > > play with that > > > now.. > > > > > > increasing lease timeout didn't help me to fix the problem, > > > but at least I seem to be able to use ceph -s now. > > > > > > few things I noticed in the meantime: > > > > > > - when I start problematic monitor, monitor slow ops start to appear for > > > quorum leader and the count is slowly increasing: > > > > > > 44 slow ops, oldest one blocked for 130 sec, mon.nodev1c has > > > slow ops > > > > > > - removing and recreating monitor didn't help > > > > > > - checking mon_status of
[ceph-users] Crashed MDS (segfault)
Dear ceph users,
we're experiencing a segfault during MDS startup (replay process) which is
making our FS inaccessible.
MDS log messages:
Oct 15 03:41:39.894584 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201
7f3c08f49700 1 -- 192.168.8.195:6800/3181891717 <== osd.26
192.168.8.209:6821/2419345 3 osd_op_reply(21 1. [getxattr]
v0'0 uv0 ondisk = -61 ((61) No data available)) v8 154+0+0 (3715233608
0 0) 0x2776340 con 0x18bd500
Oct 15 03:41:39.894584 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201
7f3c00589700 10 MDSIOContextBase::complete: 18C_IO_Inode_Fetched
Oct 15 03:41:39.894658 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201
7f3c00589700 10 mds.0.cache.ino(0x100) _fetched got 0 and 544
Oct 15 03:41:39.894658 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201
7f3c00589700 10 mds.0.cache.ino(0x100) magic is 'ceph fs volume v011'
(expecting 'ceph fs volume v011')
Oct 15 03:41:39.894735 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201
7f3c00589700 10 mds.0.cache.snaprealm(0x100 seq 1 0x1799c00) open_parents
[1,head]
Oct 15 03:41:39.894735 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201
7f3c00589700 10 mds.0.cache.ino(0x100) _fetched [inode 0x100 [...2,head]
~mds0/ auth v275131 snaprealm=0x1799c00 f(v0 1=1+0) n(v76166 rc2020-07-17
15:29:27.00 b41838692297 -3184=-3168+-16)/n() (iversion lock) 0x18bf800]
Oct 15 03:41:39.894821 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201
7f3c00589700 10 MDSIOContextBase::complete: 18C_IO_Inode_Fetched
Oct 15 03:41:39.894821 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201
7f3c00589700 10 mds.0.cache.ino(0x1) _fetched got 0 and 482
Oct 15 03:41:39.894891 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201
7f3c00589700 10 mds.0.cache.ino(0x1) magic is 'ceph fs volume v011'
(expecting 'ceph fs volume v011')
Oct 15 03:41:39.894958 mds1 ceph-mds: -472> 2019-10-15 00:40:30.205
7f3c00589700 -1 *** Caught signal (Segmentation fault) **#012 in thread
7f3c00589700 thread_name:fn_anonymous#012#012 ceph version 13.2.6
(7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable)#012 1:
(()+0x11390) [0x7f3c0e48a390]#012 2: (operator<<(std::ostream&, SnapRealm
const&)+0x42) [0x72cb92]#012 3: (SnapRealm::merge_to(SnapRealm*)+0x308)
[0x72f488]#012 4: (CInode::decode_snap_blob(ceph::buffer::list&)+0x53)
[0x6e1f63]#012 5:
(CInode::decode_store(ceph::buffer::list::iterator&)+0x76) [0x702b86]#012
6: (CInode::_fetched(ceph::buffer::list&, ceph::buffer::list&,
Context*)+0x1b2) [0x702da2]#012 7: (MDSIOContextBase::complete(int)+0x119)
[0x74fcc9]#012 8: (Finisher::finisher_thread_entry()+0x12e)
[0x7f3c0ebffece]#012 9: (()+0x76ba) [0x7f3c0e4806ba]#012 10: (clone()+0x6d)
[0x7f3c0dca941d]#012 NOTE: a copy of the executable, or `objdump -rdS
` is needed to interpret this.
Oct 15 03:41:39.895400 mds1 ceph-mds: --- logging levels ---
Oct 15 03:41:39.895473 mds1 ceph-mds:0/ 5 none
Oct 15 03:41:39.895473 mds1 ceph-mds:0/ 1 lockdep
Cluster status information:
cluster:
id: b8205875-e56f-4280-9e52-6aab9c758586
health: HEALTH_WARN
1 filesystem is degraded
1 nearfull osd(s)
11 pool(s) nearfull
services:
mon: 3 daemons, quorum mon1,mon2,mon3
mgr: mon1(active), standbys: mon2, mon3
mds: fs_padrao-1/1/1 up {0=mds1=up:replay(laggy or crashed)}
osd: 90 osds: 90 up, 90 in
data:
pools: 11 pools, 1984 pgs
objects: 75.99 M objects, 285 TiB
usage: 457 TiB used, 181 TiB / 639 TiB avail
pgs: 1896 active+clean
87 active+clean+scrubbing+deep+repair
1active+clean+scrubbing
io:
client: 89 KiB/s wr, 0 op/s rd, 3 op/s wr
Has anyone seen anything like this?
Regards,
Gustavo.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] problem returning mon back to cluster
On Mon, Oct 14, 2019 at 11:52:55PM +0200, Paul Emmerich wrote: > How big is the mon's DB? As in just the total size of the directory you > copied > > FWIW I recently had to perform mon surgery on a 14.2.4 (or was it > 14.2.2?) cluster with 8 GB mon size and I encountered no such problems > while syncing a new mon which took 10 minutes or so. Hi Paul, yup I forgot to mention this.. It doesn't seem to be too big, just about 100MB. I also noticed that while third monitor tries to join the cluster, leader starts flapping between "leader" and "electing", so I suppose it's quorum forming problem.. I tried bumping debug_ms and debug_paxos but couldn't make head or tails of it.. can paste the logs somewhere if it can help BR nik > > Paul > > -- > Paul Emmerich > > Looking for help with your Ceph cluster? Contact us at https://croit.io > > croit GmbH > Freseniusstr. 31h > 81247 München > www.croit.io > Tel: +49 89 1896585 90 > > On Mon, Oct 14, 2019 at 9:41 PM Nikola Ciprich > wrote: > > > > On Mon, Oct 14, 2019 at 04:31:22PM +0200, Nikola Ciprich wrote: > > > On Mon, Oct 14, 2019 at 01:40:19PM +0200, Harald Staub wrote: > > > > Probably same problem here. When I try to add another MON, "ceph > > > > health" becomes mostly unresponsive. One of the existing ceph-mon > > > > processes uses 100% CPU for several minutes. Tried it on 2 test > > > > clusters (14.2.4, 3 MONs, 5 storage nodes with around 2 hdd osds > > > > each). To avoid errors like "lease timeout", I temporarily increase > > > > "mon lease", from 5 to 50 seconds. > > > > > > > > Not sure how bad it is from a customer PoV. But it is a problem by > > > > itself to be several minutes without "ceph health", when there is an > > > > increased risk of losing the quorum ... > > > > > > Hi Harry, > > > > > > thanks a lot for your reply! not sure we're experiencing the same issue, > > > i don't have it on any other cluster.. when this is happening to you, does > > > only ceph health stop working, or it also blocks all clients IO? > > > > > > BR > > > > > > nik > > > > > > > > > > > > > > Harry > > > > > > > > On 13.10.19 20:26, Nikola Ciprich wrote: > > > > >dear ceph users and developers, > > > > > > > > > >on one of our production clusters, we got into pretty unpleasant > > > > >situation. > > > > > > > > > >After rebooting one of the nodes, when trying to start monitor, whole > > > > >cluster > > > > >seems to hang, including IO, ceph -s etc. When this mon is stopped > > > > >again, > > > > >everything seems to continue. Traying to spawn new monitor leads to > > > > >the same problem > > > > >(even on different node). > > > > > > > > > >I had to give up after minutes of outage, since it's unacceptable. I > > > > >think we had this > > > > >problem once in the past on this cluster, but after some (but much > > > > >shorter) time, monitor > > > > >joined and it worked fine since then. > > > > > > > > > >All cluster nodes are centos 7 machines, I have 3 monitors (so 2 are > > > > >now running), I'm > > > > >using ceph 13.2.6 > > > > > > > > > >Network connection seems to be fine. > > > > > > > > > >Anyone seen similar problem? I'd be very grateful for tips on how to > > > > >debug and solve this.. > > > > > > > > > >for those interested, here's log of one of running monitors with > > > > >debug_mon set to 10/10: > > > > > > > > > >https://storage.lbox.cz/public/d258d0 > > > > > > > > > >if I could provide more info, please let me know > > > > > > > > > >with best regards > > > > > > > > > >nikola ciprich > > > > just to add quick update, I was able to reproduce the issue by transferring > > monitor > > directories to test environmen with same IP adressing, so I can safely play > > with that > > now.. > > > > increasing lease timeout didn't help me to fix the problem, > > but at least I seem to be able to use ceph -s now. > > > > few things I noticed in the meantime: > > > > - when I start problematic monitor, monitor slow ops start to appear for > > quorum leader and the count is slowly increasing: > > > > 44 slow ops, oldest one blocked for 130 sec, mon.nodev1c has > > slow ops > > > > - removing and recreating monitor didn't help > > > > - checking mon_status of problematic monitor shows it remains in the > > "synchronizing" state > > > > I tried increasing debug_ms and debug_paxos but didn't see anything usefull > > there.. > > > > will report further when I got something. I anyone has any idea in the > > meantime, please > > let me know. > > > > BR > > > > nik > > > > > > > > > > -- > > - > > Ing. Nikola CIPRICH > > LinuxBox.cz, s.r.o. > > 28. rijna 168, 709 00 Ostrava > > > > tel.: +420 591 166 214 > > fax:+420 596 621 273 > > mobil: +420 777 093 799 > > > > www.linuxbox.cz > > > > mobil servis: +420 737 238 656 > > email servis: ser...@linuxbox.cz > > - > > ___ > > ceph-users mailing list
Re: [ceph-users] problem returning mon back to cluster
Probably same problem here. When I try to add another MON, "ceph health" becomes mostly unresponsive. One of the existing ceph-mon processes uses 100% CPU for several minutes. Tried it on 2 test clusters (14.2.4, 3 MONs, 5 storage nodes with around 2 hdd osds each). To avoid errors like "lease timeout", I temporarily increase "mon lease", from 5 to 50 seconds. Not sure how bad it is from a customer PoV. But it is a problem by itself to be several minutes without "ceph health", when there is an increased risk of losing the quorum ... Harry On 13.10.19 20:26, Nikola Ciprich wrote: dear ceph users and developers, on one of our production clusters, we got into pretty unpleasant situation. After rebooting one of the nodes, when trying to start monitor, whole cluster seems to hang, including IO, ceph -s etc. When this mon is stopped again, everything seems to continue. Traying to spawn new monitor leads to the same problem (even on different node). I had to give up after minutes of outage, since it's unacceptable. I think we had this problem once in the past on this cluster, but after some (but much shorter) time, monitor joined and it worked fine since then. All cluster nodes are centos 7 machines, I have 3 monitors (so 2 are now running), I'm using ceph 13.2.6 Network connection seems to be fine. Anyone seen similar problem? I'd be very grateful for tips on how to debug and solve this.. for those interested, here's log of one of running monitors with debug_mon set to 10/10: https://storage.lbox.cz/public/d258d0 if I could provide more info, please let me know with best regards nikola ciprich ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Pool statistics via API
Hi Ernesto, I just opened the Dashboard and there is no menu at the top-right. Also no "?". I have a menu at the top-left which has the following items: Cluster health, Cluster, Block and Filesystems. Running Ceph version 12.2.8-89. Kind regards, Sinan Polat > Op 11 oktober 2019 om 22:09 schreef Sinan Polat : > > Hi Ernesto, > > Thanks for the information! I didn’t know about the existence of the REST > Dashboard API. I will check that out, Thanks again! > > Sinan > > Op 11 okt. 2019 om 21:06 heeft Ernesto Puerta mailto:epuer...@redhat.com > het volgende geschreven: > > > > > Hi Sinan, > > > > If it's in the Dashboard, it sure comes from the Dashboard REST API > > (which is an API completely unrelated to the RESTful Module). > > > > To check the Dashboard REST API, log in there and click on the > > top-right "?" menu, and in the dropdown, click on "API". That will lead you > > to the Swagger/OpenAPI spec of the Dashboard. You will likely want to > > explore the "/pool" and "/block" endpoints. The API page will give you > > ready-to-use curl commands (the only thing you'd need to renew, once > > expired, is the authorization token). > > > > Kind regards, > > > > Ernesto Puerta > > He / Him / His > > > > Senior Software Engineer, Ceph > > Red Hat https://www.redhat.com/ > > > > > > https://www.redhat.com/ > > > > > > > > > > On Thu, Oct 10, 2019 at 2:16 PM Sinan Polat > mailto:si...@turka.nl > wrote: > > > > > > > > > > Hi, > > > > > > Currently I am getting the pool statistics (especially > > > USED/MAX AVAIL) via the command line: > > > ceph df -f json-pretty| jq '.pools[] | select(.name == > > > "poolname") | .stats.max_avail' > > > ceph df -f json-pretty| jq '.pools[] | select(.name == > > > "poolname") | .stats.bytes_used' > > > > > > Command "ceph df" does not show the (total) size of the > > > provisioned RBD images. It only shows the real usage. > > > > > > I managed to get the total size of provisioned images using > > > the Python rbd module https://docs.ceph.com/docs/master/rbd/api/librbdpy/ > > > > > > https://docs.ceph.com/docs/master/rbd/api/librbdpy/ > > > Using the same Python module I also would like to get the > > > USED/MAX AVAIL per pool. That should be possible using > > > rbd.RBD().pool_stats_get, but unfortunately my python-rbd version doesn't > > > support that (running 12.2.8). > > > > > > So I went ahead and enabled the dashboard to see if the data > > > is present in the dashboard and it seems it is. Next step is to enable the > > > restful module and access this information, right? But unfortunately the > > > restful api doesn't provide this information. > > > > > > My question is, how can I access the USED/MAX AVAIL > > > information of a pool without using the ceph command line and without > > > upgrading my python-rbd package? > > > > > > Kind regards > > > Sinan Polat > > > > > > ___ > > > ceph-users mailing list > > > ceph-users@lists.ceph.com mailto:ceph-users@lists.ceph.com > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > > > > > > > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph Negative Objects Number
Hi Lazuardi, never seen that. Just wondering what Ceph version are you running? Thanks, Igor On 10/8/2019 3:52 PM, Lazuardi Nasution wrote: Hi, I get following weird negative objects number on tiering. Why is this happening? How to get back to normal? Best regards, [root@management-a ~]# ceph df detail GLOBAL: SIZE AVAIL RAW USED %RAW USED OBJECTS 446T 184T 261T 58.62 22092k POOLS: NAME ID CATEGORY QUOTA OBJECTS QUOTA BYTES USED %USED MAX AVAIL OBJECTS DIRTY READ WRITE RAW USED rbd 1 - N/A N/A 0 0 25838G 0 0 0 1 0 volumes 2 - N/A N/A 82647G 76.18 25838G 21177891 20681k 5897M 2447M 242T images 3 - N/A N/A 3683G 12.48 25838G 705881 689k 37844k 10630k 11049G backups 4 - N/A N/A 0 0 25838G 0 0 0 0 0 vms 5 - N/A N/A 3003G 10.41 25838G 772845 754k 623M 812M 9010G rbd_tiering 11 - N/A N/A 333 0 3492G 4 0 1 2 999 volumes_tiering 12 - N/A N/A 9761M 0 3492G -1233 338 2340M 1982M 0 images_tiering 13 - N/A N/A 293k 0 3492G 129 0 34642k 3600k 880k backups_tiering 14 - N/A N/A 83 0 3492G 1 0 2 2 249 vms_tiering 15 - N/A N/A 2758M 0 3492G -32567 116 31942M 2875M 0 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph Negative Objects Number
Hi, Any body has same problem with my case? Best regards, On Tue, Oct 8, 2019, 19:52 Lazuardi Nasution wrote: > Hi, > > I get following weird negative objects number on tiering. Why is this > happening? How to get back to normal? > > Best regards, > > [root@management-a ~]# ceph df detail > GLOBAL: > SIZE AVAIL RAW USED %RAW USED OBJECTS > 446T 184T 261T 58.62 22092k > POOLS: > NAMEID CATEGORY QUOTA OBJECTS QUOTA BYTES > USED %USED MAX AVAIL OBJECTS DIRTY READ > WRITE RAW USED > rbd 1 -N/A N/A > 0 025838G0 0 0 >10 > volumes 2 -N/A N/A > 82647G 76.1825838G 21177891 20681k 5897M > 2447M 242T > images 3 -N/A N/A > 3683G 12.4825838G 705881 689k 37844k > 10630k 11049G > backups 4 -N/A N/A > 0 025838G0 0 0 >00 > vms 5 -N/A N/A > 3003G 10.4125838G 772845 754k 623M > 812M9010G > rbd_tiering 11 -N/A N/A >333 0 3492G4 0 1 >2 999 > volumes_tiering 12 -N/A N/A > 9761M 0 3492G-1233338 2340M > 1982M0 > images_tiering 13 -N/A N/A > 293k 0 3492G 129 0 34642k > 3600k 880k > backups_tiering 14 -N/A N/A > 83 0 3492G1 0 2 >2 249 > vms_tiering 15 -N/A N/A > 2758M 0 3492G -32567116 31942M > 2875M0 > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Openstack VM IOPS drops dramatically during Ceph recovery
On Thu, Oct 10, 2019 at 2:23 PM huxia...@horebdata.cn wrote: > > Hi, folks, > > I have a middle-size Ceph cluster as cinder backup for openstack (queens). > Duing testing, one Ceph node went down unexpected and powered up again ca 10 > minutes later, Ceph cluster starts PG recovery. To my surprise, VM IOPS > drops dramatically during Ceph recovery, from ca. 13K IOPS to about 400, a > factor of 1/30, and I did put a stringent throttling on backfill and > recovery, with the following ceph parameters > > osd_max_backfills = 1 > osd_recovery_max_active = 1 > osd_client_op_priority=63 > osd_recovery_op_priority=1 > osd_recovery_sleep = 0.5 > > The most weird thing is, > 1) when there is no IO activity from any VM (ALL VMs are quiet except the > recovery IO), the recovery bandwidth is ca. 10MiB/s, 2 objects/s. Seems like > recovery throttle setting is working properly > 2) when using FIO testing inside a VM, the recovery bandwith is going up > quickly, reaching above 200MiB/s, 60 objects/s. FIO IOPS performance inside > VM, however, is only at 400 IOPS/s (8KiB block size), around 3MiB/s. Obvious > recovery throttling DOES NOT work properly > 3) If i stop the FIO testing in VM, the recovery bandwith then goes down to > 10MiB/s, 2 objects/s again, strange enough. > > How can this weird behavior happen? I just wonder, is there a method to > configure recovery bandwith to a specific value, or the number of recovery > objects per second? this may give better control of bakcfilling/recovery, > instead of the faulty logic or relative osd_client_op_priority vs > osd_recovery_op_priority. > > any ideas or suggests to make the recovery under control? > > best regards, > > Samuel Not sure which version of Ceph you are on, but add these to your /etc/ceph/ceph.conf on all your OSDs and restart them. osd op queue = wpq osd op queue cut off = high That should really help and make backfills and recovery be non-impactful. This will be the default in Octopus. Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] problem returning mon back to cluster
On Mon, Oct 14, 2019 at 01:40:19PM +0200, Harald Staub wrote: > Probably same problem here. When I try to add another MON, "ceph > health" becomes mostly unresponsive. One of the existing ceph-mon > processes uses 100% CPU for several minutes. Tried it on 2 test > clusters (14.2.4, 3 MONs, 5 storage nodes with around 2 hdd osds > each). To avoid errors like "lease timeout", I temporarily increase > "mon lease", from 5 to 50 seconds. > > Not sure how bad it is from a customer PoV. But it is a problem by > itself to be several minutes without "ceph health", when there is an > increased risk of losing the quorum ... Hi Harry, thanks a lot for your reply! not sure we're experiencing the same issue, i don't have it on any other cluster.. when this is happening to you, does only ceph health stop working, or it also blocks all clients IO? BR nik > > Harry > > On 13.10.19 20:26, Nikola Ciprich wrote: > >dear ceph users and developers, > > > >on one of our production clusters, we got into pretty unpleasant situation. > > > >After rebooting one of the nodes, when trying to start monitor, whole cluster > >seems to hang, including IO, ceph -s etc. When this mon is stopped again, > >everything seems to continue. Traying to spawn new monitor leads to the same > >problem > >(even on different node). > > > >I had to give up after minutes of outage, since it's unacceptable. I think > >we had this > >problem once in the past on this cluster, but after some (but much shorter) > >time, monitor > >joined and it worked fine since then. > > > >All cluster nodes are centos 7 machines, I have 3 monitors (so 2 are now > >running), I'm > >using ceph 13.2.6 > > > >Network connection seems to be fine. > > > >Anyone seen similar problem? I'd be very grateful for tips on how to debug > >and solve this.. > > > >for those interested, here's log of one of running monitors with debug_mon > >set to 10/10: > > > >https://storage.lbox.cz/public/d258d0 > > > >if I could provide more info, please let me know > > > >with best regards > > > >nikola ciprich > > > > > > > > > > > > > > > -- - Ing. Nikola CIPRICH LinuxBox.cz, s.r.o. 28. rijna 168, 709 00 Ostrava tel.: +420 591 166 214 fax:+420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: ser...@linuxbox.cz - pgp6aNC92DcqR.pgp Description: PGP signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] tcmu-runner: mismatched sizes for rbd image size
@Mike Did you have the chance to update download.ceph.com repositories for the new version? I just tested the packages from shaman in our DEV environment and it seems to fix the work - after updating the packages i was not able to reproduce the error again and tcmu-runner starts up without any errors ;) Von: Mike Christie Gesendet: Donnerstag, 3. Oktober 2019 00:20:51 An: Kilian Ries; dilla...@redhat.com Cc: ceph-users@lists.ceph.com Betreff: Re: [ceph-users] tcmu-runner: mismatched sizes for rbd image size On 10/02/2019 02:15 PM, Kilian Ries wrote: > Ok i just compared my local python files and the git commit you sent me > - it really looks like i have the old files installed. All the changes > are missing in my local files. > > > > Where can i get a new ceph-iscsi-config package that has the fixe > included? I have installed version: They are on shaman only right now: https://4.chacra.ceph.com/r/ceph-iscsi-config/master/24deeb206ed2354d44b0f33d7d26d475e1014f76/centos/7/flavors/default/noarch/ https://4.chacra.ceph.com/r/ceph-iscsi-cli/master/4802654a6963df6bf5f4a968782cfabfae835067/centos/7/flavors/default/noarch/ The shaman rpms above have one bug we just fixed in ceph-iscsi-config where if DNS is not setup correctly gwcli commands can take minutes. I am going to try and get download.ceph.com updated. > > ceph-iscsi-config-2.6-2.6.el7.noarch > > *Von:* ceph-users im Auftrag von > Kilian Ries > *Gesendet:* Mittwoch, 2. Oktober 2019 21:04:45 > *An:* dilla...@redhat.com > *Cc:* ceph-users@lists.ceph.com > *Betreff:* Re: [ceph-users] tcmu-runner: mismatched sizes for rbd image > size > > > Yes, i created all four luns with these sizes: > > > lun0 - 5120G > > lun1 - 5121G > > lun2 - 5122G > > lun3 - 5123G > > > Its always one GB more per LUN... Is there any newer ceph-iscsi-config > package than i have installed? > > > ceph-iscsi-config-2.6-2.6.el7.noarch > > > Then i could try to update the package and see if the error is fixed ... > > > *Von:* Jason Dillaman > *Gesendet:* Mittwoch, 2. Oktober 2019 16:00:03 > *An:* Kilian Ries > *Cc:* ceph-users@lists.ceph.com > *Betreff:* Re: [ceph-users] tcmu-runner: mismatched sizes for rbd image > size > > On Wed, Oct 2, 2019 at 9:50 AM Kilian Ries wrote: >> >> Hi, >> >> >> i'm running a ceph mimic cluster with 4x ISCSI gateway nodes. Cluster was >> setup via ceph-ansible v3.2-stable. I just checked my nodes and saw that >> only two of the four configured iscsi gw nodes are working correct. I first >> noticed via gwcli: >> >> >> ### >> >> >> $gwcli -d ls >> >> Traceback (most recent call last): >> >> File "/usr/bin/gwcli", line 191, in >> >> main() >> >> File "/usr/bin/gwcli", line 103, in main >> >> root_node.refresh() >> >> File "/usr/lib/python2.7/site-packages/gwcli/gateway.py", line 87, in >> refresh >> >> raise GatewayError >> >> gwcli.utils.GatewayError >> >> >> ### >> >> >> I investigated and noticed that both "rbd-target-api" and "rbd-target-gw" >> are not running. I were not able to restart them via systemd. I then found >> that even tcmu-runner is not running and it exits with the following error: >> >> >> >> ### >> >> >> tcmu_rbd_check_image_size:827 rbd/production.lun1: Mismatched sizes. RBD >> image size 5498631880704. Requested new size 5497558138880. >> >> >> ### >> >> >> Now i have the situation that two nodes are running correct and two cant >> start tcmu-runner. I don't know where the image size mismatches are coming >> from - i haven't configured or resized any of the images. >> >> >> Is there any chance to get my two iscsi gw nodes back working? > > It sounds like you are potentially hitting [1]. The ceph-iscsi-config > library thinks your image size is 5TiB but you actually have a 5121GiB > (~5.001TiB) RBD image. Any clue how your RBD image got to be 1GiB > larger than an even 5TiB? > >> >> >> >> The following packets are installed: >> >> >> rpm -qa |egrep "ceph|iscsi|tcmu|rst|kernel" >> >> >> libtcmu-1.4.0-106.gd17d24e.el7.x86_64 >> >> ceph-iscsi-cli-2.7-2.7.el7.noarch >> >> kernel-3.10.0-957.5.1.el7.x86_64 >> >> ceph-base-13.2.5-0.el7.x86_64 >> >> ceph-iscsi-config-2.6-2.6.el7.noarch >> >> ceph-common-13.2.5-0.el7.x86_64 >> >> ceph-selinux-13.2.5-0.el7.x86_64 >> >> kernel-tools-libs-3.10.0-957.5.1.el7.x86_64 >> >> python-cephfs-13.2.5-0.el7.x86_64 >> >> ceph-osd-13.2.5-0.el7.x86_64 >> >> kernel-headers-3.10.0-957.5.1.el7.x86_64 >> >> kernel-tools-3.10.0-957.5.1.el7.x86_64 >> >> kernel-3.10.0-957.1.3.el7.x86_64 >> >> libcephfs2-13.2.5-0.el7.x86_64 >> >> kernel-3.10.0-862.14.4.el7.x86_64 >> >> tcmu-runner-1.4.0-106.gd17d24e.el7.x86_64 >> >> >> >> Thanks, >> >> Greets >> >> >> Kilian >> >> >> ___ >> ceph-users mailing list >> ceph-users@lists.ceph.com >>
Re: [ceph-users] Ceph Negative Objects Number
Hi Igor, It is the old Jewel (v10.2.11). This case is happen after I do cache-try-flush-evict-all or cache-flush-evict-all on respected tier pool. Best regards, On Mon, Oct 14, 2019 at 7:38 PM Igor Fedotov wrote: > Hi Lazuardi, > > never seen that. Just wondering what Ceph version are you running? > > > Thanks, > > Igor > On 10/8/2019 3:52 PM, Lazuardi Nasution wrote: > > Hi, > > I get following weird negative objects number on tiering. Why is this > happening? How to get back to normal? > > Best regards, > > [root@management-a ~]# ceph df detail > GLOBAL: > SIZE AVAIL RAW USED %RAW USED OBJECTS > 446T 184T 261T 58.62 22092k > POOLS: > NAMEID CATEGORY QUOTA OBJECTS QUOTA BYTES > USED %USED MAX AVAIL OBJECTS DIRTY READ > WRITE RAW USED > rbd 1 -N/A N/A > 0 025838G0 0 0 >10 > volumes 2 -N/A N/A > 82647G 76.1825838G 21177891 20681k 5897M > 2447M 242T > images 3 -N/A N/A > 3683G 12.4825838G 705881 689k 37844k > 10630k 11049G > backups 4 -N/A N/A > 0 025838G0 0 0 >00 > vms 5 -N/A N/A > 3003G 10.4125838G 772845 754k 623M > 812M9010G > rbd_tiering 11 -N/A N/A >333 0 3492G4 0 1 >2 999 > volumes_tiering 12 -N/A N/A > 9761M 0 3492G-1233338 2340M > 1982M0 > images_tiering 13 -N/A N/A > 293k 0 3492G 129 0 34642k > 3600k 880k > backups_tiering 14 -N/A N/A > 83 0 3492G1 0 2 >2 249 > vms_tiering 15 -N/A N/A > 2758M 0 3492G -32567116 31942M > 2875M0 > > ___ > ceph-users mailing > listceph-us...@lists.ceph.comhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] problem returning mon back to cluster
On Mon, Oct 14, 2019 at 04:31:22PM +0200, Nikola Ciprich wrote: > On Mon, Oct 14, 2019 at 01:40:19PM +0200, Harald Staub wrote: > > Probably same problem here. When I try to add another MON, "ceph > > health" becomes mostly unresponsive. One of the existing ceph-mon > > processes uses 100% CPU for several minutes. Tried it on 2 test > > clusters (14.2.4, 3 MONs, 5 storage nodes with around 2 hdd osds > > each). To avoid errors like "lease timeout", I temporarily increase > > "mon lease", from 5 to 50 seconds. > > > > Not sure how bad it is from a customer PoV. But it is a problem by > > itself to be several minutes without "ceph health", when there is an > > increased risk of losing the quorum ... > > Hi Harry, > > thanks a lot for your reply! not sure we're experiencing the same issue, > i don't have it on any other cluster.. when this is happening to you, does > only ceph health stop working, or it also blocks all clients IO? > > BR > > nik > > > > > > Harry > > > > On 13.10.19 20:26, Nikola Ciprich wrote: > > >dear ceph users and developers, > > > > > >on one of our production clusters, we got into pretty unpleasant situation. > > > > > >After rebooting one of the nodes, when trying to start monitor, whole > > >cluster > > >seems to hang, including IO, ceph -s etc. When this mon is stopped again, > > >everything seems to continue. Traying to spawn new monitor leads to the > > >same problem > > >(even on different node). > > > > > >I had to give up after minutes of outage, since it's unacceptable. I think > > >we had this > > >problem once in the past on this cluster, but after some (but much > > >shorter) time, monitor > > >joined and it worked fine since then. > > > > > >All cluster nodes are centos 7 machines, I have 3 monitors (so 2 are now > > >running), I'm > > >using ceph 13.2.6 > > > > > >Network connection seems to be fine. > > > > > >Anyone seen similar problem? I'd be very grateful for tips on how to debug > > >and solve this.. > > > > > >for those interested, here's log of one of running monitors with debug_mon > > >set to 10/10: > > > > > >https://storage.lbox.cz/public/d258d0 > > > > > >if I could provide more info, please let me know > > > > > >with best regards > > > > > >nikola ciprich just to add quick update, I was able to reproduce the issue by transferring monitor directories to test environmen with same IP adressing, so I can safely play with that now.. increasing lease timeout didn't help me to fix the problem, but at least I seem to be able to use ceph -s now. few things I noticed in the meantime: - when I start problematic monitor, monitor slow ops start to appear for quorum leader and the count is slowly increasing: 44 slow ops, oldest one blocked for 130 sec, mon.nodev1c has slow ops - removing and recreating monitor didn't help - checking mon_status of problematic monitor shows it remains in the "synchronizing" state I tried increasing debug_ms and debug_paxos but didn't see anything usefull there.. will report further when I got something. I anyone has any idea in the meantime, please let me know. BR nik -- - Ing. Nikola CIPRICH LinuxBox.cz, s.r.o. 28. rijna 168, 709 00 Ostrava tel.: +420 591 166 214 fax:+420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: ser...@linuxbox.cz - pgpVNlq5CstSn.pgp Description: PGP signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
