Re: [ceph-users] Mounting Ceph RBD image to XenServer 7 as SR

[Mike Jacobacci]

Yes, I would like to know too… I decided nott to update the kernel as it could 
possibly affect xenserver’s stability and/or performance.

Cheers,
Mike
> On Jun 30, 2016, at 11:54 PM, Josef Johansson  wrote:
> 
> Also, is it possible to recompile the rbd kernel module in XenServer? I am 
> under the impression that it's open source as well.
> 
> Regards,
> Josef
> 
> 
> On Fri, 1 Jul 2016, 04:52 Mike Jacobacci,  > wrote:
> Thanks Somnath and Christian,
> 
> Yes, it looks like the latest version of XenServer still runs on an old 
> kernel (3.10).  I know the method Christian linked, but it doesn’t work if 
> XenServer is installed from iso.  It is really annoying there has been no 
> movement on this for 3 years… I really like XenServer and am excited to use 
> Ceph, I want this to work.  
> 
> Since there are no VM’s on it yet, I think I will upgrade the kernel and see 
> what happens.
> 
> Cheers,
> Mike
> 
> 
>> On Jun 30, 2016, at 7:40 PM, Somnath Roy > > wrote:
>> 
>> It seems your client kernel is pretty old ?
>> Either upgrade your kernel to 3.15 or later or you need to disable 
>> CRUSH_TUNABLES3.
>> ceph osd crush tunables bobtail or ceph osd crush tunables legacy should 
>> help. This will start rebalancing and also you will lose improvement added 
>> in Firefly. So, better to upgrade client kernel IMO.
>>  
>> Thanks & Regards
>> Somnath
>>  
>> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com 
>> ] On Behalf Of Mike Jacobacci
>> Sent: Thursday, June 30, 2016 7:27 PM
>> To: Jake Young
>> Cc: ceph-users@lists.ceph.com 
>> Subject: Re: [ceph-users] Mounting Ceph RBD image to XenServer 7 as SR
>>  
>> Thanks Jake!  I enabled the epel 7 repo and was able to get ceph-common 
>> installed.  Here is what happens when I try to map the drive:
>>  
>> rbd map rbd/enterprise-vm0 --name client.admin -m mon0 -k 
>> /etc/ceph/ceph.client.admin.keyring 
>> rbd: sysfs write failed
>> In some cases useful info is found in syslog - try "dmesg | tail" or so.
>> rbd: map failed: (5) Input/output error
>>  
>> dmesg | tail:
>>  
>> [35034.469236] libceph: mon0 192.168.10.187:6789 
>>  socket error on read
>> [35044.469183] libceph: mon0 192.168.10.187:6789 
>>  feature set mismatch, my 4a042a42 < server's 
>> 2004a042a42, missing 200
>> [35044.469199] libceph: mon0 192.168.10.187:6789 
>>  socket error on read
>> [35054.469076] libceph: mon0 192.168.10.187:6789 
>>  feature set mismatch, my 4a042a42 < server's 
>> 2004a042a42, missing 200
>> [35054.469083] libceph: mon0 192.168.10.187:6789 
>>  socket error on read
>> [35064.469287] libceph: mon0 192.168.10.187:6789 
>>  feature set mismatch, my 4a042a42 < server's 
>> 2004a042a42, missing 200
>> [35064.469302] libceph: mon0 192.168.10.187:6789 
>>  socket error on read
>> [35074.469162] libceph: mon0 192.168.10.187:6789 
>>  feature set mismatch, my 4a042a42 < server's 
>> 2004a042a42, missing 200
>> [35074.469178] libceph: mon0 192.168.10.187:6789 
>>  socket error on read
>>  
>>  
>>  
>>  
>> On Jun 30, 2016, at 6:15 PM, Jake Young > > wrote:
>>  
>> See https://www.mail-archive.com/ceph-users@lists.ceph.com/msg17112.html 
>> 
>> 
>> 
>> On Thursday, June 30, 2016, Mike Jacobacci > > wrote:
>> So after adding the ceph repo and enabling the cents-7 repo… It fails trying 
>> to install ceph-common:
>>  
>> Loaded plugins: fastestmirror
>> Loading mirror speeds from cached hostfile
>>  * base: mirror.web-ster.com 
>> Resolving Dependencies
>> --> Running transaction check
>> ---> Package ceph-common.x86_64 1:10.2.2-0.el7 will be installed
>> --> Processing Dependency: python-cephfs = 1:10.2.2-0.el7 for package: 
>> 1:ceph-common-10.2.2-0.el7.x86_64
>> --> Processing Dependency: python-rados = 1:10.2.2-0.el7 for package: 
>> 1:ceph-common-10.2.2-0.el7.x86_64
>> --> Processing Dependency: librbd1 = 1:10.2.2-0.el7 for package: 
>> 1:ceph-common-10.2.2-0.el7.x86_64
>> --> Processing Dependency: libcephfs1 = 1:10.2.2-0.el7 for package: 
>> 1:ceph-common-10.2.2-0.el7.x86_64
>> --> Processing Dependency: python-rbd = 1:10.2.2-0.el7 for package: 
>> 1:ceph-common-10.2.2-0.el7.x86_64
>> --> Processing Dependency: librados2 = 1:10.2.2-0.el7 for package: 
>> 1:ceph-common-10.2.2-0.el7.x86_64
>> --> Processing Dependency: python-requests for package: 
>> 1:ceph-common-10.2.2-0.el7.x86_64
>> --> 

Re: [ceph-users] rbd cache command thru admin socket

[Deneau, Tom]

Thanks, Jason--

Turns out AppArmor was indeed enabled (I was not aware of that).
Disabled it and now I see the socket but it seems to only be there
temporarily while some client app is running.

The original reason I wanted to use this socket was that I am also
using an rbd images thru kvm and I wanted to be able to flush and 
invalidate the rbd cache as an experiment.

I would have thought the socket would get created and stay there as
long as kvm is active (since kvm is using librbd).  But even when I 
access the rbd disk from the VM, I don't see any socket created at all.

-- Tom


> -Original Message-
> From: Jason Dillaman [mailto:jdill...@redhat.com]
> Sent: Thursday, June 30, 2016 6:15 PM
> To: Deneau, Tom 
> Cc: ceph-users 
> Subject: Re: [ceph-users] rbd cache command thru admin socket
> 
> Can you check the permissions on "/var/run/ceph/" and ensure that the user
> your client runs under has permissions to access the directory?
> If the permissions are OK, do you have SElinux or AppArmor enabled and
> enforcing?
> 
> On Thu, Jun 30, 2016 at 5:37 PM, Deneau, Tom  wrote:
> > I was following the instructions in
> >
> > https://www.sebastien-han.fr/blog/2015/09/02/ceph-validate-that-the-rb
> > d-cache-is-active/
> >
> > because I wanted to look at some of the rbd cache state and possibly
> > flush and invalidate it
> >
> > My ceph.conf has
> > [client]
> > rbd default features = 1
> > rbd default format = 2
> > admin socket = /var/run/ceph/$cluster-$type.$id.$pid.$cctid.asok
> > log file = /var/log/ceph/
> >
> > I have a client only node (no osds) and on that node I ran fio with the
> librbd engine, which worked fine.  But I did not see anything in
> /var/run/ceph on that client node.  Is there something else I have to do?
> >
> > -- Tom Deneau, AMD
> >
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> 
> 
> --
> Jason
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] suse_enterprise_storage3_rbd_LIO_vmware_performance_bad

[Nick Fisk]

To summarise,

LIO is just not working very well at the moment because of the ABORT Tasks 
problem, this will hopefully be fixed at some point. I'm not sure if SUSE works 
around this, but see below for other pain points with RBD + ESXi + iSCSI

TGT is easy to get going, but performance isn't the best and failover is an 
absolute pain as TGT won't stop if it has ongoing IO. You normally end up in a 
complete mess if you try and do HA, unless you can cover a number of different 
failure scenarios.

SCST probably works the best at the moment. Yes, you have to compile it into a 
new kernel, but it performs well, doesn't fall over, supports the VAAI 
extensions and can be configured HA in an ALUA or VIP failover modes. There 
might be a couple of corner cases with the ALUA mode with Active/Standby paths, 
with possible data corruption that need to be tested/explored.

However, there are a number of pain points with iSCSI + ESXi + RBD and they all 
mainly centre on write latency. It seems VMFS was designed around the fact that 
Enterprise storage arrays service writes in 10-100us, whereas Ceph will service 
them in 2-10ms.

1. Thin Provisioning makes things slow. I believe the main cause is that when 
growing and zeroing the new blocks, metadata needs to be updated and the block 
zero'd. Both issue small IO which would normally not be a problem, but with 
Ceph it becomes a bottleneck to overall IO on the datastore.

2. Snapshots effectively turn all IO into 64kb IO's. Again a traditional SAN 
will coalesce these back into a stream of larger IO's before committing to 
disk. However with Ceph each IO takes 2-10ms and so everything seems slow. The 
future feature of persistent RBD cache may go a long way to helping with this.

3. >2TB VMDK's with snapshots use a different allocation mode, which happens in 
4kb chunks instead of 64kb ones. This makes the problem 16 times worse than 
above.

4. Any of the above will also apply when migrating machines around, so VM's can 
takes hours/days to move.

5. If you use FILEIO, you can't use thin provisioning. If you use BLOCKIO, you 
get thin provisioning, but no pagecache or readahead, so performance can nose 
dive if this is needed.

6. iSCSI is very complicated (especially ALUA) and sensitive. Get used to 
seeing APD/PDL even when you think you have finally got everything working 
great.


Normal IO from eager zeroed VM's with no snapshots, however should perform ok. 
So depends what your workload is.


And then comes NFS. It's very easy to setup, very easy to configure for HA, and 
works pretty well overall. You don't seem to get any of the IO size penalties 
when using snapshots. If you mount with discard, thin provisioning is done by 
Ceph. You can defragment the FS on the proxy node and several other things that 
you can't do with VMFS. Just make sure you run the server in sync mode to avoid 
data loss.

The only downside is that every IO causes an IO to the FS and one to the FS 
journal, so you effectively double your IO. But if your Ceph backend can 
support it, then it shouldn't be too much of a problem.

Now to the original poster, assuming the iSCSI node is just kernel mounting the 
RBD, I would run iostat on it, to try and see what sort of latency you are 
seeing at that point. Also do the same with esxtop +u, and look at the write 
latency there, both whilst running the fio in the VM. This should hopefully let 
you see if there is just a gradual increase as you go from hop to hop or if 
there is an obvious culprit.

Can you also confirm your kernel version? 

With 1GB networking I think you will struggle to get your write latency much 
below 10-15ms, but from your example ~30ms is still a bit high. I wonder if the 
default queue depths on your iSCSI target are too low as well?

Nick

> -Original Message-
> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of
> Oliver Dzombic
> Sent: 01 July 2016 09:27
> To: ceph-users@lists.ceph.com
> Subject: Re: [ceph-users]
> suse_enterprise_storage3_rbd_LIO_vmware_performance_bad
> 
> Hi,
> 
> my experience:
> 
> ceph + iscsi ( multipath ) + vmware == worst
> 
> Better you search for another solution.
> 
> vmware + nfs + vmware might have a much better performance.
> 
> 
> 
> If you are able to get vmware run with iscsi and ceph, i would be
> >>very<< intrested in what/how you did that.
> 
> --
> Mit freundlichen Gruessen / Best regards
> 
> Oliver Dzombic
> IP-Interactive
> 
> mailto:i...@ip-interactive.de
> 
> Anschrift:
> 
> IP Interactive UG ( haftungsbeschraenkt ) Zum Sonnenberg 1-3
> 63571 Gelnhausen
> 
> HRB 93402 beim Amtsgericht Hanau
> Geschäftsführung: Oliver Dzombic
> 
> Steuer Nr.: 35 236 3622 1
> UST ID: DE274086107
> 
> 
> Am 01.07.2016 um 07:04 schrieb mq:
> > Hi list
> > I have tested suse enterprise storage3 using 2 iscsi  gateway attached
> > to  vmware. The performance is bad.  I have turn off  VAAI following
> > the
> >
> 

Re: [ceph-users] mds0: Behind on trimming (58621/30)

[Kenneth Waegeman]

On 01/07/16 12:59, John Spray wrote:

On Fri, Jul 1, 2016 at 11:35 AM, Kenneth Waegeman
 wrote:

Hi all,

While syncing a lot of files to cephfs, our mds cluster got haywire: the
mdss have a lot of segments behind on trimming:  (58621/30)
Because of this the mds cluster gets degraded. RAM usage is about 50GB. The
mdses were respawning and replaying continiously, and I had to stop all
syncs , unmount all clients and increase the beacon_grace to keep the
cluster up .

[root@mds03 ~]# ceph status
 cluster 92bfcf0a-1d39-43b3-b60f-44f01b630e47
  health HEALTH_WARN
 mds0: Behind on trimming (58621/30)
  monmap e1: 3 mons at
{mds01=10.141.16.1:6789/0,mds02=10.141.16.2:6789/0,mds03=10.141.16.3:6789/0}
 election epoch 170, quorum 0,1,2 mds01,mds02,mds03
   fsmap e78658: 1/1/1 up {0=mds03=up:active}, 2 up:standby
  osdmap e19966: 156 osds: 156 up, 156 in
 flags sortbitwise
   pgmap v10213164: 4160 pgs, 4 pools, 253 TB data, 203 Mobjects
 357 TB used, 516 TB / 874 TB avail
 4151 active+clean
5 active+clean+scrubbing
4 active+clean+scrubbing+deep
   client io 0 B/s rd, 0 B/s wr, 63 op/s rd, 844 op/s wr
   cache io 68 op/s promote


Now it finally is up again, it is trimming very slowly (+-120 segments /
min)

Hmm, so it sounds like something was wrong that got cleared by either
the MDS restart or the client unmount, and now it's trimming at a
healthier rate.

What client (kernel or fuse, and version)?

kernel client of centos 7.2, 3.10.0-327.18.2.el7


Can you confirm that the RADOS cluster itself was handling operations
reasonably quickly?  Is your metadata pool using the same drives as
your data?  Were the OSDs saturated with IO?
Metadata pool is a pool of SSDS. Data is ecpool with a cache layer of 
seperate ssds. There was indeed load on the OSDS, and the ceph health 
command produced regularly Cache at/near full ratio warnings too




While the cluster was accumulating untrimmed segments, did you also
have a "client xyz failing to advanced oldest_tid" warning?

We did not see that warning.


It would be good to clarify whether the MDS was trimming slowly, or
not at all.  If you can reproduce this situation, get it to a "behind
on trimming" state, and the stop the client IO (but leave it mounted).
See if the (x/30) number stays the same.  Then, does it start to
decrease when you unmount the client?  That would indicate a
misbehaving client.
mds trimming still at (37927/30), so have to wait some more hours before 
i can try to reproduce it. (Nothing can be done to speed this up?)
There was a moment were the mds was active and I didn't saw the segments 
going down.. I did ran ceph daemon mds.mds03 flush journal. But this was 
before i changed the beacon_grace so it respawned again at that moment, 
so I'm not quite sure if there was another issue then.


Thanks again!

Kenneth


John


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] mds0: Behind on trimming (58621/30)

[Yan, Zheng]

On Fri, Jul 1, 2016 at 6:59 PM, John Spray  wrote:
> On Fri, Jul 1, 2016 at 11:35 AM, Kenneth Waegeman
>  wrote:
>> Hi all,
>>
>> While syncing a lot of files to cephfs, our mds cluster got haywire: the
>> mdss have a lot of segments behind on trimming:  (58621/30)
>> Because of this the mds cluster gets degraded. RAM usage is about 50GB. The
>> mdses were respawning and replaying continiously, and I had to stop all
>> syncs , unmount all clients and increase the beacon_grace to keep the
>> cluster up .
>>
>> [root@mds03 ~]# ceph status
>> cluster 92bfcf0a-1d39-43b3-b60f-44f01b630e47
>>  health HEALTH_WARN
>> mds0: Behind on trimming (58621/30)
>>  monmap e1: 3 mons at
>> {mds01=10.141.16.1:6789/0,mds02=10.141.16.2:6789/0,mds03=10.141.16.3:6789/0}
>> election epoch 170, quorum 0,1,2 mds01,mds02,mds03
>>   fsmap e78658: 1/1/1 up {0=mds03=up:active}, 2 up:standby
>>  osdmap e19966: 156 osds: 156 up, 156 in
>> flags sortbitwise
>>   pgmap v10213164: 4160 pgs, 4 pools, 253 TB data, 203 Mobjects
>> 357 TB used, 516 TB / 874 TB avail
>> 4151 active+clean
>>5 active+clean+scrubbing
>>4 active+clean+scrubbing+deep
>>   client io 0 B/s rd, 0 B/s wr, 63 op/s rd, 844 op/s wr
>>   cache io 68 op/s promote
>>
>>
>> Now it finally is up again, it is trimming very slowly (+-120 segments /
>> min)
>
> Hmm, so it sounds like something was wrong that got cleared by either
> the MDS restart or the client unmount, and now it's trimming at a
> healthier rate.
>
> What client (kernel or fuse, and version)?
>
> Can you confirm that the RADOS cluster itself was handling operations
> reasonably quickly?  Is your metadata pool using the same drives as
> your data?  Were the OSDs saturated with IO?
>
> While the cluster was accumulating untrimmed segments, did you also
> have a "client xyz failing to advanced oldest_tid" warning?

This does not prevent MDS from trimming log segment.

>
> It would be good to clarify whether the MDS was trimming slowly, or
> not at all.  If you can reproduce this situation, get it to a "behind
> on trimming" state, and the stop the client IO (but leave it mounted).
> See if the (x/30) number stays the same.  Then, does it start to
> decrease when you unmount the client?  That would indicate a
> misbehaving client.

Behind on trimming on single MDS cluster should be caused by either
slow rados operations or MDS trim too few log segments on each tick.

Kenneth, could you try setting mds_log_max_expiring to a large value
(such as 200)

Regards
Yan, Zheng

>
> John
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] performance issue with jewel on ubuntu xenial (kernel)

[Yoann Moulin]

Hello,

>>> I found a performance drop between kernel 3.13.0-88 (default kernel on 
>>> Ubuntu
>>> Trusty 14.04) and kernel 4.4.0.24.14 (default kernel on Ubuntu Xenial 
>>> 16.04)
>>>
>>> ceph version is Jewel (10.2.2).
>>> All tests have been done under Ubuntu 14.04
>>
>> Knowing that you also have an internalis cluster on almost identical
>> hardware, can you please let the list know whether you see the same
>> behavior (severely reduced throughput on a 4.4 kernel, vs. 3.13) on
>> that cluster as well?
>
> ceph version is infernalis (9.2.0)
>
> Ceph osd Benchmark:
>
> Kernel 3.13.0-88-generic : ceph tell osd.ID => average ~84MB/s
> Kernel 4.2.0-38-generic  : ceph tell osd.ID => average ~90MB/s
> Kernel 4.4.0-24-generic  : ceph tell osd.ID => average ~75MB/s
>
> The slow down is not as much as I have with Jewel but it is still present.

 But this is not on precisely identical hardware, is it?
>>>
>>> All the benchmarks were run on strictly identical hardware setups per node.
>>> Clusters differ slightly in sizes (infernalis vs jewel) but nodes and OSDs 
>>> are identical.
>>
>> One thing differ in the osd configuration, on the Jewel cluster, we have 
>> journal
>> on disk, on the Infernalis cluster, we have journal on SSD (S3500)
>>
>> I can restart my test on a Jewel cluster with journal on SSD if needed.
>> I can do as well a test on an Infernalis cluster with journal on disk.
> 
> I'd suggest that the second option is probably more meaningful to test.

I did new benchmarks on 3 clusters. Each cluster has 3 nodes strictly identical.
Each node has 10 OSDs. Journals are on the disk.

bench5 : Ubuntu 14.04 / Ceph Infernalis
bench6 : Ubuntu 14.04 / Ceph Jewel
bench7 : Ubuntu 16.04 / Ceph jewel

this is the average of 2 runs of "ceph tell osd.* bench" on each cluster (2 x 30
OSDs)

bench5 / 14.04 / Infernalis / kernel 3.13 :  54.35 MB/s
bench6 / 14.04 / Jewel  / kernel 3.13 :  86.47 MB/s

bench5 / 14.04 / Infernalis / kernel 4.2  :  63.38 MB/s
bench6 / 14.04 / Jewel  / kernel 4.2  : 107.75 MB/s
bench7 / 16.04 / Jewel  / kernel 4.2  : 101.54 MB/s

bench5 / 14.04 / Infernalis / kernel 4.4  :  53.61 MB/s
bench6 / 14.04 / Jewel  / kernel 4.4  :  65.82 MB/s
bench7 / 16.04 / Jewel  / kernel 4.4  :  61.57 MB/s

If needed, I have the raw output of "ceph tell osd.* bench"

> What I find curious is that no-one else on the list has apparently run
> into this. Any Ubuntu xenial users out there, or perhaps folks on
> trusty who choose to install linux-image-generic-lts-xenial?

Anyone to try on their side if they have the same behaviour ?

Cheers,

-- 
Yoann Moulin
EPFL IC-IT
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CEPH Replication

[Adrien Gillard]

To start safely you need a replication factor of 3 and at least 4 nodes
(think size+1) to allow for smooth maintenance on your nodes.

On Fri, Jul 1, 2016 at 2:31 PM, Ashley Merrick 
wrote:

> Hello,
>
> Okie makes perfect sense.
>
> So if run CEPH with a replication of 3, is it still required to run an odd
> number of OSD Nodes.
>
> Or could I run 4 OSD Nodes to start with, with a replication of 3, with
> each replication on a separate server.
>
> ,Ashley Merrick
>
> -Original Message-
> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of
> Tomasz Kuzemko
> Sent: 01 July 2016 13:28
> To: ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] CEPH Replication
>
> Still in case of object corruption you will not be able to determine which
> copy is valid. Ceph does not provide data integrity with filestore (it's
> planned for bluestore).
>
> On 01.07.2016 14:20, David wrote:
> > It will work but be aware 2x replication is not a good idea if your
> > data is important. The exception would be if the OSD's are DC class
> > SSD's that you monitor closely.
> >
> > On Fri, Jul 1, 2016 at 1:09 PM, Ashley Merrick  > > wrote:
> >
> > Hello,
> >
> > Perfect, I want to keep on separate node's, so wanted to make sure
> > the expected behaviour was that it would do that.
> >
> > And no issues with running an odd number of nodes for a replication
> > of 2? I know you have quorum, just wanted to make sure would not
> > effect when running an even replication.
> >
> > Will be adding nodes in future as require, but will always keep an
> > uneven number.
> >
> > ,Ashley
> >
> > -Original Message-
> > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com
> > ] On Behalf Of
> > c...@jack.fr.eu.org 
> > Sent: 01 July 2016 13:07
> > To: ceph-users@lists.ceph.com 
> > Subject: Re: [ceph-users] CEPH Replication
> >
> > It will put each object on 2 OSD, on 2 separate node All nodes, and
> > all OSDs will have the same used space (approx)
> >
> > If you want to allow both copies of an object to put stored on the
> > same node, you should use osd_crush_chooseleaf_type = 0 (see
> >
> http://docs.ceph.com/docs/master/rados/operations/crush-map/#crush-map-bucket-types
> > and
> >
> > http://docs.ceph.com/docs/hammer/rados/configuration/pool-pg-config-re
> > f/)
> >
> >
> > On 01/07/2016 13:49, Ashley Merrick wrote:
> > > Hello,
> > >
> > > Looking at setting up a new CEPH Cluster, starting with the
> following.
> > >
> > > 3 x CEPH OSD Servers
> > >
> > > Each Server:
> > >
> > > 20Gbps Network
> > > 12 OSD's
> > > SSD Journal
> > >
> > > Looking at running with replication of 2, will there be any issues
> > using 3 nodes with a replication of two, this should "technically"
> > give me ½ the available total capacity of the 3 node's?
> > >
> > > Will the CRUSH map automaticly setup each 12 OSD's as a separate
> > group, so that the two replicated objects are put on separate OSD
> > servers?
> > >
> > > Thanks,
> > > Ashley
> > >
> > >
> > >
> > >
> > > ___
> > > ceph-users mailing list
> > > ceph-users@lists.ceph.com 
> > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > >
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com 
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com 
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
> >
> >
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
>
> --
> Tomasz Kuzemko
> tomasz.kuze...@corp.ovh.com
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CEPH Replication

[Ashley Merrick]

Hello,

Okie makes perfect sense.

So if run CEPH with a replication of 3, is it still required to run an odd 
number of OSD Nodes.

Or could I run 4 OSD Nodes to start with, with a replication of 3, with each 
replication on a separate server.

,Ashley Merrick

-Original Message-
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Tomasz 
Kuzemko
Sent: 01 July 2016 13:28
To: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] CEPH Replication

Still in case of object corruption you will not be able to determine which copy 
is valid. Ceph does not provide data integrity with filestore (it's planned for 
bluestore).

On 01.07.2016 14:20, David wrote:
> It will work but be aware 2x replication is not a good idea if your 
> data is important. The exception would be if the OSD's are DC class 
> SSD's that you monitor closely.
> 
> On Fri, Jul 1, 2016 at 1:09 PM, Ashley Merrick  > wrote:
> 
> Hello,
> 
> Perfect, I want to keep on separate node's, so wanted to make sure
> the expected behaviour was that it would do that.
> 
> And no issues with running an odd number of nodes for a replication
> of 2? I know you have quorum, just wanted to make sure would not
> effect when running an even replication.
> 
> Will be adding nodes in future as require, but will always keep an
> uneven number.
> 
> ,Ashley
> 
> -Original Message-
> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com
> ] On Behalf Of
> c...@jack.fr.eu.org 
> Sent: 01 July 2016 13:07
> To: ceph-users@lists.ceph.com 
> Subject: Re: [ceph-users] CEPH Replication
> 
> It will put each object on 2 OSD, on 2 separate node All nodes, and
> all OSDs will have the same used space (approx)
> 
> If you want to allow both copies of an object to put stored on the
> same node, you should use osd_crush_chooseleaf_type = 0 (see
> 
> http://docs.ceph.com/docs/master/rados/operations/crush-map/#crush-map-bucket-types
> and
> 
> http://docs.ceph.com/docs/hammer/rados/configuration/pool-pg-config-re
> f/)
> 
> 
> On 01/07/2016 13:49, Ashley Merrick wrote:
> > Hello,
> >
> > Looking at setting up a new CEPH Cluster, starting with the following.
> >
> > 3 x CEPH OSD Servers
> >
> > Each Server:
> >
> > 20Gbps Network
> > 12 OSD's
> > SSD Journal
> >
> > Looking at running with replication of 2, will there be any issues
> using 3 nodes with a replication of two, this should "technically"
> give me ½ the available total capacity of the 3 node's?
> >
> > Will the CRUSH map automaticly setup each 12 OSD's as a separate
> group, so that the two replicated objects are put on separate OSD
> servers?
> >
> > Thanks,
> > Ashley
> >
> >
> >
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com 
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com 
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com 
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> 
> 
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 

--
Tomasz Kuzemko
tomasz.kuze...@corp.ovh.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CEPH Replication

[Tomasz Kuzemko]

Still in case of object corruption you will not be able to determine
which copy is valid. Ceph does not provide data integrity with filestore
(it's planned for bluestore).

On 01.07.2016 14:20, David wrote:
> It will work but be aware 2x replication is not a good idea if your data
> is important. The exception would be if the OSD's are DC class SSD's
> that you monitor closely.
> 
> On Fri, Jul 1, 2016 at 1:09 PM, Ashley Merrick  > wrote:
> 
> Hello,
> 
> Perfect, I want to keep on separate node's, so wanted to make sure
> the expected behaviour was that it would do that.
> 
> And no issues with running an odd number of nodes for a replication
> of 2? I know you have quorum, just wanted to make sure would not
> effect when running an even replication.
> 
> Will be adding nodes in future as require, but will always keep an
> uneven number.
> 
> ,Ashley
> 
> -Original Message-
> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com
> ] On Behalf Of
> c...@jack.fr.eu.org 
> Sent: 01 July 2016 13:07
> To: ceph-users@lists.ceph.com 
> Subject: Re: [ceph-users] CEPH Replication
> 
> It will put each object on 2 OSD, on 2 separate node All nodes, and
> all OSDs will have the same used space (approx)
> 
> If you want to allow both copies of an object to put stored on the
> same node, you should use osd_crush_chooseleaf_type = 0 (see
> 
> http://docs.ceph.com/docs/master/rados/operations/crush-map/#crush-map-bucket-types
> and
> http://docs.ceph.com/docs/hammer/rados/configuration/pool-pg-config-ref/)
> 
> 
> On 01/07/2016 13:49, Ashley Merrick wrote:
> > Hello,
> >
> > Looking at setting up a new CEPH Cluster, starting with the following.
> >
> > 3 x CEPH OSD Servers
> >
> > Each Server:
> >
> > 20Gbps Network
> > 12 OSD's
> > SSD Journal
> >
> > Looking at running with replication of 2, will there be any issues
> using 3 nodes with a replication of two, this should "technically"
> give me ½ the available total capacity of the 3 node's?
> >
> > Will the CRUSH map automaticly setup each 12 OSD's as a separate
> group, so that the two replicated objects are put on separate OSD
> servers?
> >
> > Thanks,
> > Ashley
> >
> >
> >
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com 
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com 
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com 
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> 
> 
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 

-- 
Tomasz Kuzemko
tomasz.kuze...@corp.ovh.com



signature.asc
Description: OpenPGP digital signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CEPH Replication

[David]

It will work but be aware 2x replication is not a good idea if your data is
important. The exception would be if the OSD's are DC class SSD's that you
monitor closely.

On Fri, Jul 1, 2016 at 1:09 PM, Ashley Merrick 
wrote:

> Hello,
>
> Perfect, I want to keep on separate node's, so wanted to make sure the
> expected behaviour was that it would do that.
>
> And no issues with running an odd number of nodes for a replication of 2?
> I know you have quorum, just wanted to make sure would not effect when
> running an even replication.
>
> Will be adding nodes in future as require, but will always keep an uneven
> number.
>
> ,Ashley
>
> -Original Message-
> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of
> c...@jack.fr.eu.org
> Sent: 01 July 2016 13:07
> To: ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] CEPH Replication
>
> It will put each object on 2 OSD, on 2 separate node All nodes, and all
> OSDs will have the same used space (approx)
>
> If you want to allow both copies of an object to put stored on the same
> node, you should use osd_crush_chooseleaf_type = 0 (see
> http://docs.ceph.com/docs/master/rados/operations/crush-map/#crush-map-bucket-types
> and
> http://docs.ceph.com/docs/hammer/rados/configuration/pool-pg-config-ref/)
>
>
> On 01/07/2016 13:49, Ashley Merrick wrote:
> > Hello,
> >
> > Looking at setting up a new CEPH Cluster, starting with the following.
> >
> > 3 x CEPH OSD Servers
> >
> > Each Server:
> >
> > 20Gbps Network
> > 12 OSD's
> > SSD Journal
> >
> > Looking at running with replication of 2, will there be any issues using
> 3 nodes with a replication of two, this should "technically" give me ½ the
> available total capacity of the 3 node's?
> >
> > Will the CRUSH map automaticly setup each 12 OSD's as a separate group,
> so that the two replicated objects are put on separate OSD servers?
> >
> > Thanks,
> > Ashley
> >
> >
> >
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CEPH Replication

[Ashley Merrick]

Hello,

Perfect, I want to keep on separate node's, so wanted to make sure the expected 
behaviour was that it would do that.

And no issues with running an odd number of nodes for a replication of 2? I 
know you have quorum, just wanted to make sure would not effect when running an 
even replication.

Will be adding nodes in future as require, but will always keep an uneven 
number.

,Ashley

-Original Message-
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of 
c...@jack.fr.eu.org
Sent: 01 July 2016 13:07
To: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] CEPH Replication

It will put each object on 2 OSD, on 2 separate node All nodes, and all OSDs 
will have the same used space (approx)

If you want to allow both copies of an object to put stored on the same node, 
you should use osd_crush_chooseleaf_type = 0 (see 
http://docs.ceph.com/docs/master/rados/operations/crush-map/#crush-map-bucket-types
and
http://docs.ceph.com/docs/hammer/rados/configuration/pool-pg-config-ref/)


On 01/07/2016 13:49, Ashley Merrick wrote:
> Hello,
> 
> Looking at setting up a new CEPH Cluster, starting with the following.
> 
> 3 x CEPH OSD Servers
> 
> Each Server:
> 
> 20Gbps Network
> 12 OSD's
> SSD Journal
> 
> Looking at running with replication of 2, will there be any issues using 3 
> nodes with a replication of two, this should "technically" give me ½ the 
> available total capacity of the 3 node's?
> 
> Will the CRUSH map automaticly setup each 12 OSD's as a separate group, so 
> that the two replicated objects are put on separate OSD servers?
> 
> Thanks,
> Ashley
> 
> 
> 
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CEPH Replication

[ceph]

It will put each object on 2 OSD, on 2 separate node
All nodes, and all OSDs will have the same used space (approx)

If you want to allow both copies of an object to put stored on the same
node, you should use osd_crush_chooseleaf_type = 0 (see
http://docs.ceph.com/docs/master/rados/operations/crush-map/#crush-map-bucket-types
and
http://docs.ceph.com/docs/hammer/rados/configuration/pool-pg-config-ref/)


On 01/07/2016 13:49, Ashley Merrick wrote:
> Hello,
> 
> Looking at setting up a new CEPH Cluster, starting with the following.
> 
> 3 x CEPH OSD Servers
> 
> Each Server:
> 
> 20Gbps Network
> 12 OSD's
> SSD Journal
> 
> Looking at running with replication of 2, will there be any issues using 3 
> nodes with a replication of two, this should "technically" give me ½ the 
> available total capacity of the 3 node's?
> 
> Will the CRUSH map automaticly setup each 12 OSD's as a separate group, so 
> that the two replicated objects are put on separate OSD servers?
> 
> Thanks,
> Ashley
> 
> 
> 
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] CEPH Replication

[Ashley Merrick]

Hello,

Looking at setting up a new CEPH Cluster, starting with the following.

3 x CEPH OSD Servers

Each Server:

20Gbps Network
12 OSD's
SSD Journal

Looking at running with replication of 2, will there be any issues using 3 
nodes with a replication of two, this should "technically" give me ½ the 
available total capacity of the 3 node's?

Will the CRUSH map automaticly setup each 12 OSD's as a separate group, so that 
the two replicated objects are put on separate OSD servers?

Thanks,
Ashley

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Can't create bucket (ERROR: endpoints not configured for upstream zone)

[Micha Krause]

Hi,

> In Infernalis there was this command:


radosgw-admin regions list

But this is missing in Jewel.


Ok, I just found out that this was renamed to zonegroup list:

root@rgw01:~ # radosgw-admin --id radosgw.rgw zonegroup list
read_default_id : -2
{
"default_info": "",
"zonegroups": [
"default"
]
}

This looks to me like there is indeed only one zonegroup or region configured.

Micha Krause
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] mds0: Behind on trimming (58621/30)

[John Spray]

On Fri, Jul 1, 2016 at 11:35 AM, Kenneth Waegeman
 wrote:
> Hi all,
>
> While syncing a lot of files to cephfs, our mds cluster got haywire: the
> mdss have a lot of segments behind on trimming:  (58621/30)
> Because of this the mds cluster gets degraded. RAM usage is about 50GB. The
> mdses were respawning and replaying continiously, and I had to stop all
> syncs , unmount all clients and increase the beacon_grace to keep the
> cluster up .
>
> [root@mds03 ~]# ceph status
> cluster 92bfcf0a-1d39-43b3-b60f-44f01b630e47
>  health HEALTH_WARN
> mds0: Behind on trimming (58621/30)
>  monmap e1: 3 mons at
> {mds01=10.141.16.1:6789/0,mds02=10.141.16.2:6789/0,mds03=10.141.16.3:6789/0}
> election epoch 170, quorum 0,1,2 mds01,mds02,mds03
>   fsmap e78658: 1/1/1 up {0=mds03=up:active}, 2 up:standby
>  osdmap e19966: 156 osds: 156 up, 156 in
> flags sortbitwise
>   pgmap v10213164: 4160 pgs, 4 pools, 253 TB data, 203 Mobjects
> 357 TB used, 516 TB / 874 TB avail
> 4151 active+clean
>5 active+clean+scrubbing
>4 active+clean+scrubbing+deep
>   client io 0 B/s rd, 0 B/s wr, 63 op/s rd, 844 op/s wr
>   cache io 68 op/s promote
>
>
> Now it finally is up again, it is trimming very slowly (+-120 segments /
> min)

Hmm, so it sounds like something was wrong that got cleared by either
the MDS restart or the client unmount, and now it's trimming at a
healthier rate.

What client (kernel or fuse, and version)?

Can you confirm that the RADOS cluster itself was handling operations
reasonably quickly?  Is your metadata pool using the same drives as
your data?  Were the OSDs saturated with IO?

While the cluster was accumulating untrimmed segments, did you also
have a "client xyz failing to advanced oldest_tid" warning?

It would be good to clarify whether the MDS was trimming slowly, or
not at all.  If you can reproduce this situation, get it to a "behind
on trimming" state, and the stop the client IO (but leave it mounted).
See if the (x/30) number stays the same.  Then, does it start to
decrease when you unmount the client?  That would indicate a
misbehaving client.

John
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] mds0: Behind on trimming (58621/30)

[Kenneth Waegeman]

Hi all,

While syncing a lot of files to cephfs, our mds cluster got haywire: the 
mdss have a lot of segments behind on trimming:  (58621/30)
Because of this the mds cluster gets degraded. RAM usage is about 50GB. 
The mdses were respawning and replaying continiously, and I had to stop 
all syncs , unmount all clients and increase the beacon_grace to keep 
the cluster up .


[root@mds03 ~]# ceph status
cluster 92bfcf0a-1d39-43b3-b60f-44f01b630e47
 health HEALTH_WARN
mds0: Behind on trimming (58621/30)
 monmap e1: 3 mons at 
{mds01=10.141.16.1:6789/0,mds02=10.141.16.2:6789/0,mds03=10.141.16.3:6789/0}

election epoch 170, quorum 0,1,2 mds01,mds02,mds03
  fsmap e78658: 1/1/1 up {0=mds03=up:active}, 2 up:standby
 osdmap e19966: 156 osds: 156 up, 156 in
flags sortbitwise
  pgmap v10213164: 4160 pgs, 4 pools, 253 TB data, 203 Mobjects
357 TB used, 516 TB / 874 TB avail
4151 active+clean
   5 active+clean+scrubbing
   4 active+clean+scrubbing+deep
  client io 0 B/s rd, 0 B/s wr, 63 op/s rd, 844 op/s wr
  cache io 68 op/s promote


Now it finally is up again, it is trimming very slowly (+-120 segments / 
min)

We've seen some 'behind on trimming' before, but never that much..
So now our production cluster is unusable for approx half a day..

What could be the problem here? We are running 10.2.1
Can something be done to not let the mds keep that much segments ?
Can we fasten the trimming process?

Thanks you very much!

Cheers,
Kenneth
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] suse_enterprise_storage3_rbd_LIO_vmware_performance_bad

[mq]

HI  
1.
2 sw  iscsi gateways(deploy on osd/monitor ) using lrbd to create,the iscsi 
target is LIO 
configuration:
{
  "auth": [
{
  "target": "iqn.2016-07.org.linux-iscsi.iscsi.x86:testvol", 
  "authentication": "none"
}
  ], 
  "targets": [
{
  "target": "iqn.2016-07.org.linux-iscsi.iscsi.x86:testvol", 
  "hosts": [
{
  "host": "node2", 
  "portal": "east"
}, 
{
  "host": "node3", 
  "portal": "west"
}
  ]
}
  ], 
  "portals": [
{
  "name": "east", 
  "addresses": [
"10.0.52.92"
  ]
}, 
{
  "name": "west", 
  "addresses": [
"10.0.52.93"
  ]
}
  ], 
  "pools": [
{
  "pool": "rbd", 
  "gateways": [
{
  "target": "iqn.2016-07.org.linux-iscsi.iscsi.x86:testvol", 
  "tpg": [
{
  "image": "testvol"
}
  ]
}
  ]
}
  ]
}

2 the ceph cluster itself’s performance is ok. i create a rbd on one of ceph 
node. fio results is nice: 4K randwrite IOPS=3013 bw=100MB/s.
so i think the ceph cluster have no bottleneck.

3  Intel S3510 SSD 480G enterprise not consumer

new test :clone a VM in wmware can reach 100MB/s. but fio and dd test in vm 
still poor.


> 在 2016年7月1日，下午4:18，Christian Balzer  写道：
> 
> 
> Hello,
> 
> On Fri, 1 Jul 2016 13:04:45 +0800 mq wrote:
> 
>> Hi list
>> I have tested suse enterprise storage3 using 2 iscsi  gateway attached
>> to  vmware. The performance is bad.  
> 
> First off, it's somewhat funny that you're testing the repackaged SUSE
> Ceph, but asking for help here (with Ceph being owned by Red Hat).
> 
> Aside from that, you're not telling us what these 2 iSCSI gateways are
> (SW, HW specs/configuration).
> 
> Having iSCSI on top of Ceph is by the very nature of things going to be
> slower than native Ceph.
> 
> Use "rbd bench" or a VM client with RBD to get a base number of what your
> Ceph cluster is capable of, this will help identifying where the slowdown
> is.
> 
>> I have turn off  VAAI following the
>> (https://kb.vmware.com/selfservice/microsites/search.do?language=en_US=displayKC=1033665
>>  
>> )
>> >  
>> )>.
>> My cluster 3 ceph nodes :2*E5-2620 64G , mem 2*1Gbps (3*10K SAS, 1*480G
>> SSD) per node, SSD as journal 1 vmware node  2*E5-2620 64G , mem 2*1Gbps 
> 
> That's a slow (latency wise) network, but not your problem.
> What SSD model? 
> A 480GB size suggests a consumer model and that would explain a lot.
> 
> Check you storage nodes with atop during the fio runs and see if you can
> spot a bottleneck.
> 
> Christian
> 
>> # ceph -s
>>cluster 0199f68d-a745-4da3-9670-15f2981e7a15
>> health HEALTH_OK
>> monmap e1: 3 mons at
>> {node1=192.168.50.91:6789/0,node2=192.168.50.92:6789/0,node3=192.168.50.93:6789/0}
>> election epoch 22, quorum 0,1,2 node1,node2,node3 osdmap e200: 9 osds: 9
>> up, 9 in flags sortbitwise
>>  pgmap v1162: 448 pgs, 1 pools, 14337 MB data, 4935 objects
>>18339 MB used, 5005 GB / 5023 GB avail
>> 448 active+clean
>>  client io 87438 kB/s wr, 0 op/s rd, 213 op/s wr
>> 
>> sudo ceph osd tree
>> ID WEIGHT  TYPE NAME  UP/DOWN REWEIGHT PRIMARY-AFFINITY
>> -1 4.90581 root default
>> -2 1.63527 host node1
>> 0 0.54509 osd.0   up  1.0  1.0
>> 1 0.54509 osd.1   up  1.0  1.0
>> 2 0.54509 osd.2   up  1.0  1.0
>> -3 1.63527 host node2
>> 3 0.54509 osd.3   up  1.0  1.0
>> 4 0.54509 osd.4   up  1.0  1.0
>> 5 0.54509 osd.5   up  1.0  1.0
>> -4 1.63527 host node3
>> 6 0.54509 osd.6   up  1.0  1.0
>> 7 0.54509 osd.7   up  1.0  1.0
>> 8 0.54509 osd.8   up  1.0  1.0
>> 
>> 
>> 
>> An linux vm in vmmare， running fio.  4k randwrite result just 64 IOPS
>> lantency is high，dd test just 11MB／s. 
>> fio -ioengine=libaio -bs=4k -direct=1 -thread -rw=randwrite -size=100G
>> -filename=/dev/sdb  -name="EBS 4KB randwrite test" -iodepth=32
>> -runtime=60 EBS 4KB randwrite test: (g=0): rw=randwrite,
>> bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32 fio-2.0.13 Starting 1
>> thread Jobs: 1 (f=1): [w] [100.0% done] [0K/131K/0K /s] [0 /32 /0  iops]
>> [eta 00m:00s] EBS 4KB randwrite test: (groupid=0, jobs=1): err= 0:
>> pid=6766: Wed Jun 29 21:28:06 2016 write: io=15696KB, bw=264627 B/s,
>> iops=64 , runt= 60737msec slat (usec): min=10 , max=213 , avg=35.54,
>> stdev=16.41 clat (msec): min=1 , max=31368 , avg=495.01, stdev=1862.52
>> lat (msec): min=2 , 

[ceph-users] confused by ceph quick install and manual install

[Chengwei Yang]

Hi List,

Sorry if this question was answered before.

I'm new to ceph and following the ceph document to setting up a ceph cluster.
However, I noticed that the manual install guide said below

http://docs.ceph.com/docs/master/install/install-storage-cluster/

> Ensure your YUM ceph.repo entry includes priority=2. See Get Packages for
> details:

But the /etc/yum.repos.d/ceph.repo created by ceph-deploy(followed quick install
quide) contains **priority=1**.

So I'm now quite confused, which one is correct?

-- 
Thanks,
Chengwei


signature.asc
Description: Digital signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Can't create bucket (ERROR: endpoints not configured for upstream zone)

[Micha Krause]

Hi,

> See  this thread,

https://www.mail-archive.com/ceph-users@lists.ceph.com/msg23852.html


Yes, I found this as well, but I don't think I have configured more than
one region.
I never touched any region settings, and I have to admit I wouldn't
even know how to check which regions I have.

In Infernalis there was this command:

radosgw-admin regions list

But this is missing in Jewel.



Micha Krause
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] suse_enterprise_storage3_rbd_LIO_vmware_performance_bad

[Oliver Dzombic]

Hi,

my experience:

ceph + iscsi ( multipath ) + vmware == worst

Better you search for another solution.

vmware + nfs + vmware might have a much better performance.



If you are able to get vmware run with iscsi and ceph, i would be
>>very<< intrested in what/how you did that.

-- 
Mit freundlichen Gruessen / Best regards

Oliver Dzombic
IP-Interactive

mailto:i...@ip-interactive.de

Anschrift:

IP Interactive UG ( haftungsbeschraenkt )
Zum Sonnenberg 1-3
63571 Gelnhausen

HRB 93402 beim Amtsgericht Hanau
Geschäftsführung: Oliver Dzombic

Steuer Nr.: 35 236 3622 1
UST ID: DE274086107


Am 01.07.2016 um 07:04 schrieb mq:
> Hi list
> I have tested suse enterprise storage3 using 2 iscsi  gateway attached
> to  vmware. The performance is bad.  I have turn off  VAAI following the
> (https://kb.vmware.com/selfservice/microsites/search.do?language=en_US=displayKC=1033665)
> .
>   
> My cluster
> 3 ceph nodes :2*E5-2620 64G , mem 2*1Gbps
> (3*10K SAS, 1*480G  SSD) per node, SSD as journal
> 1 vmware node  2*E5-2620 64G , mem 2*1Gbps
>  
> # ceph -s
> cluster 0199f68d-a745-4da3-9670-15f2981e7a15
>  health HEALTH_OK
>  monmap e1: 3 mons at
> {node1=192.168.50.91:6789/0,node2=192.168.50.92:6789/0,node3=192.168.50.93:6789/0}
> election epoch 22, quorum 0,1,2 node1,node2,node3
>  osdmap e200: 9 osds: 9 up, 9 in
> flags sortbitwise
>   pgmap v1162: 448 pgs, 1 pools, 14337 MB data, 4935 objects
> 18339 MB used, 5005 GB / 5023 GB avail
>  448 active+clean
>   client io 87438 kB/s wr, 0 op/s rd, 213 op/s wr
>  
> sudo ceph osd tree
> ID WEIGHT  TYPE NAME  UP/DOWN REWEIGHT PRIMARY-AFFINITY
> -1 4.90581 root default
> -2 1.63527 host node1
> 0 0.54509 osd.0   up  1.0  1.0
> 1 0.54509 osd.1   up  1.0  1.0
> 2 0.54509 osd.2   up  1.0  1.0
> -3 1.63527 host node2
> 3 0.54509 osd.3   up  1.0  1.0
> 4 0.54509 osd.4   up  1.0  1.0
> 5 0.54509 osd.5   up  1.0  1.0
> -4 1.63527 host node3
> 6 0.54509 osd.6   up  1.0  1.0
> 7 0.54509 osd.7   up  1.0  1.0
> 8 0.54509 osd.8   up  1.0  1.0
>  
>  
>  
> An linux vm in vmmare， running fio.  4k randwrite result just 64 IOPS
> lantency is high，dd test just 11MB／s.
>  
> fio -ioengine=libaio -bs=4k -direct=1 -thread -rw=randwrite -size=100G
> -filename=/dev/sdb  -name="EBS 4KB randwrite test" -iodepth=32 -runtime=60
> EBS 4KB randwrite test: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K,
> ioengine=libaio, iodepth=32
> fio-2.0.13
> Starting 1 thread
> Jobs: 1 (f=1): [w] [100.0% done] [0K/131K/0K /s] [0 /32 /0  iops] [eta
> 00m:00s]
> EBS 4KB randwrite test: (groupid=0, jobs=1): err= 0: pid=6766: Wed Jun
> 29 21:28:06 2016
>   write: io=15696KB, bw=264627 B/s, iops=64 , runt= 60737msec
> slat (usec): min=10 , max=213 , avg=35.54, stdev=16.41
> clat (msec): min=1 , max=31368 , avg=495.01, stdev=1862.52
>  lat (msec): min=2 , max=31368 , avg=495.04, stdev=1862.52
> clat percentiles (msec):
>  |  1.00th=[7],  5.00th=[8], 10.00th=[8], 20.00th=[9],
>  | 30.00th=[9], 40.00th=[   10], 50.00th=[  198], 60.00th=[  204],
>  | 70.00th=[  208], 80.00th=[  217], 90.00th=[  799], 95.00th=[ 1795],
>  | 99.00th=[ 7177], 99.50th=[12649], 99.90th=[16712], 99.95th=[16712],
>  | 99.99th=[16712]
> bw (KB/s)  : min=   36, max=11960, per=100.00%, avg=264.77,
> stdev=1110.81
> lat (msec) : 2=0.03%, 4=0.23%, 10=40.93%, 20=0.48%, 50=0.03%
> lat (msec) : 100=0.08%, 250=39.55%, 500=5.63%, 750=2.91%, 1000=1.35%
> lat (msec) : 2000=4.03%, >=2000=4.77%
>   cpu  : usr=0.02%, sys=0.22%, ctx=2973, majf=0,
> minf=18446744073709538907
>   IO depths: 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=0.4%, 32=99.2%,
>>=64=0.0%
>  submit: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>=64=0.0%
>  complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>=64=0.0%
>  issued: total=r=0/w=3924/d=0, short=r=0/w=0/d=0
>  
> Run status group 0 (all jobs):
>   WRITE: io=15696KB, aggrb=258KB/s, minb=258KB/s, maxb=258KB/s,
> mint=60737msec, maxt=60737msec
>  
> Disk stats (read/write):
>   sdb: ios=83/3921, merge=0/0, ticks=60/1903085, in_queue=1931694,
> util=100.00%
> 
> anyone can give me some suggestion to improve the performance ?
> 
> Regards
> 
> MQ
> 
> 
> 
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] suse_enterprise_storage3_rbd_LIO_vmware_performance_bad

[Christian Balzer]

Hello,

On Fri, 1 Jul 2016 13:04:45 +0800 mq wrote:

> Hi list
> I have tested suse enterprise storage3 using 2 iscsi  gateway attached
> to  vmware. The performance is bad.  

First off, it's somewhat funny that you're testing the repackaged SUSE
Ceph, but asking for help here (with Ceph being owned by Red Hat).

Aside from that, you're not telling us what these 2 iSCSI gateways are
(SW, HW specs/configuration).

Having iSCSI on top of Ceph is by the very nature of things going to be
slower than native Ceph.

Use "rbd bench" or a VM client with RBD to get a base number of what your
Ceph cluster is capable of, this will help identifying where the slowdown
is.

>I have turn off  VAAI following the
> (https://kb.vmware.com/selfservice/microsites/search.do?language=en_US=displayKC=1033665)
> .
> My cluster 3 ceph nodes :2*E5-2620 64G , mem 2*1Gbps (3*10K SAS, 1*480G
> SSD) per node, SSD as journal 1 vmware node  2*E5-2620 64G , mem 2*1Gbps 

That's a slow (latency wise) network, but not your problem.
What SSD model? 
A 480GB size suggests a consumer model and that would explain a lot.

Check you storage nodes with atop during the fio runs and see if you can
spot a bottleneck.

Christian

> # ceph -s
> cluster 0199f68d-a745-4da3-9670-15f2981e7a15
>  health HEALTH_OK
>  monmap e1: 3 mons at
> {node1=192.168.50.91:6789/0,node2=192.168.50.92:6789/0,node3=192.168.50.93:6789/0}
> election epoch 22, quorum 0,1,2 node1,node2,node3 osdmap e200: 9 osds: 9
> up, 9 in flags sortbitwise
>   pgmap v1162: 448 pgs, 1 pools, 14337 MB data, 4935 objects
> 18339 MB used, 5005 GB / 5023 GB avail
>  448 active+clean
>   client io 87438 kB/s wr, 0 op/s rd, 213 op/s wr
>  
> sudo ceph osd tree
> ID WEIGHT  TYPE NAME  UP/DOWN REWEIGHT PRIMARY-AFFINITY
> -1 4.90581 root default
> -2 1.63527 host node1
> 0 0.54509 osd.0   up  1.0  1.0
> 1 0.54509 osd.1   up  1.0  1.0
> 2 0.54509 osd.2   up  1.0  1.0
> -3 1.63527 host node2
> 3 0.54509 osd.3   up  1.0  1.0
> 4 0.54509 osd.4   up  1.0  1.0
> 5 0.54509 osd.5   up  1.0  1.0
> -4 1.63527 host node3
> 6 0.54509 osd.6   up  1.0  1.0
> 7 0.54509 osd.7   up  1.0  1.0
> 8 0.54509 osd.8   up  1.0  1.0
>  
>  
>  
> An linux vm in vmmare， running fio.  4k randwrite result just 64 IOPS
> lantency is high，dd test just 11MB／s. 
> fio -ioengine=libaio -bs=4k -direct=1 -thread -rw=randwrite -size=100G
> -filename=/dev/sdb  -name="EBS 4KB randwrite test" -iodepth=32
> -runtime=60 EBS 4KB randwrite test: (g=0): rw=randwrite,
> bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32 fio-2.0.13 Starting 1
> thread Jobs: 1 (f=1): [w] [100.0% done] [0K/131K/0K /s] [0 /32 /0  iops]
> [eta 00m:00s] EBS 4KB randwrite test: (groupid=0, jobs=1): err= 0:
> pid=6766: Wed Jun 29 21:28:06 2016 write: io=15696KB, bw=264627 B/s,
> iops=64 , runt= 60737msec slat (usec): min=10 , max=213 , avg=35.54,
> stdev=16.41 clat (msec): min=1 , max=31368 , avg=495.01, stdev=1862.52
>  lat (msec): min=2 , max=31368 , avg=495.04, stdev=1862.52
> clat percentiles (msec):
>  |  1.00th=[7],  5.00th=[8], 10.00th=[8],
> 20.00th=[9], | 30.00th=[9], 40.00th=[   10], 50.00th=[  198],
> 60.00th=[  204], | 70.00th=[  208], 80.00th=[  217], 90.00th=[  799],
> 95.00th=[ 1795], | 99.00th=[ 7177], 99.50th=[12649], 99.90th=[16712],
> 99.95th=[16712], | 99.99th=[16712]
> bw (KB/s)  : min=   36, max=11960, per=100.00%, avg=264.77,
> stdev=1110.81 lat (msec) : 2=0.03%, 4=0.23%, 10=40.93%, 20=0.48%,
> 50=0.03% lat (msec) : 100=0.08%, 250=39.55%, 500=5.63%, 750=2.91%,
> 1000=1.35% lat (msec) : 2000=4.03%, >=2000=4.77%
>   cpu  : usr=0.02%, sys=0.22%, ctx=2973, majf=0,
> minf=18446744073709538907 IO depths: 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%,
> 16=0.4%, 32=99.2%, >=64=0.0% submit: 0=0.0%, 4=100.0%, 8=0.0%,
> 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete  : 0=0.0%, 4=100.0%,
> 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% issued:
> total=r=0/w=3924/d=0, short=r=0/w=0/d=0 
> Run status group 0 (all jobs):
>   WRITE: io=15696KB, aggrb=258KB/s, minb=258KB/s, maxb=258KB/s,
> mint=60737msec, maxt=60737msec 
> Disk stats (read/write):
>   sdb: ios=83/3921, merge=0/0, ticks=60/1903085, in_queue=1931694,
> util=100.00%
> 
> anyone can give me some suggestion to improve the performance ?
> 
> Regards
> 
> MQ
> 
> 


-- 
Christian BalzerNetwork/Systems Engineer
ch...@gol.com   Global OnLine Japan/Rakuten Communications
http://www.gol.com/
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Is anyone seeing iissues with task_numa_find_cpu?

[Christoph Adomeit]

Hi,

is there meanwhile a proven solution to this issue ?

What can be done do fix the scheduler bug ? 1 Patch, 3 Patches, 20 Patches ?

Thanks
  Christoph

On Wed, Jun 29, 2016 at 12:02:11PM +0200, Stefan Priebe - Profihost AG wrote:
> Hi,
> 
> to be precise i've far more patches attached to the sched part (around
> 20) of the kernel. So maybe that's the reason why it helps to me.
> 
> Could you please post a complete stack trace? Also Qemu / KVM triggers this.
> 
> Stefan
> 
> Am 29.06.2016 um 11:41 schrieb Campbell Steven:
> > Hi Alex/Stefan,
> > 
> > I'm in the middle of testing 4.7rc5 on our test cluster to confirm
> > once and for all this particular issue has been completely resolved by
> > Peter's recent patch to sched/fair.c refereed to by Stefan above. For
> > us anyway the patches that Stefan applied did not solve the issue and
> > neither did any 4.5.x or 4.6.x released kernel thus far, hopefully it
> > does the trick for you. We could get about 4 hours uptime before
> > things went haywire for us.
> > 
> > It's interesting how it seems the CEPH workload triggers this bug so
> > well as it's quite a long standing issue that's only just been
> > resolved, another user chimed in on the lkml thread a couple of days
> > ago as well and again his trace had ceph-osd in it as well.
> > 
> > https://lkml.org/lkml/headers/2016/6/21/491
> > 
> > Campbell
> > 
> > On 29 June 2016 at 18:29, Stefan Priebe - Profihost AG
> >  wrote:
> >>
> >> Am 29.06.2016 um 04:30 schrieb Alex Gorbachev:
> >>> Hi Stefan,
> >>>
> >>> On Tue, Jun 28, 2016 at 1:46 PM, Stefan Priebe - Profihost AG
> >>>  wrote:
>  Please be aware that you may need even more patches. Overall this needs 3
>  patches. Where the first two try to fix a bug and the 3rd one fixes the
>  fixes + even more bugs related to the scheduler. I've no idea on which 
>  patch
>  level Ubuntu is.
> >>>
> >>> Stefan, would you be able to please point to the other two patches
> >>> beside https://lkml.org/lkml/diff/2016/6/22/102/1 ?
> >>
> >> Sorry sure yes:
> >>
> >> 1. 2b8c41daba32 ("sched/fair: Initiate a new task's util avg to a
> >> bounded value")
> >>
> >> 2.) 40ed9cba24bb7e01cc380a02d3f04065b8afae1d ("sched/fair: Fix
> >> post_init_entity_util_avg() serialization")
> >>
> >> 3.) the one listed at lkml.
> >>
> >> Stefan
> >>
> >>>
> >>> Thank you,
> >>> Alex
> >>>
> 
>  Stefan
> 
>  Excuse my typo sent from my mobile phone.
> 
>  Am 28.06.2016 um 17:59 schrieb Tim Bishop :
> 
>  Yes - I noticed this today on Ubuntu 16.04 with the default kernel. No
>  useful information to add other than it's not just you.
> 
>  Tim.
> 
>  On Tue, Jun 28, 2016 at 11:05:40AM -0400, Alex Gorbachev wrote:
> 
>  After upgrading to kernel 4.4.13 on Ubuntu, we are seeing a few of
> 
>  these issues where an OSD would fail with the stack below.  I logged a
> 
>  bug at https://bugzilla.kernel.org/show_bug.cgi?id=121101 and there is
> 
>  a similar description at https://lkml.org/lkml/2016/6/22/102, but the
> 
>  odd part is we have turned off CFQ and blk-mq/scsi-mq and are using
> 
>  just the noop scheduler.
> 
> 
>  Does the ceph kernel code somehow use the fair scheduler code block?
> 
> 
>  Thanks
> 
>  --
> 
>  Alex Gorbachev
> 
>  Storcium
> 
> 
>  Jun 28 09:46:41 roc04r-sca090 kernel: [137912.684974] CPU: 30 PID:
> 
>  10403 Comm: ceph-osd Not tainted 4.4.13-040413-generic #201606072354
> 
>  Jun 28 09:46:41 roc04r-sca090 kernel: [137912.684991] Hardware name:
> 
>  Supermicro X9DRi-LN4+/X9DR3-LN4+/X9DRi-LN4+/X9DR3-LN4+, BIOS 3.2
> 
>  03/04/2015
> 
>  Jun 28 09:46:41 roc04r-sca090 kernel: [137912.685009] task:
> 
>  880f79df8000 ti: 880f79fb8000 task.ti: 880f79fb8000
> 
>  Jun 28 09:46:41 roc04r-sca090 kernel: [137912.685024] RIP:
> 
>  0010:[]  []
> 
>  task_numa_find_cpu+0x22e/0x6f0
> 
>  Jun 28 09:46:41 roc04r-sca090 kernel: [137912.685051] RSP:
> 
>  0018:880f79fbb818  EFLAGS: 00010206
> 
>  Jun 28 09:46:41 roc04r-sca090 kernel: [137912.685063] RAX:
> 
>   RBX: 880f79fbb8b8 RCX: 
> 
>  Jun 28 09:46:41 roc04r-sca090 kernel: [137912.685076] RDX:
> 
>   RSI:  RDI: 8810352d4800
> 
>  Jun 28 09:46:41 roc04r-sca090 kernel: [137912.685107] RBP:
> 
>  880f79fbb880 R08: 0001020cf87c R09: 00ff00ff
> 
>  Jun 28 09:46:41 roc04r-sca090 kernel: [137912.685150] R10:
> 
>  0009 R11: 0006 R12: 8807c3adc4c0
> 
>  Jun 28 09:46:41 roc04r-sca090 kernel: [137912.685194] R13:
> 
>  0006 R14: 033e R15: fec7
> 
> 

Re: [ceph-users] Mounting Ceph RBD image to XenServer 7 as SR

[Josef Johansson]

Also, is it possible to recompile the rbd kernel module in XenServer? I am
under the impression that it's open source as well.

Regards,
Josef

On Fri, 1 Jul 2016, 04:52 Mike Jacobacci,  wrote:

> Thanks Somnath and Christian,
>
> Yes, it looks like the latest version of XenServer still runs on an old
> kernel (3.10).  I know the method Christian linked, but it doesn’t work if
> XenServer is installed from iso.  It is really annoying there has been no
> movement on this for 3 years… I really like XenServer and am excited to use
> Ceph, I want this to work.
>
> Since there are no VM’s on it yet, I think I will upgrade the kernel and
> see what happens.
>
> Cheers,
> Mike
>
>
> On Jun 30, 2016, at 7:40 PM, Somnath Roy  wrote:
>
> It seems your client kernel is pretty old ?
> Either upgrade your kernel to 3.15 or later or you need to disable
> CRUSH_TUNABLES3.
> ceph osd crush tunables bobtail or ceph osd crush tunables legacy should
> help. This will start rebalancing and also you will lose improvement added
> in Firefly. So, better to upgrade client kernel IMO.
>
> Thanks & Regards
> Somnath
>
> *From:* ceph-users [mailto:ceph-users-boun...@lists.ceph.com
> ] *On Behalf Of *Mike Jacobacci
> *Sent:* Thursday, June 30, 2016 7:27 PM
> *To:* Jake Young
> *Cc:* ceph-users@lists.ceph.com
> *Subject:* Re: [ceph-users] Mounting Ceph RBD image to XenServer 7 as SR
>
> Thanks Jake!  I enabled the epel 7 repo and was able to get ceph-common
> installed.  Here is what happens when I try to map the drive:
>
> rbd map rbd/enterprise-vm0 --name client.admin -m mon0 -k
> /etc/ceph/ceph.client.admin.keyring
> rbd: sysfs write failed
> In some cases useful info is found in syslog - try "dmesg | tail" or so.
> rbd: map failed: (5) Input/output error
>
> dmesg | tail:
>
> [35034.469236] libceph: mon0 192.168.10.187:6789 socket error on read
> [35044.469183] libceph: mon0 192.168.10.187:6789 feature set mismatch, my
> 4a042a42 < server's 2004a042a42, missing 200
> [35044.469199] libceph: mon0 192.168.10.187:6789 socket error on read
> [35054.469076] libceph: mon0 192.168.10.187:6789 feature set mismatch, my
> 4a042a42 < server's 2004a042a42, missing 200
> [35054.469083] libceph: mon0 192.168.10.187:6789 socket error on read
> [35064.469287] libceph: mon0 192.168.10.187:6789 feature set mismatch, my
> 4a042a42 < server's 2004a042a42, missing 200
> [35064.469302] libceph: mon0 192.168.10.187:6789 socket error on read
> [35074.469162] libceph: mon0 192.168.10.187:6789 feature set mismatch, my
> 4a042a42 < server's 2004a042a42, missing 200
> [35074.469178] libceph: mon0 192.168.10.187:6789 socket error on read
>
>
>
>
>
> On Jun 30, 2016, at 6:15 PM, Jake Young  wrote:
>
> See https://www.mail-archive.com/ceph-users@lists.ceph.com/msg17112.html
>
>
> On Thursday, June 30, 2016, Mike Jacobacci  wrote:
> So after adding the ceph repo and enabling the cents-7 repo… It fails
> trying to install ceph-common:
>
> Loaded plugins: fastestmirror
> Loading mirror speeds from cached hostfile
>  * base: mirror.web-ster.com
> Resolving Dependencies
> --> Running transaction check
> ---> Package ceph-common.x86_64 1:10.2.2-0.el7 will be installed
> --> Processing Dependency: python-cephfs = 1:10.2.2-0.el7 for package:
> 1:ceph-common-10.2.2-0.el7.x86_64
> --> Processing Dependency: python-rados = 1:10.2.2-0.el7 for package:
> 1:ceph-common-10.2.2-0.el7.x86_64
> --> Processing Dependency: librbd1 = 1:10.2.2-0.el7 for package:
> 1:ceph-common-10.2.2-0.el7.x86_64
> --> Processing Dependency: libcephfs1 = 1:10.2.2-0.el7 for package:
> 1:ceph-common-10.2.2-0.el7.x86_64
> --> Processing Dependency: python-rbd = 1:10.2.2-0.el7 for package:
> 1:ceph-common-10.2.2-0.el7.x86_64
> --> Processing Dependency: librados2 = 1:10.2.2-0.el7 for package:
> 1:ceph-common-10.2.2-0.el7.x86_64
> --> Processing Dependency: python-requests for package:
> 1:ceph-common-10.2.2-0.el7.x86_64
> --> Processing Dependency: libboost_program_options-mt.so.1.53.0()(64bit)
> for package: 1:ceph-common-10.2.2-0.el7.x86_64
> --> Processing Dependency: librgw.so.2()(64bit) for package:
> 1:ceph-common-10.2.2-0.el7.x86_64
> --> Processing Dependency: libradosstriper.so.1()(64bit) for package:
> 1:ceph-common-10.2.2-0.el7.x86_64
> --> Processing Dependency: libbabeltrace-ctf.so.1()(64bit) for package:
> 1:ceph-common-10.2.2-0.el7.x86_64
> --> Processing Dependency: libboost_regex-mt.so.1.53.0()(64bit) for
> package: 1:ceph-common-10.2.2-0.el7.x86_64
> --> Processing Dependency: libboost_iostreams-mt.so.1.53.0()(64bit) for
> package: 1:ceph-common-10.2.2-0.el7.x86_64
> --> Processing Dependency: librbd.so.1()(64bit) for package:
> 1:ceph-common-10.2.2-0.el7.x86_64
> --> Processing Dependency: libtcmalloc.so.4()(64bit) for package:
> 1:ceph-common-10.2.2-0.el7.x86_64
> --> Processing Dependency: librados.so.2()(64bit) for package:
> 

[ceph-users] Swift or S3?

[stephane.davy]

Hello Ceph users,

We are going to build a Ceph cluster as a backend storage for a backup 
solution. We have the possibility of using S3 or Swift from the client side. 
Which protocol is better implemented / supported on the rados gateway?

Thanks,

Stéphane

_

Ce message et ses pieces jointes peuvent contenir des informations 
confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce 
message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages 
electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou 
falsifie. Merci.

This message and its attachments may contain confidential or privileged 
information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete 
this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been 
modified, changed or falsified.
Thank you.

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com