Re: [ceph-users] Public network faster than cluster network

Il giorno gio 10 mag 2018 alle ore 09:48 Christian Balzer ha scritto: > Without knowing what your use case is (lots of large reads or writes, or > the more typical smallish I/Os) it's hard to give specific advice. 99% VM hosting. Everything else would be negligible and I don't

[ceph-users] How to normally expand OSD’s capacity?

Hi All, We are wondering if there is any way to expand OSD’s capacity. We are studying about this and conducted an experiment. However, in the result, the size of expanded capacity is counted on the USED part rather than the AVAIL one. The following shows the process of our experiment: 1.

[ceph-users] How to normally expand OSD’s capacity?

Hi All, We are wondering if there is any way to expand OSD’s capacity. We are studying about this and conducted an experiment. However, in the result, the size of expanded capacity is counted on the USED part rather than the AVAIL one. The following shows the process of our experiment: 1.

Re: [ceph-users] ceph-deploy: is it a requirement that the name of each node of the ceph cluster must be resolved to the public IP ?

Monitors can use only exactly one IP address. ceph-deploy uses some heuristics based on hostname resolution and ceph public addr configuration to guess which one to use during setup. (Which I've always found to be a quite annoying feature.) The mon's IP must be reachable from all ceph daemons and

Re: [ceph-users] GDPR encryption at rest

On 02/05/18 16:12, David Turner wrote: > I've heard conflicting opinions if GDPR requires data to be encrypted > at rest Encryption both in transit and at rest is part of data protection by design: it is about making sure that you have control over the data that you hold/are processing and that if

Re: [ceph-users] ceph-deploy: is it a requirement that the name of each node of the ceph cluster must be resolved to the public IP ?

I configured the "public network" attribute in the ceph configuration file. But it looks like to me that in the "auth get client.admin" command [*] issued by ceph-deploy the address of the management network is used (I guess because c-mon-02 gets resolved to the IP management address) Cheers,

Re: [ceph-users] ceph-deploy: is it a requirement that the name of each node of the ceph cluster must be resolved to the public IP ?

check ceph.conf, it controls to which mon IP the client tries to connect. 2018-05-10 12:57 GMT+02:00 Massimo Sgaravatto : > I configured the "public network" attribute in the ceph configuration file. > > But it looks like to me that in the "auth get client.admin"

[ceph-users] ceph-deploy: is it a requirement that the name of each node of the ceph cluster must be resolved to the public IP ?

I have a ceph cluster that I manually deployed, and now I am trying to see if I can use ceph-deploy to deploy new nodes (in particular the object gw). The network configuration is the following: Each MON node has two network IP: one on a "management network" (not used for ceph related stuff) and

Re: [ceph-users] How to normally expand OSD’s capacity?

You usually don't do that because you are supposed to use the whole disk. Paul 2018-05-10 12:31 GMT+02:00 Yi-Cian Pu : > Hi All, > > > > We are wondering if there is any way to expand OSD’s capacity. We are > studying about this and conducted an experiment. However,

Re: [ceph-users] ceph-deploy: is it a requirement that the name of each node of the ceph cluster must be resolved to the public IP ?

This [*] is my ceph.conf 10.70.42.9 is the public address And it is indeed the IP used by the MON daemon: [root@c-mon-02 ~]# netstat -anp | grep 6789 tcp0 0 10.70.42.9:6789 0.0.0.0:* LISTEN 3835/ceph-mon tcp0 0 10.70.42.9:33592

Re: [ceph-users] slow requests are blocked

Hi JC! Thanks for your answer first. 1. I have added output of  ceph health detail to Zabbix in case of warning. So every time I will see with which OSD the problem is. 2. I have default level of all logs. As I see here http://docs.ceph.com/docs/master/rados/troubleshooting/log-and-debug/

Re: [ceph-users] How to normally expand OSD’s capacity?

I do not believe there is any way to change the size of any part of a bluestore OSD configuration. On Thu, May 10, 2018 at 6:37 AM Paul Emmerich wrote: > You usually don't do that because you are supposed to use the whole disk. > > > Paul > > 2018-05-10 12:31 GMT+02:00

Re: [ceph-users] Public network faster than cluster network

Il giorno gio 10 mag 2018 alle ore 02:30 Christian Balzer ha scritto: > This cosmic imbalance would clearly lead to the end of the universe. > Seriously, think it through, what do you _think_ will happen? I thought what David told: "For a write on a replicated pool with size 3

Re: [ceph-users] Public network faster than cluster network

Hello, On Thu, 10 May 2018 07:24:20 + Gandalf Corvotempesta wrote: > > > Lastly, more often than not segregated networks are not needed, add > > unnecessary complexity and the resources spent on them would be better > > used to have just one fast and redundant network instead. > >

Re: [ceph-users] How to normally expand OSD’s capacity?

On 10.05.2018 12:24, Yi-Cian Pu wrote: Hi All, We are wondering if there is any way to expand OSD’s capacity. We are studying about this and conducted an experiment. However, in the result, the size of expanded capacity is counted on the USED part rather than the AVAIL one. The following

Re: [ceph-users] ceph mds memory usage 20GB : is it normal ?

I am now seeing the exact same issues you are reporting. A heap release did nothing for me. The only odd thing I'm doing is migrating data in cephfs from one pool to another. The process looks something like the following: TARGET_DIR=/media/cephfs/labs/ TARGET_POOL="cephfs_ec_data" setfattr -n

[ceph-users] Scrubbing impacting write latency since Luminous

Hi All, I've just upgraded our main cluster to Luminous and have noticed that where before the cluster 64k write latency was always hovering around 2ms regardless of what scrubbing was going on, since the upgrade to Luminous, scrubbing takes the average latency up to around 5-10ms and deep

[ceph-users] RBD Buffer I/O errors cleared by flatten?

Hi All, recently I saw a number of rbd backed VMs in my openstack cloud fail to reboot after a hypervisor crash with errors simialr to: [5.279393] blk_update_request: I/O error, dev vda, sector 2048 [5.281427] Buffer I/O error on dev vda1, logical block 0, lost async page write [

Re: [ceph-users] RBD Buffer I/O errors cleared by flatten?

My immediate guess is that your caps are incorrect for your OpenStack Ceph user. Please refer to step 6 from the Luminous upgrade guide to ensure your RBD users have permission to blacklist dead peers [1] [1] http://docs.ceph.com/docs/master/releases/luminous/#upgrade-from-jewel-or-kraken On

Re: [ceph-users] RBD Buffer I/O errors cleared by flatten?

It only bites you if you have a hard failure of a VM (i.e. the RBD image wasn't cleanly closed and the lock wasn't cleanly released). In that case, the next librbd client to attempt to acquire the lock will notice the dead lock owner and will attempt to blacklist it from the cluster to ensure it

Re: [ceph-users] RBD Buffer I/O errors cleared by flatten?

On Thu, May 10, 2018 at 09:55:15AM -0700, Jason Dillaman wrote: :My immediate guess is that your caps are incorrect for your OpenStack :Ceph user. Please refer to step 6 from the Luminous upgrade guide to :ensure your RBD users have permission to blacklist dead peers [1] : :[1]

Re: [ceph-users] RBD Buffer I/O errors cleared by flatten?

also, I should point out that if you've already upgraded to Luminous, you can just use the new RBD caps profiles (a la mon 'profile rbd' osd 'profile rbd') [1]. The explicit blacklist caps mentioned in the upgrade guide are only required since pre-Luminous clusters didn't support the RBD caps

Re: [ceph-users] slow requests are blocked

2. When logging the 1/5 is what's written to the log file/what's temporarily stored in memory. If you want to increase logging, you need to increase both numbers to 20/20 or 10/10. You can also just set it to 20 or 10 and ceph will set them to the same number. I personally do both numbers to

Re: [ceph-users] ceph mds memory usage 20GB : is it normal ?

Hello Brady, On Thu, May 10, 2018 at 7:35 AM, Brady Deetz wrote: > I am now seeing the exact same issues you are reporting. A heap release did > nothing for me. I'm not sure it's the same issue... > [root@mds0 ~]# ceph daemon mds.mds0 config get mds_cache_memory_limit > { >

[ceph-users] howto: multiple ceph filesystems

Hello guys, My company is about to rebuild its whole infrastructure, so I was called in order to help on the planning. We are essentially an corporate mail provider, so we handle daily lots of clients using dovecot and roundcube and in order to do so we want to design a better plant of our

Re: [ceph-users] howto: multiple ceph filesystems

On Thu, May 10, 2018 at 7:38 PM, João Paulo Sacchetto Ribeiro Bastos wrote: > Hello guys, > > My company is about to rebuild its whole infrastructure, so I was called in > order to help on the planning. We are essentially an corporate mail > provider, so we handle daily

Re: [ceph-users] ceph mds memory usage 20GB : is it normal ?

[ceph-admin@mds0 ~]$ ps aux | grep ceph-mds ceph1841 3.5 94.3 133703308 124425384 ? Ssl Apr04 1808:32 /usr/bin/ceph-mds -f --cluster ceph --id mds0 --setuser ceph --setgroup ceph [ceph-admin@mds0 ~]$ sudo ceph daemon mds.mds0 cache status { "pool": { "items": 173261056,

Re: [ceph-users] ceph mds memory usage 20GB : is it normal ?

On Thu, May 10, 2018 at 12:00 PM, Brady Deetz wrote: > [ceph-admin@mds0 ~]$ ps aux | grep ceph-mds > ceph1841 3.5 94.3 133703308 124425384 ? Ssl Apr04 1808:32 > /usr/bin/ceph-mds -f --cluster ceph --id mds0 --setuser ceph --setgroup ceph > > > [ceph-admin@mds0 ~]$ sudo

Re: [ceph-users] RBD Cache and rbd-nbd

On Thu, May 10, 2018 at 12:03 PM, Marc Schöchlin wrote: > Hello list, > > i map ~30 rbds per xenserver host by using rbd-nbd to run virtual machines > on these devices. > > I have the following questions: > > Is it possible to use rbd cache for rbd-nbd? I assume that this is

[ceph-users] RBD Cache and rbd-nbd

Hello list, i map ~30 rbds  per xenserver host by using rbd-nbd to run virtual machines on these devices. I have the following questions: * Is it possible to use rbd cache for rbd-nbd? I assume that this is true, but  the

[ceph-users] Nfs-ganesha 2.6 packages in ceph repo

Hi All I'm testing out the nfs-ganesha-2.6.1-0.1.el7.x86_64.rpm package from http://download.ceph.com/nfs-ganesha/rpm-V2.6-stable/luminous/x86_64/ It's failing to load /usr/lib64/ganesha/libfsalceph.so With libcephfs-12.2.1 installed I get the following error in my ganesha log: load_fsal :NFS

Re: [ceph-users] howto: multiple ceph filesystems

Hey John, thanks for you answer. For sure the hardware robustness will be nice enough. My true concern was actually the two FS ecosystem coexistence. In fact I realized that we may not use this as well because it may be represent a high overhead, despite the fact that it's a experiental feature

Re: [ceph-users] howto: multiple ceph filesystems

Another option you could do is to use a placement rule. You could create a general pool for most data to go to and a special pool for specific folders on the filesystem. Particularly I think of a pool for replica vs EC vs flash for specific folders in the filesystem. If the pool and OSDs wasn't

[ceph-users] Inaccurate client io stats

Hi everyone, I've got a 3-node cluster running without any issue. However, I found out that since upgraded to luminous, the client io stat is far too way off from the real one. Have no idea how to troubleshoot this after went through all the logs. Any help would be appreciated. Got more than

Re: [ceph-users] Nfs-ganesha 2.6 packages in ceph repo

Hi David, for what it's worth, we are running with nfs-ganesha 2.6.1 from Ceph repos on CentOS 7.4 with the following set of versions: libcephfs2-12.2.4-0.el7.x86_64 nfs-ganesha-2.6.1-0.1.el7.x86_64 nfs-ganesha-ceph-2.6.1-0.1.el7.x86_64 Of course, we plan to upgrade to 12.2.5 soon-ish... Am

[ceph-users] Ceph osd crush weight to utilization incorrect on one node

Hi, We have a large 1PB ceph cluster. We recently added 6 nodes with 16 2TB disks each to the cluster. All the 5 nodes rebalanced well without any issues and the sixth/last node OSDs started acting weird as I increase weight of one osd the utilization doesn't change but a different osd on the