Re: [ceph-users] amount of PGs/pools/OSDs for your openstack / Ceph
Our use case is not Openstack but we have a cluster with similar size to what you are looking at. Our cluster has 540 OSDs with 4PB of raw storage spread across 9 nodes at this point. 2 pools - 512 PGs - 3 way redundancy - 32768 PGs - RS(6,3) erasure coding (99.9% of data in this pool) The reason we chose to go with ~550PGs/OSD currently is to reduce the number of data moves that will happen when OSDs are added to the cluster and the number of PGs need to be expanded. We have enough memory on the nodes to handle the high number of PGs. 512GB for 60 OSDs/ node. For testing the cluster about 2.5TB of data was written to the EC pool using "rados bench" at 2-3GB/s of sustained throughput. The cluster is being used with librados and objects are directly stored in the pools. Did not hit any major issues with simulated scenarios like drive replacement and recovery. We also tested with double the number of PGs in each pool - 1024 and 65536. The cluster started showing instability at that point. Whenever an OSD went down, cascading failures started to occur during recovery i.e more OSDs would fail during the peering process when a failed OSD tried to rejoin the cluster. Keeping the OSD usage balanced becomes very important as the cluster fills up. A few OSDs that have much higher usage than the others can stop all writes into the cluster and it is very hard to recover from it when the usage is very close to the capacity thresholds. Subhachandra On Sat, Apr 7, 2018 at 7:01 PM, Christian Wuerdig < christian.wuer...@gmail.com> wrote: > The general recommendation is to target around 100 PG/OSD. Have you tried > the https://ceph.com/pgcalc/ tool? > > On Wed, 4 Apr 2018 at 21:38, Osama Hasebou wrote: > >> Hi Everyone, >> >> I would like to know what kind of setup had the Ceph community been using >> for their Openstack's Ceph configuration when it comes to number of Pools & >> OSDs and their PGs. >> >> Ceph documentation briefly mentions it for small cluster size, and I >> would like to know from your experience, how much PGs have you created for >> your openstack pools in reality for a ceph cluster ranging from 1-2 PB >> capacity or 400-600 number of OSDs that performs well without issues. >> >> Hope to hear from you! >> >> Thanks. >> >> Regards, >> Ossi >> >> ___ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] User deletes bucket with partial multipart uploads in, objects still in quota
I believe there is a command in radosgw-admin to change the owner of a bucket which might be able to resolve the incorrect quota issue. I don't know if that will work since the bucket doesn't think it exists. Perhaps creating a new bucket of the same name and trying to run commands against that to both change the owner and delete the objects might help. It also might be necessary to rebuild the index for the faux bucket before it will actually do anything. This is a guess and not from experience with situations like this. On Wed, Apr 4, 2018 at 5:34 AM Matthew Vernon wrote: > On 04/04/18 10:30, Matthew Vernon wrote: > > Hi, > > > > We have an rgw user who had a bunch of partial multipart uploads in a > > bucket, which they then deleted. radosgw-admin bucket list doesn't show > > the bucket any more, but user stats --sync-stats still has (I think) > > the contents of that bucket counted against the users' quota. > > > > So, err, how do I cause a) the users' quota usage to not include this > > deleted bucket b) the associated storage to actually be cleared (since I > > infer the failure to do so is causing the quota issue)? > > Sorry, should have said: this is running jewel. > > Regards, > > Matthew > > > -- > The Wellcome Sanger Institute is operated by Genome Research > Limited, a charity registered in England with number 1021457 and a > company registered in England with number 2742969, whose registered > office is 215 Euston Road, London, NW1 2BE. > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Question to avoid service stop when osd is full
The proper way to prevent this is to set your full ratios safe and monitor your disk usage. That will allow you to either clean up old data or add new storage before you get to 95 full on any OSDs. What I mean by setting your full ratios safe is that if your use case can fill 20% of your disk space within a couple days, then having your warnings start at 75% is too high because you can easily fill up the rest of your space within a couple days and then need more storage before you have it ready. There is no method to allow read-only while OSDs are full. On Mon, Apr 9, 2018 at 6:58 AM 渥美 慶彦 wrote: > Hi, > > I have 2 questions. > > I want to use ceph for OpenStack's volume backend by creating 2 ceph pools. > One pool consists of osds on SSD, and the other consists of osds on HDD. > The storage capacity of SSD pool is much smaller than that of HDD pool, > so I want to make configuration not to stop all IO even if one osd on > SSD becomes full. > Is this possible? > > "osd full ratio" is default to 0.95, and if one osd becomes full, then > all osd will stop. > Is there any configuration to allow us to read-only while one or more > osds are full? > > best regards, > > -- > > Atsumi Yoshihiko > E-mail:atsumi.yoshih...@po.ntt-tx.co.jp > > > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph scrub logs: _scan_snaps no head for $object?
I have this on a rbd pool with images/snapshots that have been created in Luminous > Hi Stefan, Mehmet, > > Are these clusters that were upgraded from prior versions, or fresh > luminous installs? > > > This message indicates that there is a stray clone object with no > associated head or snapdir object. That normally should never > happen--it's presumably the result of a (hopefully old) bug. The scrub > process doesn't even clean them up, which maybe says something about how > common it is/was... > > sage > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph scrub logs: _scan_snaps no head for $object?
I have found one image, how do I know what snapshot version to delete? I have multiple -Original Message- From: c...@elchaka.de [mailto:c...@elchaka.de] Sent: zondag 8 april 2018 13:30 To: ceph-users Subject: Re: [ceph-users] Ceph scrub logs: _scan_snaps no head for $object? Am 8. April 2018 05:44:11 MESZ schrieb Marc Roos : > >Hi Mehmet, > >The data is already lost in these snapshots? I cannot say that. Cause i did Not need the Snapshots. But you can try to Clone the vm in the state of the Snapshot ( i am using proxmox). > And how did you identify >the snapshot? It looks like I have these only in the rbd pool. You have to use "rbd info" to identify which Image is Related to this. Search for " 239f5274b0dc51" - Mehmet > > > > >-Original Message- >From: c...@elchaka.de [mailto:c...@elchaka.de] >Sent: zondag 8 april 2018 10:44 >To: ceph-users@lists.ceph.com >Subject: Re: [ceph-users] Ceph scrub logs: _scan_snaps no head for >$object? > >Hi Marc, > >Am 7. April 2018 18:32:40 MESZ schrieb Marc Roos >: >> >>How do you resolve these issues? >> > >In my Case i could get rid of this by deleting the existing Snapshots. > >- Mehmet >> >>Apr 7 22:39:21 c03 ceph-osd: 2018-04-07 22:39:21.928484 7f0826524700 >>-1 >>osd.13 pg_epoch: 19008 pg[17.13( v 19008'6019891 >>(19008'6018375,19008'6019891] local-lis/les=18980/18981 n=3825 >>ec=3636/3636 lis/c 18980/18980 les/c/f 18981/18982/0 >18980/18980/18903) >> >>[4,13,0] r=1 lpr=18980 luod=0'0 crt=19008'6019891 lcod 19008'6019890 >>active] _scan_snaps no head for >>17:cbf61056:::rbd_data.239f5274b0dc51.0ff2:15 (have MIN) >>___ >>ceph-users mailing list >>ceph-users@lists.ceph.com >>http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >___ >ceph-users mailing list >ceph-users@lists.ceph.com >http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph luminous - troubleshooting performance issues overall DSK 100%, busy 1%
Disk controller seem fine Any other suggestions will be really appreciated megacli -AdpBbuCmd -aAll BBU status for Adapter: 0 BatteryType: BBU Voltage: 3925 mV Current: 0 mA Temperature: 17 C Battery State: Optimal BBU Firmware Status: Charging Status : None Voltage : OK Temperature : OK Learn Cycle Requested : No Learn Cycle Active : No Learn Cycle Status : OK Learn Cycle Timeout : No I2c Errors Detected : No Battery Pack Missing: No Battery Replacement required: No Remaining Capacity Low : No Periodic Learn Required : No Transparent Learn : No No space to cache offload : No Pack is about to fail & should be replaced : No Cache Offload premium feature required : No Module microcode update required: No megacli -AdpDiag -a0 Performing Diagnostic on Controller 0. It will take 20 seconds to complete. Please wait... Diagnostic Completed on Controller 0. Exit Code: 0x00 On Fri, 6 Apr 2018 at 15:11, David Turner wrote: > First and foremost, have you checked your disk controller. Of most import > would be your cache battery. Any time I have a single node acting up, the > controller is Suspect #1. > > On Thu, Apr 5, 2018 at 11:23 AM Steven Vacaroaia wrote: > >> Hi, >> >> I have a strange issue - OSDs from a specific server are introducing huge >> performance issue >> >> This is a brand new installation on 3 identical servers - >> DELL R620 with PERC H710 , bluestore DB and WAL on SSD, 10GB dedicated >> private/public networks >> >> >> When I add the OSD I see gaps like below and huge latency >> >> atop provides no clear culprit EXCEPT very low network and specific disk >> utilization BUT 100% DSK for ceph-osd process which stay like that ( 100%) >> for the duration of the test >> ( see below) >> >> Not sure why ceph-osd process DSK stays at 100% while all the specific >> DSK ( for sdb, sde ..etc) are 1% busy ? >> >> Any help/ instructions for how to troubleshooting this will be >> appreciated >> >> (apologies if the format is not being kept) >> >> >> CPU | sys 4% | user 1% | | irq 1% | >>| idle794% | wait 0% | | >> | steal 0% | guest 0% | curf 2.20GHz | | >> curscal ?% | >> CPL | avg10.00 | | avg50.00 | avg15 0.00 | >>| | | csw547/s | >> | intr 832/s | | | numcpu 8 | >> | >> MEM | tot62.9G | free 61.4G | cache 520.6M | dirty 0.0M | >> buff7.5M | slab 98.9M | slrec 64.8M | shmem 8.8M | shrss >> 0.0M | shswp 0.0M | vmbal 0.0M | | hptot 0.0M | >> hpuse 0.0M | >> SWP | tot 6.0G | free6.0G | | | >>| | | | >> | | | vmcom 1.5G | | vmlim >> 37.4G | >> LVM | dm-0 | busy 1% | | read 0/s | >> write 54/s | | KiB/r 0 | KiB/w455 | MBr/s >> 0.0 | | MBw/s 24.0 | avq 3.69 | | avio >> 0.14 ms | >> DSK | sdb | busy 1% | | read 0/s | >> write 102/s | | KiB/r 0 | KiB/w240 | MBr/s >> 0.0 | | MBw/s 24.0 | avq 6.69 | | avio >> 0.08 ms | >> DSK | sda | busy 0% | | read 0/s | >> write 12/s | | KiB/r 0 | KiB/w 4 | MBr/s >> 0.0 | | MBw/s0.1 | avq 1.00 | | avio >> 0.05 ms | >> DSK | sde | busy 0% | | read 0/s | >> write0/s | | KiB/r 0 | KiB/w 0 | MBr/s >> 0.0 | | MBw/s0.0 | avq 1.00 | | avio >> 2.50 ms | >> NET | transport | tcpi 718/s | tcpo 972/s | udpi 0/s | >>| udpo 0/s | tcpao0/s | tcppo0/s | tcprs 21/s >> | tcpie0/s | tcpor0/s | | udpnp0/s | udpie >> 0/s | >> NET | network | ipi719/s | | ipo399/s | >> ipfrw0/s | | deliv 719/s | | >> | | | icmpi0/s | | icmpo >> 0/s | >> NET | eth5 1% | pcki 2214/s | pcko 939/s | | sp >> 10 Gbps | si 154 Mbps | so 52 Mbps | | coll 0/s | >> mlti 0/s | erri 0/s | erro 0/s | drpi 0/s | drpo >> 0/s | >> NET | eth4 0% | pcki 712/s | pcko54/s | | sp >> 10 Gbps | si 50 Mbps | so 90 Kbps |
Re: [ceph-users] Admin socket on a pure client: is it possible?
On 04/09/2018 04:01 PM, Fulvio Galeazzi wrote: > Hallo, > > I am wondering whether I could have the admin socket functionality > enabled on a server which is a pure Ceph client (no MDS/MON/OSD/whatever > running on such server). Is this at all possible? How should ceph.conf > be configured? Documentation pages led me to write something like this: > > . > [client] > admin socket = /var/run/ceph/$cluster-guest.asok > log file = /var/log/ceph/client-guest.log > . > but the .asok is absent. Please enlighten me as I must be missing > something very basic. > Yes, this would work: [client] admin socket = /var/run/ceph/$cluster-$type.$id.$pid.$cctid.asok If not, try the [global] section. But what type of client is it? RBD? RADOS? Kernel RBD? > The use-case would be to integrate with a piece of code (namely, a > Juju charm) which assumes to be able to talk with Ceph cluster via an > admin socket: problem is that such Juju charm/bundle also assumes to > manage its own Ceph cluster, whereas I'd like to have it interface to an > independent, external, ceph-ansible-managed Ceph cluster. > > Would it suffice to install ceph-mgr on such client? But then, I don't > want such ceph-mgr to form quorum with the "real" ceph-mgr(s) installed > on my MON nodes. > > Do you think it's possible to achieve such a configuration? > > Thanks! > > Fulvio > > > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Admin socket on a pure client: is it possible?
Hallo, I am wondering whether I could have the admin socket functionality enabled on a server which is a pure Ceph client (no MDS/MON/OSD/whatever running on such server). Is this at all possible? How should ceph.conf be configured? Documentation pages led me to write something like this: . [client] admin socket = /var/run/ceph/$cluster-guest.asok log file = /var/log/ceph/client-guest.log . but the .asok is absent. Please enlighten me as I must be missing something very basic. The use-case would be to integrate with a piece of code (namely, a Juju charm) which assumes to be able to talk with Ceph cluster via an admin socket: problem is that such Juju charm/bundle also assumes to manage its own Ceph cluster, whereas I'd like to have it interface to an independent, external, ceph-ansible-managed Ceph cluster. Would it suffice to install ceph-mgr on such client? But then, I don't want such ceph-mgr to form quorum with the "real" ceph-mgr(s) installed on my MON nodes. Do you think it's possible to achieve such a configuration? Thanks! Fulvio smime.p7s Description: S/MIME Cryptographic Signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Scrubbing for RocksDB
Hi list, we were wondering if and how the consistency of OSD journals (BlueStore) is checked. Our cluster runs on Luminous (12.2.2) and we had migrated all our filestore OSDs to bluestore a couple of months ago. During that process we placed each rocksDB on a separate partition on a RAID1 consisting of two SSDs. Our cluster was healthy, we deep-scrub the whole cluster once a week without any errors etc. Then we decided to restructure the disk layout on one of the hosts, we didn't want that RAID of SSDs anymore. So we failed one disk (diskB), wiped it and assigned a new volume group to it, now containing one logical volume per OSD. We started the journal migration as mentioned in [1] by copying the data from diskA (degraded RAID1) to diskB (LVM) with dd. The first journal migration worked like a charm, but for the next four partitions the dd command reported errors like these: ---cut here--- FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Sense Key : Medium Error [current] Add. Sense: Read retries exhausted CDB: Read(10) 28 00 0a 08 8b a0 00 04 00 00 blk_update_request: critical medium error, dev sdk, sector 168332406 Buffer I/O error on dev md126p6, logical block 1363854, async page read ---cut here--- Four of six partitions reported these errors, a look into smartctl confirmed that this SSD is corrupt and has non-recoverable errors. That's why we had to rebuild the respective OSDs from scratch, but at least without rearranging the whole cluster (also mentioned in [1]). So my question is, why can't I find anything in the ceph logs about this? The scrubbing and deep-scrubbing only check the PGs on the data device for consistency, but what about the journal? Is there any tool we haven't found yet or any mechanism that would detect an I/O error? Of course there is a possibility that the respective blocks on the corrupt partitions haven't been updated for some time, but IMHO there should be something to check the journal's consistency and report it in the ceph logs, something like a journal-scrub, maybe. Has someone experienced similar issues and can shed some light on this? Any insights would be very helpful. Regards, Eugen [1] http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-February/024913.html -- Eugen Block voice : +49-40-559 51 75 NDE Netzdesign und -entwicklung AG fax : +49-40-559 51 77 Postfach 61 03 15 D-22423 Hamburg e-mail : ebl...@nde.ag Vorsitzende des Aufsichtsrates: Angelika Mozdzen Sitz und Registergericht: Hamburg, HRB 90934 Vorstand: Jens-U. Mozdzen USt-IdNr. DE 814 013 983 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Ceph Dashboard v2 update
Hi all, a month has passed since the Dashboard v2 was merged into the master branch, so I thought it might be helpful to write a summary/update (with screenshots) of what we've been up to since then: https://www.openattic.org/posts/ceph-dashboard-v2-update/ Let us know what you think! Cheers, Lenz -- SUSE Linux GmbH - Maxfeldstr. 5 - 90409 Nuernberg (Germany) GF:Felix Imendörffer,Jane Smithard,Graham Norton,HRB 21284 (AG Nürnberg) signature.asc Description: OpenPGP digital signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Issue with fstrim and Nova hw_disk_discard=unmap
Hallo Jason, thanks again for your time and apologies for long silence but I was busy upgrading to Luminous and converting Filestore->Bluestore. In the meantime, the staging cluster where I was making tests was both upgraded to Ceph-Luminous and upgraded to OpenStack-Pike: good news is that now fstrim works as expected so I think it's not worth it (and difficult/impossible) to investigate further. I may post some more info once I have a maintenance window to upgrade the production cluster (I have to touch nova.conf, and I want to do that during a maintenance). By the way, I am unable to configure Ceph such that the admin socket is made available on the (pure) client node, am going to open a separate issue for this. Thanks! Fulvio Original Message Subject: Re: [ceph-users] Issue with fstrim and Nova hw_disk_discard=unmap From: Jason Dillaman To: Fulvio Galeazzi CC: Ceph Users Date: 03/15/2018 01:35 PM OK, last suggestion just to narrow the issue down: ensure you have a functional admin socket and librbd log file as documented here [1]. With the VM running, before you execute "fstrim", run "ceph --admin-daemon /path/to/the/asok/file conf set debug_rbd 20" on the hypervisor host, execute "fstrim" within the VM, and then restore the log settings via "ceph --admin-daemon /path/to/the/asok/file conf set debug_rbd 0/5". Grep the log file for "aio_discard" to verify if QEMU is passing the discard down to librbd. [1] http://docs.ceph.com/docs/master/rbd/rbd-openstack/ On Thu, Mar 15, 2018 at 6:53 AM, Fulvio Galeazzi wrote: Hallo Jason, I am really thankful for your time! Changed the volume features: rbd image 'volume-80838a69-e544-47eb-b981-a4786be89736': . features: layering, exclusive-lock, deep-flatten I had to create several dummy files before seeing and increase with "rbd du": to me, this is sort of indication that dirty blocks are, at least, reused if not properly released. Then I did "rm * ; sync ; fstrim / ; sync" but the size did not go down. Is there a way to instruct Ceph to perform what is not currently happening automatically (namely, scan the object-map of a volume and force cleanup of released blocks)? Or the problem is exactly that such blocks are not seen by Ceph as reusable? By the way, I think I forgot to mention that underlying OSD disks are taken from a FibreChannel storage (DELL MD3860, which is not capable of presenting JBOD so I present single disks as RAID0) and XFS formatted. Thanks! Fulvio Original Message Subject: Re: [ceph-users] Issue with fstrim and Nova hw_disk_discard=unmap From: Jason Dillaman To: Fulvio Galeazzi CC: Ceph Users Date: 03/14/2018 02:10 PM Hmm -- perhaps as an experiment, can you disable the object-map and fast-diff features to see if they are incorrectly reporting the object as in-use after a discard? $ rbd --cluster cephpa1 -p cinder-ceph feature disable volume-80838a69-e544-47eb-b981-a4786be89736 object-map,fast-diff On Wed, Mar 14, 2018 at 3:29 AM, Fulvio Galeazzi wrote: Hallo Jason, sure here it is! rbd --cluster cephpa1 -p cinder-ceph info volume-80838a69-e544-47eb-b981-a4786be89736 rbd image 'volume-80838a69-e544-47eb-b981-a4786be89736': size 15360 MB in 3840 objects order 22 (4096 kB objects) block_name_prefix: rbd_data.9e7ffe238e1f29 format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten flags: Thanks Fulvio Original Message Subject: Re: [ceph-users] Issue with fstrim and Nova hw_disk_discard=unmap From: Jason Dillaman To: Fulvio Galeazzi CC: Ceph Users Date: 03/13/2018 06:33 PM Can you provide the output from "rbd info /volume-80838a69-e544-47eb-b981-a4786be89736"? On Tue, Mar 13, 2018 at 12:30 PM, Fulvio Galeazzi wrote: Hallo! Discards appear like they are being sent to the device. How big of a temporary file did you create and then delete? Did you sync the file to disk before deleting it? What version of qemu-kvm are you running? I made several test with commands like (issuing sync after each operation): dd if=/dev/zero of=/tmp/fileTest bs=1M count=200 oflag=direct What I see is that if I repeat the command with count<=200 the size does not increase. Let's try now with count>200: NAMEPROVISIONED USED volume-80838a69-e544-47eb-b981-a4786be89736 15360M 2284M dd if=/dev/zero of=/tmp/fileTest bs=1M count=750 oflag=direct dd if=/dev/zero of=/tmp/fileTest2 bs=1M count=750 oflag=direct sync NAMEPROVISIONED USED volume-80838a69-e544-47eb-b981-a4786be89736 15360M 2528M rm /tmp/fileTest* sync sudo fstrim -v / /: 14.1 GiB (15145271296 bytes) trimmed NAMEPROVISIONED USED volume-80838a69-e544-47eb-b981-a4786be89736
[ceph-users] Question to avoid service stop when osd is full
Hi, I have 2 questions. I want to use ceph for OpenStack's volume backend by creating 2 ceph pools. One pool consists of osds on SSD, and the other consists of osds on HDD. The storage capacity of SSD pool is much smaller than that of HDD pool, so I want to make configuration not to stop all IO even if one osd on SSD becomes full. Is this possible? "osd full ratio" is default to 0.95, and if one osd becomes full, then all osd will stop. Is there any configuration to allow us to read-only while one or more osds are full? best regards, -- Atsumi Yoshihiko E-mail:atsumi.yoshih...@po.ntt-tx.co.jp ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Fwd: Separate --block.wal --block.db bluestore not working as expected.
Hi, Just a little question regarding this operation : [root@osdhost osd]# ceph-volume lvm prepare --bluestore --data /dev/sdc --block.wal /dev/sda2 --block.db /dev/sda1 On a previous post, I understood that if both wal and db are stored on the same separate device, then we could use a single partition for both...which means we could do : # ceph-volume lvm prepare --bluestore --data /dev/sdc --block.wal /dev/sda1 --block.db /dev/sda1 and so on with other uniq wal/db partition for other OSD... Did I get that correctly ? Thanks, Hervé Le 07/04/2018 à 17:59, Gary Verhulp a écrit : I’m trying to create bluestore osds with separate --block.wal --block.db devices on a write intensive SSD I’ve split the SSD (/dev/sda) into two partditions sda1 and sda2 for db and wal I seems to me the osd uuid is getting changed and I’m only able to start the last OSD Do I need to create a new partition or logical volume on the SSD for each OSD? I’m sure this is a simple fail in my understanding of how it is supposed to be provisioned. Any advice would be appreciated. Thanks, Gary [root@osdhost osd]# ceph-volume lvm prepare --bluestore --data /dev/sdc --block.wal /dev/sda2 --block.db /dev/sda1 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Move ceph admin node to new other server
Hello, We have use 1 server for deploy (called ceph-admin-node) for 3 mon and 4 OSD node. We have created a folder called *ceph-deploy* to deploy all node members. May we move this folder to other server? This folder contains all following files: total 1408 -rw--- 1 root root 113 Oct 26 16:48 ceph.bootstrap-mds.keyring -rw--- 1 root root 71 Oct 26 16:48 ceph.bootstrap-mgr.keyring -rw--- 1 root root 113 Oct 26 16:48 ceph.bootstrap-osd.keyring -rw--- 1 root root 113 Oct 26 16:48 ceph.bootstrap-rgw.keyring -rw--- 1 root root 129 Oct 26 16:48 ceph.client.admin.keyring -rw-r--r-- 1 root root 818 Oct 26 12:20 ceph.conf -rw-r--r-- 1 root root 1405043 Apr 2 14:16 ceph-deploy-ceph.log -rw--- 1 root root 73 Oct 26 12:19 ceph.mon.keyring Any guide for this task? Thanks, ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com