Re: [ceph-users] amount of PGs/pools/OSDs for your openstack / Ceph

[Subhachandra Chandra]

Our use case is not Openstack but we have a cluster with similar size to
what you are looking at. Our cluster has 540 OSDs with 4PB of raw storage
spread across 9 nodes at this point.

2 pools
   - 512 PGs - 3 way redundancy
   - 32768 PGs - RS(6,3) erasure coding (99.9% of data in this pool)

The reason we chose to go with ~550PGs/OSD currently is to reduce the
number of data moves that will happen when OSDs are added to the cluster
and the number of PGs need to be expanded. We have enough memory on the
nodes to handle the high number of PGs. 512GB for 60 OSDs/ node. For
testing the cluster about 2.5TB of data was written to the EC pool using
"rados bench" at 2-3GB/s of sustained throughput. The cluster is being used
with librados and objects are directly stored in the pools. Did not hit any
major issues with simulated scenarios like drive replacement and recovery.

We also tested with double the number of PGs in each pool - 1024 and 65536.
The cluster started showing instability at that point. Whenever an OSD went
down, cascading failures started to occur during recovery i.e more OSDs
would fail during the peering process when a failed OSD tried to rejoin the
cluster.

Keeping the OSD usage balanced becomes very important as the cluster fills
up. A few OSDs that have much higher usage than the others can stop all
writes into the cluster and it is very hard to recover from it when the
usage is very close to the capacity thresholds.

Subhachandra


On Sat, Apr 7, 2018 at 7:01 PM, Christian Wuerdig <
christian.wuer...@gmail.com> wrote:

> The general recommendation is to target around 100 PG/OSD. Have you tried
> the https://ceph.com/pgcalc/ tool?
>
> On Wed, 4 Apr 2018 at 21:38, Osama Hasebou  wrote:
>
>> Hi Everyone,
>>
>> I would like to know what kind of setup had the Ceph community been using
>> for their Openstack's Ceph configuration when it comes to number of Pools &
>> OSDs and their PGs.
>>
>> Ceph documentation briefly mentions it for small cluster size, and I
>> would like to know from your experience, how much PGs have you created for
>> your openstack pools in reality for a ceph cluster ranging from 1-2 PB
>> capacity or 400-600 number of OSDs that performs well without issues.
>>
>> Hope to hear from you!
>>
>> Thanks.
>>
>> Regards,
>> Ossi
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] User deletes bucket with partial multipart uploads in, objects still in quota

[David Turner]

I believe there is a command in radosgw-admin to change the owner of a
bucket which might be able to resolve the incorrect quota issue.  I don't
know if that will work since the bucket doesn't think it exists.  Perhaps
creating a new bucket of the same name and trying to run commands against
that to both change the owner and delete the objects might help.  It also
might be necessary to rebuild the index for the faux bucket before it will
actually do anything.  This is a guess and not from experience with
situations like this.

On Wed, Apr 4, 2018 at 5:34 AM Matthew Vernon  wrote:

> On 04/04/18 10:30, Matthew Vernon wrote:
> > Hi,
> >
> > We have an rgw user who had a bunch of partial multipart uploads in a
> > bucket, which they then deleted. radosgw-admin bucket list doesn't show
> > the bucket any more, but  user stats --sync-stats still has (I think)
> > the contents of that bucket counted against the users' quota.
> >
> > So, err, how do I cause a) the users' quota usage to not include this
> > deleted bucket b) the associated storage to actually be cleared (since I
> > infer the failure to do so is causing the quota issue)?
>
> Sorry, should have said: this is running jewel.
>
> Regards,
>
> Matthew
>
>
> --
>  The Wellcome Sanger Institute is operated by Genome Research
>  Limited, a charity registered in England with number 1021457 and a
>  company registered in England with number 2742969, whose registered
>  office is 215 Euston Road, London, NW1 2BE.
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Question to avoid service stop when osd is full

[David Turner]

The proper way to prevent this is to set your full ratios safe and monitor
your disk usage.  That will allow you to either clean up old data or add
new storage before you get to 95 full on any OSDs.  What I mean by setting
your full ratios safe is that if your use case can fill 20% of your disk
space within a couple days, then having your warnings start at 75% is too
high because you can easily fill up the rest of your space within a couple
days and then need more storage before you have it ready.

There is no method to allow read-only while OSDs are full.

On Mon, Apr 9, 2018 at 6:58 AM 渥美 慶彦 
wrote:

> Hi,
>
> I have 2 questions.
>
> I want to use ceph for OpenStack's volume backend by creating 2 ceph pools.
> One pool consists of osds on SSD, and the other consists of osds on HDD.
> The storage capacity of SSD pool is much smaller than that of HDD pool,
> so I want to make configuration not to stop all IO even if one osd on
> SSD becomes full.
> Is this possible?
>
> "osd full ratio" is default to 0.95, and if one osd becomes full, then
> all osd will stop.
> Is there any configuration to allow us to read-only while one or more
> osds are full?
>
> best regards,
>
> --
> 
> Atsumi Yoshihiko
> E-mail：atsumi.yoshih...@po.ntt-tx.co.jp
> 
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph scrub logs: _scan_snaps no head for $object?

[Marc Roos]

I have this on a rbd pool with images/snapshots that have been created 
in Luminous

> Hi Stefan, Mehmet,
> 
> Are these clusters that were upgraded from prior versions, or fresh 
> luminous installs?
> 
> 
> This message indicates that there is a stray clone object with no 
> associated head or snapdir object.  That normally should never 
> happen--it's presumably the result of a (hopefully old) bug.  The 
scrub 
> process doesn't even clean them up, which maybe says something about 
how 
> common it is/was...
> 
> sage
> 
 



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph scrub logs: _scan_snaps no head for $object?

[Marc Roos]

 
I have found one image, how do I know what snapshot version to delete? I 
have multiple




-Original Message-
From: c...@elchaka.de [mailto:c...@elchaka.de] 
Sent: zondag 8 april 2018 13:30
To: ceph-users
Subject: Re: [ceph-users] Ceph scrub logs: _scan_snaps no head for 
$object?



Am 8. April 2018 05:44:11 MESZ schrieb Marc Roos 
:
>
>Hi Mehmet,
> 
>The data is already lost in these snapshots?

I cannot say that. Cause i did Not need the Snapshots. But you can try 
to Clone the vm in the state of the Snapshot ( i am using proxmox).  
  
> And how did you identify
>the snapshot? It looks like I have these only in the rbd pool. 

You have to use "rbd info" to identify which Image is Related to this. 
Search for " 239f5274b0dc51"

- Mehmet 
>
>
>
>
>-Original Message-
>From: c...@elchaka.de [mailto:c...@elchaka.de]
>Sent: zondag 8 april 2018 10:44
>To: ceph-users@lists.ceph.com
>Subject: Re: [ceph-users] Ceph scrub logs: _scan_snaps no head for 
>$object?
>
>Hi Marc,
>
>Am 7. April 2018 18:32:40 MESZ schrieb Marc Roos
>:
>>
>>How do you resolve these issues?
>>
>
>In my Case i could get rid of this by deleting the existing Snapshots.
>
>- Mehmet   
>>
>>Apr  7 22:39:21 c03 ceph-osd: 2018-04-07 22:39:21.928484 7f0826524700
>>-1
>>osd.13 pg_epoch: 19008 pg[17.13( v 19008'6019891 
>>(19008'6018375,19008'6019891] local-lis/les=18980/18981 n=3825
>>ec=3636/3636 lis/c 18980/18980 les/c/f 18981/18982/0
>18980/18980/18903)
>>
>>[4,13,0] r=1 lpr=18980 luod=0'0 crt=19008'6019891 lcod 19008'6019890 
>>active] _scan_snaps no head for
>>17:cbf61056:::rbd_data.239f5274b0dc51.0ff2:15 (have MIN) 
>>___
>>ceph-users mailing list
>>ceph-users@lists.ceph.com
>>http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>___
>ceph-users mailing list
>ceph-users@lists.ceph.com
>http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph luminous - troubleshooting performance issues overall DSK 100%, busy 1%

[Steven Vacaroaia]

Disk controller seem fine

Any other suggestions will be really appreciated

megacli -AdpBbuCmd -aAll

BBU status for Adapter: 0

BatteryType: BBU
Voltage: 3925 mV
Current: 0 mA
Temperature: 17 C
Battery State: Optimal
BBU Firmware Status:

  Charging Status  : None
  Voltage : OK
  Temperature : OK
  Learn Cycle Requested   : No
  Learn Cycle Active  : No
  Learn Cycle Status  : OK
  Learn Cycle Timeout : No
  I2c Errors Detected : No
  Battery Pack Missing: No
  Battery Replacement required: No
  Remaining Capacity Low  : No
  Periodic Learn Required : No
  Transparent Learn   : No
  No space to cache offload   : No
  Pack is about to fail & should be replaced : No
  Cache Offload premium feature required  : No
  Module microcode update required: No


 megacli -AdpDiag -a0

Performing Diagnostic on Controller 0.
 It will take 20 seconds to complete. Please wait...
Diagnostic Completed on Controller 0.

Exit Code: 0x00



On Fri, 6 Apr 2018 at 15:11, David Turner  wrote:

> First and foremost, have you checked your disk controller.  Of most import
> would be your cache battery.  Any time I have a single node acting up, the
> controller is Suspect #1.
>
> On Thu, Apr 5, 2018 at 11:23 AM Steven Vacaroaia  wrote:
>
>> Hi,
>>
>> I have a strange issue - OSDs from a specific server are introducing huge
>> performance issue
>>
>> This is a brand new installation on 3 identical servers -
>>  DELL R620 with PERC H710 , bluestore  DB and WAL on SSD, 10GB dedicated
>> private/public networks
>>
>>
>> When I add the OSD I see gaps like below and huge latency
>>
>> atop provides no  clear culprit EXCEPT very low network and specific disk
>> utilization BUT 100% DSK for ceph-osd process  which stay like that ( 100%)
>> for the duration of the test
>> ( see below)
>>
>> Not sure why ceph-osd process  DSK stays at 100% while all the specific
>> DSK ( for sdb, sde ..etc) are 1% busy ?
>>
>> Any help/ instructions for how to troubleshooting this will be
>> appreciated
>>
>> (apologies if the format is not being kept)
>>
>>
>> CPU | sys   4%  | user  1%  |   | irq   1%  |
>>| idle794%  | wait  0%  |  |
>>  |  steal 0% |  guest 0% |  curf 2.20GHz |   |
>> curscal   ?% |
>> CPL | avg10.00  |   | avg50.00  | avg15   0.00  |
>>|   |   | csw547/s |
>>  |  intr   832/s |   |   |  numcpu 8 |
>>  |
>> MEM | tot62.9G  | free   61.4G  | cache 520.6M  | dirty   0.0M  |
>> buff7.5M  | slab   98.9M  | slrec  64.8M  | shmem   8.8M |  shrss
>>  0.0M |  shswp   0.0M |  vmbal   0.0M |   |  hptot   0.0M |
>> hpuse   0.0M |
>> SWP | tot 6.0G  | free6.0G  |   |   |
>>|   |   |  |
>>  |   |   |  vmcom   1.5G |   |  vmlim
>> 37.4G |
>> LVM | dm-0  | busy  1%  |   | read 0/s  |
>> write   54/s  |   | KiB/r  0  | KiB/w455 |  MBr/s
>> 0.0 |   |  MBw/s   24.0 |  avq 3.69 |   |  avio
>> 0.14 ms |
>> DSK |  sdb  | busy  1%  |   | read 0/s  |
>> write  102/s  |   | KiB/r  0  | KiB/w240 |  MBr/s
>> 0.0 |   |  MBw/s   24.0 |  avq 6.69 |   |  avio
>> 0.08 ms |
>> DSK |  sda  | busy  0%  |   | read 0/s  |
>> write   12/s  |   | KiB/r  0  | KiB/w  4 |  MBr/s
>> 0.0 |   |  MBw/s0.1 |  avq 1.00 |   |  avio
>> 0.05 ms |
>> DSK |  sde  | busy  0%  |   | read 0/s  |
>> write0/s  |   | KiB/r  0  | KiB/w  0 |  MBr/s
>> 0.0 |   |  MBw/s0.0 |  avq 1.00 |   |  avio
>> 2.50 ms |
>> NET | transport | tcpi   718/s  | tcpo   972/s  | udpi 0/s  |
>>| udpo 0/s  | tcpao0/s  | tcppo0/s |  tcprs   21/s
>> |  tcpie0/s |  tcpor0/s |   |  udpnp0/s |  udpie
>> 0/s |
>> NET | network   | ipi719/s  |   | ipo399/s  |
>> ipfrw0/s  |   | deliv  719/s  |  |
>>  |   |   |  icmpi0/s |   |  icmpo
>>   0/s |
>> NET | eth5  1%  | pcki  2214/s  | pcko   939/s  |   | sp
>>  10 Gbps  | si  154 Mbps  | so   52 Mbps  |  |  coll 0/s |
>> mlti 0/s |  erri 0/s |  erro 0/s |  drpi 0/s |  drpo
>>  0/s |
>> NET | eth4  0%  | pcki   712/s  | pcko54/s  |   | sp
>>  10 Gbps  

Re: [ceph-users] Admin socket on a pure client: is it possible?

[Wido den Hollander]

On 04/09/2018 04:01 PM, Fulvio Galeazzi wrote:
> Hallo,
> 
>   I am wondering whether I could have the admin socket functionality
> enabled on a server which is a pure Ceph client (no MDS/MON/OSD/whatever
> running on such server). Is this at all possible? How should ceph.conf
> be configured? Documentation pages led me to write something like this:
> 
> .
> [client]
> admin socket = /var/run/ceph/$cluster-guest.asok
> log file = /var/log/ceph/client-guest.log
> .
>  but the .asok is absent. Please enlighten me as I must be missing
> something very basic.
> 

Yes, this would work:

[client]
admin socket = /var/run/ceph/$cluster-$type.$id.$pid.$cctid.asok

If not, try the [global]  section.

But what type of client is it? RBD? RADOS? Kernel RBD?

>   The use-case would be to integrate with a piece of code (namely, a
> Juju charm) which assumes to be able to talk with Ceph cluster via an
> admin socket: problem is that such Juju charm/bundle also assumes to
> manage its own Ceph cluster, whereas I'd like to have it interface to an
> independent, external, ceph-ansible-managed Ceph cluster.
> 
>   Would it suffice to install ceph-mgr on such client? But then, I don't
> want such ceph-mgr to form quorum with the "real" ceph-mgr(s) installed
> on my MON nodes.
> 
>   Do you think it's possible to achieve such a configuration?
> 
>   Thanks!
> 
>     Fulvio
> 
> 
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Admin socket on a pure client: is it possible?

[Fulvio Galeazzi]

Hallo,

  I am wondering whether I could have the admin socket functionality 
enabled on a server which is a pure Ceph client (no MDS/MON/OSD/whatever 
running on such server). Is this at all possible? How should ceph.conf 
be configured? Documentation pages led me to write something like this:


.
[client]
admin socket = /var/run/ceph/$cluster-guest.asok
log file = /var/log/ceph/client-guest.log
.
 but the .asok is absent. Please enlighten me as I must be missing 
something very basic.


  The use-case would be to integrate with a piece of code (namely, a 
Juju charm) which assumes to be able to talk with Ceph cluster via an 
admin socket: problem is that such Juju charm/bundle also assumes to 
manage its own Ceph cluster, whereas I'd like to have it interface to an 
independent, external, ceph-ansible-managed Ceph cluster.


  Would it suffice to install ceph-mgr on such client? But then, I 
don't want such ceph-mgr to form quorum with the "real" ceph-mgr(s) 
installed on my MON nodes.


  Do you think it's possible to achieve such a configuration?

  Thanks!

Fulvio



smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Scrubbing for RocksDB

[Eugen Block]

Hi list,

we were wondering if and how the consistency of OSD journals  
(BlueStore) is checked.


Our cluster runs on Luminous (12.2.2) and we had migrated all our  
filestore OSDs to bluestore a couple of months ago. During that  
process we placed each rocksDB on a separate partition on a RAID1  
consisting of two SSDs. Our cluster was healthy, we deep-scrub the  
whole cluster once a week without any errors etc.


Then we decided to restructure the disk layout on one of the hosts, we  
didn't want that RAID of SSDs anymore. So we failed one disk (diskB),  
wiped it and assigned a new volume group to it, now containing one  
logical volume per OSD. We started the journal migration as mentioned  
in [1] by copying the data from diskA (degraded RAID1) to diskB (LVM)  
with dd. The first journal migration worked like a charm, but for the  
next four partitions the dd command reported errors like these:


---cut here---
FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Sense Key : Medium Error [current]
Add. Sense: Read retries exhausted
CDB: Read(10) 28 00 0a 08 8b a0 00 04 00 00
blk_update_request: critical medium error, dev sdk, sector 168332406
Buffer I/O error on dev md126p6, logical block 1363854, async page read
---cut here---

Four of six partitions reported these errors, a look into smartctl  
confirmed that this SSD is corrupt and has non-recoverable errors.  
That's why we had to rebuild the respective OSDs from scratch, but at  
least without rearranging the whole cluster (also mentioned in [1]).


So my question is, why can't I find anything in the ceph logs about  
this? The scrubbing and deep-scrubbing only check the PGs on the data  
device for consistency, but what about the journal? Is there any tool  
we haven't found yet or any mechanism that would detect an I/O error?  
Of course there is a possibility that the respective blocks on the  
corrupt partitions haven't been updated for some time, but IMHO there  
should be something to check the journal's consistency and report it  
in the ceph logs, something like a journal-scrub, maybe.


Has someone experienced similar issues and can shed some light on  
this? Any insights would be very helpful.


Regards,
Eugen

[1]  
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-February/024913.html


--
Eugen Block voice   : +49-40-559 51 75
NDE Netzdesign und -entwicklung AG  fax : +49-40-559 51 77
Postfach 61 03 15
D-22423 Hamburg e-mail  : ebl...@nde.ag

Vorsitzende des Aufsichtsrates: Angelika Mozdzen
  Sitz und Registergericht: Hamburg, HRB 90934
  Vorstand: Jens-U. Mozdzen
   USt-IdNr. DE 814 013 983

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Ceph Dashboard v2 update

[Lenz Grimmer]

Hi all,

a month has passed since the Dashboard v2 was merged into the master
branch, so I thought it might be helpful to write a summary/update (with
screenshots) of what we've been up to since then:

  https://www.openattic.org/posts/ceph-dashboard-v2-update/

Let us know what you think!

Cheers,

Lenz

-- 
SUSE Linux GmbH - Maxfeldstr. 5 - 90409 Nuernberg (Germany)
GF:Felix Imendörffer,Jane Smithard,Graham Norton,HRB 21284 (AG Nürnberg)



signature.asc
Description: OpenPGP digital signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Issue with fstrim and Nova hw_disk_discard=unmap

[Fulvio Galeazzi]

Hallo Jason,
thanks again for your time and apologies for long silence but I was 
busy upgrading to Luminous and converting Filestore->Bluestore.


  In the meantime, the staging cluster where I was making tests was 
both upgraded to Ceph-Luminous and upgraded to OpenStack-Pike: good news 
is that now fstrim works as expected so I think it's not worth it (and 
difficult/impossible) to investigate further.
I may post some more info once I have a maintenance window to upgrade 
the production cluster (I have to touch nova.conf, and I want to do that 
during a maintenance).


  By the way, I am unable to configure Ceph such that the admin socket 
is made available on the (pure) client node, am going to open a separate 
issue for this.


  Thanks!

Fulvio

 Original Message 
Subject: Re: [ceph-users] Issue with fstrim and Nova hw_disk_discard=unmap
From: Jason Dillaman 
To: Fulvio Galeazzi 
CC: Ceph Users 
Date: 03/15/2018 01:35 PM


OK, last suggestion just to narrow the issue down: ensure you have a
functional admin socket and librbd log file as documented here [1].
With the VM running, before you execute "fstrim", run "ceph
--admin-daemon /path/to/the/asok/file conf set debug_rbd 20" on the
hypervisor host, execute "fstrim" within the VM, and then restore the
log settings via "ceph --admin-daemon /path/to/the/asok/file conf set
debug_rbd 0/5".  Grep the log file for "aio_discard" to verify if QEMU
is passing the discard down to librbd.


[1] http://docs.ceph.com/docs/master/rbd/rbd-openstack/

On Thu, Mar 15, 2018 at 6:53 AM, Fulvio Galeazzi
 wrote:

Hallo Jason, I am really thankful for your time!

   Changed the volume features:

rbd image 'volume-80838a69-e544-47eb-b981-a4786be89736':
.
 features: layering, exclusive-lock, deep-flatten

I had to create several dummy files before seeing and increase with "rbd
du": to me, this is sort of indication that dirty blocks are, at least,
reused if not properly released.

   Then I did "rm * ; sync ; fstrim / ; sync" but the size did not go down.
   Is there a way to instruct Ceph to perform what is not currently happening
automatically (namely, scan the object-map of a volume and force cleanup of
released blocks)? Or the problem is exactly that such blocks are not seen by
Ceph as reusable?

   By the way, I think I forgot to mention that underlying OSD disks are
taken from a FibreChannel storage (DELL MD3860, which is not capable of
presenting JBOD so I present single disks as RAID0) and XFS formatted.

   Thanks!

 Fulvio

 Original Message 
Subject: Re: [ceph-users] Issue with fstrim and Nova hw_disk_discard=unmap
From: Jason Dillaman 
To: Fulvio Galeazzi 
CC: Ceph Users 
Date: 03/14/2018 02:10 PM


Hmm -- perhaps as an experiment, can you disable the object-map and
fast-diff features to see if they are incorrectly reporting the object
as in-use after a discard?

$ rbd --cluster cephpa1 -p cinder-ceph feature disable
volume-80838a69-e544-47eb-b981-a4786be89736 object-map,fast-diff

On Wed, Mar 14, 2018 at 3:29 AM, Fulvio Galeazzi
 wrote:


Hallo Jason, sure here it is!

rbd --cluster cephpa1 -p cinder-ceph info
volume-80838a69-e544-47eb-b981-a4786be89736
rbd image 'volume-80838a69-e544-47eb-b981-a4786be89736':
  size 15360 MB in 3840 objects
  order 22 (4096 kB objects)
  block_name_prefix: rbd_data.9e7ffe238e1f29
  format: 2
  features: layering, exclusive-lock, object-map, fast-diff,
deep-flatten
  flags:

Thanks

  Fulvio


 Original Message 
Subject: Re: [ceph-users] Issue with fstrim and Nova
hw_disk_discard=unmap
From: Jason Dillaman 
To: Fulvio Galeazzi 
CC: Ceph Users 
Date: 03/13/2018 06:33 PM


Can you provide the output from "rbd info /volume-80838a69-e544-47eb-b981-a4786be89736"?

On Tue, Mar 13, 2018 at 12:30 PM, Fulvio Galeazzi
 wrote:



Hallo!


Discards appear like they are being sent to the device.  How big of a
temporary file did you create and then delete? Did you sync the file
to disk before deleting it? What version of qemu-kvm are you running?





I made several test with commands like (issuing sync after each
operation):

dd if=/dev/zero of=/tmp/fileTest bs=1M count=200 oflag=direct

What I see is that if I repeat the command with count<=200 the size
does
not
increase.

Let's try now with count>200:

NAMEPROVISIONED  USED
volume-80838a69-e544-47eb-b981-a4786be89736  15360M 2284M

dd if=/dev/zero of=/tmp/fileTest bs=1M count=750 oflag=direct
dd if=/dev/zero of=/tmp/fileTest2 bs=1M count=750 oflag=direct
sync

NAME

[ceph-users] Question to avoid service stop when osd is full

[渥美 慶彦]

Hi,

I have 2 questions.

I want to use ceph for OpenStack's volume backend by creating 2 ceph pools.
One pool consists of osds on SSD, and the other consists of osds on HDD.
The storage capacity of SSD pool is much smaller than that of HDD pool,
so I want to make configuration not to stop all IO even if one osd on 
SSD becomes full.

Is this possible?

"osd full ratio" is default to 0.95, and if one osd becomes full, then 
all osd will stop.
Is there any configuration to allow us to read-only while one or more 
osds are full?


best regards,

--

Atsumi Yoshihiko
E-mail：atsumi.yoshih...@po.ntt-tx.co.jp



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Fwd: Separate --block.wal --block.db bluestore not working as expected.

[Hervé Ballans]

Hi,

Just a little question regarding this operation :

[root@osdhost osd]# ceph-volume lvm prepare --bluestore --data /dev/sdc 
--block.wal /dev/sda2 --block.db /dev/sda1


On a previous post, I understood that if both wal and db are stored on 
the same separate device, then we could use a single partition for 
both...which means we could do :


# ceph-volume lvm prepare --bluestore --data /dev/sdc --block.wal 
/dev/sda1 --block.db /dev/sda1


and so on with other uniq wal/db partition for other OSD...

Did I get that correctly ?

Thanks,

Hervé


Le 07/04/2018 à 17:59, Gary Verhulp a écrit :


I’m trying to create bluestore osds with separate --block.wal 
--block.db devices on a write intensive SSD


I’ve split the SSD (/dev/sda) into two partditions sda1 and sda2 for 
db and wal


I seems to me the osd uuid is getting changed and I’m only able to 
start the last OSD


Do I need to create a new partition or logical volume on the SSD for 
each OSD?


I’m sure this is a simple fail in my understanding of how it is 
supposed to be provisioned.


Any advice would be appreciated.

Thanks,

Gary

[root@osdhost osd]# ceph-volume lvm prepare --bluestore --data 
/dev/sdc --block.wal /dev/sda2 --block.db /dev/sda1




___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Move ceph admin node to new other server

[Nghia Than]

Hello,

We have use 1 server for deploy (called ceph-admin-node) for 3 mon and 4
OSD node.

We have created a folder called *ceph-deploy* to deploy all node members.
May we move this folder to other server?

This folder contains all following files:

total 1408
-rw--- 1 root root 113 Oct 26 16:48 ceph.bootstrap-mds.keyring
-rw--- 1 root root  71 Oct 26 16:48 ceph.bootstrap-mgr.keyring
-rw--- 1 root root 113 Oct 26 16:48 ceph.bootstrap-osd.keyring
-rw--- 1 root root 113 Oct 26 16:48 ceph.bootstrap-rgw.keyring
-rw--- 1 root root 129 Oct 26 16:48 ceph.client.admin.keyring
-rw-r--r-- 1 root root 818 Oct 26 12:20 ceph.conf
-rw-r--r-- 1 root root 1405043 Apr  2 14:16 ceph-deploy-ceph.log
-rw--- 1 root root  73 Oct 26 12:19 ceph.mon.keyring

Any guide for this task?

Thanks,
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com