Re: [ceph-users] Log format in Ceph

[Sinan Polat]

Hi Stefan,

I do not want to know the reason. I want to parse Ceph logs (and use it in 
Elastic). But without knowing the log format I can’t parse. I know that the 
first and second ‘words’ are date + timestamp, but what about the 3rd-5th words 
of a log line?

Sinan 

> Op 8 jan. 2020 om 09:48 heeft Stefan Kooman  het volgende 
> geschreven:
> 
> Quoting Sinan Polat (si...@turka.nl):
>> Hi,
>> 
>> 
>> I couldn't find any documentation or information regarding the log format in
>> Ceph. For example, I have 2 log lines (see below). For each 'word' I would 
>> like
>> to know what it is/means.
>> 
>> As far as I know, I can break the log lines into:
>> [date] [timestamp] [unknown] [unknown] [unknown] [pthread] [colon char]
>> [unknown] [PRIORITY] [message]
>> 
>> Can anyone fill in the [unknown] fields, or redirect me to some
>> documentation/information?
> 
> Issue "ceph daemon osd.3 dump_historic_slow_ops" on the storage node
> hosting this OSD and you will get JSON output with the reason
> (flag_point) of the slow op and the series of events.
> 
> Gr. Stefan
> 
> 
> -- 
> | BIT BV  https://www.bit.nl/Kamer van Koophandel 09090351
> | GPG: 0xD14839C6   +31 318 648 688 / i...@bit.nl

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Log format in Ceph

[Sinan Polat]

Hi,


I couldn't find any documentation or information regarding the log format in
Ceph. For example, I have 2 log lines (see below). For each 'word' I would like
to know what it is/means.

As far as I know, I can break the log lines into:
[date] [timestamp] [unknown] [unknown] [unknown] [pthread] [colon char]
[unknown] [PRIORITY] [message]

Can anyone fill in the [unknown] fields, or redirect me to some
documentation/information?

2020-01-07 15:45:15.593092 osd.3 osd.3 10.36.212.72:6800/5645 2117 : cluster
[WRN] slow request 30.762632 seconds old, received at 2020-01-07
15:44:44.830356: osd_op(client.2127384.0:772793 1.25
1:a71849c8:::rbd_data.20760c15c9284.0014:head [stat,write
8323072~65536] snapc 0=[] ondisk+write+known_if_redirected e191) currently
waiting for rw locks

2020-01-08 03:23:48.297619 mgr.bms-cephmon03-lab client.1199560
10.36.212.93:0/2512770604 2398154 : cluster [DBG] pgmap v2398247: 320 pgs: 320
active+clean; 96.4GiB data, 292GiB used, 2.38TiB / 2.67TiB avail; 0B/s rd,
161KiB/s wr, 19op/s

Thanks!

Sinan___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph luminous bluestore poor random write performances

[Sinan Polat]

Hi,

Your performance is not that bad, is it? What performance do you expect?

I just ran the same test.
12 Node, SATA SSD Only:
   READ: bw=63.8MiB/s (66.9MB/s), 63.8MiB/s-63.8MiB/s (66.9MB/s-66.9MB/s),
io=3070MiB (3219MB), run=48097-48097msec
  WRITE: bw=21.3MiB/s (22.4MB/s), 21.3MiB/s-21.3MiB/s (22.4MB/s-22.4MB/s),
io=1026MiB (1076MB), run=48097-48097msec

6 Node, SAS Only:
   READ: bw=22.1MiB/s (23.2MB/s), 22.1MiB/s-22.1MiB/s (23.2MB/s-23.2MB/s),
io=3070MiB (3219MB), run=138650-138650msec
  WRITE: bw=7578KiB/s (7759kB/s), 7578KiB/s-7578KiB/s (7759kB/s-7759kB/s),
io=1026MiB (1076MB), run=138650-138650msec

This is OpenStack Queens with Ceph FileStore (Luminous).

Kind regards,
Sinan Polat

> Op 2 januari 2020 om 10:59 schreef Stefan Kooman :
> 
> 
> Quoting Ignazio Cassano (ignaziocass...@gmail.com):
> > Hello All,
> > I installed ceph luminous with openstack, an using fio in a virtual machine
> > I got slow random writes:
> > 
> > fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test
> > --filename=random_read_write.fio --bs=4k --iodepth=64 --size=4G
> > --readwrite=randrw --rwmixread=75
> 
> Do you use virtio-scsi with a SCSI queue per virtual CPU core? How many
> cores do you have? I suspect that the queue depth is hampering
> throughput here ... but is throughput performance really interesting
> anyway for your use case? Low latency generally matters most.
> 
> Gr. Stefan
> 
> 
> -- 
> | BIT BV  https://www.bit.nl/Kamer van Koophandel 09090351
> | GPG: 0xD14839C6   +31 318 648 688 / i...@bit.nl
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Consumer-grade SSD in Ceph

[Sinan Polat]

Thanks for all the replies. In summary; consumer grade SSD is a no go.

What is an alternative to SM863a? Since it is quite hard to get these due non 
non-stock.

Thanks!
Sinan

> Op 23 dec. 2019 om 08:50 heeft Eneko Lacunza  het 
> volgende geschreven:
> 
> Hi Sinan,
> 
> Just to reiterate: don't do this. Consumer SSDs will destroy your enterprise 
> SSD's performance.
> 
> Our office cluster is made of consumer-grade servers: cheap gaming 
> motherboards, memory, ryzen processors, desktop HDDs. But SSD drives are 
> Enterprise, we had awful experiences with consumer SSDs (some perform worse 
> that HDDs with Ceph).
> 
> Cheers
> Eneko
> 
>> El 19/12/19 a las 20:20, Sinan Polat escribió:
>> Hi all,
>> 
>> Thanks for the replies. I am not worried about their lifetime. We will be 
>> adding only 1 SSD disk per physical server. All SSD’s are enterprise drives. 
>> If the added consumer grade disk will fail, no problem.
>> 
>> I am more curious regarding their I/O performance. I do want to have 50% 
>> drop in performance.
>> 
>> So anyone any experience with 860 EVO or Crucial MX500 in a Ceph setup?
>> 
>> Thanks!
>> 
>>> Op 19 dec. 2019 om 19:18 heeft Mark Nelson  het 
>>> volgende geschreven:
>>> 
>>> The way I try to look at this is:
>>> 
>>> 
>>> 1) How much more do the enterprise grade drives cost?
>>> 
>>> 2) What are the benefits? (Faster performance, longer life, etc)
>>> 
>>> 3) How much does it cost to deal with downtime, diagnose issues, and 
>>> replace malfunctioning hardware?
>>> 
>>> 
>>> My personal take is that enterprise drives are usually worth it. There may 
>>> be consumer grade drives that may be worth considering in very specific 
>>> scenarios if they still have power loss protection and high write 
>>> durability.  Even when I was in academia years ago with very limited 
>>> budgets, we got burned with consumer grade SSDs to the point where we had 
>>> to replace them all.  You have to be very careful and know exactly what you 
>>> are buying.
>>> 
>>> 
>>> Mark
>>> 
>>> 
>>>> On 12/19/19 12:04 PM, jes...@krogh.cc wrote:
>>>> I dont think “usually” is good enough in a production setup.
>>>> 
>>>> 
>>>> 
>>>> Sent from myMail for iOS
>>>> 
>>>> 
>>>> Thursday, 19 December 2019, 12.09 +0100 from Виталий Филиппов 
>>>> :
>>>> 
>>>>Usually it doesn't, it only harms performance and probably SSD
>>>>lifetime
>>>>too
>>>> 
>>>>> I would not be running ceph on ssds without powerloss protection. I
>>>>> delivers a potential data loss scenario
>>>> 
>>>> 
>>>> ___
>>>> ceph-users mailing list
>>>> ceph-users@lists.ceph.com
>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> 
> -- 
> Zuzendari Teknikoa / Director Técnico
> Binovo IT Human Project, S.L.
> Telf. 943569206
> Astigarragako bidea 2, 2º izq. oficina 11; 20180 Oiartzun (Gipuzkoa)
> www.binovo.es
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Restarting firewall causes slow requests

[Sinan Polat]

Hi,

Restarting the firewall (systemctl restart firewalld) on an OSD node causes slow
requests. Is this expected behavior?

Cluster is running Ceph 12.2.

Thanks!
Sinan___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Consumer-grade SSD in Ceph

[Sinan Polat]

Hi all,

Thanks for the replies. I am not worried about their lifetime. We will be 
adding only 1 SSD disk per physical server. All SSD’s are enterprise drives. If 
the added consumer grade disk will fail, no problem.

I am more curious regarding their I/O performance. I do want to have 50% drop 
in performance.

So anyone any experience with 860 EVO or Crucial MX500 in a Ceph setup?

Thanks!

> Op 19 dec. 2019 om 19:18 heeft Mark Nelson  het volgende 
> geschreven:
> 
> The way I try to look at this is:
> 
> 
> 1) How much more do the enterprise grade drives cost?
> 
> 2) What are the benefits? (Faster performance, longer life, etc)
> 
> 3) How much does it cost to deal with downtime, diagnose issues, and replace 
> malfunctioning hardware?
> 
> 
> My personal take is that enterprise drives are usually worth it. There may be 
> consumer grade drives that may be worth considering in very specific 
> scenarios if they still have power loss protection and high write durability. 
>  Even when I was in academia years ago with very limited budgets, we got 
> burned with consumer grade SSDs to the point where we had to replace them 
> all.  You have to be very careful and know exactly what you are buying.
> 
> 
> Mark
> 
> 
>> On 12/19/19 12:04 PM, jes...@krogh.cc wrote:
>> I dont think “usually” is good enough in a production setup.
>> 
>> 
>> 
>> Sent from myMail for iOS
>> 
>> 
>> Thursday, 19 December 2019, 12.09 +0100 from Виталий Филиппов 
>> :
>> 
>>Usually it doesn't, it only harms performance and probably SSD
>>lifetime
>>too
>> 
>>> I would not be running ceph on ssds without powerloss protection. I
>>> delivers a potential data loss scenario
>> 
>> 
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Consumer-grade SSD in Ceph

[Sinan Polat]

Hi,

I am aware that
https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/
holds a list with benchmark of quite some different ssd models. Unfortunately it
doesn't have benchmarks for recent ssd models.

A client is planning to expand a running cluster (Luminous, FileStore, SSD only,
Replicated). I/O Utilization is close to 0, but capacity wise the cluster is
almost nearfull. To save costs the cluster will be expanded will customer-grade
SSD's, but I am unable to find benchmarks of recent SSD models.

Does anyone has experience with Samsung 860 EVO, 860 PRO and Crucial MX500 in a
Ceph cluster?

Thanks!
Sinan___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Pool statistics via API

[Sinan Polat]

Hi Ernesto,

I just opened the Dashboard and there is no menu at the top-right. Also no "?".
I have a menu at the top-left which has the following items: Cluster health,
Cluster, Block and Filesystems.

Running Ceph version 12.2.8-89.

Kind regards,
Sinan Polat

> Op 11 oktober 2019 om 22:09 schreef Sinan Polat :
> 
> Hi Ernesto,
> 
> Thanks for the information! I didn’t know about the existence of the REST
> Dashboard API. I will check that out, Thanks again!
> 
> Sinan
> 
> Op 11 okt. 2019 om 21:06 heeft Ernesto Puerta  mailto:epuer...@redhat.com > het volgende geschreven:
> 
> 
> > > Hi Sinan,
> > 
> > If it's in the Dashboard, it sure comes from the Dashboard REST API
> > (which is an API completely unrelated to the RESTful Module).
> > 
> > To check the Dashboard REST API, log in there and click on the
> > top-right "?" menu, and in the dropdown, click on "API". That will lead you
> > to the Swagger/OpenAPI spec of the Dashboard. You will likely want to
> > explore the "/pool" and "/block" endpoints. The API page will give you
> > ready-to-use curl commands (the only thing you'd need to renew, once
> > expired, is the authorization token).
> > 
> > Kind regards,
> > 
> > Ernesto Puerta
> > He / Him / His
> > 
> > Senior Software Engineer, Ceph
> > Red Hat https://www.redhat.com/
> > 
> >  
> >  https://www.redhat.com/
> > 
> > 
> > 
> > 
> > On Thu, Oct 10, 2019 at 2:16 PM Sinan Polat  > mailto:si...@turka.nl > wrote:
> > 
> > > > > 
> > > Hi,
> > > 
> > > Currently I am getting the pool statistics (especially
> > > USED/MAX AVAIL) via the command line:
> > > ceph df -f json-pretty| jq '.pools[] | select(.name ==
> > > "poolname") | .stats.max_avail'
> > > ceph df -f json-pretty| jq '.pools[] | select(.name ==
> > > "poolname") | .stats.bytes_used'
> > > 
> > > Command "ceph df" does not show the (total) size of the
> > > provisioned RBD images. It only shows the real usage.
> > > 
> > > I managed to get the total size of provisioned images using
> > > the Python rbd module https://docs.ceph.com/docs/master/rbd/api/librbdpy/
> > > 
> > >  https://docs.ceph.com/docs/master/rbd/api/librbdpy/
> > > Using the same Python module I also would like to get the
> > > USED/MAX AVAIL per pool. That should be possible using
> > > rbd.RBD().pool_stats_get, but unfortunately my python-rbd version doesn't
> > > support that (running 12.2.8).
> > > 
> > > So I went ahead and enabled the dashboard to see if the data
> > > is present in the dashboard and it seems it is. Next step is to enable the
> > > restful module and access this information, right? But unfortunately the
> > > restful api doesn't provide this information.
> > > 
> > > My question is, how can I access the USED/MAX AVAIL
> > > information of a pool without using the ceph command line and without
> > > upgrading my python-rbd package?
> > > 
> > > Kind regards
> > > Sinan Polat
> > > 
> > > ___
> > > ceph-users mailing list
> > > ceph-users@lists.ceph.com mailto:ceph-users@lists.ceph.com
> > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > > 
> > > > > 
> > > 


 

> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Pool statistics via API

[Sinan Polat]

Hi Ernesto,

Thanks for the information! I didn’t know about the existence of the REST 
Dashboard API. I will check that out, Thanks again!

Sinan

> Op 11 okt. 2019 om 21:06 heeft Ernesto Puerta  het 
> volgende geschreven:
> 
> Hi Sinan,
> 
> If it's in the Dashboard, it sure comes from the Dashboard REST API (which is 
> an API completely unrelated to the RESTful Module).
> 
> To check the Dashboard REST API, log in there and click on the top-right "?" 
> menu, and in the dropdown, click on "API". That will lead you to the 
> Swagger/OpenAPI spec of the Dashboard. You will likely want to explore the 
> "/pool" and "/block" endpoints. The API page will give you ready-to-use curl 
> commands (the only thing you'd need to renew, once expired, is the 
> authorization token).
> 
> Kind regards,
> 
> Ernesto Puerta
> He / Him / His
> Senior Software Engineer, Ceph
> Red Hat
> 
> 
> 
>> On Thu, Oct 10, 2019 at 2:16 PM Sinan Polat  wrote:
>> Hi,
>> 
>> Currently I am getting the pool statistics (especially USED/MAX AVAIL) via 
>> the command line:
>> ceph df -f json-pretty| jq '.pools[] | select(.name == "poolname") | 
>> .stats.max_avail'
>> ceph df -f json-pretty| jq '.pools[] | select(.name == "poolname") | 
>> .stats.bytes_used'
>> 
>> Command "ceph df" does not show the (total) size of the provisioned RBD 
>> images. It only shows the real usage.
>> 
>> I managed to get the total size of provisioned images using the Python rbd 
>> module https://docs.ceph.com/docs/master/rbd/api/librbdpy/
>> 
>> Using the same Python module I also would like to get the USED/MAX AVAIL per 
>> pool. That should be possible using rbd.RBD().pool_stats_get, but 
>> unfortunately my python-rbd version doesn't support that (running 12.2.8).
>> 
>> So I went ahead and enabled the dashboard to see if the data is present in 
>> the dashboard and it seems it is. Next step is to enable the restful module 
>> and access this information, right? But unfortunately the restful api 
>> doesn't provide this information.
>> 
>> My question is, how can I access the USED/MAX AVAIL information of a pool 
>> without using the ceph command line and without upgrading my python-rbd 
>> package?
>> 
>> Kind regards
>> Sinan Polat
>> 
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Pool statistics via API

[Sinan Polat]

Hi,

Currently I am getting the pool statistics (especially USED/MAX AVAIL) via the
command line:
ceph df -f json-pretty| jq '.pools[] | select(.name == "poolname") |
.stats.max_avail'
ceph df -f json-pretty| jq '.pools[] | select(.name == "poolname") |
.stats.bytes_used'

Command "ceph df" does not show the (total) size of the provisioned RBD images.
It only shows the real usage.

I managed to get the total size of provisioned images using the Python rbd
module https://docs.ceph.com/docs/master/rbd/api/librbdpy/

https://docs.ceph.com/docs/master/rbd/api/librbdpy/
Using the same Python module I also would like to get the USED/MAX AVAIL per
pool. That should be possible using rbd.RBD().pool_stats_get, but unfortunately
my python-rbd version doesn't support that (running 12.2.8).

So I went ahead and enabled the dashboard to see if the data is present in the
dashboard and it seems it is. Next step is to enable the restful module and
access this information, right? But unfortunately the restful api doesn't
provide this information.

My question is, how can I access the USED/MAX AVAIL information of a pool
without using the ceph command line and without upgrading my python-rbd package?

Kind regards
Sinan Polat https://docs.ceph.com/docs/master/rados/api/python/___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Questions regarding backing up Ceph

[Sinan Polat]

Hi,

Why not using backup tools that can do native OpenStack backups?

We are also using Ceph as the cinder backend on our OpenStack platform. We use 
CommVault to make our backups.

- Sinan

> Op 24 jul. 2019 om 17:48 heeft Wido den Hollander  het 
> volgende geschreven:
> 
> 
> 
>> On 7/24/19 4:06 PM, Fabian Niepelt wrote:
>> Hi, thanks for the reply.
>> 
>> Am Mittwoch, den 24.07.2019, 15:26 +0200 schrieb Wido den Hollander:
>>> 
>>> On 7/24/19 1:37 PM, Fabian Niepelt wrote:
 Hello ceph-users,
 
 I am currently building a Ceph cluster that will serve as a backend for
 Openstack and object storage using RGW. The cluster itself is finished and
 integrated with Openstack and virtual machines for testing are being
 deployed.
 Now I'm a bit stumped on how to effectively backup the Ceph pools.
 My requirements are two weekly backups, of which one must be offline after
 finishing backing up (systems turned powerless). We are expecting about
 250TB to
 500TB of data for now. The backups must protect against accidental pool
 deletion/corruption or widespread infection of a cryptovirus. In short:
 Complete
 data loss in the production Ceph cluster.
 
 At the moment, I am facing two issues:
 
 1. For the cinder pool, I looked into creating snapshots using the ceph CLI
 (so
 they don't turn up in Openstack and cannot be accidentally deleted by 
 users)
 and
 exporting their diffs. But volumes with snapshots created this way cannot 
 be
 removed from Openstack. Does anyone have an idea how to do this better?
>>> 
>>> You mean that while you leave the snapshot there OpenStack can't remove it?
>> 
>> Yes, that is correct. cinder-volume cannot remove a volume that still has a
>> snapshot. If the snapshot is created by openstack, it will remove the 
>> snapshot
>> before removing the volume. But snapshotting directly from ceph will forego
>> Openstack so it will never know about that snapshot's existence.
>> 
> 
> Ah, yes. That means you would need to remove it manually.
> 
 Alternatively, I could do a full export each week, but I am not sure if 
 that
 would be fast enough..
 
>>> 
>>> It probably won't, but the full backup is still the safest way imho.
>>> However: Does this scale?
>>> 
>>> You can export multiple RBD images in parallel and store them somewhere
>>> else, but it will still take a long time.
>>> 
>>> The export needs to be stored somewhere and then picked up. Or you could
>>> use some magic with Netcat to stream the RBD export to a destination host.
>>> 
>> 
>> Scaling is also my biggest worry about this.
>> 
 2. My search so far has only turned up backing up RBD pools, but how could 
 I
 backup the pools that are used for object storage?
 
>>> 
>>> Not easily. I think you mean RGW? You could try the RGW MultiSite, but
>>> it's difficult.
>>> 
>>> A complete DR with Ceph to restore it back to how it was at a given
>>> point in time is a challenge.
>>> 
>> 
>> Yes, I would like to backup the pools used by the RGW.
> 
> Not really an option. You would need to use the RGW MultiSite to
> replicate all data to a second environment.
> 
>> 
 Of course, I'm also open to completely other ideas on how to backup Ceph 
 and
 would appreciate hearing how you people are doing your backups.
>>> 
>>> A lot of time the backups are created inside the VMs on File level. And
>>> there is a second OpenStack+Ceph system which runs a mirror of the VMs
>>> or application. If one burns down it's not the end of the world.
>>> 
>>> Trying to backup a Ceph cluster sounds very 'enterprise' and is
>>> difficult to scale as well.
>>> 
>> 
>> Are those backups saved in Ceph as well? I cannot solely rely on Ceph 
>> because we
>> want to protect ourselves against failures in Ceph or a human accidentally or
>> maliciously deletes all pools.
>> 
> 
> That second OpenStack+Ceph environment is completely different. All the
> VMs are set up twice and using replication and backups on application
> level such things are redundant.
> 
> Think about MySQL replication for example.
> 
>> From what I'm reading, it seems to be better to maybe implement a backup
>> solution outside of Ceph that our Openstack users can use and not deal with
>> backing up Ceph at all, except its configs to get it running after total
>> desaster...
>> 
> 
> You could backup OpenStack's MySQL database, the ceph config and then
> backup the data inside the VMs.
> 
> It's very difficult to backup data for DR to a certain point of time
> when you go into the >100TB scale.
> 
> Wido
> 
>>> Wido
>>> 
 Any help is much appreciated.
 
 Greetings
 Fabian
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 
> ___
> ceph-users mailing list
> 

Re: [ceph-users] Expected IO in luminous Ceph Cluster

[Sinan Polat]

Hi Felix,

I have 2 Pools, a SSD only and a SAS only pool.

SSD pool is spread over 12 OSD servers.
SAS pool is spread over 6 OSD servers.


See results (SSD Only Pool):

# sysbench --file-fsync-freq=1 --threads=16 fileio --file-total-size=1G
--file-test-mode=rndrw --file-rw-ratio=2 run
sysbench 1.0.17 (using system LuaJIT 2.0.4)

Running the test with following options:
Number of threads: 16
Initializing random number generator from current time


Extra file open flags: (none)
128 files, 8MiB each
1GiB total file size
Block size 16KiB
Number of IO requests: 0
Read/Write ratio for combined random IO test: 2.00
Periodic FSYNC enabled, calling fsync() each 1 requests.
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random r/w test
Initializing worker threads...

Threads started!


File operations:
reads/s:  508.38
writes/s: 254.19
fsyncs/s: 32735.14

Throughput:
read, MiB/s:  7.94
written, MiB/s:   3.97

General statistics:
total time:  10.0103s
total number of events:  36

Latency (ms):
 min:0.00
 avg:0.48
 max:   10.18
 95th percentile:2.11
 sum:   159830.07

Threads fairness:
events (avg/stddev):   20833.5000/335.70
execution time (avg/stddev):   9.9894/0.00
#

See results (SAS Only Pool):
# sysbench --file-fsync-freq=1 --threads=16 fileio --file-total-size=1G
--file-test-mode=rndrw --file-rw-ratio=2 run
sysbench 1.0.17 (using system LuaJIT 2.0.4)

Running the test with following options:
Number of threads: 16
Initializing random number generator from current time


Extra file open flags: (none)
128 files, 8MiB each
1GiB total file size
Block size 16KiB
Number of IO requests: 0
Read/Write ratio for combined random IO test: 2.00
Periodic FSYNC enabled, calling fsync() each 1 requests.
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random r/w test
Initializing worker threads...

Threads started!


File operations:
reads/s:  490.11
writes/s: 245.10
fsyncs/s: 31565.00

Throughput:
read, MiB/s:  7.66
written, MiB/s:   3.83

General statistics:
total time:  10.0143s
total number of events:  321477

Latency (ms):
 min:0.00
 avg:0.50
 max:   20.50
 95th percentile:2.30
 sum:   159830.82

Threads fairness:
events (avg/stddev):   20092.3125/186.66
execution time (avg/stddev):   9.9894/0.00
#


Kind regards,
Sinan Polat



> Op 7 juni 2019 om 12:47 schreef "Stolte, Felix" :
> 
> 
> Hi Sinan,
> 
> that would be great. The numbers should differ a lot, since you have an all
> flash pool, but it would be interesting, what we could expect from such a
> configuration.
> 
> Regards
> Felix
> 
> -
> -
> Forschungszentrum Juelich GmbH
> 52425 Juelich
> Sitz der Gesellschaft: Juelich
> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
> Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
> Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
> Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
> Prof. Dr. Sebastian M. Schmidt
> -
> -----
>  
> 
> Am 07.06.19, 12:02 schrieb "Sinan Polat" :
> 
> Hi Felix,
> 
> I can run your commands inside an OpenStack VM. Tthe storage cluster
> contains of 12 OSD servers, holding each 8x 960GB SSD. Luminous FileStore.
> Replicated 3.
> 
> Would it help you to run your command on my cluster?
> 
> Sinan
> 
> > Op 7 jun. 2019 om 08:52 heeft Stolte, Felix  het
> > volgende geschreven:
> > 
> > I have no performance data before we migrated to bluestore. You should
> > start a separate topic regarding your question.
> > 
> > Could anyone with an more or less equally sized cluster post the output
> > of a sysbench with the following parameters (either from inside an
> 

Re: [ceph-users] Expected IO in luminous Ceph Cluster

[Sinan Polat]

Hi Felix,

I can run your commands inside an OpenStack VM. Tthe storage cluster contains 
of 12 OSD servers, holding each 8x 960GB SSD. Luminous FileStore. Replicated 3.

Would it help you to run your command on my cluster?

Sinan

> Op 7 jun. 2019 om 08:52 heeft Stolte, Felix  het 
> volgende geschreven:
> 
> I have no performance data before we migrated to bluestore. You should start 
> a separate topic regarding your question.
> 
> Could anyone with an more or less equally sized cluster post the output of a 
> sysbench with the following parameters (either from inside an openstack vm or 
> a mounted rbd)?
> 
> sysbench --file-fsync-freq=1 --threads=16 fileio --file-total-size=1G 
>--file-test-mode=rndrw --file-rw-ratio=2 prepare
> 
> sysbench --file-fsync-freq=1 --threads=16 fileio --file-total-size=1G 
>--file-test-mode=rndrw --file-rw-ratio=2 run
> 
> Thanks in advance.
> 
> Regards
> Felix
> 
> -
> -
> Forschungszentrum Juelich GmbH
> 52425 Juelich
> Sitz der Gesellschaft: Juelich
> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
> Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
> Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
> Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
> Prof. Dr. Sebastian M. Schmidt
> -
> -
> 
> 
> Am 06.06.19, 15:09 schrieb "Marc Roos" :
> 
> 
>I am also thinking of moving the wal/db to ssd of the sata hdd's. Did 
>you do tests before and after this change, and know what the difference 
>is iops? And is the advantage more or less when your sata hdd's are 
>slower? 
> 
> 
>-Original Message-
>From: Stolte, Felix [mailto:f.sto...@fz-juelich.de] 
>Sent: donderdag 6 juni 2019 10:47
>To: ceph-users
>Subject: [ceph-users] Expected IO in luminous Ceph Cluster
> 
>Hello folks,
> 
>we are running a ceph cluster on Luminous consisting of 21 OSD Nodes 
>with 9 8TB SATA drives and 3 Intel 3700 SSDs for Bluestore WAL and DB 
>(1:3 Ratio). OSDs have 10Gb for Public and Cluster Network. The cluster 
>is running stable for over a year. We didn’t had a closer look on IO 
>until one of our customers started to complain about a VM we migrated 
>from VMware with Netapp Storage to our Openstack Cloud with ceph 
>storage. He sent us a sysbench report from the machine, which I could 
>reproduce on other VMs as well as on a mounted RBD on physical hardware:
> 
>sysbench --file-fsync-freq=1 --threads=16 fileio --file-total-size=1G 
>--file-test-mode=rndrw --file-rw-ratio=2 run sysbench 1.0.11 (using 
>system LuaJIT 2.1.0-beta3)
> 
>Running the test with following options:
>Number of threads: 16
>Initializing random number generator from current time
> 
>Extra file open flags: 0
>128 files, 8MiB each
>1GiB total file size
>Block size 16KiB
>Number of IO requests: 0
>Read/Write ratio for combined random IO test: 2.00 Periodic FSYNC 
>enabled, calling fsync() each 1 requests.
>Calling fsync() at the end of test, Enabled.
>Using synchronous I/O mode
>Doing random r/w test
> 
>File operations:
>reads/s:  36.36
>writes/s: 18.18
>fsyncs/s: 2318.59
> 
>Throughput:
>read, MiB/s:  0.57
>written, MiB/s:   0.28
> 
>General statistics:
>total time:  10.0071s
>total number of events:  23755
> 
>Latency (ms):
> min:  0.01
> avg:  6.74
> max:   1112.58
> 95th percentile: 26.68
> sum: 160022.67
> 
>Threads fairness:
>events (avg/stddev):   1484.6875/52.59
>execution time (avg/stddev):   10.0014/0.00
> 
>Are these numbers reasonable for a cluster of our size?
> 
>Best regards
>Felix
>IT-Services
>Telefon 02461 61-9243
>E-Mail: f.sto...@fz-juelich.de
>
>-
>
>-
>Forschungszentrum Juelich GmbH
>52425 Juelich
>Sitz der Gesellschaft: Juelich
>Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 
>Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
>Geschaeftsfuehrung: Prof. Dr.-Ing. 

Re: [ceph-users] ceph-ansible as non-root user

[Sinan Polat]

Hi,

Does your ansible user has sudo rights? Without password prompt?

Kind regards,
Sinan Polat

> Op 23 apr. 2019 om 05:00 heeft ST Wong (ITSC)  het 
> volgende geschreven:
> 
> Hi all,
>  
> We tried to deploy a new CEPH cluster using latest ceph-ansible, run as an 
> non-root user (e.g. ansible), and got following error during gathering facts:
>  
> -- cut here --
> TASK [ceph-facts : create a local fetch directory if it does not exist] 
> 
> Tuesday 23 April 2019  09:42:30 +0800 (0:00:01.155)   0:02:03.781 
> *
> fatal: [mon1.mydomain -> localhost]: FAILED! => changed=false
>   module_stderr: |-
> Sorry, try again.
> [sudo via ansible, key=xx] password: Sorry, try again.
> [sudo via ansible, key=xx] password: sudo: 3 
> incorrect password attempts
>   module_stdout: ''
>   msg: |-
> MODULE FAILURE
> See stdout/stderr for the exact error
>   rc: 1
> -- cut here --
>  
> However, it’s okay if running as root user on the admin server.  
> We tried to setup CEPH cluster sometime last user using similar method and 
> there was no problem to deploy as non-root user.  
> Did we miss anything or if there is change in deployment requirement?  
>  
> We’re using ansible 2.7.10.
>  
> Thanks a lot.
> /st wong
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph expansion/deploy via ansible

[Sinan Polat]

I have deployed, expanded and upgraded multiple Ceph clusters using 
ceph-ansible. Works great.

What information are you looking for?

--
Sinan

> Op 17 apr. 2019 om 16:24 heeft Francois Lafont 
>  het volgende geschreven:
> 
> Hi,
> 
> +1 for ceph-ansible too. ;)
> 
> -- 
> François (flaf)
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] 'Missing' capacity

[Sinan Polat]

Probably inbalance of data across your OSDs.

Could you show ceph osd df.

>From there take the disk with lowest available space. Multiply that number 
>with number of OSDs. How much is it?

Kind regards,
Sinan Polat

> Op 16 apr. 2019 om 05:21 heeft Igor Podlesny  het 
> volgende geschreven:
> 
>> On Tue, 16 Apr 2019 at 06:43, Mark Schouten  wrote:
>> [...]
>> So where is the rest of the free space? :X
> 
> Makes sense to see:
> 
> sudo ceph osd df tree
> 
> -- 
> End of message. Next message?
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] BlueStore / OpenStack Rocky performance issues

[Sinan Polat]

Hi Eric,

40% slower performance compared to ..? Could you please share the current
performance. How many OSD nodes do you have?

Regards,
Sinan

> Op 21 februari 2019 om 14:19 schreef "Smith, Eric" :
> 
> 
> Hey folks – I recently deployed Luminous / BlueStore on SSDs to back an
> OpenStack cluster that supports our build / deployment infrastructure and I’m
> getting 40% slower build times. Any thoughts on what I may need to do with
> Ceph to speed things up? I have 30 SSDs backing an 11 compute node cluster.
> 
>  
> 
> Eric
> 


 

> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Explanation of perf dump of rbd

[Sinan Polat]

Thanks for the clarification!

Great that the next release will include the feature. We are running on Red Hat 
Ceph, so we might have to wait longer before having the feature available.

Another related (simple) question:
We are using
/var/run/ceph/$cluster-$type.$id.$pid.$cctid.asok
in ceph.conf, can we include the volume name in the path?

Sinan

> Op 1 feb. 2019 om 00:44 heeft Jason Dillaman  het 
> volgende geschreven:
> 
>> On Thu, Jan 31, 2019 at 12:16 PM Paul Emmerich  
>> wrote:
>> 
>> "perf schema" has a description field that may or may not contain
>> additional information.
>> 
>> My best guess for these fields would be bytes read/written since
>> startup of this particular librbd instance. (Based on how these
>> counters usually work)
> 
> Correct -- they should be strictly increasing while the image is
> in-use. If you periodically scrape the values (along w/ the current
> timestamp), you can convert these values to the rates between the
> current and previous metrics.
> 
> On a semi-related subject: the forthcoming Nautilus release will
> include new "rbd perf image iotop" and "rbd perf image iostat"
> commands to monitor metrics by RBD image.
> 
>> Paul
>> 
>> --
>> Paul Emmerich
>> 
>> Looking for help with your Ceph cluster? Contact us at https://croit.io
>> 
>> croit GmbH
>> Freseniusstr. 31h
>> 81247 München
>> www.croit.io
>> Tel: +49 89 1896585 90
>> 
>>> On Thu, Jan 31, 2019 at 3:41 PM Sinan Polat  wrote:
>>> 
>>> Hi,
>>> 
>>> I finally figured out how to measure the statistics of a specific RBD 
>>> volume;
>>> 
>>> $ ceph --admin-daemon  perf dump
>>> 
>>> 
>>> It outputs a lot, but I don't know what it means, is there any 
>>> documentation about the output?
>>> 
>>> For now the most important values are:
>>> 
>>> - bytes read
>>> 
>>> - bytes written
>>> 
>>> 
>>> I think I need to look at this:
>>> 
>>> {
>>> "rd": 1043,
>>> "rd_bytes": 28242432,
>>> "rd_latency": {
>>> "avgcount": 1768,
>>> "sum": 2.375461133,
>>> "avgtime": 0.001343586
>>> },
>>> "wr": 76,
>>> "wr_bytes": 247808,
>>> "wr_latency": {
>>> "avgcount": 76,
>>> "sum": 0.970222300,
>>> "avgtime": 0.012766082
>>> }
>>> }
>>> 
>>> 
>>> But what is 28242432 (rd_bytes) and 247808 (wr_bytes). Is that 28242432 
>>> bytes read and 247808 bytes written during the last minute/hour/day? Or is 
>>> it since mounted, or...?
>>> 
>>> 
>>> Thanks!
>>> 
>>> 
>>> Sinan
>>> 
>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> 
> 
> -- 
> Jason

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Explanation of perf dump of rbd

[Sinan Polat]

Hi,


I finally figured out how to measure the statistics of a specific RBD volume;

$ ceph --admin-daemon  perf dump


It outputs a lot, but I don't know what it means, is there any documentation
about the output?

For now the most important values are:

- bytes read

- bytes written


I think I need to look at this:

{
"rd": 1043,
"rd_bytes": 28242432,
"rd_latency": {
"avgcount": 1768,
"sum": 2.375461133,
"avgtime": 0.001343586
},
"wr": 76,
"wr_bytes": 247808,
"wr_latency": {
"avgcount": 76,
"sum": 0.970222300,
"avgtime": 0.012766082
}
}


But what is 28242432 (rd_bytes) and 247808 (wr_bytes). Is that 28242432 bytes
read and 247808 bytes written during the last minute/hour/day? Or is it since
mounted, or...?


Thanks!


Sinan___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] utilization of rbd volume

[Sinan Polat]

Hi Jason,

Thanks for your reply.

Unfortunately we do not have access to the clients.

We are running Red Hat Ceph 2.x which is based on Jewel, that means we cannot 
pinpoint who or what is causing the load on the cluster, am I right?

Thanks!
Sinan

> Op 28 dec. 2018 om 15:14 heeft Jason Dillaman  het 
> volgende geschreven:
> 
> With the current releases of Ceph, the only way to accomplish this is
> by gathering the IO stats on each client node. However, with the
> future Nautilus release, this data will now be available directly from
> the OSDs.
> 
>> On Fri, Dec 28, 2018 at 6:18 AM Sinan Polat  wrote:
>> 
>> Hi all,
>> 
>> We have a couple of hundreds RBD volumes/disks in our Ceph cluster, each RBD 
>> disk is mounted by a different client. Currently we see quite high IOPS 
>> happening on the cluster, but we don't know which client/RBD is causing it.
>> 
>> Is it somehow easily to see the utilization per RBD disk?
>> 
>> Thanks!
>> Sinan
>> 
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> 
> 
> -- 
> Jason

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] utilization of rbd volume

[Sinan Polat]

Hi all,

We have a couple of hundreds RBD volumes/disks in our Ceph cluster, each RBD
disk is mounted by a different client. Currently we see quite high IOPS
happening on the cluster, but we don't know which client/RBD is causing it.

Is it somehow easily to see the utilization per RBD disk?

Thanks!
Sinan___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Memory configurations

[Sinan Polat]

I don’t know about the memory, but your CPU’s would be overkill. For what would 
you need 20 cores (40 threads)?

When using 2 sockets I would go for 2 memory modules. Does it even work with 
just 1 module?

Regards,
Sinan

> Op 21 nov. 2018 om 22:30 heeft Georgios Dimitrakakis  
> het volgende geschreven:
> 
> Hello,
> 
> I would like to see people's opinion about memory configurations.
> Would you prefer 2x8GB over 1x16GB or the opposite?
> 
> In addition what are the latest memory recommendations? Should we should keep 
> the rule of thumb of 1GB per TB
> or now with Bluestore things have changed?
> 
> I am planning to build from scratch a small new cluster with 3replicas on 
> 3OSD nodes with 4TB (2x2TB SSDs) per node, for now. The plan is to expand the 
> cluster in the near future by adding more 2TB SSDs. OSD nodes will also be 
> monitor nodes.
> The primary usage will be for RBD provision to OpenStack.
> 
> Are 16GB enough for this and on what configuration (2x8GB vs 1x16GB)? Should 
> I go better for 32GB and what is the preferred configuration (4x8GB vs 
> 1x32GB) in this case.
> 
> As for the CPUs the plan is for dual socket with 2x Intel Xeon Silver 4114 
> 2.2G, 10C/20T
> 
> Looking forward for your opinions!
> 
> Best regards,
> 
> G.
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Degraded data redundancy (low space): 1 pg backfill_toofull

[Sinan Polat]

Ceph has tried to (re)balance your data, backfill_toofull means no available 
space to move data, but you have plenty of space.

Why do you have so little pgs? I would increase the amount of pgs, but before 
doing so lets see what others will say.

Sinan

> Op 28 jul. 2018 om 11:50 heeft Sebastian Igerl  het 
> volgende geschreven:
> 
> Hi,
> 
> i added 4 more OSDs on my 4 node Test Cluster and now i'm in HEALTH_ERR 
> state. Right now its still recovering, but still, should this happen ? None 
> of my OSDs are full. Maybe i need more PGs ? But since my %USE is < 40% it 
> should be still ok to recover without HEALTH_ERR ?
> 
>   data:
> pools:   7 pools, 484 pgs
> objects: 2.70 M objects, 10 TiB
> usage:   31 TiB used, 114 TiB / 146 TiB avail
> pgs: 2422839/8095065 objects misplaced (29.930%)
>  343 active+clean
>  101 active+remapped+backfill_wait
>  39  active+remapped+backfilling
>  1   active+remapped+backfill_wait+backfill_toofull
> 
>   io:
> recovery: 315 MiB/s, 78 objects/s
> 
> 
> 
> 
> 
> ceph osd df
> ID CLASS WEIGHT  REWEIGHT SIZEUSE AVAIL   %USE  VAR  PGS
>  0   hdd 2.72890  1.0 2.7 TiB 975 GiB 1.8 TiB 34.89 1.62  31
>  1   hdd 2.72899  1.0 2.7 TiB 643 GiB 2.1 TiB 23.00 1.07  36
>  8   hdd 7.27739  1.0 7.3 TiB 1.7 TiB 5.5 TiB 23.85 1.11  83
> 12   hdd 7.27730  1.0 7.3 TiB 1.1 TiB 6.2 TiB 14.85 0.69  81
> 16   hdd 7.27730  1.0 7.3 TiB 2.0 TiB 5.3 TiB 27.68 1.29  74
> 20   hdd 9.09569  1.0 9.1 TiB 108 GiB 9.0 TiB  1.16 0.05  43
>  2   hdd 2.72899  1.0 2.7 TiB 878 GiB 1.9 TiB 31.40 1.46  36
>  3   hdd 2.72899  1.0 2.7 TiB 783 GiB 2.0 TiB 28.02 1.30  39
>  9   hdd 7.27739  1.0 7.3 TiB 2.0 TiB 5.3 TiB 27.58 1.28  85
> 13   hdd 7.27730  1.0 7.3 TiB 2.2 TiB 5.1 TiB 30.10 1.40  78
> 17   hdd 7.27730  1.0 7.3 TiB 2.1 TiB 5.2 TiB 28.23 1.31  84
> 21   hdd 9.09569  1.0 9.1 TiB 192 GiB 8.9 TiB  2.06 0.10  41
>  4   hdd 2.72899  1.0 2.7 TiB 927 GiB 1.8 TiB 33.18 1.54  34
>  5   hdd 2.72899  1.0 2.7 TiB 1.0 TiB 1.7 TiB 37.57 1.75  28
> 10   hdd 7.27739  1.0 7.3 TiB 2.2 TiB 5.0 TiB 30.66 1.43  87
> 14   hdd 7.27730  1.0 7.3 TiB 1.8 TiB 5.5 TiB 24.23 1.13  89
> 18   hdd 7.27730  1.0 7.3 TiB 2.5 TiB 4.8 TiB 33.83 1.57  93
> 22   hdd 9.09569  1.0 9.1 TiB 210 GiB 8.9 TiB  2.26 0.10  44
>  6   hdd 2.72899  1.0 2.7 TiB 350 GiB 2.4 TiB 12.51 0.58  21
>  7   hdd 2.72899  1.0 2.7 TiB 980 GiB 1.8 TiB 35.07 1.63  35
> 11   hdd 7.27739  1.0 7.3 TiB 2.8 TiB 4.4 TiB 39.14 1.82  99
> 15   hdd 7.27730  1.0 7.3 TiB 1.6 TiB 5.6 TiB 22.49 1.05  82
> 19   hdd 7.27730  1.0 7.3 TiB 2.1 TiB 5.2 TiB 28.49 1.32  77
> 23   hdd 9.09569  1.0 9.1 TiB 285 GiB 8.8 TiB  3.06 0.14  52
> TOTAL 146 TiB  31 TiB 114 TiB 21.51
> MIN/MAX VAR: 0.05/1.82  STDDEV: 11.78
> 
> 
> 
> 
> Right after adding the osds it showed degraded for a few minutes, since all 
> my pools have a redundancy of 3 and i'm adding osd i'm a bit confused why 
> this happens ? I get why it's misplaced, but undersized and degraded ?
> 
> pgs: 4611/8095032 objects degraded (0.057%)
>  2626460/8095032 objects misplaced (32.445%)
>  215 active+clean
>  192 active+remapped+backfill_wait
>  26  active+recovering+undersized+remapped
>  17  active+recovery_wait+undersized+degraded+remapped
>  16  active+recovering
>  11  active+recovery_wait+degraded
>  6   active+remapped+backfilling
>  1   active+remapped+backfill_toofull
> 
> 
> Maybe someone can give me some pointers on what i'm missing to understand 
> whats happening here ?
> 
> Thanks!
> 
> Sebastian
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] IO rate-limiting with Ceph RBD (and libvirt)

[Sinan Polat]

FYI: I/O limiting in combination with OpenStack 10/12 + Ceph doesn’t work 
properly. Bug: https://bugzilla.redhat.com/show_bug.cgi?id=1476830

> Op 22 mrt. 2018 om 07:59 heeft Wido den Hollander  het 
> volgende geschreven:
> 
> 
> 
>> On 03/21/2018 06:48 PM, Andre Goree wrote:
>> I'm trying to determine the best way to go about configuring IO
>> rate-limiting for individual images within an RBD pool.
>> 
>> Here [1], I've found that OpenStack appears to use Libvirt's "iotune"
>> parameter, however I seem to recall reading about being able to do so
>> via Ceph's settings.
>> 
>> Is there a place in Ceph to set IO limits on individual images within an
>> RBD pool?  Thanks in advance for the advice.
> 
> No, there is not. Right now you will need to limit this through
> libvirt/Qemu indeed.
> 
> People are thinking about a QoS mechanism inside the OSDs, but that's
> not there yet, so don't count on it.
> 
> Wido
> 
>> 
>> [1] https://ceph.com/geen-categorie/openstack-ceph-rbd-and-qos/
>> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Configuring Ceph using multiple networks

[Sinan Polat]

Why do your put your mons inside your cluster network, shouldn't they reisde 
within the public network?

Cluster network is only for replica data / for traffic between your osds.

> Op 7 okt. 2017 om 14:32 heeft Kashif Mumtaz  het 
> volgende geschreven:
> 
> 
> 
> I have successfully installed Luminous on Ubutnu  16.04 by one network.
>  
> Now I am trying to install same by using two networks this.(on different 
> machine)
> public network =   192.168.10.0/24
> cluster network =  172.16.50.0/24
>  
> Eeach node has two interfaces. One in public network other in cluster network.
>  
>  
> While creating the initial monitors  facing the the below error
>  
> “Some monitors have still not reached quorum”
>  
> Below is /etc/hosts file on each host
> 172.16.50.1 mon1
> 172.16.50.2 mon2
> 172.16.50.3 mon3
> 172.16.50.4 osd1
> 172.16.50.5 osd2
>  
> This is ceph.conf  file
> mon1:/home/cephadmin/my-cluster# cat ceph.conf
> [global]
> fsid = 7f0ffa2b-7528-407b-8a63-360741d80939
> mon_initial_members = mon1, mon2, mon3
> mon_host = 172.16.50.1,172.16.50.2,172.16.50.3
> auth_cluster_required = cephx
> auth_service_required = cephx
> auth_client_required = cephx
>  
> # Public and cluster network
> public network =   192.168.10.0/24
> cluster network =  172.16.50.0/24
> # Write an object 2 times
> osd pool default size = 2
> # 1 for a multi node cluster in a single rack
> osd crush chooseleaf type = 1
>  
>  
>  
> While searching on net , I found same error discussed in below post
>  
> https://www.spinics.net/lists/ceph-users/msg24603.html
>  
> I am observing same error in my case  that during ceph-deploy mon 
> create-initial  all devices name are not resolving properly.
>  
> Log:
> [2017-10-07 00:31:18,975][mon1][DEBUG ] status for monitor: mon.mon1
> [2017-10-07 00:31:18,976][mon1][DEBUG ] {
> [2017-10-07 00:31:18,976][mon1][DEBUG ]   "election_epoch": 0,
> [2017-10-07 00:31:18,976][mon1][DEBUG ]   "extra_probe_peers": [
> [2017-10-07 00:31:18,976][mon1][DEBUG ] "172.16.50.1:6789/0",
> [2017-10-07 00:31:18,977][mon1][DEBUG ] "172.16.50.2:6789/0",
> [2017-10-07 00:31:18,977][mon1][DEBUG ] "172.16.50.3:6789/0
> [2017-10-07 00:31:18,982][mon1][DEBUG ] "addr": 
> "192.168.10.31:6789/0",
> [2017-10-07 00:31:18,982][mon1][DEBUG ] "name": "mon1",
> [2017-10-07 00:31:18,982][mon1][DEBUG ] "public_addr": 
> "192.168.10.31:6789/0",
> [2017-10-07 00:31:18,982][mon1][DEBUG ] "rank": 0
> [2017-10-07 00:31:18,983][mon1][DEBUG ]   },
> [2017-10-07 00:31:18,983][mon1][DEBUG ]   {
> [2017-10-07 00:31:18,983][mon1][DEBUG ] "addr": "0.0.0.0:0/1",
> [2017-10-07 00:31:18,983][mon1][DEBUG ] "name": "mon2",
> [2017-10-07 00:31:18,983][mon1][DEBUG ] "public_addr": "0.0.0.0:0/1",
> [2017-10-07 00:31:18,983][mon1][DEBUG ] "rank": 1
> [2017-10-07 00:31:18,984][mon1][DEBUG ]   },
> [2017-10-07 00:31:18,984][mon1][DEBUG ]   {
> [2017-10-07 00:31:18,984][mon1][DEBUG ] "addr": "0.0.0.0:0/2",
> [2017-10-07 00:31:18,984][mon1][DEBUG ] "name": "mon3",
> [2017-10-07 00:31:18,984][mon1][DEBUG ] "public_addr": "0.0.0.0:0/2",
> [2017-10-07 00:31:18,985][mon1][DEBUG ] "rank": 2
>  
>  
>  
> I configured the DNS also as suggested solution in above link, but  issue did 
> not resolve.
>  
> Can some one help in this regard?
>  
> I am not sue in DNSlooup name should be resolve to node’s public IP or 
> cluster IP?
>  
> In /etc/hosts file which IP of nodes should use cluster or public ?
>  
>  
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] PGs get placed in the same datacenter (Trying to make a hybrid NVMe/HDD pool with 6 servers, 2 in each datacenter)

[Sinan Polat]

You are talking about the min_size, which should be 2 according to your text.

Please be aware, the min_size in your CRUSH is _not_ the replica size. The 
replica size is set with your pools.

> Op 7 okt. 2017 om 19:39 heeft Peter Linder  het 
> volgende geschreven:
> 
>> On 10/7/2017 7:36 PM, Дробышевский, Владимир wrote:
>> Hello!
>> 
>> 2017-10-07 19:12 GMT+05:00 Peter Linder 
>> :
>> 
>>> The idea is to select an nvme osd, and
>>> then select the rest from hdd osds in different datacenters (see crush
>>> map below for hierarchy). 
>>> 
>> It's a little bit aside of the question, but why do you want to mix SSDs and 
>> HDDs in the same pool? Do you have read-intensive workload and going to use 
>> primary-affinity to get all reads from nvme?
>>  
>> 
> Yes, this is pretty much the idea, getting the performance from NVMe 
> reads, while still maintaining triple redundancy and a reasonable cost. 
> 
> 
>> -- 
>> Regards,
>> Vladimir
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Usage not balanced over OSDs

[Sinan Polat]

Hi,

 

I have 52 OSD's in my cluster, all with the same disk size and same weight.

 

When I perform a:

ceph osd df

 

The disk with the least available space: 863G

The disk with the most available space: 1055G

 

I expect the available space or the usage on the disks to be the same, since
they have the same weight, but there is a difference of almost 200GB.

 

Due to this, the MAX AVAIL in ceph df is lower than expected (the MAX AVAIL
is based on the disk with the least available space).

 

-  How can I balance the disk usage over the disks, so the usage /
available space on each disk is more or less the same?

-  What will happen if I hit the MAX AVAIL, while most of the disks
still have space?

 

Thanks!

Sinan

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] What's 'failsafe full'

[Sinan Polat]

Hi

 

According to: 
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2013-July/003140.html

 

You can set it with:

on the OSDs you may (not) want to change "osd failsafe full ratio" and "osd 
failsafe nearfull ratio".

 

Van: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] Namens dE
Verzonden: woensdag 13 september 2017 16:39
Aan: ceph-users@lists.ceph.com
Onderwerp: [ceph-users] What's 'failsafe full'

 

Hello everyone,

Just started with Ceph here.

I was reading the documentation here -- 

http://docs.ceph.com/docs/master/rados/operations/health-checks/#osd-out-of-order-full

And just started to wonder what's failsafe_full... I know it's some kind of 
ratio, but how do I change it? I didn't find anything on google 

 .

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] MAX AVAIL in ceph df

[Sinan Polat]

Hi,

 

How is the MAX AVAIL calculated in 'ceph df'? Since I am missing some space.

 

I have 26 OSD's, each is 1484GB (according to df). I have 3 replica's.
Shouldn't the MAX AVAIL be: (26*1484)/3 = 12.861GB?

Instead 'ceph df' is showing 7545G for the pool that is using the 26 OSD's.

 

What is wrong with my calculation?

 

Thanks!

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Ceph df incorrect pool size (MAX AVAIL)

[Sinan Polat]

Hi,

 

I am quite new to Ceph, so forgive me for any stupid questions.

 

Setup:

-  2 datacenters, each datacenter has 26 OSD's, which makes in total
52 OSD's.

-  ceph osd df shows that every disk is 1484GB.

-  I have 2 rulesets and 4 pools, 1 ruleset + 2 pools per
datacenter. Each ruleset has 26 OSD's.

-  52 OSD * 1484GB makes: 77.168GB.

 

To my understanding - ceph df - should show 4 pools and the MAX AVAIL of
each pool should be the size of all OSDs together in the ruleset:

77168GB / 3 replicas = 25.722GB / 2 rulesets = 12.861GB.

 

But in my case, ceph df shows indeed the 4 pools, but the MAX AVAIL of each
pool is 7545G and not 12861GB. What is wrong with my calculation?

 

When I multiply 7545G with 3 replicas = 22.635 * 2 rulesets = 45.270GB.

 

Thanks!

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ruleset vs replica count

[Sinan Polat]

Hi David,

 

Thank you for your reply, I will read about the min_size 1 value.

 

What about my initial question, anyone?

 

Thanks!

 

Van: David Turner [mailto:drakonst...@gmail.com] 
Verzonden: donderdag 24 augustus 2017 19:45
Aan: Sinan Polat; ceph-us...@ceph.com
Onderwerp: Re: [ceph-users] Ruleset vs replica count

 

> min_size 1

STOP THE MADNESS.  Search the ML to realize why you should never user a 
min_size of 1.

 

I'm curious as well as to what this sort of configuration will do for how many 
copies are stored between DCs.

 

On Thu, Aug 24, 2017 at 1:03 PM Sinan Polat <si...@turka.nl> wrote:

Hi,

 

In a Multi Datacenter Cluster I have the following rulesets:

--

rule ams5_ssd {

ruleset 1

type replicated

min_size 1

max_size 10

step take ams5-ssd

step chooseleaf firstn 2 type host

step emit

step take ams6-ssd

step chooseleaf firstn -2 type host

step emit

}

rule ams6_ssd {

ruleset 2

type replicated

min_size 1

max_size 10

step take ams6-ssd

step chooseleaf firstn 2 type host

step emit

step take ams5-ssd

step chooseleaf firstn -2 type host

step emit

}

--

 

The replication size is set to 3.

 

When for example ruleset 1 is used, how is the replication being done? Does it 
store 2 replica’s in ams5-ssd and store 1 replica in ams6-ssd? Or does it store 
3 replicas in ams5-ssd and 3 replicas in ams6-ssd?

 

Thanks!

 

Sinan

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Ruleset vs replica count

[Sinan Polat]

Hi,

 

In a Multi Datacenter Cluster I have the following rulesets:

--

rule ams5_ssd {

ruleset 1

type replicated

min_size 1

max_size 10

step take ams5-ssd

step chooseleaf firstn 2 type host

step emit

step take ams6-ssd

step chooseleaf firstn -2 type host

step emit

}

rule ams6_ssd {

ruleset 2

type replicated

min_size 1

max_size 10

step take ams6-ssd

step chooseleaf firstn 2 type host

step emit

step take ams5-ssd

step chooseleaf firstn -2 type host

step emit

}

--

 

The replication size is set to 3.

 

When for example ruleset 1 is used, how is the replication being done? Does
it store 2 replica's in ams5-ssd and store 1 replica in ams6-ssd? Or does it
store 3 replicas in ams5-ssd and 3 replicas in ams6-ssd?

 

Thanks!

 

Sinan

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] mon osd down out subtree limit default

[Sinan Polat]

The docs is providing the following information:
The smallest CRUSH unit type that Ceph will notautomatically mark out. For 
instance, if set to host and if all OSDs of a host are down, Ceph will not 
automatically mark out these OSDs.

But what does it exactly mean? Anyone who can explain it? Thanks!

> Op 21 aug. 2017 om 17:56 heeft Scottix  het volgende 
> geschreven:
> 
> Great to hear.
> 
> Best
> 
>> On Mon, Aug 21, 2017 at 8:54 AM John Spray  wrote:
>> On Mon, Aug 21, 2017 at 4:34 PM, Scottix  wrote:
>> > I don't want to hijack another thread so here is my question.
>> > I just learned about this option from another thread and from my
>> > understanding with our Ceph cluster that we have setup, the default value 
>> > is
>> > not good. Which is "rack" and I should have it on "host".
>> > Which comes to my point why is it set to rack? To be on the safer side
>> > wouldn't the option make more sense as host as default? Then if you are 
>> > rack
>> > aware then you can change the default.
>> 
>> Yes!
>> 
>> As it happens, we (Sage was in the room, not sure who else) talked
>> about this recently, and the idea was to make the default conditional
>> depending on system size.  So for smallish systems, we would set it to
>> host, and on larger systems it would be rack.
>> 
>> John
>> 
>> >
>> > Best,
>> > Scott
>> >
>> >
>> > ___
>> > ceph-users mailing list
>> > ceph-users@lists.ceph.com
>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> >
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph cluster with SSDs

[Sinan Polat]

What has DWPD to do with performance / IOPS? The SSD will just fail earlier, 
but it should not have any affect on the performance, right?

Correct me if I am wrong, just want to learn.


> Op 20 aug. 2017 om 06:03 heeft Christian Balzer  het volgende 
> geschreven:
> 
> DWPD

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Book & questions

[Sinan Polat]

Hi all,

 

I am quite new with Ceph Storage. Currently we have a Ceph environment
running, but in a few months we will be setting up a new Ceph storage
environment.

 

I have read a lot of information on the Ceph website, but the more
information the better for me. What book(s) would you suggest?

 

I found the following books:

Learning Ceph - Karan Singh (Jan 2015)

Ceph Cookbook - Karan Singh (Feb 2016)

Mastering Ceph - Nick Fisk (May 2017)

 

Another question;

Ceph is self-healing, it will distribute the replicas to the available OSD's
in case of a failure of one of the OSD's. Lets say my setup is configured to
have 3 replicas, this means when there is a failure of one the OSD's it will
start healing. I want that when an OSD fails and only 2 replicas are left,
it shouldn't do anything, only when also the 2nd OSD fails it should start
replicating/healing. Which configuration setting do I need to use, is it the
min size option?

 

Thanks!

 

 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com