Re: [ceph-users] Fwd: HW failure cause client IO drops
Good morning, the OSDs are usually marked out after 10 minutes, that's when rebalancing starts. But the I/O should not drop during that time, this could be related to your pool configuration. If you have a replicated pool of size 3 and also set min_size to 3 the I/O would pause if a node or OSD fails. So more information about the cluster would help, can you share that? ceph osd tree ceph osd pool ls detail Were all pools affected or just specific pools? Regards, Eugen Zitat von M Ranga Swami Reddy : Hello - Recevenlt we had an issue with storage node's battery failure, which cause ceph client IO dropped to '0' bytes. Means ceph cluster couldn't perform IO operations on the cluster till the node takes out. This is not expected from Ceph, as some HW fails, those respective OSDs should mark as out/down and IO should go as is.. Please let me know if anyone seen the similar behavior and is this issue resolved? Thanks Swami ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Multi-site replication speed
> On Apr 15, 2019, at 5:18 PM, Brian Topping wrote: > > If I am correct, how do I trigger the full sync? Apologies for the noise on this thread. I came to discover the `radosgw-admin [meta]data sync init` command. That’s gotten me with something that looked like this for several hours: > [root@master ~]# radosgw-admin sync status > realm 54bb8477-f221-429a-bbf0-76678c767b5f (example) > zonegroup 8e33f5e9-02c8-4ab8-a0ab-c6a37c2bcf07 (us) >zone b6e32bc8-f07e-4971-b825-299b5181a5f0 (secondary) > metadata sync preparing for full sync > full sync: 64/64 shards > full sync: 0 entries to sync > incremental sync: 0/64 shards > metadata is behind on 64 shards > behind shards: > [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63] > data sync source: 35835cb0-4639-43f4-81fd-624d40c7dd6f (master) > preparing for full sync > full sync: 1/128 shards > full sync: 0 buckets to sync > incremental sync: 127/128 shards > data is behind on 1 shards > behind shards: [0] I also had the data sync showing a list of “behind shards”, but both of them sat in “preparing for full sync” for several hours, so I tried `radosgw-admin [meta]data sync run`. My sense is that was a bad idea, but neither of the commands seem to be documented and the thread I found them on indicated they wouldn’t damage the source data. QUESTIONS at this point: 1) What is the best sequence of commands to properly start the sync? Does init just set things up and do nothing until a run is started? 2) Are there commands I should run before that to clear out any previous bad runs? Thanks very kindly for any assistance. As I didn’t really see any documentation outside of setting up the realms/zones/groups, it seems like this would be useful information for others that follow. best, Brian___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Is it possible to run a standalone Bluestore instance?
Hi, I'd like to run a standalone Bluestore instance so as to test and tune its performance. Are there any tools about it, or any suggestions? Best, Can Zhang ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] 'Missing' capacity
Probably inbalance of data across your OSDs. Could you show ceph osd df. >From there take the disk with lowest available space. Multiply that number >with number of OSDs. How much is it? Kind regards, Sinan Polat > Op 16 apr. 2019 om 05:21 heeft Igor Podlesny het > volgende geschreven: > >> On Tue, 16 Apr 2019 at 06:43, Mark Schouten wrote: >> [...] >> So where is the rest of the free space? :X > > Makes sense to see: > > sudo ceph osd df tree > > -- > End of message. Next message? > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] 'Missing' capacity
On Tue, 16 Apr 2019 at 06:43, Mark Schouten wrote: [...] > So where is the rest of the free space? :X Makes sense to see: sudo ceph osd df tree -- End of message. Next message? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] 'Missing' capacity
Hi, I have a cluster with 97TiB of storage and one pool on it, size 3, using 17.4TiB, totalling at 52.5TiB in use on the cluster. However, I feel that that should leave me with 45TiB/3=15TiB available, but Ceph tells me the pool only has 4.57TiB max available, as you can see below. root@proxmox01:~# ceph osd pool ls detail pool 3 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 2048 pgp_num 2048 last_change 21954 flags hashpspool min_write_recency_for_promote 1 stripe_width 0 application rbd removed_snaps [1~55,57~2,5a~1,5d~1b,7a~15,90~1,98~6] root@proxmox01:~# ceph df detail GLOBAL: SIZE AVAIL RAW USED %RAW USED OBJECTS 97.4TiB 45.0TiB 52.5TiB 53.84 4.59M POOLS: NAME ID QUOTA OBJECTS QUOTA BYTES USED %USED MAX AVAIL OBJECTS DIRTY READ WRITE RAW USED rbd 3 N/A N/A 17.4TiB 79.20 4.57TiB 4587951 4.59M 1.16GiB 4.44GiB 52.2TiB So where is the rest of the free space? :X -- Mark Schouten Tuxis, Ede, https://www.tuxis.nl T: +31 318 200208 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Multi-site replication speed
I’m starting to wonder if I actually have things configured and working correctly, but the light traffic I am seeing is that of an incremental replication. That would make sense, the cluster being replicated does not have a lot of traffic on it yet. Obviously, without the full replication, the incremental is pretty useless. Here’s the status coming from the secondary side: > [root@secondary ~]# radosgw-admin sync status > realm 54bb8477-f221-429a-bbf0-76678c767b5f (example) > zonegroup 8e33f5e9-02c8-4ab8-a0ab-c6a37c2bcf07 (us) >zone b6e32bc8-f07e-4971-b825-299b5181a5f0 (secondary) > metadata sync syncing > full sync: 0/64 shards > incremental sync: 64/64 shards > metadata is caught up with master > data sync source: 35835cb0-4639-43f4-81fd-624d40c7dd6f (master) > syncing > full sync: 0/128 shards > incremental sync: 128/128 shards > data is caught up with source If I am correct, how do I trigger the full sync? Thanks!! Brian ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] showing active config settings
On Tue, Apr 16, 2019 at 7:38 AM solarflow99 wrote: > > Then why doesn't this work? > > # ceph tell 'osd.*' injectargs '--osd-recovery-max-active 4' > osd.0: osd_recovery_max_active = '4' (not observed, change may require > restart) > osd.1: osd_recovery_max_active = '4' (not observed, change may require > restart) > osd.2: osd_recovery_max_active = '4' (not observed, change may require > restart) > osd.3: osd_recovery_max_active = '4' (not observed, change may require > restart) > osd.4: osd_recovery_max_active = '4' (not observed, change may require > restart) > > # ceph -n osd.1 --show-config | grep osd_recovery_max_active > osd_recovery_max_active = 3 Did you try "config diff" as Paul suggested? > > > > On Wed, Apr 10, 2019 at 7:21 AM Eugen Block wrote: >> >> > I always end up using "ceph --admin-daemon >> > /var/run/ceph/name-of-socket-here.asok config show | grep ..." to get what >> > is in effect now for a certain daemon. >> > Needs you to be on the host of the daemon of course. >> >> Me too, I just wanted to try what OP reported. And after trying that, >> I'll keep it that way. ;-) >> >> >> Zitat von Janne Johansson : >> >> > Den ons 10 apr. 2019 kl 13:37 skrev Eugen Block : >> > >> >> > If you don't specify which daemon to talk to, it tells you what the >> >> > defaults would be for a random daemon started just now using the same >> >> > config as you have in /etc/ceph/ceph.conf. >> >> >> >> I tried that, too, but the result is not correct: >> >> >> >> host1:~ # ceph -n osd.1 --show-config | grep osd_recovery_max_active >> >> osd_recovery_max_active = 3 >> >> >> > >> > I always end up using "ceph --admin-daemon >> > /var/run/ceph/name-of-socket-here.asok config show | grep ..." to get what >> > is in effect now for a certain daemon. >> > Needs you to be on the host of the daemon of course. >> > >> > -- >> > May the most significant bit of your life be positive. >> >> >> >> ___ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Cheers, Brad ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph Object storage for physically separating tenants storage infrastructure
On Sat, Apr 13, 2019 at 9:42 AM Varun Singh wrote: > > Thanks Greg. A followup question. Will Zone, ZoneGroup and Realm come > into picture? While reading the documentation, I inferred that by > setting different Realms, I should be able to achieve the desired > result. Is that incorrect? I think they will come in and you're correct, but I haven't worked with RGW in years so it's a bit out of my wheelhouse. -Greg > > -- > Regards, > Varun Singh > > On Sat, Apr 13, 2019 at 12:50 AM Gregory Farnum wrote: > > > > Yes, you would do this by setting up separate data pools for segregated > > clients, giving those pools a CRUSH rule placing them on their own servers, > > and if using S3 assigning the clients to them using either wholly separate > > instances or perhaps separate zones and the S3 placement options. > > -Greg > > > > On Fri, Apr 12, 2019 at 3:04 AM Varun Singh wrote: > >> > >> Hi, > >> We have a requirement to build an object storage solution with thin > >> layer of customization on top. This is to be deployed in our own data > >> centre. We will be using the objects stored in this system at various > >> places in our business workflow. The solution should support > >> multi-tenancy. Multiple tenants can come and store their objects in > >> it. However, there is also a requirement that a tenant may want to use > >> their own machines. In that case, their objects should be stored and > >> replicated within their machines. But those machines should still be > >> part of our system. This is because we will still need access to the > >> objects for our business workflows. It's just that their data should > >> not be stored and replicated outside of their systems. Is it something > >> that can be achieved using Ceph? Thanks a lot in advance. > >> > >> -- > >> Regards, > >> Varun Singh > > > > > > > >> > > -- > Confidentiality Notice and Disclaimer: This email (including any > attachments) contains information that may be confidential, privileged > and/or copyrighted. If you are not the intended recipient, please notify > the sender immediately and destroy this email. Any unauthorized use of the > contents of this email in any manner whatsoever, is strictly prohibited. If > improper activity is suspected, all available information may be used by > the sender for possible disciplinary action, prosecution, civil claim or > any remedy or lawful purpose. Email transmission cannot be guaranteed to be > secure or error-free, as information could be intercepted, lost, arrive > late, or contain viruses. The sender is not liable whatsoever for damage > resulting from the opening of this message and/or the use of the > information contained in this message and/or attachments. Expressions in > this email cannot be treated as opined by the sender company management – > they are solely expressed by the sender unless authorized. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] showing active config settings
Then why doesn't this work? # ceph tell 'osd.*' injectargs '--osd-recovery-max-active 4' osd.0: osd_recovery_max_active = '4' (not observed, change may require restart) osd.1: osd_recovery_max_active = '4' (not observed, change may require restart) osd.2: osd_recovery_max_active = '4' (not observed, change may require restart) osd.3: osd_recovery_max_active = '4' (not observed, change may require restart) osd.4: osd_recovery_max_active = '4' (not observed, change may require restart) # ceph -n osd.1 --show-config | grep osd_recovery_max_active osd_recovery_max_active = 3 On Wed, Apr 10, 2019 at 7:21 AM Eugen Block wrote: > > I always end up using "ceph --admin-daemon > > /var/run/ceph/name-of-socket-here.asok config show | grep ..." to get > what > > is in effect now for a certain daemon. > > Needs you to be on the host of the daemon of course. > > Me too, I just wanted to try what OP reported. And after trying that, > I'll keep it that way. ;-) > > > Zitat von Janne Johansson : > > > Den ons 10 apr. 2019 kl 13:37 skrev Eugen Block : > > > >> > If you don't specify which daemon to talk to, it tells you what the > >> > defaults would be for a random daemon started just now using the same > >> > config as you have in /etc/ceph/ceph.conf. > >> > >> I tried that, too, but the result is not correct: > >> > >> host1:~ # ceph -n osd.1 --show-config | grep osd_recovery_max_active > >> osd_recovery_max_active = 3 > >> > > > > I always end up using "ceph --admin-daemon > > /var/run/ceph/name-of-socket-here.asok config show | grep ..." to get > what > > is in effect now for a certain daemon. > > Needs you to be on the host of the daemon of course. > > > > -- > > May the most significant bit of your life be positive. > > > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Default Pools
On Mon, Apr 15, 2019 at 1:52 PM Brent Kennedy wrote: > > I was looking around the web for the reason for some of the default pools in > Ceph and I cant find anything concrete. Here is our list, some show no use > at all. Can any of these be deleted ( or is there an article my googlefu > failed to find that covers the default pools? > > We only use buckets, so I took out .rgw.buckets, .users and > .rgw.buckets.index… > > Name > .log > .rgw.root > .rgw.gc > .rgw.control > .rgw > .users.uid > .users.email > .rgw.buckets.extra > default.rgw.control > default.rgw.meta > default.rgw.log > default.rgw.buckets.non-ec All of these are created by RGW when you run it, not by the core Ceph system. I think they're all used (although they may report sizes of 0, as they mostly make use of omap). > metadata Except this one used to be created-by-default for CephFS metadata, but that hasn't been true in many releases. So I guess you're looking at an old cluster? (In which case it's *possible* some of those RGW pools are also unused now but were needed in the past; I haven't kept good track of them.) -Greg ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Default Pools
I was looking around the web for the reason for some of the default pools in Ceph and I cant find anything concrete. Here is our list, some show no use at all. Can any of these be deleted ( or is there an article my googlefu failed to find that covers the default pools? We only use buckets, so I took out .rgw.buckets, .users and .rgw.buckets.index. Name .log .rgw.root .rgw.gc .rgw.control .rgw .users.uid .users.email .rgw.buckets.extra default.rgw.control default.rgw.meta default.rgw.log default.rgw.buckets.non-ec metadata Regards, -Brent Existing Clusters: Test: Luminous 12.2.11 with 3 osd servers, 1 mon/man, 1 gateway ( all virtual on SSD ) US Production(HDD): Luminous 12.2.11 with 11 osd servers, 3 mons, 3 gateways behind haproxy LB UK Production(HDD): Luminous 12.2.11 with 15 osd servers, 3 mons/man, 3 gateways behind haproxy LB US Production(SSD): Luminous 12.2.11 with 6 osd servers, 3 mons/man, 3 gateways behind haproxy LB ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Unhandled exception from module 'dashboard' while running on mgr.xxxx: IOError
Hello,
After we upgraded ceph version from 12.2.7 to 12.2.11 dashboard start
reporting issues
2019-04-15 10:46:37.332514 [ERR] Unhandled exception from module
'dashboard' while running on mgr.x-a1: IOError("Port 43795 not free
on 'xxx.xxx.xxx.xx'",)
2019-04-15 10:41:37.078304 [ERR] Unhandled exception from module
'dashboard' while running on mgr.x-a1: IOError("Port 43795 not free
on 'xxx.xxx.xxx.xx'",)
2019-04-15 10:36:36.574722 [ERR] Unhandled exception from module
'dashboard' while running on mgr.x-a1: IOError("Port 43795 not free
on 'xxx.xxx.xxx.xx'",)
2019-04-15 10:33:39.587100 [ERR] Unhandled exception from module
'dashboard' while running on mgr.x-a1: IOError("Port 43795 not free
on 'xxx.xxx.xxx.xx'",)
While checking the logs shows the following
2019-04-15 10:44:57.754542 7f99ea8a2700 1 mgr send_beacon standby
2019-04-15 10:44:59.754948 7f99ea8a2700 1 mgr send_beacon standby
2019-04-15 10:45:01.755320 7f99ea8a2700 1 mgr send_beacon standby
2019-04-15 10:45:01.800317 7f99ea0a1700 -1 received signal: Terminated
from PID: 1 task name: /usr/lib/systemd/systemd --system --deserialize
21 UID: 0
2019-04-15 10:45:01.800332 7f99ea0a1700 -1 mgr handle_signal *** Got
signal Terminated ***
2019-04-15 10:46:31.937324 7fe6c5af37c0 0 set uid:gid to 167:167
(ceph:ceph)
2019-04-15 10:46:31.937340 7fe6c5af37c0 0 ceph version 12.2.11
(26dc3775efc7bb286a1d6d66faee0ba30ea23eee) luminous (stable), process
ceph-mgr, pid 1366334
2019-04-15 10:46:31.937717 7fe6c5af37c0 0 pidfile_write: ignore empty
--pid-file
2019-04-15 10:46:31.967287 7fe6c5af37c0 1 mgr send_beacon standby
2019-04-15 10:46:31.977735 7fe6bcc62700 1 mgr init Loading python
module 'dashboard'
2019-04-15 10:46:32.094982 7fe6bcc62700 1 mgr init Loading python
module 'restful'
2019-04-15 10:46:32.267235 7fe6bcc62700 1 mgr init Loading python
module 'status'
2019-04-15 10:46:32.296668 7fe6bcc62700 1 mgr load Constructed class
from module: dashboard
2019-04-15 10:46:33.967673 7fe6b9c5c700 1 mgr send_beacon standby
2019-04-15 10:46:35.968093 7fe6b9c5c700 1 mgr send_beacon standby
2019-04-15 10:46:37.332497 7fe6a112e700 -1 log_channel(cluster) log
[ERR] : Unhandled exception from module 'dashboard' while running on
mgr.ceph-ash-mon-a1: IOError("Port 43795 not free on 'xxx.xxx.xxx.xx'",)
2019-04-15 10:46:37.332517 7fe6a112e700 -1 dashboard.serve:
2019-04-15 10:46:37.332522 7fe6a112e700 -1 Traceback (most recent call
last):
File "/usr/lib64/ceph/mgr/dashboard/module.py", line 112, in serve
cherrypy.engine.start()
File "/usr/lib/python2.7/site-packages/cherrypy/process/wspbus.py",
line 250, in start
raise e_info
ChannelFailures: IOError("Port 43795 not free on 'xxx.xxx.xxx.xx'",)
2019-04-15 10:46:37.968522 7fe6b9c5c700 1 mgr send_beacon standby
2019-04-15 10:46:39.968878 7fe6b9c5c700 1 mgr send_beacon standby
2019-04-15 10:46:41.969271 7fe6b9c5c700 1 mgr send_beacon standby
2019-04-15 10:46:43.969677 7fe6b9c5c700 1 mgr send_beacon standby
2019-04-15 10:46:45.970064 7fe6b9c5c700 1 mgr send_beacon standby
Do you know why?
I tried to fail the mgr. but it again starts automatically.
--
Thank you,
Ramsh
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] v12.2.12 Luminous released
Thanks! -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Mon, Apr 15, 2019 at 12:58 PM Abhishek Lekshmanan wrote: > > Paul Emmerich writes: > > > I think the most notable change here is the backport of the new bitmap > > allocator, but that's missing completely from the change log. > > Updated the changelog in docs and the blog. The earlier script was > ignoring entries that didn't link to backport tracker following back to > a master tracker. > > > > > > Paul > > > > -- > > Paul Emmerich > > > > Looking for help with your Ceph cluster? Contact us at https://croit.io > > > > croit GmbH > > Freseniusstr. 31h > > 81247 München > > www.croit.io > > Tel: +49 89 1896585 90 > > > > On Fri, Apr 12, 2019 at 6:48 PM Abhishek Lekshmanan > > wrote: > >> > >> > >> We are happy to announce the next bugfix release for v12.2.x Luminous > >> stable release series. We recommend all luminous users to upgrade to > >> this release. Many thanks to everyone who contributed backports and a > >> special mention to Yuri for the QE efforts put in to this release. > >> > >> Notable Changes > >> --- > >> * In 12.2.11 and earlier releases, keyring caps were not checked for > >> validity, > >> so the caps string could be anything. As of 12.2.12, caps strings are > >> validated and providing a keyring with an invalid caps string to, e.g., > >> `ceph auth add` will result in an error. > >> > >> For the complete changelog, please refer to the release blog entry at > >> https://ceph.com/releases/v12-2-12-luminous-released/ > >> > >> Getting ceph: > >> > >> * Git at git://github.com/ceph/ceph.git > >> * Tarball at http://download.ceph.com/tarballs/ceph-12.2.12.tar.gz > >> * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ > >> * Release git sha1: 1436006594665279fe734b4c15d7e08c13ebd777 > >> > >> -- > >> Abhishek Lekshmanan > >> SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, > >> HRB 21284 (AG Nürnberg) > > > > -- > Abhishek Lekshmanan > SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, > HRB 21284 (AG Nürnberg) ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] PGLog.h: 777: FAILED assert(log.complete_to != log.log.end())
I have an OSD process that throws an assert whenever I boot it (see
traceback below).
I have successfully run ceph-bluestore-tool with the commands repair and
fsck, including with the --deep flag, but this did not fix the problem.
Any ideas how to fix this, apart from deleting the whole OSD and
starting over?
Example traceback:
root@d5:~# /usr/bin/ceph-osd -f --cluster ceph --id 16 --setuser ceph
--setgroup ceph
2019-04-15 13:18:21.318 7f9f265a6e00 -1 Public network was set, but
cluster network was not set
2019-04-15 13:18:21.318 7f9f265a6e00 -1 Using public network also
for cluster network
starting osd.16 at - osd_data /var/lib/ceph/osd/ceph-16
/var/lib/ceph/osd/ceph-16/journal
2019-04-15 13:18:48.142 7f9f265a6e00 -1 osd.16 14840 log_to_monitors
{default=true}
/build/ceph-13.2.5/src/osd/PGLog.h: In function 'void
PGLog::reset_complete_to(pg_info_t*)' thread 7f9efba21700 time
2019-04-15 13:18:49.541809
/build/ceph-13.2.5/src/osd/PGLog.h: 777: FAILED assert(log.complete_to
!= log.log.end())
ceph version 13.2.5 (cbff874f9007f1869bfd3821b7e33b2a6ffd4988) mimic
(stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x102) [0x7f9f1d976c12]
2: (()+0x282dd7) [0x7f9f1d976dd7]
3: (PGLog::reset_complete_to(pg_info_t*)+0x113) [0x55d1409c7c73]
4: (PG::activate(ObjectStore::Transaction&, unsigned int, std::map,
std::allocator > >, std::less,
std::allocator, std::allocator > >
>>> &, std::map,
std::allocator > >,
std::less, std::allocator,
std::allocator > > > > >*,
PG::RecoveryCtx*)+0x2cc) [0x55d14099a52c]
5: (PG::RecoveryState::ReplicaActive::react(PG::Activate const&)+0xbf)
[0x55d14099cccf]
6: (boost::statechart::simple_state::react_impl(boost::statechart::event_base
const&, void const*)+0x42e) [0x55d1409fbf4e]
7:
(boost::statechart::simple_state,
(boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base
const&, void const*)+0x12e) [0x55d1409f92ae]
8:
(boost::statechart::state_machine,
boost::statechart::null_exception_translator>::process_queued_events()+0xb3)
[0x55d1409cefb3]
9:
(boost::statechart::state_machine,
boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base
const&)+0x87) [0x55d1409cf217]
10: (PG::do_peering_event(std::shared_ptr,
PG::RecoveryCtx*)+0x143) [0x55d1409b50b3]
11: (OSD::dequeue_peering_evt(OSDShard*, PG*,
std::shared_ptr, ThreadPool::TPHandle&)+0xcf)
[0x55d1408f59ff]
12: (PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr&,
ThreadPool::TPHandle&)+0x50) [0x55d140b63080]
13: (OSD::ShardedOpWQ::_process(unsigned int,
ceph::heartbeat_handle_d*)+0x598) [0x55d140905768]
14: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x46e)
[0x7f9f1d97bb4e]
15: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x7f9f1d97dbd0]
16: (()+0x76db) [0x7f9f1c05a6db]
17: (clone()+0x3f) [0x7f9f1b02388f]
Thanks in advance for any help,
Egil
signature.asc
Description: OpenPGP digital signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] BlueStore bitmap allocator under Luminous and Mimic
On 4/15/19 2:55 PM, Igor Fedotov wrote: > Hi Wido, > > the main driver for this backport were multiple complains on write ops > latency increasing over time. > > E.g. see thread named: "ceph osd commit latency increase over time, > until restart" here. > > Or http://tracker.ceph.com/issues/38738 > > Most symptoms showed Stupid Allocator as a root cause for that. > > Hence we've got a decision to backport bitmap allocator which should > work a fix/workaround. > I see, that makes things clear. Anything users should take into account when setting: [osd] bluestore_allocator = bitmap bluefs_allocator = bitmap Writing this here for archival purposes so that users who have the same question can find it easily. Wido > > Thanks, > > Igor > > > On 4/15/2019 3:39 PM, Wido den Hollander wrote: >> Hi, >> >> With the release of 12.2.12 the bitmap allocator for BlueStore is now >> available under Mimic and Luminous. >> >> [osd] >> bluestore_allocator = bitmap >> bluefs_allocator = bitmap >> >> Before setting this in production: What might the implications be and >> what should be thought of? >> >> From what I've read the bitmap allocator seems to be better in read >> performance and uses less memory. >> >> In Nautilus bitmap is the default, but L and M still default to stupid. >> >> Since the bitmap allocator was backported there must be a use-case to >> use the bitmap allocator instead of stupid. >> >> Thanks! >> >> Wido >> ___ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Fwd: HW failure cause client IO drops
Hello - Recevenlt we had an issue with storage node's battery failure, which cause ceph client IO dropped to '0' bytes. Means ceph cluster couldn't perform IO operations on the cluster till the node takes out. This is not expected from Ceph, as some HW fails, those respective OSDs should mark as out/down and IO should go as is.. Please let me know if anyone seen the similar behavior and is this issue resolved? Thanks Swami ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] HW failure cause client IO drops
Hello - Recevenlt we had an issue with storage node's battery failure, which cause ceph client IO dropped to '0' bytes. Means ceph cluster couldn't perform IO operations on the cluster till the node takes out. This is not expected from Ceph, as some HW fails, those respective OSDs should mark as out/down and IO should go as is.. Please let me know if anyone seen the similar behavior and is this issue resolved? Thanks Swami ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] BlueStore bitmap allocator under Luminous and Mimic
Hi Wido, the main driver for this backport were multiple complains on write ops latency increasing over time. E.g. see thread named: "ceph osd commit latency increase over time, until restart" here. Or http://tracker.ceph.com/issues/38738 Most symptoms showed Stupid Allocator as a root cause for that. Hence we've got a decision to backport bitmap allocator which should work a fix/workaround. Thanks, Igor On 4/15/2019 3:39 PM, Wido den Hollander wrote: Hi, With the release of 12.2.12 the bitmap allocator for BlueStore is now available under Mimic and Luminous. [osd] bluestore_allocator = bitmap bluefs_allocator = bitmap Before setting this in production: What might the implications be and what should be thought of? From what I've read the bitmap allocator seems to be better in read performance and uses less memory. In Nautilus bitmap is the default, but L and M still default to stupid. Since the bitmap allocator was backported there must be a use-case to use the bitmap allocator instead of stupid. Thanks! Wido ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] BlueStore bitmap allocator under Luminous and Mimic
Hi, With the release of 12.2.12 the bitmap allocator for BlueStore is now available under Mimic and Luminous. [osd] bluestore_allocator = bitmap bluefs_allocator = bitmap Before setting this in production: What might the implications be and what should be thought of? >From what I've read the bitmap allocator seems to be better in read performance and uses less memory. In Nautilus bitmap is the default, but L and M still default to stupid. Since the bitmap allocator was backported there must be a use-case to use the bitmap allocator instead of stupid. Thanks! Wido ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Decreasing pg_num
On 4/15/19 1:13 PM, Alfredo Daniel Rezinovsky wrote: > > On 15/4/19 06:54, Jasper Spaans wrote: >> On 14/04/2019 17:05, Alfredo Daniel Rezinovsky wrote: >>> autoscale-status reports some of my PG_NUMs are way too big >>> >>> I have 256 and need 32 >>> >>> POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET >>> RATIO PG_NUM NEW PG_NUM AUTOSCALE >>> rbd 1214G 3.0 56490G >>> 0.0645 256 32 warn >>> >>> If I try to decrease the pg_num I get: >>> >>> # ceph osd pool set rbd pg_num 32 >>> Error EPERM: nautilus OSDs are required to decrease pg_num >>> >>> But all my osds are nautilus >> This is somewhat hidden in the upgrade docs but I missed it as well the >> first time - did you run >> >> ceph osd require-osd-release nautilus >> >> ? >> > That was. It worked. With very few misplaced objects > > Should I also decrease pgp_num ? > Yes, both should be decreased to the same value. Wido > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Decreasing pg_num
On 15/4/19 06:54, Jasper Spaans wrote: On 14/04/2019 17:05, Alfredo Daniel Rezinovsky wrote: autoscale-status reports some of my PG_NUMs are way too big I have 256 and need 32 POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO PG_NUM NEW PG_NUM AUTOSCALE rbd 1214G 3.0 56490G 0.0645 256 32 warn If I try to decrease the pg_num I get: # ceph osd pool set rbd pg_num 32 Error EPERM: nautilus OSDs are required to decrease pg_num But all my osds are nautilus This is somewhat hidden in the upgrade docs but I missed it as well the first time - did you run ceph osd require-osd-release nautilus ? That was. It worked. With very few misplaced objects Should I also decrease pgp_num ? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] v12.2.12 Luminous released
Paul Emmerich writes: > I think the most notable change here is the backport of the new bitmap > allocator, but that's missing completely from the change log. Updated the changelog in docs and the blog. The earlier script was ignoring entries that didn't link to backport tracker following back to a master tracker. > > > Paul > > -- > Paul Emmerich > > Looking for help with your Ceph cluster? Contact us at https://croit.io > > croit GmbH > Freseniusstr. 31h > 81247 München > www.croit.io > Tel: +49 89 1896585 90 > > On Fri, Apr 12, 2019 at 6:48 PM Abhishek Lekshmanan wrote: >> >> >> We are happy to announce the next bugfix release for v12.2.x Luminous >> stable release series. We recommend all luminous users to upgrade to >> this release. Many thanks to everyone who contributed backports and a >> special mention to Yuri for the QE efforts put in to this release. >> >> Notable Changes >> --- >> * In 12.2.11 and earlier releases, keyring caps were not checked for >> validity, >> so the caps string could be anything. As of 12.2.12, caps strings are >> validated and providing a keyring with an invalid caps string to, e.g., >> `ceph auth add` will result in an error. >> >> For the complete changelog, please refer to the release blog entry at >> https://ceph.com/releases/v12-2-12-luminous-released/ >> >> Getting ceph: >> >> * Git at git://github.com/ceph/ceph.git >> * Tarball at http://download.ceph.com/tarballs/ceph-12.2.12.tar.gz >> * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ >> * Release git sha1: 1436006594665279fe734b4c15d7e08c13ebd777 >> >> -- >> Abhishek Lekshmanan >> SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, >> HRB 21284 (AG Nürnberg) > -- Abhishek Lekshmanan SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Save the date: Ceph Day for Research @ CERN -- Sept 16, 2019
Hey Cephalopods! This is an early heads up that we are planning a Ceph Day event at CERN in Geneva, Switzerland on September 16, 2019 [1]. For this Ceph Day, we want to focus on use-cases and solutions for research, academia, or other non-profit applications [2]. Registration and call for proposals will be available by mid-May. All the Best, Dan van der Ster CERN IT Department Ceph Governing Board, Academic Liaison [1] Sept 16 is the day after CERN Open Days, where there will be plenty to visit on our campus if you arrive a couple of days before https://home.cern/news/news/cern/cern-open-days-explore-future-us [2] Marine biologists studying actual Cephalopods with Ceph are especially welcome ;-) ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Decreasing pg_num
On 14/04/2019 17:05, Alfredo Daniel Rezinovsky wrote: > autoscale-status reports some of my PG_NUMs are way too big > > I have 256 and need 32 > > POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET > RATIO PG_NUM NEW PG_NUM AUTOSCALE > rbd 1214G 3.0 56490G > 0.0645 256 32 warn > > If I try to decrease the pg_num I get: > > # ceph osd pool set rbd pg_num 32 > Error EPERM: nautilus OSDs are required to decrease pg_num > > But all my osds are nautilus This is somewhat hidden in the upgrade docs but I missed it as well the first time - did you run ceph osd require-osd-release nautilus ? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] restful mgr API does not start due to Python SocketServer error
Hi, I recently upgraded a cluster to 13.2.5 and right now the RestFul API won't start due to this stacktrace: 2019-04-15 11:32:18.632 7f8797cb6700 0 mgr[restful] Traceback (most recent call last): File "/usr/lib64/ceph/mgr/restful/module.py", line 254, in serve self._serve() File "/usr/lib64/ceph/mgr/restful/module.py", line 336, in _serve self.server.serve_forever() File "/usr/lib/python2.7/site-packages/werkzeug/serving.py", line 436, in serve_forever HTTPServer.serve_forever(self) File "/usr/lib64/python2.7/SocketServer.py", line 236, in serve_forever poll_interval) File "/usr/lib64/python2.7/SocketServer.py", line 155, in _eintr_retry return func(*args) ValueError: filedescriptor out of range in select() Searching on the internet I found that it might be related to Python's SocketServer, but I could not confirm that yet. I also created an issue: http://tracker.ceph.com/issues/39292 Has anybody seen this before? Running: - Mimic 13.2.5 - CentOS 7 - 2000 OSDs Wido ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
