Re: [ceph-users] 12.2.4 Both Ceph MDS nodes crashed. Please help.
On Wed, May 2, 2018 at 7:19 AM, Sean Sullivan wrote: > Forgot to reply to all: > > Sure thing! > > I couldn't install the ceph-mds-dbg packages without upgrading. I just > finished upgrading the cluster to 12.2.5. The issue still persists in 12.2.5 > > From here I'm not really sure how to do generate the backtrace so I hope I > did it right. For others on Ubuntu this is what I did: > > * firstly up the debug_mds to 20 and debug_ms to 1: > ceph tell mds.* injectargs '--debug-mds 20 --debug-ms 1' > > * install the debug packages > ceph-mds-dbg in my case > > * I also added these options to /etc/ceph/ceph.conf just in case they > restart. > > * Now allow pids to dump (stolen partly from redhat docs and partly from > ubuntu) > echo -e 'DefaultLimitCORE=infinity\nPrivateTmp=true' | tee -a > /etc/systemd/system.conf > sysctl fs.suid_dumpable=2 > sysctl kernel.core_pattern=/tmp/core > systemctl daemon-reload > systemctl restart ceph-mds@$(hostname -s) > > * A crash was created in /var/crash by apport but gdb cant read it. I used > apport-unpack and then ran GDB on what is inside: > core dump should be in /tmp/core > apport-unpack /var/crash/$(ls /var/crash/*mds*) /root/crash_dump/ > cd /root/crash_dump/ > gdb $(cat ExecutablePath) CoreDump -ex 'thr a a bt' | tee > /root/ceph_mds_$(hostname -s)_backtrace > > * This left me with the attached backtraces (which I think are wrong as I > see a lot of ?? yet gdb says > /usr/lib/debug/.build-id/1d/23dc5ef4fec1dacebba2c6445f05c8fe6b8a7c.debug was > loaded) > > kh10-8 mds backtrace -- https://pastebin.com/bwqZGcfD > kh09-8 mds backtrace -- https://pastebin.com/vvGiXYVY > Try running ceph-mds inside gdb. It should be easy to locate the bug once we have correct coredump file. Regards Yan, Zheng > > The log files are pretty large (one 4.1G and the other 200MB) > > kh10-8 (200MB) mds log -- > https://griffin-objstore.opensciencedatacloud.org/logs/ceph-mds.kh10-8.log > kh09-8 (4.1GB) mds log -- > https://griffin-objstore.opensciencedatacloud.org/logs/ceph-mds.kh09-8.log > > On Tue, May 1, 2018 at 12:09 AM, Patrick Donnelly > wrote: >> >> Hello Sean, >> >> On Mon, Apr 30, 2018 at 2:32 PM, Sean Sullivan >> wrote: >> > I was creating a new user and mount point. On another hardware node I >> > mounted CephFS as admin to mount as root. I created /aufstest and then >> > unmounted. From there it seems that both of my mds nodes crashed for >> > some >> > reason and I can't start them any more. >> > >> > https://pastebin.com/1ZgkL9fa -- my mds log >> > >> > I have never had this happen in my tests so now I have live data here. >> > If >> > anyone can lend a hand or point me in the right direction while >> > troubleshooting that would be a godsend! >> >> Thanks for keeping the list apprised of your efforts. Since this is so >> easily reproduced for you, I would suggest that you next get higher >> debug logs (debug_mds=20/debug_ms=1) from the MDS. And, since this is >> a segmentation fault, a backtrace with debug symbols from gdb would >> also be helpful. >> >> -- >> Patrick Donnelly > > > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Place on separate hosts?
And also make sure the OSD<-> mapping is correct with "ceph osd tree". :) On Fri, May 4, 2018 at 1:44 AM Matthew Vernon wrote: > Hi, > > On 04/05/18 08:25, Tracy Reed wrote: > > On Fri, May 04, 2018 at 12:18:15AM PDT, Tracy Reed spake thusly: > >> > https://jcftang.github.io/2012/09/06/going-from-replicating-across-osds-to-replicating-across-hosts-in-a-ceph-cluster/ > > > >> How can I tell which way mine is configured? I could post the whole > >> crushmap if necessary but it's a bit large to copy and paste. > > > > To further answer my own question (sorry for the spam) the above linked > > doc says this should do what I want: > > > > step chooseleaf firstn 0 type host > > > > which is what I already have in my crush map. So it looks like the > > default is as I want it. In which case I wonder why I had the problem > > previously... I guess the only way to know for sure is to stop one osd > > node and see what happens. > > You can ask ceph which OSDs a particular pg is on: > > root@sto-1-1:~# ceph pg map 71.983 > osdmap e435728 pg 71.983 (71.983) -> up [1948,2984,511] acting > [1948,2984,511] > > ...then you can check these are on different hosts. > > HTH, > > Matthew > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] issues on CT + EC pool
Hi, I have a big-ish cluster that, amongst other things, has a radosgw configured to have an EC data pool (k=12, m=4). The cluster is currently running Jewel (10.2.7). That pool spans 244 HDDs and has 2048 PGs. from the df detail: .rgw.buckets.ec 26 -N/A N/A 76360G 28.66 185T 97908947 95614k 73271k 185M 101813G ct-radosgw 37 -N/A N/A 4708G 70.69 1952G 5226185 2071k 591M 1518M9416G The ct-radosgw should be size 3, but currently due to an unrelated issue (pdu failure) is size 2. Whenever I flush data from the cache tier to the base tier the OSDs start updating their local leveldb database, using up 100% IO, until they: a) are set as down for no answer, and/or b) suicide timeout. I have other pools targeting those same OSDs but until now nothing has happened when the IO goes to the other pools. Any ideas on where to proceed? thanks, ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] mgr dashboard differs from ceph status
On Fri, May 4, 2018 at 1:59 AM John Spray wrote: > On Fri, May 4, 2018 at 7:21 AM, Tracy Reed wrote: > > My ceph status says: > > > > cluster: > > id: b2b00aae-f00d-41b4-a29b-58859aa41375 > > health: HEALTH_OK > > > > services: > > mon: 3 daemons, quorum ceph01,ceph03,ceph07 > > mgr: ceph01(active), standbys: ceph-ceph07, ceph03 > > osd: 78 osds: 78 up, 78 in > > > > data: > > pools: 4 pools, 3240 pgs > > objects: 4384k objects, 17533 GB > > usage: 53141 GB used, 27311 GB / 80452 GB avail > > pgs: 3240 active+clean > > > > io: > > client: 4108 kB/s rd, 10071 kB/s wr, 27 op/s rd, 331 op/s wr > > > > but my mgr dashboard web interface says: > > > > > > Health > > Overall status: HEALTH_WARN > > > > PG_AVAILABILITY: Reduced data availability: 2563 pgs inactive > > > > > > Anyone know why the discrepency? Hopefully the dashboard is very > > mistaken! Everything seems to be operating normally. If I had 2/3 of my > > pgs inactive I'm sure all of my rbd backing my VMs would be blocked etc. > > A situation like this probably indicates that something is going wrong > with the mon->mgr synchronisation of health state (it's all calculated > in one place and the mon updates the mgr every few seconds). > > 1. Look for errors in your monitor logs > 2. You'll probably find that everything gets back in sync if you > restart a mgr daemon > > John > Isn't that the wrong direction for sync issues, though? I mean, the manager is where the PG reports actually go. So if the cluster's still running, the monitor says it's active+clean, and the *dashboard* says the PGs are inactive, it sounds like the monitor has the correct view and something has gone wrong between the rest of the manager guts and the dashboard display. -Greg > > > I'm running ceph-12.2.4-0.el7.x86_64 on CentOS 7. Almost all filestore > > except for one OSD which recently had to be replaced which I made > > bluestore. I plan to slowly migrate everything over to bluestore over > > the course of the next month. > > > > Thanks! > > > > -- > > Tracy Reed > > http://tracyreed.org > > Digital signature attached for your safety. > > > > ___ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] OSD doesnt start after reboot
yes correct,but the main issue is, the osd configuration gets lost after every reboot On Fri, May 4, 2018 at 6:11 PM, Alfredo Deza wrote: > On Fri, May 4, 2018 at 1:22 AM, Akshita Parekh > wrote: > > Steps followed during installing ceph- > > 1) Installing rpms > > > > Then the steps given in - > > http://docs.ceph.com/docs/master/start/quick-ceph-deploy/ , apart from > step > > 2 and 3 > > > > Then ceph-deploy osd prepare osd1:/dev/sda1 > > ceph-deploy osd activate osd1:/dev/sda1 > > > > It said conf files were different and no monitor found.So i > copied > > ceph.conf from admin to osd node. I am using only 1 osd currently.Osd got > > activated . Then i enabled the ceph.target and ceph-osd.target services. > > After rebooting the folders created in /var/lib/ceph/osd/cluster-id > like > > current superblock etc gets erased. Now i am trying to reinstall the file > > system(created using fuse). But it says file system is busy,stop MDS > > services.MDS services are stopped already. > > Sounds like you tried installing/re-installing a few times, with > different contents and got into a place where your servers have stale > information. > > You should start from scratch again, probably with new servers or at > least try to ensure everything was fully uninstalled and removed. > > > > > > > > > > > > > > > > > > On Fri, May 4, 2018 at 9:50 AM, David Turner > wrote: > >> > >> Please keep the mailing list in your responses. What steps did you > follow > >> when configuring your osds. > >> > >> > >> On Fri, May 4, 2018, 12:14 AM Akshita Parekh > >> wrote: > >>> > >>> Ceph v10.2.0 -jewel , Why ceph disk or ceph volume is required to > >>> configure disks?encryption where? > >>> > >>> On Thu, May 3, 2018 at 6:24 PM, David Turner > >>> wrote: > > Which version of ceph, filestore or bluestore, did you use ceph-disk, > ceph-volume, or something else to configure the osds, did you use > lvm, is > there encryption or any other later involved? > > On Thu, May 3, 2018, 6:45 AM Akshita Parekh > > wrote: > > > > Hi All, > > > > > > after every reboot the current superblock etc folders get deleted > from > > /var/lib/ceph/osd/ceph-0(1,etc) > > .I have to prepare and activate osd after every reboot. Any > > suggestions? > > > > ceph.target and ceph-osd are enabled. > > > > Thanks in advance! > > ___ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >>> > >>> > > > > > > ___ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] 12.2.4 Both Ceph MDS nodes crashed. Please help.
Most of this is over my head but the last line of the logs on both mds servers show something similar to: 0> 2018-05-01 15:37:46.871932 7fd10163b700 -1 *** Caught signal (Segmentation fault) ** in thread 7fd10163b700 thread_name:mds_rank_progr When I search for this in ceph user and devel mailing list the only mention I can see is from 12.0.3: https://marc.info/?l=ceph-devel&m=149726392820648&w=2 -- ceph-devel I don't see any mention of journal.cc in my logs however so I hope they are not related. I also have not experienced any major loss in my cluster as of yet and cephfs-journal-tool shows my journals as healthy. To trigger this bug I created a cephfs directory and user called aufstest. Here is the part of the log with the crash mentioning aufstest. https://pastebin.com/EL5ALLuE I created a new bug ticket on ceph.com with all of the current info as I believe this isn't a problem with my setup specifically and anyone else trying this will have the same issue. https://tracker.ceph.com/issues/23972 I hope this is the correct path. If anyone can guide me in the right direction for troubleshooting this further I would be grateful. On Tue, May 1, 2018 at 6:19 PM, Sean Sullivan wrote: > Forgot to reply to all: > > > Sure thing! > > I couldn't install the ceph-mds-dbg packages without upgrading. I just > finished upgrading the cluster to 12.2.5. The issue still persists in 12.2.5 > > From here I'm not really sure how to do generate the backtrace so I hope I > did it right. For others on Ubuntu this is what I did: > > * firstly up the debug_mds to 20 and debug_ms to 1: > ceph tell mds.* injectargs '--debug-mds 20 --debug-ms 1' > > * install the debug packages > ceph-mds-dbg in my case > > * I also added these options to /etc/ceph/ceph.conf just in case they > restart. > > * Now allow pids to dump (stolen partly from redhat docs and partly from > ubuntu) > echo -e 'DefaultLimitCORE=infinity\nPrivateTmp=true' | tee -a > /etc/systemd/system.conf > sysctl fs.suid_dumpable=2 > sysctl kernel.core_pattern=/tmp/core > systemctl daemon-reload > systemctl restart ceph-mds@$(hostname -s) > > * A crash was created in /var/crash by apport but gdb cant read it. I used > apport-unpack and then ran GDB on what is inside: > > apport-unpack /var/crash/$(ls /var/crash/*mds*) /root/crash_dump/ > cd /root/crash_dump/ > gdb $(cat ExecutablePath) CoreDump -ex 'thr a a bt' | tee > /root/ceph_mds_$(hostname -s)_backtrace > > * This left me with the attached backtraces (which I think are wrong as I > see a lot of ?? yet gdb says /usr/lib/debug/.build-id/1d/ > 23dc5ef4fec1dacebba2c6445f05c8fe6b8a7c.debug was loaded) > > kh10-8 mds backtrace -- https://pastebin.com/bwqZGcfD > kh09-8 mds backtrace -- https://pastebin.com/vvGiXYVY > > > The log files are pretty large (one 4.1G and the other 200MB) > > kh10-8 (200MB) mds log -- https://griffin-objstore.op > ensciencedatacloud.org/logs/ceph-mds.kh10-8.log > kh09-8 (4.1GB) mds log -- https://griffin-objstore.op > ensciencedatacloud.org/logs/ceph-mds.kh09-8.log > > On Tue, May 1, 2018 at 12:09 AM, Patrick Donnelly > wrote: > >> Hello Sean, >> >> On Mon, Apr 30, 2018 at 2:32 PM, Sean Sullivan >> wrote: >> > I was creating a new user and mount point. On another hardware node I >> > mounted CephFS as admin to mount as root. I created /aufstest and then >> > unmounted. From there it seems that both of my mds nodes crashed for >> some >> > reason and I can't start them any more. >> > >> > https://pastebin.com/1ZgkL9fa -- my mds log >> > >> > I have never had this happen in my tests so now I have live data here. >> If >> > anyone can lend a hand or point me in the right direction while >> > troubleshooting that would be a godsend! >> >> Thanks for keeping the list apprised of your efforts. Since this is so >> easily reproduced for you, I would suggest that you next get higher >> debug logs (debug_mds=20/debug_ms=1) from the MDS. And, since this is >> a segmentation fault, a backtrace with debug symbols from gdb would >> also be helpful. >> >> -- >> Patrick Donnelly >> > > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Luminous radosgw S3/Keystone integration issues
Hi Dan,
We agreed in upstream RGW to make this change. Do you intend to
submit this as a PR?
regards
Matt
On Fri, May 4, 2018 at 10:57 AM, Dan van der Ster wrote:
> Hi Valery,
>
> Did you eventually find a workaround for this? I *think* we'd also
> prefer rgw to fallback to external plugins, rather than checking them
> before local. But I never understood the reasoning behind the change
> from jewel to luminous.
>
> I saw that there is work towards a cache for ldap [1] and I assume a
> similar approach would be useful for keystone as well.
>
> In the meantime, would a patch like [2] work?
>
> Cheers, Dan
>
> [1] https://github.com/ceph/ceph/pull/20624
>
> [2] diff --git a/src/rgw/rgw_auth_s3.h b/src/rgw/rgw_auth_s3.h
> index 6bcdebaf1c..3c343adf66 100644
> --- a/src/rgw/rgw_auth_s3.h
> +++ b/src/rgw/rgw_auth_s3.h
> @@ -129,20 +129,17 @@ public:
>add_engine(Control::SUFFICIENT, anonymous_engine);
> }
>
> +/* The local auth. */
> +if (cct->_conf->rgw_s3_auth_use_rados) {
> + add_engine(Control::SUFFICIENT, local_engine);
> +}
> +
> /* The external auth. */
> Control local_engine_mode;
> if (! external_engines.is_empty()) {
>add_engine(Control::SUFFICIENT, external_engines);
> -
> - local_engine_mode = Control::FALLBACK;
> -} else {
> - local_engine_mode = Control::SUFFICIENT;
> }
>
> -/* The local auth. */
> -if (cct->_conf->rgw_s3_auth_use_rados) {
> - add_engine(local_engine_mode, local_engine);
> -}
>}
>
>const char* get_name() const noexcept override {
>
>
> On Thu, Feb 1, 2018 at 4:44 PM, Valery Tschopp
> wrote:
>> Hi,
>>
>> We are operating a Luminous 12.2.2 radosgw, with the S3 Keystone
>> authentication enabled.
>>
>> Some customers are uploading millions of objects per bucket at once,
>> therefore the radosgw is doing millions of s3tokens POST requests to the
>> Keystone. All those s3tokens requests to Keystone are the same (same
>> customer, same EC2 credentials). But because there is no cache in radosgw
>> for the EC2 credentials, every incoming S3 operation generates a call to the
>> external auth Keystone. It can generate hundreds of s3tokens requests per
>> second to Keystone.
>>
>> We had already this problem with Jewel, but we implemented a workaround. The
>> EC2 credentials of the customer were added directly in the local auth engine
>> of radosgw. So for this particular heavy user, the radosgw local
>> authentication was checked first, and no external auth request to Keystone
>> was necessary.
>>
>> But the default behavior for the S3 authentication have change in Luminous.
>>
>> In Luminous, if you enable the S3 Keystone authentication, every incoming S3
>> operation will first check for anonymous authentication, then external
>> authentication (Keystone and/or LDAP), and only then local authentication.
>> See https://github.com/ceph/ceph/blob/master/src/rgw/rgw_auth_s3.h#L113-L141
>>
>> Is there a way to get the old authentication behavior (anonymous -> local ->
>> external) to work again?
>>
>> Or is it possible to implement a caching mechanism (similar to the Token
>> cache) for the EC2 credentials?
>>
>> Cheers,
>> Valery
>>
>> --
>> SWITCH
>> Valéry Tschopp, Software Engineer
>> Werdstrasse 2, P.O. Box, 8021 Zurich, Switzerland
>> email: valery.tsch...@switch.ch phone: +41 44 268 1544
>>
>> 30 years of pioneering the Swiss Internet.
>> Celebrate with us at https://swit.ch/30years
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103
http://www.redhat.com/en/technologies/storage
tel. 734-821-5101
fax. 734-769-8938
cel. 734-216-5309
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Luminous radosgw S3/Keystone integration issues
Hi Valery,
Did you eventually find a workaround for this? I *think* we'd also
prefer rgw to fallback to external plugins, rather than checking them
before local. But I never understood the reasoning behind the change
from jewel to luminous.
I saw that there is work towards a cache for ldap [1] and I assume a
similar approach would be useful for keystone as well.
In the meantime, would a patch like [2] work?
Cheers, Dan
[1] https://github.com/ceph/ceph/pull/20624
[2] diff --git a/src/rgw/rgw_auth_s3.h b/src/rgw/rgw_auth_s3.h
index 6bcdebaf1c..3c343adf66 100644
--- a/src/rgw/rgw_auth_s3.h
+++ b/src/rgw/rgw_auth_s3.h
@@ -129,20 +129,17 @@ public:
add_engine(Control::SUFFICIENT, anonymous_engine);
}
+/* The local auth. */
+if (cct->_conf->rgw_s3_auth_use_rados) {
+ add_engine(Control::SUFFICIENT, local_engine);
+}
+
/* The external auth. */
Control local_engine_mode;
if (! external_engines.is_empty()) {
add_engine(Control::SUFFICIENT, external_engines);
-
- local_engine_mode = Control::FALLBACK;
-} else {
- local_engine_mode = Control::SUFFICIENT;
}
-/* The local auth. */
-if (cct->_conf->rgw_s3_auth_use_rados) {
- add_engine(local_engine_mode, local_engine);
-}
}
const char* get_name() const noexcept override {
On Thu, Feb 1, 2018 at 4:44 PM, Valery Tschopp wrote:
> Hi,
>
> We are operating a Luminous 12.2.2 radosgw, with the S3 Keystone
> authentication enabled.
>
> Some customers are uploading millions of objects per bucket at once,
> therefore the radosgw is doing millions of s3tokens POST requests to the
> Keystone. All those s3tokens requests to Keystone are the same (same
> customer, same EC2 credentials). But because there is no cache in radosgw
> for the EC2 credentials, every incoming S3 operation generates a call to the
> external auth Keystone. It can generate hundreds of s3tokens requests per
> second to Keystone.
>
> We had already this problem with Jewel, but we implemented a workaround. The
> EC2 credentials of the customer were added directly in the local auth engine
> of radosgw. So for this particular heavy user, the radosgw local
> authentication was checked first, and no external auth request to Keystone
> was necessary.
>
> But the default behavior for the S3 authentication have change in Luminous.
>
> In Luminous, if you enable the S3 Keystone authentication, every incoming S3
> operation will first check for anonymous authentication, then external
> authentication (Keystone and/or LDAP), and only then local authentication.
> See https://github.com/ceph/ceph/blob/master/src/rgw/rgw_auth_s3.h#L113-L141
>
> Is there a way to get the old authentication behavior (anonymous -> local ->
> external) to work again?
>
> Or is it possible to implement a caching mechanism (similar to the Token
> cache) for the EC2 credentials?
>
> Cheers,
> Valery
>
> --
> SWITCH
> Valéry Tschopp, Software Engineer
> Werdstrasse 2, P.O. Box, 8021 Zurich, Switzerland
> email: valery.tsch...@switch.ch phone: +41 44 268 1544
>
> 30 years of pioneering the Swiss Internet.
> Celebrate with us at https://swit.ch/30years
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] OSD doesnt start after reboot
On Fri, May 4, 2018 at 1:22 AM, Akshita Parekh wrote: > Steps followed during installing ceph- > 1) Installing rpms > > Then the steps given in - > http://docs.ceph.com/docs/master/start/quick-ceph-deploy/ , apart from step > 2 and 3 > > Then ceph-deploy osd prepare osd1:/dev/sda1 > ceph-deploy osd activate osd1:/dev/sda1 > > It said conf files were different and no monitor found.So i copied > ceph.conf from admin to osd node. I am using only 1 osd currently.Osd got > activated . Then i enabled the ceph.target and ceph-osd.target services. > After rebooting the folders created in /var/lib/ceph/osd/cluster-id like > current superblock etc gets erased. Now i am trying to reinstall the file > system(created using fuse). But it says file system is busy,stop MDS > services.MDS services are stopped already. Sounds like you tried installing/re-installing a few times, with different contents and got into a place where your servers have stale information. You should start from scratch again, probably with new servers or at least try to ensure everything was fully uninstalled and removed. > > > > > > > > On Fri, May 4, 2018 at 9:50 AM, David Turner wrote: >> >> Please keep the mailing list in your responses. What steps did you follow >> when configuring your osds. >> >> >> On Fri, May 4, 2018, 12:14 AM Akshita Parekh >> wrote: >>> >>> Ceph v10.2.0 -jewel , Why ceph disk or ceph volume is required to >>> configure disks?encryption where? >>> >>> On Thu, May 3, 2018 at 6:24 PM, David Turner >>> wrote: Which version of ceph, filestore or bluestore, did you use ceph-disk, ceph-volume, or something else to configure the osds, did you use lvm, is there encryption or any other later involved? On Thu, May 3, 2018, 6:45 AM Akshita Parekh wrote: > > Hi All, > > > after every reboot the current superblock etc folders get deleted from > /var/lib/ceph/osd/ceph-0(1,etc) > .I have to prepare and activate osd after every reboot. Any > suggestions? > > ceph.target and ceph-osd are enabled. > > Thanks in advance! > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> >>> > > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] mgr dashboard differs from ceph status
I get this too, since I last rebooted a server (one of three). ceph -s says: cluster: id: a8c34694-a172-4418-a7dd-dd8a642eb545 health: HEALTH_OK services: mon: 3 daemons, quorum box1,box2,box3 mgr: box3(active), standbys: box1, box2 osd: N osds: N up, N in rgw: 3 daemons active mgr dashboard says: Overall status: HEALTH_WARN MON_DOWN: 1/3 mons down, quorum box1,box3 I wasn't going to worry too much. I'll check logs and restart an mgr then. Sean On Fri, 4 May 2018, John Spray said: > On Fri, May 4, 2018 at 7:21 AM, Tracy Reed wrote: > > My ceph status says: > > > > cluster: > > id: b2b00aae-f00d-41b4-a29b-58859aa41375 > > health: HEALTH_OK > > > > services: > > mon: 3 daemons, quorum ceph01,ceph03,ceph07 > > mgr: ceph01(active), standbys: ceph-ceph07, ceph03 > > osd: 78 osds: 78 up, 78 in > > > > data: > > pools: 4 pools, 3240 pgs > > objects: 4384k objects, 17533 GB > > usage: 53141 GB used, 27311 GB / 80452 GB avail > > pgs: 3240 active+clean > > > > io: > > client: 4108 kB/s rd, 10071 kB/s wr, 27 op/s rd, 331 op/s wr > > > > but my mgr dashboard web interface says: > > > > > > Health > > Overall status: HEALTH_WARN > > > > PG_AVAILABILITY: Reduced data availability: 2563 pgs inactive > > > > > > Anyone know why the discrepency? Hopefully the dashboard is very > > mistaken! Everything seems to be operating normally. If I had 2/3 of my > > pgs inactive I'm sure all of my rbd backing my VMs would be blocked etc. > > A situation like this probably indicates that something is going wrong > with the mon->mgr synchronisation of health state (it's all calculated > in one place and the mon updates the mgr every few seconds). > > 1. Look for errors in your monitor logs > 2. You'll probably find that everything gets back in sync if you > restart a mgr daemon > > John > > > I'm running ceph-12.2.4-0.el7.x86_64 on CentOS 7. Almost all filestore > > except for one OSD which recently had to be replaced which I made > > bluestore. I plan to slowly migrate everything over to bluestore over > > the course of the next month. > > > > Thanks! > > > > -- > > Tracy Reed > > http://tracyreed.org > > Digital signature attached for your safety. > > > > ___ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] mgr dashboard differs from ceph status
On Fri, May 4, 2018 at 7:21 AM, Tracy Reed wrote: > My ceph status says: > > cluster: > id: b2b00aae-f00d-41b4-a29b-58859aa41375 > health: HEALTH_OK > > services: > mon: 3 daemons, quorum ceph01,ceph03,ceph07 > mgr: ceph01(active), standbys: ceph-ceph07, ceph03 > osd: 78 osds: 78 up, 78 in > > data: > pools: 4 pools, 3240 pgs > objects: 4384k objects, 17533 GB > usage: 53141 GB used, 27311 GB / 80452 GB avail > pgs: 3240 active+clean > > io: > client: 4108 kB/s rd, 10071 kB/s wr, 27 op/s rd, 331 op/s wr > > but my mgr dashboard web interface says: > > > Health > Overall status: HEALTH_WARN > > PG_AVAILABILITY: Reduced data availability: 2563 pgs inactive > > > Anyone know why the discrepency? Hopefully the dashboard is very > mistaken! Everything seems to be operating normally. If I had 2/3 of my > pgs inactive I'm sure all of my rbd backing my VMs would be blocked etc. A situation like this probably indicates that something is going wrong with the mon->mgr synchronisation of health state (it's all calculated in one place and the mon updates the mgr every few seconds). 1. Look for errors in your monitor logs 2. You'll probably find that everything gets back in sync if you restart a mgr daemon John > I'm running ceph-12.2.4-0.el7.x86_64 on CentOS 7. Almost all filestore > except for one OSD which recently had to be replaced which I made > bluestore. I plan to slowly migrate everything over to bluestore over > the course of the next month. > > Thanks! > > -- > Tracy Reed > http://tracyreed.org > Digital signature attached for your safety. > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Place on separate hosts?
Hi, On 04/05/18 08:25, Tracy Reed wrote: > On Fri, May 04, 2018 at 12:18:15AM PDT, Tracy Reed spake thusly: >> https://jcftang.github.io/2012/09/06/going-from-replicating-across-osds-to-replicating-across-hosts-in-a-ceph-cluster/ > >> How can I tell which way mine is configured? I could post the whole >> crushmap if necessary but it's a bit large to copy and paste. > > To further answer my own question (sorry for the spam) the above linked > doc says this should do what I want: > > step chooseleaf firstn 0 type host > > which is what I already have in my crush map. So it looks like the > default is as I want it. In which case I wonder why I had the problem > previously... I guess the only way to know for sure is to stop one osd > node and see what happens. You can ask ceph which OSDs a particular pg is on: root@sto-1-1:~# ceph pg map 71.983 osdmap e435728 pg 71.983 (71.983) -> up [1948,2984,511] acting [1948,2984,511] ...then you can check these are on different hosts. HTH, Matthew signature.asc Description: OpenPGP digital signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Place on separate hosts?
Le vendredi 04 mai 2018 à 00:25 -0700, Tracy Reed a écrit : > On Fri, May 04, 2018 at 12:18:15AM PDT, Tracy Reed spake thusly: > > https://jcftang.github.io/2012/09/06/going-from-replicating-across- > > osds-to-replicating-across-hosts-in-a-ceph-cluster/ > > > > How can I tell which way mine is configured? I could post the whole > > crushmap if necessary but it's a bit large to copy and paste. > > To further answer my own question (sorry for the spam) the above > linked > doc says this should do what I want: > > step chooseleaf firstn 0 type host > > which is what I already have in my crush map. So it looks like the > default is as I want it. In which case I wonder why I had the problem > previously... I guess the only way to know for sure is to stop one > osd > node and see what happens. You can test the crush rules. See http://docs.ceph.com/docs/master/rados/operations/crush-map-edits/ Examples from my own notes: ceph osd getcrushmap -o crushmap crushtool -i crushmap --test --rule 0 --num-rep 4 --show-utilization crushtool -i crushmap --test --rule 0 --num-rep 4 --show-mappings --show-choose-tries --show-statistics | less etc. This helped me validate the placement on different hosts and datacenters. -- Nicolas Huillard ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] ceph mgr module not working
On Fri, May 4, 2018 at 7:26 AM, Tracy Reed wrote: > Hello all, > > I can seemingly enable the balancer ok: > > $ ceph mgr module enable balancer > > but if I try to check its status: > > $ ceph balancer status > Error EINVAL: unrecognized command This generally indicates that something went wrong loading the module -- hopefully it would be complaining about this in "ceph status", but you could also go and look at the log file for the active ceph-mgr daemon and grep it for "balancer". John > or turn it on: > > $ ceph balancer on > Error EINVAL: unrecognized command > > $ which ceph > /bin/ceph > $ rpm -qf /bin/ceph > ceph-common-12.2.4-0.el7.x86_64 > > So it's not like I'm running an old version of the ceph command which > wouldn't know about the balancer. > > I'm running ceph-12.2.4-0.el7.x86_64 on CentOS 7. Almost all filestore > except for one OSD which recently had to be replaced which I made > bluestore. I plan to slowly migrate everything over to bluestore over > the course of the next month. > > Thanks! > > -- > Tracy Reed > http://tracyreed.org > Digital signature attached for your safety. > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Place on separate hosts?
On Fri, May 04, 2018 at 12:18:15AM PDT, Tracy Reed spake thusly: > https://jcftang.github.io/2012/09/06/going-from-replicating-across-osds-to-replicating-across-hosts-in-a-ceph-cluster/ > How can I tell which way mine is configured? I could post the whole > crushmap if necessary but it's a bit large to copy and paste. To further answer my own question (sorry for the spam) the above linked doc says this should do what I want: step chooseleaf firstn 0 type host which is what I already have in my crush map. So it looks like the default is as I want it. In which case I wonder why I had the problem previously... I guess the only way to know for sure is to stop one osd node and see what happens. -- Tracy Reed http://tracyreed.org Digital signature attached for your safety. signature.asc Description: PGP signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Place on separate hosts?
On Fri, May 04, 2018 at 12:08:35AM PDT, Tracy Reed spake thusly: > I've been using ceph for nearly a year and one of the things I ran into > quite a while back was that it seems like ceph is placing copies of > objects on different OSDs but sometimes those OSDs can be on the same > host by default. Is that correct? I discovered this by taking down one > host and having some pgs become inactive. Actually, this (admittedly ancient) document: https://jcftang.github.io/2012/09/06/going-from-replicating-across-osds-to-replicating-across-hosts-in-a-ceph-cluster/ says "As the default CRUSH map replicates across OSD’s I wanted to try replicating data across hosts just to see what would happen." This would seem to align with my experience as far as the default goes. However, this: http://docs.ceph.com/docs/master/rados/operations/crush-map/ says: "When you deploy OSDs they are automatically placed within the CRUSH map under a host node named with the hostname for the host they are running on. This, combined with the default CRUSH failure domain, ensures that replicas or erasure code shards are separated across hosts and a single host failure will not affect availability." How can I tell which way mine is configured? I could post the whole crushmap if necessary but it's a bit large to copy and paste. -- Tracy Reed http://tracyreed.org Digital signature attached for your safety. signature.asc Description: PGP signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Place on separate hosts?
I've been using ceph for nearly a year and one of the things I ran into
quite a while back was that it seems like ceph is placing copies of
objects on different OSDs but sometimes those OSDs can be on the same
host by default. Is that correct? I discovered this by taking down one
host and having some pgs become inactive.
So I guess you could say I want my failure domain to be the host, not
the OSD.
How would I accomplish this? I understand it involves changing the crush
map. I've been reading over
http://docs.ceph.com/docs/master/rados/operations/crush-map/ and it
still isn't clear to me what needs to change. I expect I need to change
the default replicated_ruleset which I'm still running:
$ ceph osd crush rule dump
[
{
"rule_id": 0,
"rule_name": "replicated_ruleset",
"ruleset": 0,
"type": 1,
"min_size": 1,
"max_size": 10,
"steps": [
{
"op": "take",
"item": -1,
"item_name": "default"
},
{
"op": "chooseleaf_firstn",
"num": 0,
"type": "host"
},
{
"op": "emit"
}
]
}
]
And that I need something like:
ceph osd crush rule create-replicated
then:
ceph osd pool set crush_rule
but I'm not sure what the values of
would be in my situation. Maybe:
ceph osd crush rule create-replicated different-host default
but I don't know what failure-domain or class should just by inspecting
my current crush map.
Suggestions are greatly appreciated!
--
Tracy Reed
http://tracyreed.org
Digital signature attached for your safety.
signature.asc
Description: PGP signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
