[ceph-users] ceph 14.2.6 problem with default args to rbd (--name)
Hello, I am fighting with rbd and CEPH_ARGS in order to make typing easier on a client. First I created a keyring on one of the ceph nodes: # ceph auth add client.rainer mon 'profile rbd' osd 'profile rbd' added key for client.rainer Then I added this keyring to /etc/ceph/ceph.keyring on a client machine. # cat /etc/ceph/ceph.keyring: [client.rainer] key = xxyxyxyxyxyxyxyxyxyxyxyxyxyxy== Now on the client I try: # rbd ls 2020-01-20 14:15:49.124 7fa9191f0700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2020-01-20 14:15:49.124 7fa9189ef700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] rbd: couldn't connect to the cluster! rbd: listing images failed: (1) Operation not permitted Using the --name arg things work good: # rbd ls --name client.rainer ... Now to make typing easier I tried: # export CEPH_ARGS="--name client.rainer --keyring=/etc/ceph/ceph.keyring" # rbd ls 2020-01-20 14:18:33.367 7f691177c700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] rbd: couldn't connect to the cluster! rbd: listing images failed: (1) Operation not permitted According to the documentation this should work, but it seems it doesn't. Something I am doing wrong or is this a bug? Thanks Rainer -- Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1 56070 Koblenz, Tel: +49261287 1312 Fax +49261287 100 1312 Web: http://userpages.uni-koblenz.de/~krienke PGP: http://userpages.uni-koblenz.de/~krienke/mypgp.html ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Strange CEPH_ARGS problems
This is not my day :-) Yes this flip beween client.rz and client.user was not intended. Its another typo. When trying to run rbd I used everywhere the same client.rz user and the same keyring /etc/ceph/ceph.client.user.keyring. Sorry Rainer Am 15.11.19 um 11:02 schrieb Janne Johansson: > Is the flip between the client name "rz" and "user" also a mistype? It's > hard to divinate if it is intentional or not since you are mixing it about. > > > Den fre 15 nov. 2019 kl 10:57 skrev Rainer Krienke > mailto:krie...@uni-koblenz.de>>: > > I found a typo in my post: > > Of course I tried > > export CEPH_ARGS="-n client.rz --keyring=" > > and not > > export CEPH_ARGS=="-n client.rz --keyring=" > > Thanks > Rainer > > Am 15.11.19 um 07:46 schrieb Rainer Krienke: > > Hello, > > > > I try to use CEPH_ARGS in order to use eg rbd with a non client.admin > > user and keyring without extra parameters. On a ceph-client with > Ubuntu > > 18.04.3 I get this: > > > > # unset CEPH_ARGS > > # rbd --name=client.user > --keyring=/etc/ceph/ceph.client.user.keyring ls > > a > > b > > c > > > > # export CEPH_ARGS=="-n client.rz --keyring=/etc > > /ceph/ceph.client.user.keyring" > > # rbd ls > > > > rbd: couldn't connect to the cluster! > > rbd: listing images failed: (22) Invalid argument > > > > # export CEPH_ARGS=="--keyring=/etc/ceph/ceph.client.user.keyring"> > Thanks > Rainer > > Am 15.11.19 um 07:46 schrieb Rainer Krienke: > > Hello, > > > > I try to use CEPH_ARGS in order to use eg rbd with a non client.admin > > user and keyring without extra parameters. On a ceph-client with > Ubuntu > > 18.04.3 I get this: > > > > # unset CEPH_ARGS > > # rbd --name=client.user > --keyring=/etc/ceph/ceph.client.user.keyring ls > > a > > b > > c > > > > # export CEPH_ARGS=="-n client.rz --keyring=/etc > > /ceph/ceph.client.user.keyring" > > # rbd ls > > > > rbd: couldn't connect to the cluster! > > rbd: listing images failed: (22) Invalid argument > > > > # export CEPH_ARGS=="--keyring=/etc/ceph/ceph.client.user.keyring" > > # rbd -n client.user ls > > a > > b > > c > > > > Is this the desired behavior? I would like to set both user name and > > keyring to be used, so that I can run rbd without any parameters. > > > > How do you do this? > > > > Thanks > > Rainer > > > > > -- > Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1 > 56070 Koblenz, Tel: +49261287 1312 Fax +49261287 100 1312 > Web: http://userpages.uni-koblenz.de/~krienke > PGP: http://userpages.uni-koblenz.de/~krienke/mypgp.html > ___ > ceph-users mailing list > ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > -- > May the most significant bit of your life be positive. > > # rbd -n client.user ls > > a > > b > > c > > > > Is this the desired behavior? I would like to set both user name and > > keyring to be used, so that I can run rbd without any parameters. > > > > How do you do this? > > > > Thanks > > Rainer > > > > > -- > Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1 > 56070 Koblenz, Tel: +49261287 1312 Fax +49261287 100 1312 > Web: http://userpages.uni-koblenz.de/~krienke > PGP: http://userpages.uni-koblenz.de/~krienke/mypgp.html > ___ > ceph-users mailing list > ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > Thanks > Rainer > > Am 15.11.19 um 07:46 schrieb Rainer Krienke: > > Hello, > > > > I try to use CEPH_ARGS in order to use eg rbd with a non client.admin > > user and keyring without extra parameters. On a ceph-client with > Ubuntu > > 18.04.3 I get this: > > > > # unset CEPH_ARGS &
Re: [ceph-users] Strange CEPH_ARGS problems
I found a typo in my post: Of course I tried export CEPH_ARGS="-n client.rz --keyring=" and not export CEPH_ARGS=="-n client.rz --keyring=" Thanks Rainer Am 15.11.19 um 07:46 schrieb Rainer Krienke: > Hello, > > I try to use CEPH_ARGS in order to use eg rbd with a non client.admin > user and keyring without extra parameters. On a ceph-client with Ubuntu > 18.04.3 I get this: > > # unset CEPH_ARGS > # rbd --name=client.user --keyring=/etc/ceph/ceph.client.user.keyring ls > a > b > c > > # export CEPH_ARGS=="-n client.rz --keyring=/etc > /ceph/ceph.client.user.keyring" > # rbd ls > > rbd: couldn't connect to the cluster! > rbd: listing images failed: (22) Invalid argument > > # export CEPH_ARGS=="--keyring=/etc/ceph/ceph.client.user.keyring" > # rbd -n client.user ls > a > b > c > > Is this the desired behavior? I would like to set both user name and > keyring to be used, so that I can run rbd without any parameters. > > How do you do this? > > Thanks > Rainer > -- Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1 56070 Koblenz, Tel: +49261287 1312 Fax +49261287 100 1312 Web: http://userpages.uni-koblenz.de/~krienke PGP: http://userpages.uni-koblenz.de/~krienke/mypgp.html ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Strange CEPH_ARGS problems
Hello, I try to use CEPH_ARGS in order to use eg rbd with a non client.admin user and keyring without extra parameters. On a ceph-client with Ubuntu 18.04.3 I get this: # unset CEPH_ARGS # rbd --name=client.user --keyring=/etc/ceph/ceph.client.user.keyring ls a b c # export CEPH_ARGS=="-n client.rz --keyring=/etc /ceph/ceph.client.user.keyring" # rbd ls rbd: couldn't connect to the cluster! rbd: listing images failed: (22) Invalid argument # export CEPH_ARGS=="--keyring=/etc/ceph/ceph.client.user.keyring" # rbd -n client.user ls a b c Is this the desired behavior? I would like to set both user name and keyring to be used, so that I can run rbd without any parameters. How do you do this? Thanks Rainer -- Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1 56070 Koblenz, Tel: +49261287 1312 Fax +49261287 100 1312 Web: http://userpages.uni-koblenz.de/~krienke PGP: http://userpages.uni-koblenz.de/~krienke/mypgp.html ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Two questions about ceph update/upgrade strategies
I have a fresh ceph 14.2.1 cluster up and running based on Ubuntu 18.04. It consists of 9 hosts (+1 admin host). The nine hosts have each 16 ceph-osd daemons running, three in these nine hosts also have a ceph-mon and a ceph-mgr daemon running. So three hosts are running osd, mon and also mgr daemons. Now I am unsure about the right way to go for ceph upgrades and linux system host updates. Ceph-Upgrade: Reading the ceph upgrade docs I ask myself how a future upgrade say to 14.2.2 should be performed correctly? The recommendation says to upgrade first monitors, then osds etc... So what is the correct way to go in a mixed setup like mine? Following the rules strictly would mean not to use ceph-deploy install, but instead to log into the mon(/osd) hosts and then upgrade only the ceph-mon package and restart this mon, and then do the same with the other monitors/osd hosts. After all mons have been successfully upgraded I should then continue with upgrading OSDs (ceph-osd package) on one host and restart all osds on this host one after another or reboot the whole host. Then proceed to the next osd-host. Is this the correct and best way to go? Linux system updates: The second point I would like to hear your opinions about is how you handle linux system updates? Since even a non ceph linux system package update might break ceph or even stop the whole linux host from booting, care has to be taken. So how do you handle this problem? Do you run host upgrades only manually in a fixed sequence eg first on a osd/mon host and if the update is successful, then run the linux system package updates on the other hosts? Do you use another strategy? Thanks Rainer -- Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1 56070 Koblenz, Tel: +49261287 1312 Fax +49261287 100 1312 Web: http://userpages.uni-koblenz.de/~krienke PGP: http://userpages.uni-koblenz.de/~krienke/mypgp.html ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Erasure code profiles and crush rules. Missing link...?
Hello, thanks for the hint. I opened a ticket with a feature request to include the ec-profile information in the output of ceph osd pool ls detail. http://tracker.ceph.com/issues/40009 Rainer Am 22.05.19 um 17:04 schrieb Jan Fajerski: > On Wed, May 22, 2019 at 03:38:27PM +0200, Rainer Krienke wrote: >> Am 22.05.19 um 15:16 schrieb Dan van der Ster: >> >> Yes this is basically what I was looking for however I had expected that >> its a little better visible in the output... > Mind opening a tracker ticket on http://tracker.ceph.com/ so we can have > this added to the non-json output of ceph osd pool ls detail? >> -- Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1 56070 Koblenz, Tel: +49261287 1312 Fax +49261287 100 1312 Web: http://userpages.uni-koblenz.de/~krienke PGP: http://userpages.uni-koblenz.de/~krienke/mypgp.html ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Erasure code profiles and crush rules. Missing link...?
Am 22.05.19 um 15:16 schrieb Dan van der Ster: Yes this is basically what I was looking for however I had expected that its a little better visible in the output... Rainer > > Is this what you're looking for? > > # ceph osd pool ls detail -f json | jq .[0].erasure_code_profile > "jera_4plus2" > > -- Dan -- Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1 56070 Koblenz, Tel: +49261287 1312 Fax +49261287 100 1312 Web: http://userpages.uni-koblenz.de/~krienke PGP: http://userpages.uni-koblenz.de/~krienke/mypgp.html ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Erasure code profiles and crush rules. Missing link...?
Hello,
I created an erasure code profile named ecprofile-42 with the following
parameters:
$ ceph osd erasure-code-profile set ecprofile-42 plugin=jerasure k=4 m=2
Next I created a new pool using the ec profile from above:
$ ceph osd pool create my_erasure_pool 64 64 erasure ecprofile-42
The pool created then has an autogenerated crush rule with the contents
as shown at the end of this mail (see: ceph osd crush rule dump
my_erasure_pool).
What I am missing in the output of the crush rule dump below are the k,m
values used for this pool or a "link" from the crushrule to the erasure
code profile that contains these settings and was used creating the pool
and thus the ec crushrule. If I had several ec profiles and pools
created with the different ec profiles how else could I see which k,m
values were used for the different pools?
For a replicated crush rule there is the size parameter which is part of
the crush-rule and indirectly tells you the number of replicas, but what
about erasure coded pools?
Probably there is somewhere the link I am looking for, but I din't find
it yet...
Thanks Rainer
#
# autogenerated crush rule my_erasure_pool:
#
$ ceph osd crush rule dump my_erasure_pool
{
"rule_id": 1,
"rule_name": "my_erasure_pool",
"ruleset": 1,
"type": 3,
"min_size": 3,
"max_size": 6,
"steps": [
{
"op": "set_chooseleaf_tries",
"num": 5
},
{
"op": "set_choose_tries",
"num": 100
},
{
"op": "take",
"item": -1,
"item_name": "default"
},
{
"op": "chooseleaf_indep",
"num": 0,
"type": "host"
},
{
"op": "emit"
}
]
}
--
Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1
56070 Koblenz, Tel: +49261287 1312 Fax +49261287 100 1312
Web: http://userpages.uni-koblenz.de/~krienke
PGP: http://userpages.uni-koblenz.de/~krienke/mypgp.html
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] ceph nautilus namespaces for rbd and rbd image access problem
Am 20.05.19 um 09:06 schrieb Jason Dillaman: >> $ rbd --namespace=testnamespace map rbd/rbdtestns --name client.rainer >> --keyring=/etc/ceph/ceph.keyring >> rbd: sysfs write failed >> rbd: error opening image rbdtestns: (1) Operation not permitted >> In some cases useful info is found in syslog - try "dmesg | tail". >> 2019-05-20 08:18:29.187 7f42ab7fe700 -1 librbd::image::RefreshRequest: >> failed to retrieve pool metadata: (1) Operation not permitted >> 2019-05-20 08:18:29.187 7f42aaffd700 -1 librbd::image::OpenRequest: >> failed to refresh image: (1) Operation not permitted >> 2019-05-20 08:18:29.187 7f42aaffd700 -1 librbd::ImageState: >> 0x561792408860 failed to open image: (1) Operation not permitted >> rbd: map failed: (22) Invalid argument > > Hmm, it looks like we overlooked updating the 'rbd' profile when PR > 27423 [1] was merged into v14.2.1. We'll get that fixed, but in the > meantime, you can add a "class rbd metadata_list" cap on the base pool > (w/o the namespace restriction) [2]. > Thanks for your answer. Well I still have Kernel 4.15 so namespaces won't work for me at the moment. Could you please explain what the magic behind "class rbd metadata_list" is? Is it thought to "simply" allow access to the basepool (rbd in my case), so I authorize access to the pool instead of a namespaces? And if this is true then I do not understand the difference of your class cap compared to a cap like osd 'allow rw pool=rbd'? -- Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1 56070 Koblenz, Tel: +49261287 1312 Fax +49261287 100 1312 Web: http://userpages.uni-koblenz.de/~krienke PGP: http://userpages.uni-koblenz.de/~krienke/mypgp.html ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] ceph nautilus namespaces for rbd and rbd image access problem
Hello, just saw this message on the client when trying and failing to map the rbd image: May 20 08:59:42 client kernel: libceph: bad option at '_pool_ns=testnamespace' Rainer Am 20.05.19 um 08:56 schrieb Rainer Krienke: > Hello, > > on a ceph Nautilus cluster (14.2.1) running on Ubuntu 18.04 I try to set > up rbd images with namespaces in order to allow different clients to > access only their "own" rbd images in different namespaces in just one > pool. The rbd image data are in an erasure encoded pool named "ecpool" > and the metadata in the default "rbd" pool. -- Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1 56070 Koblenz, Tel: +49261287 1312 Fax +49261287 100 1312 Web: http://userpages.uni-koblenz.de/~krienke PGP: http://userpages.uni-koblenz.de/~krienke/mypgp.html ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] ceph nautilus namespaces for rbd and rbd image access problem
Hello, on a ceph Nautilus cluster (14.2.1) running on Ubuntu 18.04 I try to set up rbd images with namespaces in order to allow different clients to access only their "own" rbd images in different namespaces in just one pool. The rbd image data are in an erasure encoded pool named "ecpool" and the metadata in the default "rbd" pool. With this setup I am experiencing trouble when I try to access a rbd image in a namespace from a (OpenSuSE Leap 15.0 with Ceph 14.2.1) client and I do not understand what I am doing wrong. Hope someone can see the problem and give me a hint: # On one of the the ceph servers $ rbd namespace create --namespace testnamespace $ rbd namespace ls NAME testnamespace $ ceph auth caps client.rainer mon 'profile rbd' osd 'profile rbd pool=rbd namespace=testnamespace' $ ceph auth get client.rainer [client.rainer] key = AQCcVt5cHC+WJhBBoRPKhErEYzxGuU8U/GA0xA++ caps mon = "profile rbd" caps osd = "profile rbd pool=rbd namespace=testnamespace" $ rbd create rbd/rbdtestns --namespace testnamespace --size 50G --data-pool=rbd-ecpool $ rbd --namespace testnamespace ls -l NAME SIZE PARENT FMT PROT LOCK rbdtestns 50 GiB 2 On the openSuSE Client: $ rbd --namespace=testnamespace map rbd/rbdtestns --name client.rainer --keyring=/etc/ceph/ceph.keyring rbd: sysfs write failed rbd: error opening image rbdtestns: (1) Operation not permitted In some cases useful info is found in syslog - try "dmesg | tail". 2019-05-20 08:18:29.187 7f42ab7fe700 -1 librbd::image::RefreshRequest: failed to retrieve pool metadata: (1) Operation not permitted 2019-05-20 08:18:29.187 7f42aaffd700 -1 librbd::image::OpenRequest: failed to refresh image: (1) Operation not permitted 2019-05-20 08:18:29.187 7f42aaffd700 -1 librbd::ImageState: 0x561792408860 failed to open image: (1) Operation not permitted rbd: map failed: (22) Invalid argument Thanks for your help Rainer -- Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1 56070 Koblenz, Tel: +49261287 1312 Fax +49261287 100 1312 Web: http://userpages.uni-koblenz.de/~krienke PGP: http://userpages.uni-koblenz.de/~krienke/mypgp.html ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] ceph -s finds 4 pools but ceph osd lspools says no pool which is the expected answer
Hello Greg,
thank you very much for your hint.
If I should see this problem again I will try to restart the ceph-mgr
daemon and see if this helps.
Rainer
>
> I don't really see how this particular error can happen and be
> long-lived, but if you restart the ceph-mgr it will probably resolve
> itself.
> ("ceph osd lspools" looks directly at the OSDMap in the monitor,
> whereas the "ceph -s" data output is generated from the manager's
> pgmap, but there's a tight link where the pgmap gets updated and
> removes dead pools on every new OSDMap the manager sees and I can't
> see how that would go wrong.)
> -Greg
>
>
>> Thanks
>> Rainer
>> --
>> Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1
>> 56070 Koblenz, Web: http://www.uni-koblenz.de/~krienke, Tel: +49261287 1312
>> PGP: http://www.uni-koblenz.de/~krienke/mypgp.html, Fax: +49261287
>> 1001312
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1
56070 Koblenz, Tel: +49261287 1312 Fax +49261287 100 1312
Web: http://userpages.uni-koblenz.de/~krienke
PGP: http://userpages.uni-koblenz.de/~krienke/mypgp.html
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] ceph -s finds 4 pools but ceph osd lspools says no pool which is the expected answer
Hello, since I had no ideas by what the wrong pool number in the ceph -s output could be caused I simply rebooted all machines of this cluster (it does not yet contain any real data) which solved the problem. So it seems that some caching problem might have caused this issue. Thanks Rainer Am 14.05.19 um 20:03 schrieb Rainer Krienke: > Hello, > > for a fresh setup ceph cluster I see a strange difference in the number > of existing pools in the output of ceph -s and what I know that should > be there: no pools at all. > > I set up a fresh Nautilus cluster with 144 OSDs on 9 hosts. Just to play > around I created a pool named rbd with > -- Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1 56070 Koblenz, Tel: +49261287 1312 Fax +49261287 100 1312 Web: http://userpages.uni-koblenz.de/~krienke PGP: http://userpages.uni-koblenz.de/~krienke/mypgp.html ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] ceph -s finds 4 pools but ceph osd lspools says no pool which is the expected answer
Hello, for a fresh setup ceph cluster I see a strange difference in the number of existing pools in the output of ceph -s and what I know that should be there: no pools at all. I set up a fresh Nautilus cluster with 144 OSDs on 9 hosts. Just to play around I created a pool named rbd with $ ceph osd pool create rbd 512 512 replicated In ceph -s I saw the pool but also saw a warning: cluster: id: a-b-c-d-e health: HEALTH_WARN too few PGs per OSD (21 < min 30) So I experimented around, removed the pool (ceph osd pool remove rbd) and it was gone in ceph osd lspools, and created a new one with some more PGs and repeated this a few times with larger PG nums. In the end in the output of ceph -s I see that 4 pools do exist: cluster: id: a-b-c-d-e health: HEALTH_OK services: mon: 3 daemons, quorum c2,c5,c8 (age 8h) mgr: c2(active, since 8h) osd: 144 osds: 144 up (since 8h), 144 in (since 8h) data: pools: 4 pools, 0 pgs objects: 0 objects, 0 B usage: 155 GiB used, 524 TiB / 524 TiB avail pgs: but: $ ceph osd lspools Since I deleted each pool I created, 0 pools is the correct answer. I could add another "ghost" pool by creating another pool named rbd with only 512 PGs and then delete it again right away. ceph -s would then show me 5 pools. This is the way I came from 3 to 4 "ghost pools". This does not seem to happen if I use 2048 PGs for the new pool which I do delete right afterwards. In this case the pool is created and ceph -s shows one pool more (5) and if delete this pool again the counter in ceph -s goes back to 4 again. How can I fix the system so that ceph -s also understands that are actually no pools? There must be some inconsistency. Any ideas? Thanks Rainer -- Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1 56070 Koblenz, Web: http://www.uni-koblenz.de/~krienke, Tel: +49261287 1312 PGP: http://www.uni-koblenz.de/~krienke/mypgp.html, Fax: +49261287 1001312 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Need some advice about Pools and Erasure Coding
I am planning to set up a ceph cluster and already implemented a test cluster where we are going to use RBD images for data storage (9 hosts, each host has 16 OSDs, each OSD 4TB). We would like to use erasure coded (EC) pools here, and so all OSD are bluestore. Since several projects are going to store data on this ceph cluster I think it would make sense to use several EC coded pools for separation of the projects and access control. Now I have some questions I hope someone can help me with: - Do I still (nautilus) need two pools for EC based RBD images, one EC data pool and a second replicated pool for metadatata? - If I do need two pools for RBD images and I want to separate the data of different projects by using different pools with EC coding then how should I handle the metadata pool which contains probably only a small amount of data compared to the data pool? Does it make sense to have *one* replicated metadata pool (eg the default rbd pool) for all projects and one EC pool for each project, or would it be better to create one replicated and one EC pool for each project? - I also thought about the different k+m settings for a EC pool, for example k=4, m=2 compared to k=8 and m=2. Both settings allow for two OSDs to fail without any data loss, but I asked myself which of the two settings would be more performant? On one hand distributing data to more OSDs allows a higher parallel access to the data, that should result in a faster access. On the other hand each OSD has a latency until it can deliver its data shard. So is there a recommandation which of my two k+m examples should be preferred? Thanks in advance for your help Rainer -- Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1 56070 Koblenz, Tel: +49261287 1312 Fax +49261287 100 1312 Web: http://userpages.uni-koblenz.de/~krienke PGP: http://userpages.uni-koblenz.de/~krienke/mypgp.html ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Problems with osd creation in Ubuntu 18.04, ceph 13.2.4-1bionic
Hello, thanks for your answer, but zapping the disk did not make any difference. I still get the same error. Looking at the debug output I found this error message that is probably the root of all trouble: # ceph-volume lvm prepare --bluestore --data /dev/sdg stderr: 2019-02-18 08:29:25.544 7fdaa50ed240 -1 bluestore(/var/lib/ceph/osd/ceph-0/) _read_fsid unparsable uuid I found the bugreport below that seems to be exactly that problem I have: http://tracker.ceph.com/issues/15386 However there seems to be no solution up to now. Does anyone have more information how to get around this problem? Thanks Rainer Am 15.02.19 um 18:12 schrieb David Turner: > I have found that running a zap before all prepare/create commands with > ceph-volume helps things run smoother. Zap is specifically there to > clear everything on a disk away to make the disk ready to be used as an > OSD. Your wipefs command is still fine, but then I would lvm zap the > disk before continuing. I would run the commands like [1] this. I also > prefer the single command lvm create as opposed to lvm prepare and lvm > activate. Try that out and see if you still run into the problems > creating the BlueStore filesystem. > > [1] ceph-volume lvm zap /dev/sdg > ceph-volume lvm prepare --bluestore --data /dev/sdg > > On Thu, Feb 14, 2019 at 10:25 AM Rainer Krienke <mailto:krie...@uni-koblenz.de>> wrote: > > Hi, > > I am quite new to ceph and just try to set up a ceph cluster. Initially > I used ceph-deploy for this but when I tried to create a BlueStore osd > ceph-deploy fails. Next I tried the direct way on one of the OSD-nodes > using ceph-volume to create the osd, but this also fails. Below you can > see what ceph-volume says. > > I ensured that there was no left over lvm VG and LV on the disk sdg > before I started the osd creation for this disk. The very same error > happens also on other disks not just for /dev/sdg. All the disk have 4TB > in size and the linux system is Ubuntu 18.04 and finally ceph is > installed in version 13.2.4-1bionic from this repo: > https://download.ceph.com/debian-mimic. > > There is a VG and two LV's on the system for the ubuntu system itself > that is installed on two separate disks configured as software raid1 and > lvm on top of the raid. But I cannot imagine that this might do any harm > to cephs osd creation. > > Does anyone have an idea what might be wrong? > > Thanks for hints > Rainer > > root@ceph1:~# wipefs -fa /dev/sdg > root@ceph1:~# ceph-volume lvm prepare --bluestore --data /dev/sdg > Running command: /usr/bin/ceph-authtool --gen-print-key > Running command: /usr/bin/ceph --cluster ceph --name > client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring > -i - osd new 14d041d6-0beb-4056-8df2-3920e2febce0 > Running command: /sbin/vgcreate --force --yes > ceph-1433ffd0-0a80-481a-91f5-d7a47b78e17b /dev/sdg > stdout: Physical volume "/dev/sdg" successfully created. > stdout: Volume group "ceph-1433ffd0-0a80-481a-91f5-d7a47b78e17b" > successfully created > Running command: /sbin/lvcreate --yes -l 100%FREE -n > osd-block-14d041d6-0beb-4056-8df2-3920e2febce0 > ceph-1433ffd0-0a80-481a-91f5-d7a47b78e17b > stdout: Logical volume "osd-block-14d041d6-0beb-4056-8df2-3920e2febce0" > created. > Running command: /usr/bin/ceph-authtool --gen-print-key > Running command: /bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-0 > --> Absolute path not found for executable: restorecon > --> Ensure $PATH environment variable contains common executable > locations > Running command: /bin/chown -h ceph:ceph > > /dev/ceph-1433ffd0-0a80-481a-91f5-d7a47b78e17b/osd-block-14d041d6-0beb-4056-8df2-3920e2febce0 > Running command: /bin/chown -R ceph:ceph /dev/dm-8 > Running command: /bin/ln -s > > /dev/ceph-1433ffd0-0a80-481a-91f5-d7a47b78e17b/osd-block-14d041d6-0beb-4056-8df2-3920e2febce0 > /var/lib/ceph/osd/ceph-0/block > Running command: /usr/bin/ceph --cluster ceph --name > client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring > mon getmap -o /var/lib/ceph/osd/ceph-0/activate.monmap > stderr: got monmap epoch 1 > Running command: /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-0/keyring > --create-keyring --name osd.0 --add-key > AQAAY2VcU968HxAAvYWMaJZmriUc4H9bCCp8XQ== > stdout: creating /var/lib/ceph/osd/ceph-0/keyring > added entity osd.0 auth auth(auid = 18446744073709551615 > key=AQAAY2VcU968HxAAvYWMaJZmriUc4H9bCCp8XQ== with 0 caps) > Running command: /bin/
[ceph-users] Problems with osd creation in Ubuntu 18.04, ceph 13.2.4-1bionic
gned long, unsigned long, ceph::buffer::list*, IOContext*, bool)+0x4a7) [0x561371346817] stderr: 8: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned long, unsigned long, ceph::buffer::list*, char*)+0x435) [0x5613713065c5] stderr: 9: (BlueFS::_replay(bool, bool)+0x214) [0x56137130c434] stderr: 10: (BlueFS::mount()+0x1f1) [0x561371310c81] stderr: 11: (BlueStore::_open_db(bool, bool)+0x17cd) [0x56137123704d] stderr: 12: (BlueStore::mkfs()+0x805) [0x561371267fe5] stderr: 13: (OSD::mkfs(CephContext*, ObjectStore*, std::__cxx11::basic_string, std::allocator > const&, uuid_d, int)+0x1b0) [0x561370e10480] stderr: 14: (main()+0x4222) [0x561370cf7462] stderr: 15: (__libc_start_main()+0xe7) [0x7f3fc3695b97] stderr: 16: (_start()+0x2a) [0x561370dc095a] stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. stderr: 0> 2019-02-14 13:45:54.840 7f3fcecb3240 -1 *** Caught signal (Aborted) ** stderr: in thread 7f3fcecb3240 thread_name:ceph-osd stderr: ceph version 13.2.4 (b10be4d44915a4d78a8e06aa31919e74927b142e) mimic (stable) stderr: 1: (()+0x92aa40) [0x561371357a40] stderr: 2: (()+0x12890) [0x7f3fc47d7890] stderr: 3: (gsignal()+0xc7) [0x7f3fc36b2e97] stderr: 4: (abort()+0x141) [0x7f3fc36b4801] stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x250) [0x7f3fc60d3530] stderr: 6: (()+0x26d5a7) [0x7f3fc60d35a7] stderr: 7: (KernelDevice::read(unsigned long, unsigned long, ceph::buffer::list*, IOContext*, bool)+0x4a7) [0x561371346817] stderr: 8: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned long, unsigned long, ceph::buffer::list*, char*)+0x435) [0x5613713065c5] stderr: 9: (BlueFS::_replay(bool, bool)+0x214) [0x56137130c434] stderr: 10: (BlueFS::mount()+0x1f1) [0x561371310c81] stderr: 11: (BlueStore::_open_db(bool, bool)+0x17cd) [0x56137123704d] stderr: 12: (BlueStore::mkfs()+0x805) [0x561371267fe5] stderr: 13: (OSD::mkfs(CephContext*, ObjectStore*, std::__cxx11::basic_string, std::allocator > const&, uuid_d, int)+0x1b0) [0x561370e10480] stderr: 14: (main()+0x4222) [0x561370cf7462] stderr: 15: (__libc_start_main()+0xe7) [0x7f3fc3695b97] stderr: 16: (_start()+0x2a) [0x561370dc095a] stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --> Was unable to complete a new OSD, will rollback changes Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it stderr: purged osd.0 --> RuntimeError: Command failed with exit code 250: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 0 --monmap /var/lib/ceph/osd/ceph-0/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-0/ --osd-uuid 14d041d6-0beb-4056-8df2-3920e2febce0 --setuser ceph --setgroup ceph -- Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1 56070 Koblenz, Tel: +49261287 1312 Fax +49261287 100 1312 Web: http://userpages.uni-koblenz.de/~krienke PGP: http://userpages.uni-koblenz.de/~krienke/mypgp.html ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
