Re: Unable to add Ceph RBD for primary storage (No such file or directory)

Wed, 07 Aug 2024 22:29:24 -0700 



Op 08/08/2024 om 07:08 schreef Muhammad Hanis Irfan Mohd Zaid:

The port TCP/3300 was block on my firewall. As this is my first time using
Ceph, I thought only port TCP/6789 is used for monitors. Thanks Jayanth for
pointing that out.



Great!


Also thanks to Wido for clarifying about the comma. Great work from you for
the RBD integration to CloudStack.





You're welcome! If I can make one suggestion: Use Round Robin DNS and 
not add the monitors on IP-Address to CloudStack.


mon-ceph.mydomain.tld A 10.0.32.71
mon-ceph.mydomain.tld A 10.0.32.72
mon-ceph.mydomain.tld A 10.0.32.73
mon-ceph.mydomain.tld A 10.0.32.74
mon-ceph.mydomain.tld A 10.0.32.7


Then use this hostname in CS. Monitor IPs can change over time when 
using cephadm and you don't want to be updating your CloudStack config 
all the time. Rely on DNS and make it easier that way.



There are a number of other reasons, too, but they require some very 
long explanations.


Wido

 I suppose this is resolved. Thanks to all.


On Thu, 8 Aug 2024, 12:42 Wido den Hollander, <w...@widodh.nl> wrote:




Op 08/08/2024 om 02:27 schreef Muhammad Hanis Irfan Mohd Zaid:

I'm running Ceph 18.2.4 reef (stable).

Can you kindly share any reference on directly adding the pool to KVM?

I'm

not really experienced in KVM and Ceph, just experimenting right now.

Yep,

the KVM host can ping and telnet to the Ceph MONs port 6789.

Here's the pool and user list from Ceph:

# ceph osd pool ls detail
pool 2 'acs_primary_1' replicated size 3 min_size 2 crush_rule 0
object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change

183

lfor 0/0/101 flags hashpspool,selfmanaged_snaps max_bytes 5497558138880
stripe_width 0 application rbd read_balance_score 2.19

# ceph auth ls
client.cloudstack
          key: <REDACTED>
          caps: [mon] profile rbd
          caps: [osd] profile rbd pool=acs_primary_1


Double checked the poolname you provide to CloudStack? This is
'acs_primary_1' and not 'rbd' my any chance?



I've tried adding the pool back and looks like the same error persisted:

2024-08-08 08:14:45,462 INFO  [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:9a5c88d1) Attempting to create

storage

pool e  b5ec036-c08a-3d3d-996d-40968077d391 (RBD) in libvirt
2024-08-08 08:14:45,482 WARN  [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:9a5c88d1) Storage pool
eb5ec036-c08a-3d3d-996  d-40968077d391 was not found running in libvirt.
Need to create it.
2024-08-08 08:14:45,482 INFO  [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:9a5c88d1) Didn't find an existing
storage poo  l eb5ec036-c08a-3d3d-996d-40968077d391 by UUID, checking for
pools with duplicate paths
2024-08-08 08:19:45,521 ERROR [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:9a5c88d1) Failed to create RBD

storage

pool: org.libvirt.LibvirtException: failed to connect to the RADOS

monitor

on: 10.0.32.71,10.0.32.72,10.0.32.73,10.0.32.74,10.0.32.75,: No such file
or directory
2024-08-08 08:19:45,522 ERROR [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:9a5c88d1) Failed to create the RBD
storage pool, cleaning up the libvirt secret
2024-08-08 08:19:45,523 WARN  [cloud.agent.Agent]
(agentRequest-Handler-4:null) (logid:9a5c88d1) Caught:
com.cloud.utils.exception.CloudRuntimeException: Failed to create storage
pool: eb5ec036-c08a-3d3d-996d-40968077d391
          at


com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:743)

          at


com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:364)

          at


com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:358)

          at


com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:42)

          at


com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:35)

          at


com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)

          at


com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1929)

          at com.cloud.agent.Agent.processRequest(Agent.java:683)
          at

com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1106)

          at com.cloud.utils.nio.Task.call(Task.java:83)
          at com.cloud.utils.nio.Task.call(Task.java:29)
          at
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
          at


java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)

          at


java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

          at java.base/java.lang.Thread.run(Thread.java:829)

Is the comma (,) at the end of the monitor list in this log supposed to

be

there? I can assure you that I don't add any comma in the UI at the end,
when adding the storage pool.



That comma is normal! (I wrote that piece of code)

Wido


https://imgur.com/a/eN45YWa

Thanks.

On Wed, 7 Aug 2024 at 20:39, Rohit Yadav <rohit.ya...@shapeblue.com>

wrote:



Based on the logs, the error is due to some kind of rbd pool
configuration. Can you try to add rbd pool on the kvm directly, see if

it

works? You can also try to check if from the KVM host you can reach your
ceph nodes/mons?

Regards.

Regards.





------------------------------
*From:* Wei ZHOU <ustcweiz...@gmail.com>
*Sent:* Wednesday, August 7, 2024 5:45:34 PM
*To:* Muhammad Hanis Irfan Mohd Zaid <hanisirfan.w...@gmail.com>
*Cc:* users@cloudstack.apache.org <users@cloudstack.apache.org>
*Subject:* Re: Unable to add Ceph RBD for primary storage (No such file
or directory)

Hi,

I just tested adding ceph pool on alma9 (it has the same
package/version installed), it worked

What's the ceph version ?

-Wei

On Wed, Aug 7, 2024 at 12:11 PM Muhammad Hanis Irfan Mohd Zaid
<hanisirfan.w...@gmail.com> wrote:


I also noticed that comma in the end. I just took a blind eye and

expected it to be as designed. I don't enter any comma in the end:
https://ibb.co/N3zMVvc


Yep, the package is already installed.

# dnf -y install libvirt-daemon-driver-storage-rbd
Last metadata expiration check: 1:29:57 ago on Wed 07 Aug 2024 04:37:02

PM +08.

Package libvirt-daemon-driver-storage-rbd-10.0.0-6.6.el9_4.x86_64 is

already installed.

Dependencies resolved.
Nothing to do.
Complete!

I'm running the KVM host in Rocky Linux 9.4 (Blue Onyx) with CloudStack

4.19.1.1




On Wed, 7 Aug 2024 at 18:02, Wei ZHOU <ustcweiz...@gmail.com> wrote:


Hi,

There is a comma (,) after 10.0.32.75 , was it a mistake ?

org.libvirt.LibvirtException: failed to connect to the RADOS monitor
on: 10.0.32.71,10.0.32.72,10.0.32.73,10.0.32.74,10.0.32.75,: No such

file

or directory

Have you installed the package "libvirt-daemon-driver-storage-rbd" on
the kvm host ?

-Wei

On Wed, Aug 7, 2024 at 11:27 AM Muhammad Hanis Irfan Mohd Zaid
<hanisirfan.w...@gmail.com> wrote:


I'm trying to add a Ceph RBD pool for primary storage use. I've 5

Ceph MONs

in my POC lab. Ping and telnet to all the Ceph MONs with port 6789

works.


I'm following the steps from this:
- https://docs.ceph.com/en/reef/rbd/rbd-cloudstack/
- https://rohityadav.cloud/blog/ceph/

Agent log when specifying 5 monitors:
2024-08-07 17:12:34,691 INFO  [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-3:null) (logid:db5277f2) Attempting to create

storage

pool eb5ec036-c08a-3d3d-996d-40968077d391 (RBD) in libvirt
2024-08-07 17:12:34,706 WARN  [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-3:null) (logid:db5277f2) Storage pool
eb5ec036-c08a-3d3d-996d-40968077d391 was not found running in

libvirt. Need

to create it.
2024-08-07 17:12:34,706 INFO  [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-3:null) (logid:db5277f2) Didn't find an

existing

storage pool eb5ec036-c08a-3d3d-996d-40968077d391 by UUID, checking

for

pools with duplicate paths
2024-08-07 17:17:34,738 ERROR [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-3:null) (logid:db5277f2) Failed to create RBD

storage

pool: org.libvirt.LibvirtException: failed to connect to the RADOS

monitor

on: 10.0.32.71,10.0.32.72,10.0.32.73,10.0.32.74,10.0.32.75,: No such

file

or directory
2024-08-07 17:17:34,739 ERROR [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-3:null) (logid:db5277f2) Failed to create the

RBD

storage pool, cleaning up the libvirt secret
2024-08-07 17:17:34,739 WARN  [cloud.agent.Agent]
(agentRequest-Handler-3:null) (logid:db5277f2) Caught:
com.cloud.utils.exception.CloudRuntimeException: Failed to create

storage

pool: eb5ec036-c08a-3d3d-996d-40968077d391
at




com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:743)

at




com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:364)

at




com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:358)

at




com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:42)

at




com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:35)

at




com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)

at




com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1929)

at com.cloud.agent.Agent.processRequest(Agent.java:683)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1106)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at




java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)

at




java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

at java.base/java.lang.Thread.run(Thread.java:829)

Agent log when specifying 1 monitors:
2024-08-07 17:06:09,791 INFO  [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-2:null) (logid:c790784b) Attempting to create

storage

pool 69b2f6e0-12c8-31a3-bdc6-71b3a1e265f2 (RBD) in libvirt
2024-08-07 17:06:09,806 WARN  [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-2:null) (logid:c790784b) Storage pool
69b2f6e0-12c8-31a3-bdc6-71b3a1e265f2 was not found running in

libvirt. Need

to create it.
2024-08-07 17:06:09,806 INFO  [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-2:null) (logid:c790784b) Didn't find an

existing

storage pool 69b2f6e0-12c8-31a3-bdc6-71b3a1e265f2 by UUID, checking

for

pools with duplicate paths
2024-08-07 17:11:09,840 ERROR [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-2:null) (logid:c790784b) Failed to create RBD

storage

pool: org.libvirt.LibvirtException: failed to connect to the RADOS

monitor

on: 10.0.32.71,: No such file or directory
2024-08-07 17:11:09,840 ERROR [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-2:null) (logid:c790784b) Failed to create the

RBD

storage pool, cleaning up the libvirt secret
2024-08-07 17:11:09,841 WARN  [cloud.agent.Agent]
(agentRequest-Handler-2:null) (logid:c790784b) Caught:
com.cloud.utils.exception.CloudRuntimeException: Failed to create

storage

pool: 69b2f6e0-12c8-31a3-bdc6-71b3a1e265f2
at




com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:743)

at




com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:364)

at




com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:358)

at




com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:42)

at




com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:35)

at




com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)

at




com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1929)

at com.cloud.agent.Agent.processRequest(Agent.java:683)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1106)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at




java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)

at




java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

at java.base/java.lang.Thread.run(Thread.java:829)

Ran these commands on my Ceph node but still no luck. I refer this:
- https://github.com/apache/cloudstack/issues/5741

ceph config set mon auth_expose_insecure_global_id_reclaim false
ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed

false

ceph config set mon auth_allow_insecure_global_id_reclaim false
ceph orch restart mon

Thanks for the help :)






























					Reply via email to