Please also check if your hosts can reach Ceph nodes on port 3300/TCP.
Additionally, please also check if the configuration works with a single Ceph
monitor address.
Regards
Jayanth
________________________________
From: Muhammad Hanis Irfan Mohd Zaid <hanisirfan.w...@gmail.com>
Sent: Thursday, August 8, 2024 5:57:31 am
To: users@cloudstack.apache.org <users@cloudstack.apache.org>
Cc: ustcweiz...@gmail.com <ustcweiz...@gmail.com>; rohit.ya...@shapeblue.com
<rohit.ya...@shapeblue.com>
Subject: Re: Unable to add Ceph RBD for primary storage (No such file or
directory)
I'm running Ceph 18.2.4 reef (stable).
Can you kindly share any reference on directly adding the pool to KVM? I'm
not really experienced in KVM and Ceph, just experimenting right now. Yep,
the KVM host can ping and telnet to the Ceph MONs port 6789.
Here's the pool and user list from Ceph:
# ceph osd pool ls detail
pool 2 'acs_primary_1' replicated size 3 min_size 2 crush_rule 0
object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 183
lfor 0/0/101 flags hashpspool,selfmanaged_snaps max_bytes 5497558138880
stripe_width 0 application rbd read_balance_score 2.19
# ceph auth ls
client.cloudstack
key: <REDACTED>
caps: [mon] profile rbd
caps: [osd] profile rbd pool=acs_primary_1
I've tried adding the pool back and looks like the same error persisted:
2024-08-08 08:14:45,462 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:9a5c88d1) Attempting to create storage
pool e b5ec036-c08a-3d3d-996d-40968077d391 (RBD) in libvirt
2024-08-08 08:14:45,482 WARN [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:9a5c88d1) Storage pool
eb5ec036-c08a-3d3d-996 d-40968077d391 was not found running in libvirt.
Need to create it.
2024-08-08 08:14:45,482 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:9a5c88d1) Didn't find an existing
storage poo l eb5ec036-c08a-3d3d-996d-40968077d391 by UUID, checking for
pools with duplicate paths
2024-08-08 08:19:45,521 ERROR [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:9a5c88d1) Failed to create RBD storage
pool: org.libvirt.LibvirtException: failed to connect to the RADOS monitor
on: 10.0.32.71,10.0.32.72,10.0.32.73,10.0.32.74,10.0.32.75,: No such file
or directory
2024-08-08 08:19:45,522 ERROR [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:9a5c88d1) Failed to create the RBD
storage pool, cleaning up the libvirt secret
2024-08-08 08:19:45,523 WARN [cloud.agent.Agent]
(agentRequest-Handler-4:null) (logid:9a5c88d1) Caught:
com.cloud.utils.exception.CloudRuntimeException: Failed to create storage
pool: eb5ec036-c08a-3d3d-996d-40968077d391
at
com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:743)
at
com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:364)
at
com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:358)
at
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:42)
at
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:35)
at
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1929)
at com.cloud.agent.Agent.processRequest(Agent.java:683)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1106)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Is the comma (,) at the end of the monitor list in this log supposed to be
there? I can assure you that I don't add any comma in the UI at the end,
when adding the storage pool.
https://imgur.com/a/eN45YWa
Thanks.
On Wed, 7 Aug 2024 at 20:39, Rohit Yadav <rohit.ya...@shapeblue.com> wrote:
> Based on the logs, the error is due to some kind of rbd pool
> configuration. Can you try to add rbd pool on the kvm directly, see if it
> works? You can also try to check if from the KVM host you can reach your
> ceph nodes/mons?
>
> Regards.
>
> Regards.
>
>
>
>
>
> ------------------------------
> *From:* Wei ZHOU <ustcweiz...@gmail.com>
> *Sent:* Wednesday, August 7, 2024 5:45:34 PM
> *To:* Muhammad Hanis Irfan Mohd Zaid <hanisirfan.w...@gmail.com>
> *Cc:* users@cloudstack.apache.org <users@cloudstack.apache.org>
> *Subject:* Re: Unable to add Ceph RBD for primary storage (No such file
> or directory)
>
> Hi,
>
> I just tested adding ceph pool on alma9 (it has the same
> package/version installed), it worked
>
> What's the ceph version ?
>
> -Wei
>
> On Wed, Aug 7, 2024 at 12:11 PM Muhammad Hanis Irfan Mohd Zaid
> <hanisirfan.w...@gmail.com> wrote:
> >
> > I also noticed that comma in the end. I just took a blind eye and
> expected it to be as designed. I don't enter any comma in the end:
> https://ibb.co/N3zMVvc
> >
> > Yep, the package is already installed.
> >
> > # dnf -y install libvirt-daemon-driver-storage-rbd
> > Last metadata expiration check: 1:29:57 ago on Wed 07 Aug 2024 04:37:02
> PM +08.
> > Package libvirt-daemon-driver-storage-rbd-10.0.0-6.6.el9_4.x86_64 is
> already installed.
> > Dependencies resolved.
> > Nothing to do.
> > Complete!
> >
> > I'm running the KVM host in Rocky Linux 9.4 (Blue Onyx) with CloudStack
> 4.19.1.1
> >
> >
> >
> > On Wed, 7 Aug 2024 at 18:02, Wei ZHOU <ustcweiz...@gmail.com> wrote:
> >>
> >> Hi,
> >>
> >> There is a comma (,) after 10.0.32.75 , was it a mistake ?
> >>
> >> org.libvirt.LibvirtException: failed to connect to the RADOS monitor
> >> on: 10.0.32.71,10.0.32.72,10.0.32.73,10.0.32.74,10.0.32.75,: No such
> file
> >> or directory
> >>
> >> Have you installed the package "libvirt-daemon-driver-storage-rbd" on
> >> the kvm host ?
> >>
> >> -Wei
> >>
> >> On Wed, Aug 7, 2024 at 11:27 AM Muhammad Hanis Irfan Mohd Zaid
> >> <hanisirfan.w...@gmail.com> wrote:
> >> >
> >> > I'm trying to add a Ceph RBD pool for primary storage use. I've 5
> Ceph MONs
> >> > in my POC lab. Ping and telnet to all the Ceph MONs with port 6789
> works.
> >> >
> >> > I'm following the steps from this:
> >> > - https://docs.ceph.com/en/reef/rbd/rbd-cloudstack/
> >> > - https://rohityadav.cloud/blog/ceph/
> >> >
> >> > Agent log when specifying 5 monitors:
> >> > 2024-08-07 17:12:34,691 INFO [kvm.storage.LibvirtStorageAdaptor]
> >> > (agentRequest-Handler-3:null) (logid:db5277f2) Attempting to create
> storage
> >> > pool eb5ec036-c08a-3d3d-996d-40968077d391 (RBD) in libvirt
> >> > 2024-08-07 17:12:34,706 WARN [kvm.storage.LibvirtStorageAdaptor]
> >> > (agentRequest-Handler-3:null) (logid:db5277f2) Storage pool
> >> > eb5ec036-c08a-3d3d-996d-40968077d391 was not found running in
> libvirt. Need
> >> > to create it.
> >> > 2024-08-07 17:12:34,706 INFO [kvm.storage.LibvirtStorageAdaptor]
> >> > (agentRequest-Handler-3:null) (logid:db5277f2) Didn't find an existing
> >> > storage pool eb5ec036-c08a-3d3d-996d-40968077d391 by UUID, checking
> for
> >> > pools with duplicate paths
> >> > 2024-08-07 17:17:34,738 ERROR [kvm.storage.LibvirtStorageAdaptor]
> >> > (agentRequest-Handler-3:null) (logid:db5277f2) Failed to create RBD
> storage
> >> > pool: org.libvirt.LibvirtException: failed to connect to the RADOS
> monitor
> >> > on: 10.0.32.71,10.0.32.72,10.0.32.73,10.0.32.74,10.0.32.75,: No such
> file
> >> > or directory
> >> > 2024-08-07 17:17:34,739 ERROR [kvm.storage.LibvirtStorageAdaptor]
> >> > (agentRequest-Handler-3:null) (logid:db5277f2) Failed to create the
> RBD
> >> > storage pool, cleaning up the libvirt secret
> >> > 2024-08-07 17:17:34,739 WARN [cloud.agent.Agent]
> >> > (agentRequest-Handler-3:null) (logid:db5277f2) Caught:
> >> > com.cloud.utils.exception.CloudRuntimeException: Failed to create
> storage
> >> > pool: eb5ec036-c08a-3d3d-996d-40968077d391
> >> > at
> >> >
> com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:743)
> >> > at
> >> >
> com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:364)
> >> > at
> >> >
> com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:358)
> >> > at
> >> >
> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:42)
> >> > at
> >> >
> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:35)
> >> > at
> >> >
> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
> >> > at
> >> >
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1929)
> >> > at com.cloud.agent.Agent.processRequest(Agent.java:683)
> >> > at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1106)
> >> > at com.cloud.utils.nio.Task.call(Task.java:83)
> >> > at com.cloud.utils.nio.Task.call(Task.java:29)
> >> > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> >> > at
> >> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> >> > at
> >> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> >> > at java.base/java.lang.Thread.run(Thread.java:829)
> >> >
> >> > Agent log when specifying 1 monitors:
> >> > 2024-08-07 17:06:09,791 INFO [kvm.storage.LibvirtStorageAdaptor]
> >> > (agentRequest-Handler-2:null) (logid:c790784b) Attempting to create
> storage
> >> > pool 69b2f6e0-12c8-31a3-bdc6-71b3a1e265f2 (RBD) in libvirt
> >> > 2024-08-07 17:06:09,806 WARN [kvm.storage.LibvirtStorageAdaptor]
> >> > (agentRequest-Handler-2:null) (logid:c790784b) Storage pool
> >> > 69b2f6e0-12c8-31a3-bdc6-71b3a1e265f2 was not found running in
> libvirt. Need
> >> > to create it.
> >> > 2024-08-07 17:06:09,806 INFO [kvm.storage.LibvirtStorageAdaptor]
> >> > (agentRequest-Handler-2:null) (logid:c790784b) Didn't find an existing
> >> > storage pool 69b2f6e0-12c8-31a3-bdc6-71b3a1e265f2 by UUID, checking
> for
> >> > pools with duplicate paths
> >> > 2024-08-07 17:11:09,840 ERROR [kvm.storage.LibvirtStorageAdaptor]
> >> > (agentRequest-Handler-2:null) (logid:c790784b) Failed to create RBD
> storage
> >> > pool: org.libvirt.LibvirtException: failed to connect to the RADOS
> monitor
> >> > on: 10.0.32.71,: No such file or directory
> >> > 2024-08-07 17:11:09,840 ERROR [kvm.storage.LibvirtStorageAdaptor]
> >> > (agentRequest-Handler-2:null) (logid:c790784b) Failed to create the
> RBD
> >> > storage pool, cleaning up the libvirt secret
> >> > 2024-08-07 17:11:09,841 WARN [cloud.agent.Agent]
> >> > (agentRequest-Handler-2:null) (logid:c790784b) Caught:
> >> > com.cloud.utils.exception.CloudRuntimeException: Failed to create
> storage
> >> > pool: 69b2f6e0-12c8-31a3-bdc6-71b3a1e265f2
> >> > at
> >> >
> com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:743)
> >> > at
> >> >
> com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:364)
> >> > at
> >> >
> com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:358)
> >> > at
> >> >
> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:42)
> >> > at
> >> >
> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:35)
> >> > at
> >> >
> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
> >> > at
> >> >
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1929)
> >> > at com.cloud.agent.Agent.processRequest(Agent.java:683)
> >> > at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1106)
> >> > at com.cloud.utils.nio.Task.call(Task.java:83)
> >> > at com.cloud.utils.nio.Task.call(Task.java:29)
> >> > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> >> > at
> >> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> >> > at
> >> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> >> > at java.base/java.lang.Thread.run(Thread.java:829)
> >> >
> >> > Ran these commands on my Ceph node but still no luck. I refer this:
> >> > - https://github.com/apache/cloudstack/issues/5741
> >> >
> >> > ceph config set mon auth_expose_insecure_global_id_reclaim false
> >> > ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed
> false
> >> > ceph config set mon auth_allow_insecure_global_id_reclaim false
> >> > ceph orch restart mon
> >> >
> >> > Thanks for the help :)
>
