Yep I was thinking to use DNS for it as per docs. But, I need to plan to improve our DNS servers for production use. This is just for POC (both CloudStack and Ceph cluster).
I'll surely take your suggestion into consideration, Wido. On Thu, 8 Aug 2024, 13:28 Wido den Hollander, <w...@widodh.nl> wrote: > > > Op 08/08/2024 om 07:08 schreef Muhammad Hanis Irfan Mohd Zaid: > > The port TCP/3300 was block on my firewall. As this is my first time > using > > Ceph, I thought only port TCP/6789 is used for monitors. Thanks Jayanth > for > > pointing that out. > > > > Great! > > > Also thanks to Wido for clarifying about the comma. Great work from you > for > > the RBD integration to CloudStack. > > > > > > You're welcome! If I can make one suggestion: Use Round Robin DNS and > not add the monitors on IP-Address to CloudStack. > > mon-ceph.mydomain.tld A 10.0.32.71 > mon-ceph.mydomain.tld A 10.0.32.72 > mon-ceph.mydomain.tld A 10.0.32.73 > mon-ceph.mydomain.tld A 10.0.32.74 > mon-ceph.mydomain.tld A 10.0.32.7 > > Then use this hostname in CS. Monitor IPs can change over time when > using cephadm and you don't want to be updating your CloudStack config > all the time. Rely on DNS and make it easier that way. > > There are a number of other reasons, too, but they require some very > long explanations. > > Wido > > I suppose this is resolved. Thanks to all. > > > > On Thu, 8 Aug 2024, 12:42 Wido den Hollander, <w...@widodh.nl> wrote: > > > >> > >> > >> Op 08/08/2024 om 02:27 schreef Muhammad Hanis Irfan Mohd Zaid: > >>> I'm running Ceph 18.2.4 reef (stable). > >>> > >>> Can you kindly share any reference on directly adding the pool to KVM? > >> I'm > >>> not really experienced in KVM and Ceph, just experimenting right now. > >> Yep, > >>> the KVM host can ping and telnet to the Ceph MONs port 6789. > >>> > >>> Here's the pool and user list from Ceph: > >>> > >>> # ceph osd pool ls detail > >>> pool 2 'acs_primary_1' replicated size 3 min_size 2 crush_rule 0 > >>> object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change > >> 183 > >>> lfor 0/0/101 flags hashpspool,selfmanaged_snaps max_bytes 5497558138880 > >>> stripe_width 0 application rbd read_balance_score 2.19 > >>> > >>> # ceph auth ls > >>> client.cloudstack > >>> key: <REDACTED> > >>> caps: [mon] profile rbd > >>> caps: [osd] profile rbd pool=acs_primary_1 > >> > >> Double checked the poolname you provide to CloudStack? This is > >> 'acs_primary_1' and not 'rbd' my any chance? > >> > >>> > >>> I've tried adding the pool back and looks like the same error > persisted: > >>> > >>> 2024-08-08 08:14:45,462 INFO [kvm.storage.LibvirtStorageAdaptor] > >>> (agentRequest-Handler-4:null) (logid:9a5c88d1) Attempting to create > >> storage > >>> pool e b5ec036-c08a-3d3d-996d-40968077d391 (RBD) in libvirt > >>> 2024-08-08 08:14:45,482 WARN [kvm.storage.LibvirtStorageAdaptor] > >>> (agentRequest-Handler-4:null) (logid:9a5c88d1) Storage pool > >>> eb5ec036-c08a-3d3d-996 d-40968077d391 was not found running in > libvirt. > >>> Need to create it. > >>> 2024-08-08 08:14:45,482 INFO [kvm.storage.LibvirtStorageAdaptor] > >>> (agentRequest-Handler-4:null) (logid:9a5c88d1) Didn't find an existing > >>> storage poo l eb5ec036-c08a-3d3d-996d-40968077d391 by UUID, checking > for > >>> pools with duplicate paths > >>> 2024-08-08 08:19:45,521 ERROR [kvm.storage.LibvirtStorageAdaptor] > >>> (agentRequest-Handler-4:null) (logid:9a5c88d1) Failed to create RBD > >> storage > >>> pool: org.libvirt.LibvirtException: failed to connect to the RADOS > >> monitor > >>> on: 10.0.32.71,10.0.32.72,10.0.32.73,10.0.32.74,10.0.32.75,: No such > file > >>> or directory > >>> 2024-08-08 08:19:45,522 ERROR [kvm.storage.LibvirtStorageAdaptor] > >>> (agentRequest-Handler-4:null) (logid:9a5c88d1) Failed to create the RBD > >>> storage pool, cleaning up the libvirt secret > >>> 2024-08-08 08:19:45,523 WARN [cloud.agent.Agent] > >>> (agentRequest-Handler-4:null) (logid:9a5c88d1) Caught: > >>> com.cloud.utils.exception.CloudRuntimeException: Failed to create > storage > >>> pool: eb5ec036-c08a-3d3d-996d-40968077d391 > >>> at > >>> > >> > com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:743) > >>> at > >>> > >> > com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:364) > >>> at > >>> > >> > com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:358) > >>> at > >>> > >> > com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:42) > >>> at > >>> > >> > com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:35) > >>> at > >>> > >> > com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78) > >>> at > >>> > >> > com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1929) > >>> at com.cloud.agent.Agent.processRequest(Agent.java:683) > >>> at > >> com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1106) > >>> at com.cloud.utils.nio.Task.call(Task.java:83) > >>> at com.cloud.utils.nio.Task.call(Task.java:29) > >>> at > >>> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > >>> at > >>> > >> > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > >>> at > >>> > >> > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > >>> at java.base/java.lang.Thread.run(Thread.java:829) > >>> > >>> Is the comma (,) at the end of the monitor list in this log supposed to > >> be > >>> there? I can assure you that I don't add any comma in the UI at the > end, > >>> when adding the storage pool. > >>> > >> > >> That comma is normal! (I wrote that piece of code) > >> > >> Wido > >> > >>> https://imgur.com/a/eN45YWa > >>> > >>> Thanks. > >>> > >>> On Wed, 7 Aug 2024 at 20:39, Rohit Yadav <rohit.ya...@shapeblue.com> > >> wrote: > >>> > >>>> Based on the logs, the error is due to some kind of rbd pool > >>>> configuration. Can you try to add rbd pool on the kvm directly, see if > >> it > >>>> works? You can also try to check if from the KVM host you can reach > your > >>>> ceph nodes/mons? > >>>> > >>>> Regards. > >>>> > >>>> Regards. > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> ------------------------------ > >>>> *From:* Wei ZHOU <ustcweiz...@gmail.com> > >>>> *Sent:* Wednesday, August 7, 2024 5:45:34 PM > >>>> *To:* Muhammad Hanis Irfan Mohd Zaid <hanisirfan.w...@gmail.com> > >>>> *Cc:* users@cloudstack.apache.org <users@cloudstack.apache.org> > >>>> *Subject:* Re: Unable to add Ceph RBD for primary storage (No such > file > >>>> or directory) > >>>> > >>>> Hi, > >>>> > >>>> I just tested adding ceph pool on alma9 (it has the same > >>>> package/version installed), it worked > >>>> > >>>> What's the ceph version ? > >>>> > >>>> -Wei > >>>> > >>>> On Wed, Aug 7, 2024 at 12:11 PM Muhammad Hanis Irfan Mohd Zaid > >>>> <hanisirfan.w...@gmail.com> wrote: > >>>>> > >>>>> I also noticed that comma in the end. I just took a blind eye and > >>>> expected it to be as designed. I don't enter any comma in the end: > >>>> https://ibb.co/N3zMVvc > >>>>> > >>>>> Yep, the package is already installed. > >>>>> > >>>>> # dnf -y install libvirt-daemon-driver-storage-rbd > >>>>> Last metadata expiration check: 1:29:57 ago on Wed 07 Aug 2024 > 04:37:02 > >>>> PM +08. > >>>>> Package libvirt-daemon-driver-storage-rbd-10.0.0-6.6.el9_4.x86_64 is > >>>> already installed. > >>>>> Dependencies resolved. > >>>>> Nothing to do. > >>>>> Complete! > >>>>> > >>>>> I'm running the KVM host in Rocky Linux 9.4 (Blue Onyx) with > CloudStack > >>>> 4.19.1.1 > >>>>> > >>>>> > >>>>> > >>>>> On Wed, 7 Aug 2024 at 18:02, Wei ZHOU <ustcweiz...@gmail.com> wrote: > >>>>>> > >>>>>> Hi, > >>>>>> > >>>>>> There is a comma (,) after 10.0.32.75 , was it a mistake ? > >>>>>> > >>>>>> org.libvirt.LibvirtException: failed to connect to the RADOS monitor > >>>>>> on: 10.0.32.71,10.0.32.72,10.0.32.73,10.0.32.74,10.0.32.75,: No such > >>>> file > >>>>>> or directory > >>>>>> > >>>>>> Have you installed the package "libvirt-daemon-driver-storage-rbd" > on > >>>>>> the kvm host ? > >>>>>> > >>>>>> -Wei > >>>>>> > >>>>>> On Wed, Aug 7, 2024 at 11:27 AM Muhammad Hanis Irfan Mohd Zaid > >>>>>> <hanisirfan.w...@gmail.com> wrote: > >>>>>>> > >>>>>>> I'm trying to add a Ceph RBD pool for primary storage use. I've 5 > >>>> Ceph MONs > >>>>>>> in my POC lab. Ping and telnet to all the Ceph MONs with port 6789 > >>>> works. > >>>>>>> > >>>>>>> I'm following the steps from this: > >>>>>>> - https://docs.ceph.com/en/reef/rbd/rbd-cloudstack/ > >>>>>>> - https://rohityadav.cloud/blog/ceph/ > >>>>>>> > >>>>>>> Agent log when specifying 5 monitors: > >>>>>>> 2024-08-07 17:12:34,691 INFO [kvm.storage.LibvirtStorageAdaptor] > >>>>>>> (agentRequest-Handler-3:null) (logid:db5277f2) Attempting to create > >>>> storage > >>>>>>> pool eb5ec036-c08a-3d3d-996d-40968077d391 (RBD) in libvirt > >>>>>>> 2024-08-07 17:12:34,706 WARN [kvm.storage.LibvirtStorageAdaptor] > >>>>>>> (agentRequest-Handler-3:null) (logid:db5277f2) Storage pool > >>>>>>> eb5ec036-c08a-3d3d-996d-40968077d391 was not found running in > >>>> libvirt. Need > >>>>>>> to create it. > >>>>>>> 2024-08-07 17:12:34,706 INFO [kvm.storage.LibvirtStorageAdaptor] > >>>>>>> (agentRequest-Handler-3:null) (logid:db5277f2) Didn't find an > >> existing > >>>>>>> storage pool eb5ec036-c08a-3d3d-996d-40968077d391 by UUID, checking > >>>> for > >>>>>>> pools with duplicate paths > >>>>>>> 2024-08-07 17:17:34,738 ERROR [kvm.storage.LibvirtStorageAdaptor] > >>>>>>> (agentRequest-Handler-3:null) (logid:db5277f2) Failed to create RBD > >>>> storage > >>>>>>> pool: org.libvirt.LibvirtException: failed to connect to the RADOS > >>>> monitor > >>>>>>> on: 10.0.32.71,10.0.32.72,10.0.32.73,10.0.32.74,10.0.32.75,: No > such > >>>> file > >>>>>>> or directory > >>>>>>> 2024-08-07 17:17:34,739 ERROR [kvm.storage.LibvirtStorageAdaptor] > >>>>>>> (agentRequest-Handler-3:null) (logid:db5277f2) Failed to create the > >>>> RBD > >>>>>>> storage pool, cleaning up the libvirt secret > >>>>>>> 2024-08-07 17:17:34,739 WARN [cloud.agent.Agent] > >>>>>>> (agentRequest-Handler-3:null) (logid:db5277f2) Caught: > >>>>>>> com.cloud.utils.exception.CloudRuntimeException: Failed to create > >>>> storage > >>>>>>> pool: eb5ec036-c08a-3d3d-996d-40968077d391 > >>>>>>> at > >>>>>>> > >>>> > >> > com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:743) > >>>>>>> at > >>>>>>> > >>>> > >> > com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:364) > >>>>>>> at > >>>>>>> > >>>> > >> > com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:358) > >>>>>>> at > >>>>>>> > >>>> > >> > com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:42) > >>>>>>> at > >>>>>>> > >>>> > >> > com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:35) > >>>>>>> at > >>>>>>> > >>>> > >> > com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78) > >>>>>>> at > >>>>>>> > >>>> > >> > com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1929) > >>>>>>> at com.cloud.agent.Agent.processRequest(Agent.java:683) > >>>>>>> at > com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1106) > >>>>>>> at com.cloud.utils.nio.Task.call(Task.java:83) > >>>>>>> at com.cloud.utils.nio.Task.call(Task.java:29) > >>>>>>> at > java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > >>>>>>> at > >>>>>>> > >>>> > >> > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > >>>>>>> at > >>>>>>> > >>>> > >> > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > >>>>>>> at java.base/java.lang.Thread.run(Thread.java:829) > >>>>>>> > >>>>>>> Agent log when specifying 1 monitors: > >>>>>>> 2024-08-07 17:06:09,791 INFO [kvm.storage.LibvirtStorageAdaptor] > >>>>>>> (agentRequest-Handler-2:null) (logid:c790784b) Attempting to create > >>>> storage > >>>>>>> pool 69b2f6e0-12c8-31a3-bdc6-71b3a1e265f2 (RBD) in libvirt > >>>>>>> 2024-08-07 17:06:09,806 WARN [kvm.storage.LibvirtStorageAdaptor] > >>>>>>> (agentRequest-Handler-2:null) (logid:c790784b) Storage pool > >>>>>>> 69b2f6e0-12c8-31a3-bdc6-71b3a1e265f2 was not found running in > >>>> libvirt. Need > >>>>>>> to create it. > >>>>>>> 2024-08-07 17:06:09,806 INFO [kvm.storage.LibvirtStorageAdaptor] > >>>>>>> (agentRequest-Handler-2:null) (logid:c790784b) Didn't find an > >> existing > >>>>>>> storage pool 69b2f6e0-12c8-31a3-bdc6-71b3a1e265f2 by UUID, checking > >>>> for > >>>>>>> pools with duplicate paths > >>>>>>> 2024-08-07 17:11:09,840 ERROR [kvm.storage.LibvirtStorageAdaptor] > >>>>>>> (agentRequest-Handler-2:null) (logid:c790784b) Failed to create RBD > >>>> storage > >>>>>>> pool: org.libvirt.LibvirtException: failed to connect to the RADOS > >>>> monitor > >>>>>>> on: 10.0.32.71,: No such file or directory > >>>>>>> 2024-08-07 17:11:09,840 ERROR [kvm.storage.LibvirtStorageAdaptor] > >>>>>>> (agentRequest-Handler-2:null) (logid:c790784b) Failed to create the > >>>> RBD > >>>>>>> storage pool, cleaning up the libvirt secret > >>>>>>> 2024-08-07 17:11:09,841 WARN [cloud.agent.Agent] > >>>>>>> (agentRequest-Handler-2:null) (logid:c790784b) Caught: > >>>>>>> com.cloud.utils.exception.CloudRuntimeException: Failed to create > >>>> storage > >>>>>>> pool: 69b2f6e0-12c8-31a3-bdc6-71b3a1e265f2 > >>>>>>> at > >>>>>>> > >>>> > >> > com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:743) > >>>>>>> at > >>>>>>> > >>>> > >> > com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:364) > >>>>>>> at > >>>>>>> > >>>> > >> > com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:358) > >>>>>>> at > >>>>>>> > >>>> > >> > com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:42) > >>>>>>> at > >>>>>>> > >>>> > >> > com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:35) > >>>>>>> at > >>>>>>> > >>>> > >> > com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78) > >>>>>>> at > >>>>>>> > >>>> > >> > com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1929) > >>>>>>> at com.cloud.agent.Agent.processRequest(Agent.java:683) > >>>>>>> at > com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1106) > >>>>>>> at com.cloud.utils.nio.Task.call(Task.java:83) > >>>>>>> at com.cloud.utils.nio.Task.call(Task.java:29) > >>>>>>> at > java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > >>>>>>> at > >>>>>>> > >>>> > >> > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > >>>>>>> at > >>>>>>> > >>>> > >> > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > >>>>>>> at java.base/java.lang.Thread.run(Thread.java:829) > >>>>>>> > >>>>>>> Ran these commands on my Ceph node but still no luck. I refer this: > >>>>>>> - https://github.com/apache/cloudstack/issues/5741 > >>>>>>> > >>>>>>> ceph config set mon auth_expose_insecure_global_id_reclaim false > >>>>>>> ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed > >>>> false > >>>>>>> ceph config set mon auth_allow_insecure_global_id_reclaim false > >>>>>>> ceph orch restart mon > >>>>>>> > >>>>>>> Thanks for the help :) > >>>> > >>> > >> > > >
