li-liwen opened a new issue, #8405:
URL: https://github.com/apache/cloudstack/issues/8405
<!--
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release and main branch are affected too.
Always add information AFTER of these HTML comments, but no need to delete
the comments.
-->
##### ISSUE TYPE
<!-- Pick one below and delete the rest -->
* Bug Report
##### COMPONENT NAME
<!--
Categorize the issue, e.g. API, VR, VPN, UI, etc.
-->
~~~
UI
Primary storage
~~~
##### CLOUDSTACK VERSION
<!--
New line separated list of affected versions, commit ID for issues on main
branch.
-->
~~~
4.18.1.0
~~~
##### CONFIGURATION
<!--
Information about the configuration if relevant, e.g. basic network,
advanced networking, etc. N/A otherwise
-->
Advanced networking
##### OS / ENVIRONMENT
<!--
Information about the environment if relevant, N/A otherwise
-->
I am using Ubuntu server 22.04 as KVM hypervisor (Libvirt 8.0.0, QEMU 6.2.0)
with ceph (18.2.1) installed on all hypervisors for a hyper-converged setup.
The ceph cluster was bootstrapped by Cephadm.
The ceph cluster contains three separated monitor nodes , and five KVM+Ceph
nodes. In the five KVM+Ceph nodes, two of them also function as monitor nodes.
##### SUMMARY
<!-- Explain the problem/feature briefly -->
Cannot add ceph cluster as primary storage through Web UI.
I am also running a Proxmox cluster and I am able to add the ceph cluster in
Proxmox with the same user and keys. In Proxmox, it works as expected.
##### STEPS TO REPRODUCE
<!--
For bugs, show exactly how to reproduce the problem, using a minimal
test-case. Use Screenshots if accurate.
For new features, show how the feature would be used.
-->
<!-- Paste example playbooks or commands between quotes below -->
1. Install Ubuntu server 22.04 LTS on all machines.
2. Install cloudstack 4.18.1.0 management server as official document.
3. Bootstrape a Ceph Reef cluster through Cephadm (I have tried to install
Cephadm with apt/curl, same result).
4. Install KVM and prepare networking on all KVM hosts.
5. Create cloudstack pool and init rbd
~~~
ceph osd pool create cloudstack
rbd pool init cloudstack
~~~
6. Create cloudstack user
~~~
ceph auth get-or-create client.cloudstack mon 'profile rbd' osd 'profile rbd
pool=cloudstack'
~~~
Additionally, I also tried propagate the ceph.conf file and keyrings to each
hosts but not working (even with admin keys).
I have read the thread #6463 and tried the following ceph configuration
commands, but still no progress:
~~~
ceph config set mon auth_expose_insecure_global_id_reclaim false
ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false
ceph config set mon auth_allow_insecure_global_id_reclaim false
~~~
The KVM hosts can use the ceph cluster directly if provided the admin
keyring. I used the following command:
~~~
qemu-img create -f rbd rbd:cloudstack/new-libvirt-image 2G
~~~
<!-- You can also paste gist.github.com links for larger files -->
##### EXPECTED RESULTS
<!-- What did you expect to happen when running the steps above? -->
~~~
The ceph cluster is added to cloudstack successfully.
~~~
##### ACTUAL RESULTS
<!-- What actually happened? -->
<!-- Paste verbatim command output between quotes below -->
This is the cloudstack troubleshooting log right after I click the OK button
in adding storage:
~~~
2023-12-24 16:13:16,511 DEBUG [c.c.a.t.Request]
(AgentManager-Handler-11:null) (logid:) Seq 7-789818784650103765: Processing:
{ Ans: , MgmtId: 206863094307331, via: 7, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.Answer":{"result":"false","details":"com.cloud.utils.exception.CloudRuntimeException:
Failed to create storage pool: bb8473b7-f815-3d24-9bfd-885edcfd229b
2023-12-24 16:13:16,511 DEBUG [c.c.a.m.AgentManagerImpl]
(qtp1239807799-4738:ctx-491707c7 ctx-ddc41096) (logid:8a783143) Details from
executing class com.cloud.agent.api.ModifyStoragePoolCommand:
com.cloud.utils.exception.CloudRuntimeException: Failed to create storage pool:
bb8473b7-f815-3d24-9bfd-885edcfd229b
2023-12-24 16:13:16,512 WARN [c.c.a.AlertManagerImpl]
(qtp1239807799-4738:ctx-491707c7 ctx-ddc41096) (logid:8a783143) alertType=[7]
dataCenterId=[1] podId=[1] clusterId=[null] message=[Unable to attach storage
pool33 to the host7].
2023-12-24 16:13:16,517 WARN [c.c.a.AlertManagerImpl]
(qtp1239807799-4738:ctx-491707c7 ctx-ddc41096) (logid:8a783143) No recipients
set in global setting 'alert.email.addresses', skipping sending alert with
subject [Unable to attach storage pool33 to the host7] and content [Unable to
attach storage pool33 to the host7].
2023-12-24 16:13:16,518 WARN
[o.a.c.s.d.l.CloudStackPrimaryDataStoreLifeCycleImpl]
(qtp1239807799-4738:ctx-491707c7 ctx-ddc41096) (logid:8a783143) Unable to
establish a connection between Host
{"id":7,"name":"kvm-192-168-13-13","type":"Routing","uuid":"ef3f53cb-14fc-4eb4-a8ea-23a4896e973f"}
and {"name":"ceph","uuid":"bb8473b7-f815-3d24-9bfd-885edcfd229b"}
com.cloud.utils.exception.CloudRuntimeException: Unable establish connection
from storage head to storage pool 33 due to
com.cloud.utils.exception.CloudRuntimeException: Failed to create storage pool:
bb8473b7-f815-3d24-9bfd-885edcfd229b
2023-12-24 16:13:16,606 DEBUG [c.c.a.t.Request]
(AgentManager-Handler-6:null) (logid:) Seq 10-4530902700111432125: Processing:
{ Ans: , MgmtId: 206863094307331, via: 10, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.Answer":{"result":"false","details":"com.cloud.utils.exception.CloudRuntimeException:
Failed to create storage pool: bb8473b7-f815-3d24-9bfd-885edcfd229b
2023-12-24 16:13:16,606 DEBUG [c.c.a.m.AgentManagerImpl]
(qtp1239807799-4738:ctx-491707c7 ctx-ddc41096) (logid:8a783143) Details from
executing class com.cloud.agent.api.ModifyStoragePoolCommand:
com.cloud.utils.exception.CloudRuntimeException: Failed to create storage pool:
bb8473b7-f815-3d24-9bfd-885edcfd229b
2023-12-24 16:13:16,607 WARN [c.c.a.AlertManagerImpl]
(qtp1239807799-4738:ctx-491707c7 ctx-ddc41096) (logid:8a783143) alertType=[7]
dataCenterId=[1] podId=[1] clusterId=[null] message=[Unable to attach storage
pool33 to the host10].
2023-12-24 16:13:16,611 WARN [c.c.a.AlertManagerImpl]
(qtp1239807799-4738:ctx-491707c7 ctx-ddc41096) (logid:8a783143) No recipients
set in global setting 'alert.email.addresses', skipping sending alert with
subject [Unable to attach storage pool33 to the host10] and content [Unable to
attach storage pool33 to the host10].
2023-12-24 16:13:16,612 WARN
[o.a.c.s.d.l.CloudStackPrimaryDataStoreLifeCycleImpl]
(qtp1239807799-4738:ctx-491707c7 ctx-ddc41096) (logid:8a783143) Unable to
establish a connection between Host
{"id":10,"name":"kvm-192-168-13-14","type":"Routing","uuid":"9f9f655a-fef4-49d7-aafa-900cabd51315"}
and {"name":"ceph","uuid":"bb8473b7-f815-3d24-9bfd-885edcfd229b"}
com.cloud.utils.exception.CloudRuntimeException: Unable establish connection
from storage head to storage pool 33 due to
com.cloud.utils.exception.CloudRuntimeException: Failed to create storage pool:
bb8473b7-f815-3d24-9bfd-885edcfd229b
2023-12-24 16:13:16,698 DEBUG [c.c.a.t.Request]
(AgentManager-Handler-12:null) (logid:) Seq 11-7044474242137589771: Processing:
{ Ans: , MgmtId: 206863094307331, via: 11, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.Answer":{"result":"false","details":"com.cloud.utils.exception.CloudRuntimeException:
Failed to create storage pool: bb8473b7-f815-3d24-9bfd-885edcfd229b
2023-12-24 16:13:16,698 DEBUG [c.c.a.m.AgentManagerImpl]
(qtp1239807799-4738:ctx-491707c7 ctx-ddc41096) (logid:8a783143) Details from
executing class com.cloud.agent.api.ModifyStoragePoolCommand:
com.cloud.utils.exception.CloudRuntimeException: Failed to create storage pool:
bb8473b7-f815-3d24-9bfd-885edcfd229b
2023-12-24 16:13:16,699 WARN [c.c.a.AlertManagerImpl]
(qtp1239807799-4738:ctx-491707c7 ctx-ddc41096) (logid:8a783143) alertType=[7]
dataCenterId=[1] podId=[1] clusterId=[null] message=[Unable to attach storage
pool33 to the host11].
2023-12-24 16:13:16,703 WARN [c.c.a.AlertManagerImpl]
(qtp1239807799-4738:ctx-491707c7 ctx-ddc41096) (logid:8a783143) No recipients
set in global setting 'alert.email.addresses', skipping sending alert with
subject [Unable to attach storage pool33 to the host11] and content [Unable to
attach storage pool33 to the host11].
2023-12-24 16:13:16,704 WARN
[o.a.c.s.d.l.CloudStackPrimaryDataStoreLifeCycleImpl]
(qtp1239807799-4738:ctx-491707c7 ctx-ddc41096) (logid:8a783143) Unable to
establish a connection between Host
{"id":11,"name":"kvm-192-168-13-15","type":"Routing","uuid":"f6018446-c9dd-428e-95db-f5b397ce9e37"}
and {"name":"ceph","uuid":"bb8473b7-f815-3d24-9bfd-885edcfd229b"}
com.cloud.utils.exception.CloudRuntimeException: Unable establish connection
from storage head to storage pool 33 due to
com.cloud.utils.exception.CloudRuntimeException: Failed to create storage pool:
bb8473b7-f815-3d24-9bfd-885edcfd229b
2023-12-24 16:13:16,704 WARN
[o.a.c.s.d.l.CloudStackPrimaryDataStoreLifeCycleImpl]
(qtp1239807799-4738:ctx-491707c7 ctx-ddc41096) (logid:8a783143) No host can
access storage pool
{"name":"ceph","uuid":"bb8473b7-f815-3d24-9bfd-885edcfd229b"} on cluster 1
2023-12-24 16:13:16,708 DEBUG [c.c.s.StorageManagerImpl]
(qtp1239807799-4738:ctx-491707c7 ctx-ddc41096) (logid:8a783143) Failed to add
data store: Failed to access storage pool
com.cloud.utils.exception.CloudRuntimeException: Failed to access storage
pool
2023-12-24 16:13:16,710 DEBUG [c.c.s.StorageManagerImpl]
(qtp1239807799-4738:ctx-491707c7 ctx-ddc41096) (logid:8a783143) Failed to clean
up storage pool: null
2023-12-24 16:13:16,710 INFO [c.c.a.ApiServer]
(qtp1239807799-4738:ctx-491707c7 ctx-ddc41096) (logid:8a783143) Failed to add
data store: Failed to access storage pool
~~~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
