Hi @Wei ZHOU <ustcweiz...@gmail.com>
Today, we attempted to remove the disconnected host using *Cloud Monkey
tools*. As the issue began after removing the host *"cshost25.bol-online.com
<http://cshost25.bol-online.com>"*, and we discovered *two records* in the
database's host table. One record was deleted successfully without any
errors, but we encountered the following error while trying to remove the
other record.
*Cloud Monkey Outputs : *
----------------------------------
(bol-cloud-admin) 🐱 > list hosts
🙈 Error: (HTTP 530, error code 9999) <nil>
(bol-cloud-admin) 🐱 > delete host id=19714 forced=true
{
"success": true
}
(bol-cloud-admin) 🐱 > delete host id=19713 forced=true
🙈 Error: (HTTP 530, error code 9999) <nil>
(bol-cloud-admin) 🐱 >
_-----------------------------------------------------------------------------------
DB query after deletion of Host ID : 19714
mysql> SELECT h.id AS host_id, h.name AS host_name, h.status, h.type,
h.private_ip_address, h.cluster_id, c.name AS cluster_name FROM host h JOIN
cluster c ON h.cluster | NU
_id = c.id;
+---------+-------------------------+--------------+--------
-+--------------------+------------+-------------------------+ 0
| host_id | host_name | status | type |
private_ip_address | cluster_id | cluster_name |
+---------+-------------------------+--------------+--------
-+--------------------+------------+-------------------------+.10.1
| 19198 | cshost3.bol-online.com | Up | Routing | 10.10.11.33
| 1 | cluster1.bol-online.com | | NU
| 19439 | cshost5.bol-online.com | Up | Routing | 10.10.11.35
| 1 | cluster1.bol-online.com |
| 19667 | cshost8.bol-online.com | Up | Routing | 10.10.11.38
| 1 | cluster1.bol-online.com | 0
| 19668 | cshost17.bol-online.com | Up | Routing | 10.10.11.47
| 5 | cluster2.bol-online.com |
| 19670 | cph2n8.bol-online.com | Disconnected | Routing | 10.10.10.118
| 1 | cluster1.bol-online.com |.10.1
| 19675 | cshost18.bol-online.com | Up | Routing | 10.10.11.48
| 5 | cluster2.bol-online.com | | NU
| 19676 | cshost19.bol-online.com | Up | Routing | 10.10.11.49
| 5 | cluster2.bol-online.com |
| 19685 | cshost10.bol-online.com | Up | Routing | 10.10.11.41
| 1 | cluster1.bol-online.com | 0
| 19686 | cshost21.bol-online.com | Up | Routing | 10.10.11.51
| 5 | cluster2.bol-online.com |
| 19687 | cshost11.bol-online.com | Up | Routing | 10.10.11.42
| 1 | cluster1.bol-online.com |.10.1
| 19691 | cshost7.bol-online.com | Up | Routing | 10.10.11.37
| 1 | cluster1.bol-online.com | | NU
| 19696 | cshost6.bol-online.com | Up | Routing | 10.10.11.36
| 1 | cluster1.bol-online.com |
| 19701 | cshost22.bol-online.com | Up | Routing | 10.10.11.52
| 6 | cluster3.bol-online.com | 0
| 19705 | cshost24.bol-online.com | Up | Routing | 10.10.11.54
| 1 | cluster1.bol-online.com |
| 19706 | cshost23.bol-online.com | Up | Routing | 10.10.11.53
| 1 | cluster1.bol-online.com |.10.1
| 19707 | cshost2.bol-online.com | Up | Routing | 10.10.11.32
| 1 | cluster1.bol-online.com | | NU
| 19709 | cshost4.bol-online.com | Up | Routing | 10.10.11.34
| 1 | cluster1.bol-online.com |
| 19710 | cshost1.bol-online.com | Up | Routing | 10.10.11.31
| 1 | cluster1.bol-online.com | 0
| 19712 | cshost9.bol-online.com | Up | Routing | 10.10.11.39
| 1 | cluster1.bol-online.com |
| 19713 | cshost25.bol-online.com | Disconnected | Routing | 10.10.11.55
| 1 | cluster1.bol-online.com |-----
| 19715 | cshost26.bol-online.com | Up | Routing | 10.10.11.56
| 10 | cluster4.bol-online.com |-+---
| 19716 | cshost27.bol-online.com | Up | Routing | 10.10.11.57
| 12 | cluster5.bol-online.com |-----
| 19717 | cshost28.bol-online.com | Up | Routing | 10.10.11.58
| 12 | cluster5.bol-online.com |-----
| 19719 | cshost29.bol-online.com | Up | Routing | 10.10.11.59
| 12 | cluster5.bol-online.com |
+---------+-------------------------+--------------+--------
-+--------------------+------------+-------------------------+
24 rows in set (0.00 sec)
_________________________________________________
*Additionally*, we enabled the debug log for *CloudMonkey*, which displayed
the following error while listing the hosts.
cloud-admin) 🐱 > list hosts id=19713
[debug] ExecLine line:list hosts id=19713
[debug] ExecCmd args: list, hosts, id=19713
[debug] NewAPIRequest API request URL:
http://10.10.11.61:8080/client/api?apiKey=_A1conzDjN-7JrbMIeVyuEPU5Gw-d-yKYXj2gn9n7c8X85wdn0j_04pQZpZtI1fyv8QKDoFma0CF7gdMh5FaBA&command=listHosts&id=19713&response=json&signature=5RD%2FzDvE3w%2FQrdSkgpiA%2BFQNHH4%3D
[debug] NewAPIRequest response status code:530
[debug] NewAPIRequest response
body:{"listhostsresponse":{"uuidList":[],"errorcode":530,"cserrorcode":9999}}
🙈 Error: (HTTP 530, error code 9999) <nil>
(bol-cloud-admin) 🐱 >
Regards
Mosharaf Hossain
On Fri, Feb 14, 2025 at 9:41 AM Md Mosharaf Hossain <
mosharaf.hoss...@bol-online.com> wrote:
> Hi
> I find the following items
>
> 2025-02-14 05:07:06,522 DEBUG [c.c.s.StorageManagerImpl]
> (CapacityChecker:ctx-86e3d26e) (logid:0cc0f918) Total over provisioned
> capacity of the pool Eskaton-hdd-rbd-cls1 id: 100 is (83.3988 TB)
> 91697987977216
> 2025-02-14 05:07:06,524 ERROR [c.c.a.AlertManagerImpl]
> (CapacityChecker:ctx-86e3d26e) (logid:0cc0f918) Caught exception in
> recalculating capacity
> java.lang.NullPointerException
> 2025-02-14 05:07:06,544 DEBUG [c.c.a.AlertManagerImpl]
> (CapacityChecker:ctx-86e3d26e) (logid:0cc0f918) System Alert: Number of
> unallocated shared network IPs is low in availability zone Gulshan-1 SAM
> Tower
> 2025-02-14 05:07:06,544 DEBUG [c.c.a.AlertManagerImpl]
> (CapacityChecker:ctx-86e3d26e) (logid:0cc0f918) Number of unallocated
> shared network IPs is low, total: 244.0, allocated: 221.0 (90.57%)
> 2025-02-14 05:07:06,544 WARN [c.c.a.AlertManagerImpl]
> (CapacityChecker:ctx-86e3d26e) (logid:0cc0f918) alertType=[24]
> dataCenterId=[1] podId=[null] clusterId=[null] message=[System Alert:
> Number of unallocated shared network IPs is low in availability zone
> Gulshan-1 SAM Tower].
> 2025-02-14 05:07:06,546 DEBUG [c.c.a.AlertManagerImpl]
> (CapacityChecker:ctx-86e3d26e) (logid:0cc0f918) Have already sent: 1 emails
> for alert type '24' -- skipping send email
> 2025-02-14 05:07:06,693 DEBUG [c.c.a.AlertManagerImpl]
> (CapacityChecker:ctx-86e3d26e) (logid:0cc0f918) Done running Capacity
> Checker ...
> 2025-02-14 05:07:07,413 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentManager-Handler-13:null) (logid:) Seq 19198-6733725867849049292:
> MgmtId 34072927583874: Resp: Routing to peer
>
>
> Regards
> Mosharaf Hossain
> Manager, Product Development
> Bangladesh Online (BOL)
>
> Level 8, SAM Tower, Plot 4, Road 22, Gulshan 1, Dhaka 1212, Bangladesh
> Tel: +880 9609 000 999, +880 2 58815559, Ext: 14191, Fax: +880 2 2222 95757
> Cell: +880 1787 680828, Web: www.bol-online.com
>
>
>
> On Thu, Feb 13, 2025 at 2:39 PM Wei ZHOU <ustcweiz...@gmail.com> wrote:
>
>> Hi,
>>
>> Can you search "java.lang.NullPointerException" and paste the full log
>> of the exception ?
>>
>> -Wei
>>
>> On Thu, Feb 13, 2025 at 9:37 AM Md Mosharaf Hossain <
>> mosharaf.hoss...@bol-online.com> wrote:
>>
>>> Hi @Wei ZHOU <ustcweiz...@gmail.com>
>>> The following report might help which I got as an error while accessing
>>> the host. Please review the log and guide me to resolve.
>>>
>>>
>>> 2025-02-13 14:33:37,395 DEBUG [c.c.a.q.QueryManagerImpl]
>>> (qtp364604394-2433:ctx-93c55557 ctx-0bef80ad ctx-5ce8a856) (logid:89bc4212)
>>> >>>Searching for hosts>>>
>>> 2025-02-13 14:33:37,412 DEBUG [c.c.a.q.QueryManagerImpl]
>>> (qtp364604394-2433:ctx-93c55557 ctx-0bef80ad ctx-5ce8a856) (logid:89bc4212)
>>> >>>Generating Response>>>
>>> 2025-02-13 14:33:37,871 ERROR [c.c.a.ApiServer]
>>> (qtp364604394-2433:ctx-93c55557 ctx-0bef80ad ctx-5ce8a856) (logid:89bc4212)
>>> unhandled exception executing api command: [Ljava
>>> .lang.String;@631ae634
>>>
>>>
>>> Regards
>>> Mosharaf Hossain
>>> Manager, Product Development
>>> Bangladesh Online (BOL)
>>>
>>> Level 8, SAM Tower, Plot 4, Road 22, Gulshan 1, Dhaka 1212, Bangladesh
>>> Tel: +880 9609 000 999, +880 2 58815559, Ext: 14191, Fax: +880 2 2222
>>> 95757
>>> Cell: +880 1787 680828, Web: www.bol-online.com
>>>
>>>
>>>
>>> On Thu, Feb 13, 2025 at 11:42 AM Md Mosharaf Hossain <
>>> mosharaf.hoss...@bol-online.com> wrote:
>>>
>>>> Hi Wei
>>>> Are you wanting to search the keyword "Exception" in the log. There
>>>> are several entry of "exception" in the log file.
>>>> What shall I need to do to proceed further?
>>>>
>>>> *Some exception log: *
>>>> 2025-02-13 10:24:38,283 DEBUG [c.c.u.d.GenericDaoBase]
>>>> (qtp364604394-2335:ctx-daa415e4 ctx-1e82f184) (logid:9d8686b6)
>>>> ClassCastException when casting value to String. Setting the value of the
>>>> object directly.
>>>> 2025-02-13 10:24:38,283 DEBUG [c.c.u.d.GenericDaoBase]
>>>> (qtp364604394-2335:ctx-daa415e4 ctx-1e82f184) (logid:9d8686b6)
>>>> ClassCastException when casting value to String. Setting the value of the
>>>> object directly.
>>>> java.lang.NullPointerException
>>>> java.lang.NullPointerException
>>>> 2025-02-13 10:34:23,422 DEBUG [c.c.u.d.GenericDaoBase]
>>>> (qtp364604394-2321:ctx-f41289d9 ctx-fc56cd3f) (logid:d6dfb17c)
>>>> ClassCastException when casting value to String. Setting the value of the
>>>> object directly.
>>>> 2025-02-13 10:34:23,422 DEBUG [c.c.u.d.GenericDaoBase]
>>>> (qtp364604394-2321:ctx-f41289d9 ctx-fc56cd3f) (logid:d6dfb17c)
>>>> ClassCastException when casting value to String. Setting the value of the
>>>> object directly.
>>>> 2025-02-13 10:34:23,422 DEBUG [c.c.u.d.GenericDaoBase]
>>>> (qtp364604394-2321:ctx-f41289d9 ctx-fc56cd3f) (logid:d6dfb17c)
>>>> ClassCastException when castin
>>>>
>>>> Regards
>>>> Mosharaf Hossain
>>>> Manager, Product Development
>>>> Bangladesh Online (BOL)
>>>>
>>>> Level 8, SAM Tower, Plot 4, Road 22, Gulshan 1, Dhaka 1212, Bangladesh
>>>> Tel: +880 9609 000 999, +880 2 58815559, Ext: 14191, Fax: +880 2 2222
>>>> 95757
>>>> Cell: +880 1787 680828, Web: www.bol-online.com
>>>>
>>>>
>>>>
>>>> On Wed, Feb 12, 2025 at 4:08 PM Wei ZHOU <ustcweiz...@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Your log does not contain any log for listing hosts.
>>>>> Can you check "Exception" in the log ?
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Feb 12, 2025 at 10:01 AM Md Mosharaf Hossain <
>>>>> mosharaf.hoss...@bol-online.com> wrote:
>>>>>
>>>>>> Hi Wei
>>>>>>
>>>>>> The requested log is attached for your review. Please check it and
>>>>>> let me know if you require any further information.
>>>>>>
>>>>>> cloudstack-management-server_10 Feb 2025.log
>>>>>> <https://drive.google.com/open?id=1weLLgB9ib_T4ddjTvmBr8zkb8cyIKpJg>
>>>>>>
>>>>>>
>>>>>> Regards
>>>>>> Mosharaf Hossain
>>>>>> Manager, Product Development
>>>>>> Bangladesh Online (BOL)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Feb 10, 2025 at 2:20 PM Md Mosharaf Hossain <
>>>>>> mosharaf.hoss...@bol-online.com> wrote:
>>>>>>
>>>>>>> Hi
>>>>>>> I have attached logs from the management server, generated while
>>>>>>> attempting to access the host list. Please review them.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Regards
>>>>>>> Mosharaf Hossain
>>>>>>> Manager, Product Development
>>>>>>> Bangladesh Online (BOL)
>>>>>>>
>>>>>>> Level 8, SAM Tower, Plot 4, Road 22, Gulshan 1, Dhaka 1212,
>>>>>>> Bangladesh
>>>>>>> Tel: +880 9609 000 999, +880 2 58815559, Ext: 14191, Fax: +880 2
>>>>>>> 2222 95757
>>>>>>> Cell: +880 1787 680828, Web: www.bol-online.com
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Feb 10, 2025 at 1:50 PM Wei ZHOU <ustcweiz...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> The management server log is located
>>>>>>>> at /var/log/cloudstack/management/management-server.log
>>>>>>>>
>>>>>>>>
>>>>>>>> -Wei
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Feb 10, 2025 at 8:47 AM Md Mosharaf Hossain <
>>>>>>>> mosharaf.hoss...@bol-online.com> wrote:
>>>>>>>>
>>>>>>>> > What type of log you need?
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > Regards
>>>>>>>> > Mosharaf Hossain
>>>>>>>> > Manager, Product Development
>>>>>>>> > Bangladesh Online (BOL)
>>>>>>>> >
>>>>>>>> > Level 8, SAM Tower, Plot 4, Road 22, Gulshan 1, Dhaka 1212,
>>>>>>>> Bangladesh
>>>>>>>> > Tel: +880 9609 000 999, +880 2 58815559, Ext: 14191, Fax: +880 2
>>>>>>>> 2222 95757
>>>>>>>> > Cell: +880 1787 680828, Web: www.bol-online.com
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > On Mon, Feb 10, 2025 at 1:27 PM Wei ZHOU <ustcweiz...@gmail.com>
>>>>>>>> wrote:
>>>>>>>> >
>>>>>>>> > > Hi,
>>>>>>>> > >
>>>>>>>> > > Can you share some logs ?
>>>>>>>> > >
>>>>>>>> > >
>>>>>>>> > > -Wei
>>>>>>>> > >
>>>>>>>> > > On Sun, Feb 9, 2025 at 12:06 PM Md Mosharaf Hossain <
>>>>>>>> > > mosharaf.hoss...@bol-online.com> wrote:
>>>>>>>> > >
>>>>>>>> > > > Dear Users:
>>>>>>>> > > >
>>>>>>>> > > > I hope this message finds you well. I am currently using
>>>>>>>> CloudStack
>>>>>>>> > > > version 4.19.1.1 and have four clusters set up in my
>>>>>>>> environment.
>>>>>>>> > > Recently,
>>>>>>>> > > > I have encountered an issue while trying to list hosts in
>>>>>>>> *Cluster 1*.
>>>>>>>> > > >
>>>>>>>> > > > When attempting to list the hosts, I receive a *500 Internal
>>>>>>>> Server
>>>>>>>> > > Error*.
>>>>>>>> > > > This issue is specific to Cluster 1, as the other clusters are
>>>>>>>> > > functioning
>>>>>>>> > > > as expected. I have tried troubleshooting the issue by
>>>>>>>> checking the
>>>>>>>> > logs
>>>>>>>> > > > and ensuring that the hosts in Cluster 1 are properly
>>>>>>>> connected, but
>>>>>>>> > the
>>>>>>>> > > > problem persists.
>>>>>>>> > > >
>>>>>>>> > > > Here are some additional details that might help in
>>>>>>>> diagnosing the
>>>>>>>> > issue:
>>>>>>>> > > >
>>>>>>>> > > > -
>>>>>>>> > > >
>>>>>>>> > > > *CloudStack Version*: 4.19.1.1
>>>>>>>> > > > -
>>>>>>>> > > >
>>>>>>>> > > > *Number of Clusters*: 4
>>>>>>>> > > > -
>>>>>>>> > > >
>>>>>>>> > > > *Affected Cluster*: Cluster 1
>>>>>>>> > > > -
>>>>>>>> > > >
>>>>>>>> > > > *Error Message*: 500 Internal Server Error
>>>>>>>> > > >
>>>>>>>> > > > Could you please provide guidance on how to resolve this
>>>>>>>> issue? Any
>>>>>>>> > > > suggestions or steps to further investigate the problem would
>>>>>>>> be
>>>>>>>> > greatly
>>>>>>>> > > > appreciated.
>>>>>>>> > > >
>>>>>>>> > > > Thank you in advance for your support. I look forward to your
>>>>>>>> response.
>>>>>>>> > > >
>>>>>>>> > > >
>>>>>>>> > > > CloudStack login -> Infrastructure -> Host: SHowing the below
>>>>>>>> error
>>>>>>>> > > > [image: image.png]
>>>>>>>> > > >
>>>>>>>> > > >
>>>>>>>> > > >
>>>>>>>> > > >
>>>>>>>> > > > Regards
>>>>>>>> > > > Mosharaf Hossain
>>>>>>>> > > > Manager, Product Development
>>>>>>>> > > > Bangladesh Online (BOL)
>>>>>>>> > > >
>>>>>>>> > > >
>>>>>>>> > >
>>>>>>>> >
>>>>>>>>
>>>>>>>
