from:"Granwille Strauss"

Thank you, in our case, we start our VMs at 60 GB, clients then can
upgrade to 120 GB as the next upgrade. However, clients also have the
option to "Downgrade" back to 60 GB based on their package. Would kinda
not make sense to prohibit clients from doing that, from a legal POV. So
this kinda creates an issue now and its good I tested it out now. I
tried the 'qemu-img resize --shrink /var/lib/libvirt/images/`
and it seems it corrupted the image file. Not sure if it had to be
converted first maybe.

I noticed that the images are not saved as .QCOW2 files, can you perhaps
confirm what format they're currently in? I would like to run some more
test and maybe see how well it goes and potentially propose a solution.

On 9/4/23 17:06, Wei ZHOU wrote:

Hi,

Normally cloud providers create vm templates with small size (e.g. 10GB).
When create vm from the template, user can override the root disk size so
that vm have a larger size (e.g. 60GB)
If the vm template has cloud-init installed, it can detect the new disk
size and auto-grow the partition to use the full disk.

In your case, if users pay for only 60GB, resizing to 120GB should be
forbidden.

-Wei

On Mon, 4 Sept 2023 at 16:51, Granwille Strauss
wrote:

Hi Wido

Thank you. Yes, I only shrink it inside the VM. But still Cloudstack
volume shows 120GB and within the VM lsblk command shows that 120 GB is
available. Since I cannot shrink the volume, how do most providers take
care of this issue? I mean it would be awkward if a client see they can
have 120 GB available when they're paying for 60 GB, for example.[image:
Powered by AdSigner]
<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>

--
Regards / Groete

<https://www.namhost.com> Granwille Strauss // Senior Systems Admin

*e:* granwi...@namhost.com
*m:* +264 81 323 1260
*w:* www.namhost.com <https://www.namhost.com/>

<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>

Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA

The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It
is forbidden to copy, forward, or in any way reveal the contents of this
message to anyone without our explicit consent. The integrity and
security of this email cannot be guaranteed over the Internet.
Therefore, the sender will not be held liable for any damage caused by
the message. For our full privacy policy and disclaimers, please go to
https://www.namhost.com/privacy-policy

smime.p7s
Description: S/MIME Cryptographic Signature

Re: How to Best Shrink Volume


Hi Wido

Thank you. Yes, I only shrink it inside the VM. But still Cloudstack 
volume shows 120GB and within the VM lsblk command shows that 120 GB is 
available. Since I cannot shrink the volume, how do most providers take 
care of this issue? I mean it would be awkward if a client see they can 
have 120 GB available when they're paying for 60 GB, for example.


On 9/4/23 15:45, Wido den Hollander wrote:



Op 04-09-2023 om 13:08 schreef Granwille Strauss:

Hi Wido

I mounted a live ISO to the VM, booted into the ISO and went into 
recover mode, had to unmount the / partition of the VM, and then 
proceed to resize it via parted and write changes. Rebooted via and 
detatched the ISO and now VM shows 60 GB correctly. However, in 
Cloudstack, still says the volume is 120 GB. But when I want to 
shrink it to 60 GB, I get this error:




In this case you did not shrink the volume, but only the partition and 
filesystem on the volume.


Failed to resize volume operation of volume UUID: 
[eec8285e-8715-441b-b418-b71b231f1bab] due to - Unable to shrink 
volumes of type QCOW2 




QCOW2 volumes can't be reduced in size. So no, this is not a bug.

And yes, the VM is stopped when I try doing this. I now want to 
confirm from the community, is this the expected behaviour or not? It 
seems Wei doubted it. If not, that means I have a bug that need to be 
looked at and further checked.




Wei is correct. QCOW2 can't be shrinked.

Wido


On 9/4/23 12:46, Wido den Hollander wrote:



Op 04/09/2023 om 09:18 schreef Granwille Strauss:
Thank you. Can you confirm if this behaviour is expected? Or am I 
experiencing a bug?


What exactly? You did this resize manually outside the knowledge of 
cloudstack, am I correct?


How did you do this live/recovery method?

Wido



On 9/4/23 09:00, Wido den Hollander wrote:



Op 04-09-2023 om 08:20 schreef Granwille Strauss:

Hi

I managed to resize the VM itself back to 60 GB via live/recovery 
method. However, now in Cloudstack it still says the volume 
remains 120  GB and when I want to shrink it I am presented with 
the errors from my previous reply. Is there a way around this? 
Such as making a database change. Also, is this the expected 
behaviour for a KVM host?




This is recorded in the table 'volumes'. If you decrease the size 
to 60GB there is should show properly.


Wido


On 9/2/23 11:22, Granwille Strauss wrote:


Hi Wei

It seems you're right you cannot shrink the volume in Cloudstack:

Failed to resize volume operation of volume UUID: 
[eec8285e-8715-441b-b418-b71b231f1bab] due to - Unable to 
shrink volumes of type QCOW2


Going from existing size of 128849018880 to size of 64424509440 
would shrink the volume.Need to sign off by supplying the 
shrinkok parameter with value of true.


On 9/2/23 10:37, Wei ZHOU wrote:

Hi Wido,

Does it work with kvm ?

my suggestion would be
- create a new vm with 60GB
- copy the data using `rsync`

-Wei

On Sat, 2 Sept 2023 at 08:25, Wido den 
Hollander  wrote:



Hi,

Yes, you should make sure the filesystem and partitions are 
within the

to be set new boundaries.

Once that's done you can shrink the volume.

I would recommend:

- Shrink EXT4 + Partitions to 59GB
- Shrink volume to 60GB
- Grow EXT4 + Partitions to 60GB

The step to 59GB is to prevent that by calc mistake you chip 
off the

last portion of your filesystem.

Wido

Op 01-09-2023 om 16:29 schreef Granwille Strauss:

Hi Guys

Anyone willing to assist me with a quick one. What's the best 
and
recommended way to shrink a VM volume via UI? The VM volume 
was first 60
GB, we then expended it to 120 GB and increased it in the VM 
via parted.
I now want to  take it back to 60GB. How do I proceed? Do I 
first shrink
the ext4 in the VM first then proceed to shrink the volume in 
Cloudstack
UI by selecting also the "Shrink OK" option? Detailed steps 
would be

appreciated, please.

--
Regards / Groete

<https://www.namhost.com>  Granwille Strauss // Senior 
Systems Admin


*e:*granwi...@namhost.com
*m:* +264 81 323 1260
*w:*www.namhost.com <https://www.namhost.com/>

<https://www.facebook.com/namhost><https://twitter.com/namhost><
https://www.instagram.com/namhostinternetservices/>< 
https://www.linkedin.com/company/namhos>< 
https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>

<
https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner 


Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA



The content of this message is confidential. If you have 
received it by
mistake, please inform us by email reply and then delete the 
message. It
is forbidden to copy, forward, or in any way reveal the 
contents of this
message to anyone without our explicit consent. The integrity 
and

security of this email cannot be guaranteed over the Internet.
Therefore, the sender will not be held liable for any damage 
caused by
the message. For our full privacy policy and disc

Re: How to Best Shrink Volume


Hi Wei


As far as I understand, you have resized the disk inside the VM to 60GB,
right ?

- Yes, I have resized the partitions in VM back to 60 GB.


However, the allocated disk size of the VM should be still 120GB, not 60GB.
You can verify it by `qemu-img info -U /mnt//`

- Yes, in VM if I run 'lsblk' I see this:

vda    253:0    0  120G  0 disk
├─vda1 253:1    0   57G  0 part /
├─vda2 253:2    0    2G  0 part [SWAP]
└─vda3 253:3    0    1G  0 part /boot

root@hostname /var/lib/libvirt/images $ qemu-img info -U 
/var/lib/libvirt/images/c611907d-4168-4436-b928-d51ef98ea8e7

image: /var/lib/libvirt/images/c611907d-4168-4436-b928-d51ef98ea8e7
file format: qcow2
virtual size: 120 GiB (128849018880 bytes)
disk size: 59.2 GiB
cluster_size: 65536
Format specific information:
    compat: 1.1
    compression type: zlib
    lazy refcounts: false
    refcount bits: 16
    corrupt: false
    extended l2: false



Shrinking a QCOW2 image is possible, but complicated. It is currently not
supported by CloudStack. The message you received is expected.


- Thank you for the confirmation. What is the standard practise that 
most providers do in this case?


On 9/4/23 13:38, Wei ZHOU wrote:

Hi,

As far as I understand, you have resized the disk inside the VM to 60GB,
right ?

However, the allocated disk size of the VM should be still 120GB, not 60GB.
You can verify it by `qemu-img info -U /mnt//`

Shrinking a QCOW2 image is possible, but complicated. It is currently not
supported by CloudStack. The message you received is expected.


-Wei


On Mon, 4 Sept 2023 at 13:11, Granwille Strauss
  wrote:


Hi Wido

I mounted a live ISO to the VM, booted into the ISO and went into recover
mode, had to unmount the / partition of the VM, and then proceed to resize
it via parted and write changes. Rebooted via and detatched the ISO and now
VM shows 60 GB correctly. However, in Cloudstack, still says the volume is
120 GB. But when I want to shrink it to 60 GB, I get this error:

Failed to resize volume operation of volume UUID:
[eec8285e-8715-441b-b418-b71b231f1bab] due to - Unable to shrink volumes of
type QCOW2

And yes, the VM is stopped when I try doing this. I now want to confirm
from the community, is this the expected behaviour or not? It seems Wei
doubted it. If not, that means I have a bug that need to be looked at and
further checked.

On 9/4/23 12:46, Wido den Hollander wrote:



Op 04/09/2023 om 09:18 schreef Granwille Strauss:

Thank you. Can you confirm if this behaviour is expected? Or am I
experiencing a bug?


What exactly? You did this resize manually outside the knowledge of
cloudstack, am I correct?

How did you do this live/recovery method?

Wido


On 9/4/23 09:00, Wido den Hollander wrote:



Op 04-09-2023 om 08:20 schreef Granwille Strauss:

Hi

I managed to resize the VM itself back to 60 GB via live/recovery method.
However, now in Cloudstack it still says the volume remains 120  GB and
when I want to shrink it I am presented with the errors from my previous
reply. Is there a way around this? Such as making a database change. Also,
is this the expected behaviour for a KVM host?


This is recorded in the table 'volumes'. If you decrease the size to 60GB
there is should show properly.

Wido

On 9/2/23 11:22, Granwille Strauss wrote:


Hi Wei

It seems you're right you cannot shrink the volume in Cloudstack:

Failed to resize volume operation of volume UUID:
[eec8285e-8715-441b-b418-b71b231f1bab] due to - Unable to shrink volumes of
type QCOW2


Going from existing size of 128849018880 to size of 64424509440 would
shrink the volume.Need to sign off by supplying the shrinkok parameter with
value of true.


On 9/2/23 10:37, Wei ZHOU wrote:

Hi Wido,

Does it work with kvm ?

my suggestion would be
- create a new vm with 60GB
- copy the data using `rsync`

-Wei

On Sat, 2 Sept 2023 at 08:25, Wido den Hollander
   wrote:

Hi,

Yes, you should make sure the filesystem and partitions are within the
to be set new boundaries.

Once that's done you can shrink the volume.

I would recommend:

- Shrink EXT4 + Partitions to 59GB
- Shrink volume to 60GB
- Grow EXT4 + Partitions to 60GB

The step to 59GB is to prevent that by calc mistake you chip off the
last portion of your filesystem.

Wido

Op 01-09-2023 om 16:29 schreef Granwille Strauss:

Hi Guys

Anyone willing to assist me with a quick one. What's the best and
recommended way to shrink a VM volume via UI? The VM volume was first 60
GB, we then expended it to 120 GB and increased it in the VM via parted.
I now want to  take it back to 60GB. How do I proceed? Do I first shrink
the ext4 in the VM first then proceed to shrink the volume in Cloudstack
UI by selecting also the "Shrink OK" option? Detailed steps would be
appreciated, please.

--
Regards / Groete

<https://www.namhost.com>  <https://www.namhost.com>   Granwille
Strauss // Senior Systems Admin

*e:*granwi...@namhost.com
*m:* +264 81 323 1260  <+264813231260&g

Re: How to Best Shrink Volume

Hi Wido

I mounted a live ISO to the VM, booted into the ISO and went into
recover mode, had to unmount the / partition of the VM, and then proceed
to resize it via parted and write changes. Rebooted via and detatched
the ISO and now VM shows 60 GB correctly. However, in Cloudstack, still
says the volume is 120 GB. But when I want to shrink it to 60 GB, I get
this error:

Failed to resize volume operation of volume UUID:
[eec8285e-8715-441b-b418-b71b231f1bab] due to - Unable to shrink
volumes of type QCOW2

And yes, the VM is stopped when I try doing this. I now want to confirm
from the community, is this the expected behaviour or not? It seems Wei
doubted it. If not, that means I have a bug that need to be looked at
and further checked.

On 9/4/23 12:46, Wido den Hollander wrote:

Op 04/09/2023 om 09:18 schreef Granwille Strauss:
Thank you. Can you confirm if this behaviour is expected? Or am I
experiencing a bug?

What exactly? You did this resize manually outside the knowledge of
cloudstack, am I correct?

How did you do this live/recovery method?

Wido

On 9/4/23 09:00, Wido den Hollander wrote:

Op 04-09-2023 om 08:20 schreef Granwille Strauss:

I managed to resize the VM itself back to 60 GB via live/recovery
method. However, now in Cloudstack it still says the volume remains
120 GB and when I want to shrink it I am presented with the errors
from my previous reply. Is there a way around this? Such as making
a database change. Also, is this the expected behaviour for a KVM
host?

This is recorded in the table 'volumes'. If you decrease the size to
60GB there is should show properly.

Wido

On 9/2/23 11:22, Granwille Strauss wrote:

Hi Wei

It seems you're right you cannot shrink the volume in Cloudstack:

Failed to resize volume operation of volume UUID:
[eec8285e-8715-441b-b418-b71b231f1bab] due to - Unable to shrink
volumes of type QCOW2

Going from existing size of 128849018880 to size of 64424509440
would shrink the volume.Need to sign off by supplying the
shrinkok parameter with value of true.

On 9/2/23 10:37, Wei ZHOU wrote:

Hi Wido,

Does it work with kvm ?

my suggestion would be
- create a new vm with 60GB
- copy the data using `rsync`

-Wei

On Sat, 2 Sept 2023 at 08:25, Wido den Hollander
wrote:

Hi,

Yes, you should make sure the filesystem and partitions are
within the

to be set new boundaries.

Once that's done you can shrink the volume.

I would recommend:

- Shrink EXT4 + Partitions to 59GB
- Shrink volume to 60GB
- Grow EXT4 + Partitions to 60GB

The step to 59GB is to prevent that by calc mistake you chip off
the

last portion of your filesystem.

Wido

Op 01-09-2023 om 16:29 schreef Granwille Strauss:

Hi Guys

Anyone willing to assist me with a quick one. What's the best and
recommended way to shrink a VM volume via UI? The VM volume was
first 60
GB, we then expended it to 120 GB and increased it in the VM
via parted.
I now want to take it back to 60GB. How do I proceed? Do I
first shrink
the ext4 in the VM first then proceed to shrink the volume in
Cloudstack
UI by selecting also the "Shrink OK" option? Detailed steps
would be

appreciated, please.

--
Regards / Groete

<https://www.namhost.com> Granwille Strauss // Senior
Systems Admin

*e:*granwi...@namhost.com
*m:* +264 81 323 1260
*w:*www.namhost.com <https://www.namhost.com/>

<https://www.facebook.com/namhost><https://twitter.com/namhost><
https://www.instagram.com/namhostinternetservices/><
https://www.linkedin.com/company/namhos><
https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>

<
https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner

Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA

message to anyone without our explicit consent. The integrity and
security of this email cannot be guaranteed over the Internet.
Therefore, the sender will not be held liable for any damage
caused by
the message. For our full privacy policy and disclaimers,
please go to

https://www.namhost.com/privacy-policy

--
Regards / Groete

<https://www.namhost.com> Granwille Strauss // Senior Systems
Admin

*e:* granwi...@namhost.com
*m:* +264 81 323 1260
*w:* www.namhost.com <https://www.namhost.com/>

<https://www.adsigner.com/v1/l/631091998d4670001fe43ec

Re: How to Best Shrink Volume

Thank you. Can you confirm if this behaviour is expected? Or am I
experiencing a bug?

On 9/4/23 09:00, Wido den Hollander wrote:

Op 04-09-2023 om 08:20 schreef Granwille Strauss:

I managed to resize the VM itself back to 60 GB via live/recovery
method. However, now in Cloudstack it still says the volume remains
120 GB and when I want to shrink it I am presented with the errors
from my previous reply. Is there a way around this? Such as making a
database change. Also, is this the expected behaviour for a KVM host?

This is recorded in the table 'volumes'. If you decrease the size to
60GB there is should show properly.

Wido

On 9/2/23 11:22, Granwille Strauss wrote:

Hi Wei

It seems you're right you cannot shrink the volume in Cloudstack:

Failed to resize volume operation of volume UUID:
[eec8285e-8715-441b-b418-b71b231f1bab] due to - Unable to shrink
volumes of type QCOW2

Going from existing size of 128849018880 to size of 64424509440
would shrink the volume.Need to sign off by supplying the shrinkok
parameter with value of true.

On 9/2/23 10:37, Wei ZHOU wrote:

Hi Wido,

Does it work with kvm ?

my suggestion would be
- create a new vm with 60GB
- copy the data using `rsync`

-Wei

On Sat, 2 Sept 2023 at 08:25, Wido den Hollander
wrote:

Hi,

Yes, you should make sure the filesystem and partitions are within
the

to be set new boundaries.

Once that's done you can shrink the volume.

I would recommend:

- Shrink EXT4 + Partitions to 59GB
- Shrink volume to 60GB
- Grow EXT4 + Partitions to 60GB

The step to 59GB is to prevent that by calc mistake you chip off the
last portion of your filesystem.

Wido

Op 01-09-2023 om 16:29 schreef Granwille Strauss:

Hi Guys

Anyone willing to assist me with a quick one. What's the best and
recommended way to shrink a VM volume via UI? The VM volume was
first 60
GB, we then expended it to 120 GB and increased it in the VM via
parted.
I now want to take it back to 60GB. How do I proceed? Do I first
shrink
the ext4 in the VM first then proceed to shrink the volume in
Cloudstack

UI by selecting also the "Shrink OK" option? Detailed steps would be
appreciated, please.

--
Regards / Groete

<https://www.namhost.com> Granwille Strauss // Senior
Systems Admin

*e:*granwi...@namhost.com
*m:* +264 81 323 1260
*w:*www.namhost.com <https://www.namhost.com/>

<https://www.facebook.com/namhost><https://twitter.com/namhost><
https://www.instagram.com/namhostinternetservices/><
https://www.linkedin.com/company/namhos><
https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>

<
https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner

Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA

https://www.namhost.com/privacy-policy

--
Regards / Groete

<https://www.namhost.com> Granwille Strauss // Senior Systems
Admin

*e:* granwi...@namhost.com
*m:* +264 81 323 1260
*w:* www.namhost.com <https://www.namhost.com/>

<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>

Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus, 7210, RSA

The content of this message is confidential. If you have received it
by mistake, please inform us by email reply and then delete the
message. It is forbidden to copy, forward, or in any way reveal the
contents of this message to anyone without our explicit consent. The
integrity and security of this email cannot be guaranteed over the
Internet. Therefore, the sender will not be held liable for any
damage caused by the message. For our full privacy policy and
disclaimers, please go to https://www.namhost.com/privacy-policy

--
Regards / Groete

<https://www.namhost.com> Granwille Strauss // Senior Systems Admin

*e:* granwi...@namhost.com
*m:* +264 81 323 1260
*w:* www.namhost.com <https://www.namhost.com/>

<https://www.facebook.com/namhost&g

Re: How to Best Shrink Volume

I managed to resize the VM itself back to 60 GB via live/recovery
method. However, now in Cloudstack it still says the volume remains 120
GB and when I want to shrink it I am presented with the errors from my
previous reply. Is there a way around this? Such as making a database
change. Also, is this the expected behaviour for a KVM host?

On 9/2/23 11:22, Granwille Strauss wrote:

Hi Wei

It seems you're right you cannot shrink the volume in Cloudstack:

Failed to resize volume operation of volume UUID:
[eec8285e-8715-441b-b418-b71b231f1bab] due to - Unable to shrink
volumes of type QCOW2

Going from existing size of 128849018880 to size of 64424509440 would
shrink the volume.Need to sign off by supplying the shrinkok
parameter with value of true.

On 9/2/23 10:37, Wei ZHOU wrote:

Hi Wido,

Does it work with kvm ?

my suggestion would be
- create a new vm with 60GB
- copy the data using `rsync`

-Wei

On Sat, 2 Sept 2023 at 08:25, Wido den Hollander wrote:

Hi,

Yes, you should make sure the filesystem and partitions are within the
to be set new boundaries.

Once that's done you can shrink the volume.

I would recommend:

- Shrink EXT4 + Partitions to 59GB
- Shrink volume to 60GB
- Grow EXT4 + Partitions to 60GB

The step to 59GB is to prevent that by calc mistake you chip off the
last portion of your filesystem.

Wido

Op 01-09-2023 om 16:29 schreef Granwille Strauss:

Hi Guys

Anyone willing to assist me with a quick one. What's the best and
recommended way to shrink a VM volume via UI? The VM volume was first 60
GB, we then expended it to 120 GB and increased it in the VM via parted.
I now want to take it back to 60GB. How do I proceed? Do I first shrink
the ext4 in the VM first then proceed to shrink the volume in Cloudstack
UI by selecting also the "Shrink OK" option? Detailed steps would be
appreciated, please.

--
Regards / Groete

<https://www.namhost.com> Granwille Strauss // Senior Systems Admin

*e:*granwi...@namhost.com
*m:* +264 81 323 1260
*w:*www.namhost.com <https://www.namhost.com/>

<https://www.facebook.com/namhost><https://twitter.com/namhost><
https://www.instagram.com/namhostinternetservices/>< https://www.linkedin.com/company/namhos><
https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>

https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner

Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA

The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It
is forbidden to copy, forward, or in any way reveal the contents of this
message to anyone without our explicit consent. The integrity and
security of this email cannot be guaranteed over the Internet.
Therefore, the sender will not be held liable for any damage caused by
the message. For our full privacy policy and disclaimers, please go to
https://www.namhost.com/privacy-policy

https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818

--
Regards / Groete

<https://www.namhost.com> Granwille Strauss // Senior Systems Admin

*e:* granwi...@namhost.com
*m:* +264 81 323 1260
*w:* www.namhost.com <https://www.namhost.com/>

<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>

Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus, 7210, RSA

--
Regards / Groete

<https://www.namhost.com> Granwille Strauss // Senior Systems Admin

*e:* granwi...@namhost.com
*m:* +264 81 323 1260
*w:* www.namhost.com <https://www.namhost.com/>

<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>

Namhost Internet S

Re: How to Best Shrink Volume

2023-09-02 Thread Granwille Strauss

Hi Wei

It seems you're right you cannot shrink the volume in Cloudstack:

Failed to resize volume operation of volume UUID:
[eec8285e-8715-441b-b418-b71b231f1bab] due to - Unable to shrink
volumes of type QCOW2

Going from existing size of 128849018880 to size of 64424509440 would
shrink the volume.Need to sign off by supplying the shrinkok parameter
with value of true.

On 9/2/23 10:37, Wei ZHOU wrote:

Hi Wido,

Does it work with kvm ?

my suggestion would be
- create a new vm with 60GB
- copy the data using `rsync`

-Wei

On Sat, 2 Sept 2023 at 08:25, Wido den Hollander wrote:

Hi,

Yes, you should make sure the filesystem and partitions are within the
to be set new boundaries.

Once that's done you can shrink the volume.

I would recommend:

- Shrink EXT4 + Partitions to 59GB
- Shrink volume to 60GB
- Grow EXT4 + Partitions to 60GB

The step to 59GB is to prevent that by calc mistake you chip off the
last portion of your filesystem.

Wido

Op 01-09-2023 om 16:29 schreef Granwille Strauss:

Hi Guys

Anyone willing to assist me with a quick one. What's the best and
recommended way to shrink a VM volume via UI? The VM volume was first 60
GB, we then expended it to 120 GB and increased it in the VM via parted.
I now want to take it back to 60GB. How do I proceed? Do I first shrink
the ext4 in the VM first then proceed to shrink the volume in Cloudstack
UI by selecting also the "Shrink OK" option? Detailed steps would be
appreciated, please.

--
Regards / Groete

<https://www.namhost.com> Granwille Strauss // Senior Systems Admin

*e:*granwi...@namhost.com
*m:* +264 81 323 1260
*w:*www.namhost.com <https://www.namhost.com/>

<https://www.facebook.com/namhost><https://twitter.com/namhost><
https://www.instagram.com/namhostinternetservices/>< https://www.linkedin.com/company/namhos><
https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>

https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner

Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA

The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It
is forbidden to copy, forward, or in any way reveal the contents of this
message to anyone without our explicit consent. The integrity and
security of this email cannot be guaranteed over the Internet.
Therefore, the sender will not be held liable for any damage caused by
the message. For our full privacy policy and disclaimers, please go to
https://www.namhost.com/privacy-policy

https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818

--
Regards / Groete

<https://www.namhost.com> Granwille Strauss // Senior Systems Admin

*e:* granwi...@namhost.com
*m:* +264 81 323 1260
*w:* www.namhost.com <https://www.namhost.com/>

<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>

Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA

The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It
is forbidden to copy, forward, or in any way reveal the contents of this
message to anyone without our explicit consent. The integrity and
security of this email cannot be guaranteed over the Internet.
Therefore, the sender will not be held liable for any damage caused by
the message. For our full privacy policy and disclaimers, please go to
https://www.namhost.com/privacy-policy

smime.p7s
Description: S/MIME Cryptographic Signature

Re: How to Best Shrink Volume

2023-09-02 Thread Granwille Strauss

Thank you Wido

I appreciate your feedback.

On 9/2/23 08:24, Wido den Hollander wrote:

Hi,

Yes, you should make sure the filesystem and partitions are within the
to be set new boundaries.

Once that's done you can shrink the volume.

I would recommend:

- Shrink EXT4 + Partitions to 59GB
- Shrink volume to 60GB
- Grow EXT4 + Partitions to 60GB

The step to 59GB is to prevent that by calc mistake you chip off the
last portion of your filesystem.

Wido

Op 01-09-2023 om 16:29 schreef Granwille Strauss:

Hi Guys

Anyone willing to assist me with a quick one. What's the best and
recommended way to shrink a VM volume via UI? The VM volume was first
60 GB, we then expended it to 120 GB and increased it in the VM via
parted. I now want to take it back to 60GB. How do I proceed? Do I
first shrink the ext4 in the VM first then proceed to shrink the
volume in Cloudstack UI by selecting also the "Shrink OK" option?
Detailed steps would be appreciated, please.

--
Regards / Groete

<https://www.namhost.com> Granwille Strauss // Senior Systems Admin

*e:* granwi...@namhost.com
*m:* +264 81 323 1260
*w:* www.namhost.com <https://www.namhost.com/>

<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>

Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA

The content of this message is confidential. If you have received it
by mistake, please inform us by email reply and then delete the
message. It is forbidden to copy, forward, or in any way reveal the
contents of this message to anyone without our explicit consent. The
integrity and security of this email cannot be guaranteed over the
Internet. Therefore, the sender will not be held liable for any
damage caused by the message. For our full privacy policy and
disclaimers, please go to https://www.namhost.com/privacy-policy

--
Regards / Groete

<https://www.namhost.com> Granwille Strauss // Senior Systems Admin

*e:* granwi...@namhost.com
*m:* +264 81 323 1260
*w:* www.namhost.com <https://www.namhost.com/>

<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>

Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA

The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It
is forbidden to copy, forward, or in any way reveal the contents of this
message to anyone without our explicit consent. The integrity and
security of this email cannot be guaranteed over the Internet.
Therefore, the sender will not be held liable for any damage caused by
the message. For our full privacy policy and disclaimers, please go to
https://www.namhost.com/privacy-policy

smime.p7s
Description: S/MIME Cryptographic Signature

How to Best Shrink Volume

2023-09-01 Thread Granwille Strauss

Hi Guys

Anyone willing to assist me with a quick one. What's the best and
recommended way to shrink a VM volume via UI? The VM volume was first 60
GB, we then expended it to 120 GB and increased it in the VM via parted.
I now want to take it back to 60GB. How do I proceed? Do I first shrink
the ext4 in the VM first then proceed to shrink the volume in Cloudstack
UI by selecting also the "Shrink OK" option? Detailed steps would be
appreciated, please.

--
Regards / Groete

<https://www.namhost.com> Granwille Strauss // Senior Systems Admin

*e:* granwi...@namhost.com
*m:* +264 81 323 1260
*w:* www.namhost.com <https://www.namhost.com/>

<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>

Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA

The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It
is forbidden to copy, forward, or in any way reveal the contents of this
message to anyone without our explicit consent. The integrity and
security of this email cannot be guaranteed over the Internet.
Therefore, the sender will not be held liable for any damage caused by
the message. For our full privacy policy and disclaimers, please go to
https://www.namhost.com/privacy-policy

smime.p7s
Description: S/MIME Cryptographic Signature

Re: Changing Existing VM Compute Offering

Would be nice when you select scale for the VM it gives you the full 
list of compute offerings to select from in order to change it. The 
functionality exists for disk offering, this could be just replicated 
for compute too.


On 8/16/23 11:29, Daan Hoogland wrote:

ugly hack, but that should work ;)

On Wed, Aug 16, 2023 at 10:56 AM Granwille Strauss
wrote:


Thank you, I went with enabling HA for the compute offering it via DB:

update service_offering set ha_enabled=1 where id=;
update vm_instance set ha_enabled=1 where service_offering_id=
On 8/16/23 10:37, Wei ZHOU wrote:

Daan is right. Please ignore my previous email.

-Wei

On Wed, 16 Aug 2023 at 10:34, Daan Hoogland  
  wrote:


Granwille,

There currently is no way, either through UI or through API

On Wed, Aug 16, 2023 at 10:25 AM Granwille Strauss  
  wrote:


Hi All

Just a quick one, how do I change an existing VMs computer offering from

X

to Y via UI? I do not want to scale the CPU resources, no. My Y compute
offering has HA enabled but currently the one its using, X does not have

HA

Enabled. I know via UI you can change disk offerings for a VMs volume. Is
it possible to do the same for compute?
--
Regards / Groete
<https://www.namhost.com>  <https://www.namhost.com>  Granwille Strauss  //  
Senior Systems Admin

*e:*granwi...@namhost.com
*m:* +264 81 323 1260 <+264813231260>
*w:*www.namhost.com
<https://www.facebook.com/namhost>  <https://www.facebook.com/namhost>  <https://twitter.com/namhost>  
<https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/>  
<https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos>  
<https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>  
<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>


<

https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner

Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA



The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It

is

forbidden to copy, forward, or in any way reveal the contents of this
message to anyone without our explicit consent. The integrity and

security

of this email cannot be guaranteed over the Internet. Therefore, the

sender

will not be held liable for any damage caused by the message. For our

full

privacy policy and disclaimers, please go 
tohttps://www.namhost.com/privacy-policy

[image: Powered by AdSigner]
<

https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818

--
Daan


--
Regards / Groete

<https://www.namhost.com>  Granwille Strauss  //  Senior Systems Admin

*e:*granwi...@namhost.com
*m:* +264 81 323 1260 <+264813231260>
*w:*www.namhost.com

<https://www.facebook.com/namhost>  <https://twitter.com/namhost>
<https://www.instagram.com/namhostinternetservices/>
<https://www.linkedin.com/company/namhos>
<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>


<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>

Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA



The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It is
forbidden to copy, forward, or in any way reveal the contents of this
message to anyone without our explicit consent. The integrity and security
of this email cannot be guaranteed over the Internet. Therefore, the sender
will not be held liable for any damage caused by the message. For our full
privacy policy and disclaimers, please go to
https://www.namhost.com/privacy-policy

[image: Powered by AdSigner]
<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>




--
Regards / Groete

<https://www.namhost.com> Granwille Strauss  // Senior Systems Admin

*e:* granwi...@namhost.com
*m:* +264 81 323 1260 
*w:* www.namhost.com <https://www.namhost.com/>

<https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>

<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>

Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA



The content of this message is confidential. If you have received it by 
mistake, please inform us by email reply and then delete the message. It 
is forbidden to copy, forward, or in any way reveal the contents of this 
message to anyone without our explicit consent. The integrity and 
security of this email cannot be guaranteed over the

Re: Changing Existing VM Compute Offering


Thank you, I went with enabling HA for the compute offering it via DB:

update service_offering set ha_enabled=1 where id=;
update vm_instance set ha_enabled=1 where service_offering_id=

On 8/16/23 10:37, Wei ZHOU wrote:

Daan is right. Please ignore my previous email.

-Wei

On Wed, 16 Aug 2023 at 10:34, Daan Hoogland  wrote:


Granwille,

There currently is no way, either through UI or through API

On Wed, Aug 16, 2023 at 10:25 AM Granwille Strauss
  wrote:


Hi All

Just a quick one, how do I change an existing VMs computer offering from

X

to Y via UI? I do not want to scale the CPU resources, no. My Y compute
offering has HA enabled but currently the one its using, X does not have

HA

Enabled. I know via UI you can change disk offerings for a VMs volume. Is
it possible to do the same for compute?
--
Regards / Groete

<https://www.namhost.com>  Granwille Strauss  //  Senior Systems Admin

*e:*granwi...@namhost.com
*m:* +264 81 323 1260 <+264813231260>
*w:*www.namhost.com

<https://www.facebook.com/namhost>  <https://twitter.com/namhost>
<https://www.instagram.com/namhostinternetservices/>
<https://www.linkedin.com/company/namhos>
<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>


<

https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner


Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA



The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It

is

forbidden to copy, forward, or in any way reveal the contents of this
message to anyone without our explicit consent. The integrity and

security

of this email cannot be guaranteed over the Internet. Therefore, the

sender

will not be held liable for any damage caused by the message. For our

full

privacy policy and disclaimers, please go to
https://www.namhost.com/privacy-policy

[image: Powered by AdSigner]
<

https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818




--
Daan


--
Regards / Groete

<https://www.namhost.com> Granwille Strauss  // Senior Systems Admin

*e:* granwi...@namhost.com
*m:* +264 81 323 1260 
*w:* www.namhost.com <https://www.namhost.com/>

<https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>

<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>

Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA



The content of this message is confidential. If you have received it by 
mistake, please inform us by email reply and then delete the message. It 
is forbidden to copy, forward, or in any way reveal the contents of this 
message to anyone without our explicit consent. The integrity and 
security of this email cannot be guaranteed over the Internet. 
Therefore, the sender will not be held liable for any damage caused by 
the message. For our full privacy policy and disclaimers, please go to 
https://www.namhost.com/privacy-policy


Powered by AdSigner 
<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>

smime.p7s
Description: S/MIME Cryptographic Signature

Re: Changing Existing VM Compute Offering

Hi Wei and Dan

VM is stopped. But I see no way to change its compute offering, do you
mind sharing the exact steps, please? I only see the option to scale the
VMs compute offering, but it remains using the same offering and does
not allow me to select a different offering instead. Similiar with how
you can do it with disk offering via volume UI.

Would changing it via database perhaps work?

On 8/16/23 10:33, Wei ZHOU wrote:

Hi,

You need to stop the vm, and then change the compute offering.

-Wei

On Wed, 16 Aug 2023 at 10:25, Granwille Strauss
wrote:

Hi All

Just a quick one, how do I change an existing VMs computer offering from X
to Y via UI? I do not want to scale the CPU resources, no. My Y compute
offering has HA enabled but currently the one its using, X does not have HA
Enabled. I know via UI you can change disk offerings for a VMs volume. Is
it possible to do the same for compute?
--
Regards / Groete

<https://www.namhost.com> Granwille Strauss // Senior Systems Admin

*e:*granwi...@namhost.com
*m:* +264 81 323 1260 <+264813231260>
*w:*www.namhost.com

<https://www.facebook.com/namhost> <https://twitter.com/namhost>
<https://www.instagram.com/namhostinternetservices/>
<https://www.linkedin.com/company/namhos>
<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>

<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>

Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA

The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It is
forbidden to copy, forward, or in any way reveal the contents of this
message to anyone without our explicit consent. The integrity and security
of this email cannot be guaranteed over the Internet. Therefore, the sender
will not be held liable for any damage caused by the message. For our full
privacy policy and disclaimers, please go to
https://www.namhost.com/privacy-policy

[image: Powered by AdSigner]
<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>

--
Regards / Groete

<https://www.namhost.com> Granwille Strauss // Senior Systems Admin

*e:* granwi...@namhost.com
*m:* +264 81 323 1260
*w:* www.namhost.com <https://www.namhost.com/>

<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>

Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA

The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It
is forbidden to copy, forward, or in any way reveal the contents of this
message to anyone without our explicit consent. The integrity and
security of this email cannot be guaranteed over the Internet.
Therefore, the sender will not be held liable for any damage caused by
the message. For our full privacy policy and disclaimers, please go to
https://www.namhost.com/privacy-policy

smime.p7s
Description: S/MIME Cryptographic Signature

Changing Existing VM Compute Offering

Hi All

Just a quick one, how do I change an existing VMs computer offering from
X to Y via UI? I do not want to scale the CPU resources, no. My Y
compute offering has HA enabled but currently the one its using, X does
not have HA Enabled. I know via UI you can change disk offerings for a
VMs volume. Is it possible to do the same for compute?

--
Regards / Groete

<https://www.namhost.com> Granwille Strauss // Senior Systems Admin

*e:* granwi...@namhost.com
*m:* +264 81 323 1260
*w:* www.namhost.com <https://www.namhost.com/>

<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>

Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA

The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It
is forbidden to copy, forward, or in any way reveal the contents of this
message to anyone without our explicit consent. The integrity and
security of this email cannot be guaranteed over the Internet.
Therefore, the sender will not be held liable for any damage caused by
the message. For our full privacy policy and disclaimers, please go to
https://www.namhost.com/privacy-policy

smime.p7s
Description: S/MIME Cryptographic Signature

Re: Write Speeds

2023-07-14 Thread Granwille Strauss


Hi Guys

Just want to get back to you all. Thank you very much for your input it 
has greatly helped me and now I have learned a few new things too. I 
want to confirm that write speeds was never the issue, it was a network 
bottleneck instead. It was not related to ACS either but more towards 
the default network configs in the VMs itself.



I started applying network optimisations at a kernel level of my VMs 
such as setting the following:



net.ipv4.tcp_congestion_control=bbr
net.ipv4.tcp_notsent_lowat=131072
net.ipv4.tcp_rmem=8192 262144 536870912
net.ipv4.tcp_wmem=4096 16384 536870912
net.ipv4.tcp_window_scaling=1
net.core.rmem_max=536870912
net.core.wmem_max=536870912
net.ipv4.tcp_fastopen=3
net.ipv4.tcp_max_syn_backlog=4096
net.ipv4.tcp_timestamps=1
net.ipv4.tcp_sack=1


And now speeds have increased from 50 mbps to 150 Mbps, how I wish I 
thought of this earlier.  Its not quite there yet, I want to further 
optimise this and try not to have any trade-offs doing so. I read a 
little and so but would appreciate it if anyone is willing to share some 
more optimisations for VMs, for network bandwidths of 1000 Mbps.


On 7/10/23 19:53, Alex Mattioli wrote:

I doesn’t necessarily get throttled, the added latency will definitely impact 
the maximum bandwidth achievable per stream, especially if you are using TCP. 
In this case a bandwidth delay calculator can help you find the maximum 
theoretical bandwidth for a given latency:
https://www.switch.ch/network/tools/tcp_throughput/?do+new+calculation=do+new+calculation





From: Granwille Strauss
Sent: Monday, July 10, 2023 6:04 PM
To:users@cloudstack.apache.org
Cc: Levin Ng; Jithin 
Raju;vivek.ku...@indiqus.com
Subject: Re: Write Speeds


Hi Levin

I skipped all the VM testing and tested iperf straight from the KVM host to 
determine a base line to the remote USA VM machine:

- KVM to Remote USA VM: 113 Mbits/sec
- USA VM to KVM Host: 35.9 Mbits/sec

I then ran the same test again, this this time the remote host was in a DC 
that's close to our DC  in the same country that we use:

- KVM to remote host: 409 Mbits/sec
- Remote host to KVM: 477 Mbits/sec

So do you think its safe to conclude that somewhere traffic from the USA VM to 
the local KVM host gets throttled? Based on the results above, the throttling 
doesn't seems to be in from ISPs inside our country.

So yeah, somewhere some ISP is throttling during the USA routes.
On 7/10/23 15:56, Levin Ng wrote:

Hi Groete,



I’m not sure what is your network setting in ACS, but test between two public 
IP with ~500Mbps  sound like u are saturated by in/out bound traffics in the 
single network path, can you do a test from outside ACS to your VM using an 
same public network segment IP, it will avoid network routing and confusion., 
what is your ACS network driver using? If vxlan, better check with network 
switch multicast performance.



The remote performances clearly shown the ISP put some limit on the line, you 
have to check with them. Unless your line is end-to-end, Metro-Ethernet etc… 
otherwise it is not always have guarantee throughput.





On the disk performance, should you share your fio test command and result 
beforehand. I’m assuming you are doing something like



fio -filename=./testfile.bin -direct=1 -iodepth 8 -thread -rw=randrw 
-rwmixread=50 -ioengine=psync -bs=4k -size=1000M -numjobs=30 -runtime=600 
-group_reporting -name=mytest





Regards,

Levin

On 10 Jul 2023 at 11:33 +0100, Granwille 
Strauss<mailto:granwi...@namhost.com>, wrote:

Hi Guys

Thank you, I have been running more tests now with the feedback you guys gave. 
Firstly, I want to break this up into two sections:

1. Network:

- So I have been running iperf tests between my VMs on their public network, 
and my iperf tests gives me speeds of ~500 Mbps, keep in mind this in between 
two local VMs on the same KVM but on public network.

- I then run iperf tests in and out from my local VMs to remote servers, this 
is where it does funny things. From the remote VM in USA, I run an iperf test 
to my local VM, the speeds show ~50 Mbps. And if I run a test from my local VM 
to a remote USA VM the same ~50 Mbps speeds are accomplished. I ran my iperf 
tests with 1 GB and 2GB flags and the results remain constant

- During all these test I kept an eye on my VR resources, which use default 
service offerings, it never spiked or reach thresholds.

Is it safe to assume that because of the MASSIVE distance between the remote VM 
and my local VMs, the speed dropping to ~50 Mbps is normal? Keep in mind the 
remote VM has 1 Gbps line too and this VM is managed by a big ISP provider in 
the USA. To me its quite a massive drop from 1000 Mbps to 50 Mbps, this kinda 
does not make sense to me. I would understand at least 150 Mbps.

2. Disk Write Speed:

- It seems the only changes that can be made is to implement disk cache 
options. And so far I see write-back seems to be common practise for most cloud 
providers, given that the

Re: Write Speeds


I Did the following:

1. Tail agent log at /var/log/cloudstack/agent/agent.log
2. Set enable.io.uring=true in agent properties
3. Restart Cloudstack-agent
4. Stopped VM and set io.policy=io.uring
5. Set io.policy=io.uring in local primary storage settings for KVM
6. Start VM up again
7. restart cloudstack agent again.

You can find the log of all these steps and what it triggered here: 
https://we.tl/t-AJdC2bXa4u


And as you requested, this is all I see in the log when I grep "uring" 
from the log:



grep "uring" /var/log/cloudstack/agent/agent.log
2023-07-11 11:59:58,994 INFO [kvm.resource.LibvirtComputingResource] 
(main:null) (logid:) IO uring driver for Qemu: disabled
2023-07-11 12:04:59,064 INFO [kvm.resource.LibvirtComputingResource] 
(main:null) (logid:) IO uring driver for Qemu: disabled
2023-07-11 13:16:07,433 INFO [kvm.resource.LibvirtComputingResource] 
(main:null) (logid:) IO uring driver for Qemu: disabled
2023-07-11 13:17:48,056 INFO [kvm.resource.LibvirtComputingResource] 
(main:null) (logid:) IO uring driver for Qemu: disabled


That follows with the following error, which you will see in the log:

2023-07-11 13:16:07,520 WARN [kvm.storage.KVMStoragePoolManager] 
(main:null) (logid:) Duplicate StorageAdaptor type PowerFlex, not 
loading com.cloud.hypervisor.kvm.storage.ScaleIOStorageAdaptor
2023-07-11 13:16:07,522 INFO [kvm.resource.LibvirtComputingResource] 
(main:null) (logid:) No libvirt.vif.driver specified. Defaults to 
BridgeVifDriver.
2023-07-11 13:16:07,725 INFO  [cloud.serializer.GsonHelper] 
(main:null) (logid:) Default Builder inited.


On 7/11/23 13:03, Slavka Peleva wrote:
Can you share the full agent.log with the VM deployment? Also, can you 
search the agent.log for "IO uring driver for Qemu:  " and share 
the whole log message?


Best regards,
Slavka

On Tue, Jul 11, 2023 at 1:55 PM Granwille Strauss 
 wrote:


Hi

Thank you for the feedback, that's exactly what I did earlier,
didn't do anything. I also set it in global config and you can
also set it for the local primary storage. Nothing happened in the
XML dumps.

On 7/11/23 12:50, Slavka Peleva wrote:

Can you set it from the VM's settings? The VM has to be stopped.
I will check the global config later.

image.png



On Tue, Jul 11, 2023 at 1:38 PM Granwille Strauss
 wrote:

Hi

I am using the latest CS 4.18.0.0

On 7/11/23 12:28, Slavka Peleva wrote:

Can you share the CS version that you're using? There are some 
differences
in enabling the IO policy depending on the version.

Best regards,
Slavka

On Tue, Jul 11, 2023 at 1:05 PM Granwille Strauss  
<mailto:granwi...@namhost.com>
wrote:


Nope, didn't work. I added enable.io.uring=true to
/etc/cloudstack/agent/agent.properties, restarted cloudstack-agent.
Shutdown one VM machine. Set io.policy=io.uring for this VM then start 
up
VM again and XML dump remains the same:

virsh dumpxml i-2-182-VM | grep drive
   
   
   


On 7/11/23 11:53, Slavka Peleva wrote:

Hi,

Can you check if the `enable.io.uring=true` property in the
agent.properties file is set? You have to restart the cloudstack-agent
service if you make any changes.

Best regards,
Slavka


--
Regards / Groete

<https://www.namhost.com>  <https://www.namhost.com>  Granwille Strauss 
 //  Senior Systems Admin

*e:*granwi...@namhost.com
*m:* +264 81 323 1260 <+264813231260>
*w:*www.namhost.com  <http://www.namhost.com>

<https://www.facebook.com/namhost>  <https://www.facebook.com/namhost>  
<https://twitter.com/namhost>  <https://twitter.com/namhost>
<https://www.instagram.com/namhostinternetservices/>  
<https://www.instagram.com/namhostinternetservices/>
<https://www.linkedin.com/company/namhos>  
<https://www.linkedin.com/company/namhos>
<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>  
<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>



<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
  
<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>

Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA



The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. 
It is
forbidden to copy, forward, or in any way reveal the contents of this
message to anyone without our explicit consent. The integrity and 
security
of this email cannot be guaranteed over the Internet. Therefore, the 
sender

Re: Write Speeds


Hi

Thank you for the feedback, that's exactly what I did earlier, didn't do 
anything. I also set it in global config and you can also set it for the 
local primary storage. Nothing happened in the XML dumps.


On 7/11/23 12:50, Slavka Peleva wrote:
Can you set it from the VM's settings? The VM has to be stopped. I 
will check the global config later.


image.png



On Tue, Jul 11, 2023 at 1:38 PM Granwille Strauss 
 wrote:


Hi

I am using the latest CS 4.18.0.0

On 7/11/23 12:28, Slavka Peleva wrote:

Can you share the CS version that you're using? There are some differences
in enabling the IO policy depending on the version.

Best regards,
Slavka

On Tue, Jul 11, 2023 at 1:05 PM Granwille Strauss  
<mailto:granwi...@namhost.com>
wrote:


Nope, didn't work. I added enable.io.uring=true to
/etc/cloudstack/agent/agent.properties, restarted cloudstack-agent.
Shutdown one VM machine. Set io.policy=io.uring for this VM then start up
VM again and XML dump remains the same:

virsh dumpxml i-2-182-VM | grep drive
   
   
   


On 7/11/23 11:53, Slavka Peleva wrote:

Hi,

Can you check if the `enable.io.uring=true` property in the
agent.properties file is set? You have to restart the cloudstack-agent
service if you make any changes.

Best regards,
Slavka


--
Regards / Groete

<https://www.namhost.com>  <https://www.namhost.com>  Granwille Strauss  // 
 Senior Systems Admin

*e:*granwi...@namhost.com
*m:* +264 81 323 1260 <+264813231260>
*w:*www.namhost.com  <http://www.namhost.com>

<https://www.facebook.com/namhost>  <https://www.facebook.com/namhost>  
<https://twitter.com/namhost>  <https://twitter.com/namhost>
<https://www.instagram.com/namhostinternetservices/>  
<https://www.instagram.com/namhostinternetservices/>
<https://www.linkedin.com/company/namhos>  
<https://www.linkedin.com/company/namhos>
<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>  
<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>



<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
  
<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>

Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA



The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It is
forbidden to copy, forward, or in any way reveal the contents of this
message to anyone without our explicit consent. The integrity and security
of this email cannot be guaranteed over the Internet. Therefore, the sender
will not be held liable for any damage caused by the message. For our full
privacy policy and disclaimers, please go to
https://www.namhost.com/privacy-policy

[image: Powered by AdSigner]
<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> 
 <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>

-- 
Regards / Groete


<https://www.namhost.com> Granwille Strauss  // Senior Systems Admin

*e:* granwi...@namhost.com
*m:* +264 81 323 1260 
*w:* www.namhost.com <https://www.namhost.com/>


<https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>


<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>

Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA



The content of this message is confidential. If you have received
it by mistake, please inform us by email reply and then delete the
message. It is forbidden to copy, forward, or in any way reveal
the contents of this message to anyone without our explicit
consent. The integrity and security of this email cannot be
guaranteed over the Internet. Therefore, the sender will not be
held liable for any damage caused by the message. For our full
privacy policy and disclaimers, please go to
https://www.namhost.com/privacy-policy

Powered by AdSigner

<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>


--
Regards / Groete

<https://www.namhost.com> Granwille Strauss  // Senior Systems Admin

*e:* granwi...@namhost.com
*m:* +264 81 323 1260 
*w:* www.namhost.com <https://www.namhost.com/>

<https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/na

Re: Write Speeds

I am using the latest CS 4.18.0.0

On 7/11/23 12:28, Slavka Peleva wrote:

Can you share the CS version that you're using? There are some differences
in enabling the IO policy depending on the version.

Best regards,
Slavka

On Tue, Jul 11, 2023 at 1:05 PM Granwille Strauss
wrote:

Nope, didn't work. I added enable.io.uring=true to
/etc/cloudstack/agent/agent.properties, restarted cloudstack-agent.
Shutdown one VM machine. Set io.policy=io.uring for this VM then start up
VM again and XML dump remains the same:

virsh dumpxml i-2-182-VM | grep drive

On 7/11/23 11:53, Slavka Peleva wrote:

Hi,

Can you check if the `enable.io.uring=true` property in the
agent.properties file is set? You have to restart the cloudstack-agent
service if you make any changes.

Best regards,
Slavka

--
Regards / Groete

<https://www.namhost.com> Granwille Strauss // Senior Systems Admin

*e:*granwi...@namhost.com
*m:* +264 81 323 1260 <+264813231260>
*w:*www.namhost.com

<https://www.facebook.com/namhost> <https://twitter.com/namhost>
<https://www.instagram.com/namhostinternetservices/>
<https://www.linkedin.com/company/namhos>
<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>

<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>

Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA

The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It is
forbidden to copy, forward, or in any way reveal the contents of this
message to anyone without our explicit consent. The integrity and security
of this email cannot be guaranteed over the Internet. Therefore, the sender
will not be held liable for any damage caused by the message. For our full
privacy policy and disclaimers, please go to
https://www.namhost.com/privacy-policy

[image: Powered by AdSigner]
<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>

--
Regards / Groete

<https://www.namhost.com> Granwille Strauss // Senior Systems Admin

*e:* granwi...@namhost.com
*m:* +264 81 323 1260
*w:* www.namhost.com <https://www.namhost.com/>

<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>

Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA

The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It
is forbidden to copy, forward, or in any way reveal the contents of this
message to anyone without our explicit consent. The integrity and
security of this email cannot be guaranteed over the Internet.
Therefore, the sender will not be held liable for any damage caused by
the message. For our full privacy policy and disclaimers, please go to
https://www.namhost.com/privacy-policy

smime.p7s
Description: S/MIME Cryptographic Signature

Re: Write Speeds

virsh dumpxml i-2-182-VM | grep drive

On 7/11/23 11:53, Slavka Peleva wrote:

Hi,

Can you check if the `enable.io.uring=true` property in the
agent.properties file is set? You have to restart the cloudstack-agent
service if you make any changes.

Best regards,
Slavka

--
Regards / Groete

<https://www.namhost.com> Granwille Strauss // Senior Systems Admin

*e:* granwi...@namhost.com
*m:* +264 81 323 1260
*w:* www.namhost.com <https://www.namhost.com/>

<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>

Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA

The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It
is forbidden to copy, forward, or in any way reveal the contents of this
message to anyone without our explicit consent. The integrity and
security of this email cannot be guaranteed over the Internet.
Therefore, the sender will not be held liable for any damage caused by
the message. For our full privacy policy and disclaimers, please go to
https://www.namhost.com/privacy-policy

smime.p7s
Description: S/MIME Cryptographic Signature

Re: Async backup


HI Alex

Sorry, I said this wrong, when making a snapshot a copy is first made to 
the primary storage, _not management disk_, and once done its moved to 
the secondary storage and then deleted from primary. This is explained 
here: 
https://github.com/apache/cloudstack/issues/6836#issuecomment-1285524454


On 7/10/23 19:50, Alex Mattioli wrote:

all snapshots make a local copy to the management server disk before sending it 
to the secondary storage locations, yes.

What exactly do you mean by that?

From: Granwille Strauss
Sent: Monday, July 10, 2023 1:21 PM
To:users@cloudstack.apache.org
Cc: Nikolaos Tsinganos
Subject: Re: Async backup


Correct me if I am wrong, all snapshots make a local copy to the management 
server disk before sending it to the secondary storage locations, yes. Async or 
not. But yes, I believe that is the case. However, when async is enabled you 
get to use and work on other tasks with the management server, while it does 
the snapshot process in the background. Whereas having async disabled, you are 
restricted from doing other task on the management server until the snapshot 
process completes, that's how I understand it.
On 7/10/23 13:04, Nikolaos Tsinganos wrote:

Thank you, Granwille for your prompt answer.



Can you elaborate on the "locally"  and "storage location" terms?

On a host that has multiple primary storages (e.g. NFS and some other storage 
over iSCSI)  but no local disks are used, what is considered as 'locally'?

Also, by "storage location"  do you mean the secondary storage?



Regards,

Nikolaos



From: Granwille Strauss<mailto:granwi...@namhost.com>

Sent: Monday, July 10, 2023 1:48 PM

To:users@cloudstack.apache.org<mailto:users@cloudstack.apache.org>

Cc: Nikolaos Tsinganos<mailto:ni...@dataways.gr>

Subject: Re: Async backup



In short and in layman's terms, it makes the volume snapshot, stores it locally 
and then in the background transfers it to the storage location. This helps 
with server resource usage. But see attached screenshot for a detailed answer.

On 7/10/23 12:38, Nikolaos Tsinganos wrote:

Hi All,



Can somebody explain what the "Async backup" option does while taking volume 
snapshot?



Regards,

Nikolaos


--

Regards / Groete
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
Granwille Strauss  //  Senior Systems Admin

e:granwi...@namhost.com<mailto:granwi...@namhost.com>
m: +264 81 323 1260
w:www.namhost.com<https://www.namhost.com/>

[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>

[https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA

The content of this message is confidential. If you have received it by 
mistake, please inform us by email reply and then delete the message. It is 
forbidden to copy, forward, or in any way reveal the contents of this message 
to anyone without our explicit consent. The integrity and security of this 
email cannot be guaranteed over the Internet. Therefore, the sender will not be 
held liable for any damage caused by the message. For our full privacy policy 
and disclaimers, please go tohttps://www.namhost.com/privacy-policy

<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>

  


--
Regards / Groete

<https://www.namhost.com> Granwille Strauss  // Senior Systems Admin

*e:* granwi...@namhost.com
*m:* +264 81 323 1260 
*w:* www.namhost.com <https://www.namhost.com/>

<https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><http

Re: Write Speeds

Anyone with any ideas why ACS is not detecting I have newer versions of 
qemu and libvirt to enable io_uring?


On 7/10/23 19:57, Jorge Luiz Correa wrote:

Hum, so strange.

I'm not a CloudStack specialist but it look like the code is simple 
and try to test the versions of qemu and libvirt:


https://github.com/apache/cloudstack/pull/5012/commits/c7c3dd3dd9b8869f45c5bd9c17af83d230ac7886

Here, at the slide bottom he shows this simple test too.

https://youtu.be/y0NYuUtm5Kk?list=PLnIKk7GjgFlYfut3ZIOrvN--_YuSPIerQ=791 
<https://youtu.be/y0NYuUtm5Kk?list=PLnIKk7GjgFlYfut3ZIOrvN--_YuSPIerQ=791>


For some reason CloudStack is not detecting your versions. My disk 
offering is simple, Thin provisioning, custom disk size, QoS = none, 
Write-cache type = no disk cache. I'm using Ubuntu Server 22.04 and 
CloudStack 4.17.2.


Em seg., 10 de jul. de 2023 às 14:22, Granwille Strauss 
 escreveu:


Jorge, I thought so too, but XML dumps does not contain it. So I
figured the oi.policy setting needs to be set in management
server. Here's my KVM details:


Compiled against library: libvirt 8.0.0
Using library: libvirt 8.0.0
Using API: QEMU 8.0.0
Running hypervisor: QEMU 6.2.0


qemu guest agents also exist on VMs. And here's a XML dump:


root@athena03 ~ $ virsh dumpxml i-2-120-VM | grep driver
  
  
root@athena03 ~ $

On 7/10/23 18:51, Jorge Luiz Correa wrote:

Granwille, no special configuration, just the CloudStack default
behavior. As I understand, CloudStack can detect automatically if
host supports this feature based on qemu and libvirt versions.

https://github.com/apache/cloudstack/issues/4883#issuecomment-813955599


What versions of kernel, qemu and libvirt are you using in KVM host?

Em seg., 10 de jul. de 2023 às 13:26, Granwille Strauss
 escreveu:

Hi Jorge

How do you actually enable io_uring via Cloustack?
My KVM does have the necessary requirements.

I enabled io.policy settings in global settings, local storage
and in the VM settings via UI. And my xml dump of VM doesn’t
include io_uring under driver for some reason.

-- 
Regards / Groete


<https://www.namhost.com/>    Granwille Strauss  // Senior
Systems Administrator

*e:* granwi...@namhost.com
*m:* +264 81 323 1260 
*w:* www.namhost.com <https://www.namhost.com/>

<https://www.facebook.com/namhost>
<https://twitter.com/namhost>
<https://www.instagram.com/namhostinternetservices/>
<https://www.linkedin.com/company/namhos>
<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>


<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>

The content of this message is confidential.

For our full privacy policy and disclaimers, please go to
https://www.namhost.com/privacy-policy

Powered by AdSigner

<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>


On 10 Jul 2023, at 5:27 PM, Granwille Strauss

<mailto:granwi...@namhost.com.invalid> wrote:



Hi Jorge

Thank you so much for this. I used your FIO config and
surprisingly it seems fine:


write-test: (g=0): rw=randrw, bs=(R) 1300MiB-1300MiB, (W)
1300MiB-1300MiB, (T) 1300MiB-1300MiB, ioengine=libaio,
iodepth=1
fio-3.19

Run status group 0 (all jobs):
   READ: bw=962MiB/s (1009MB/s), 962MiB/s-962MiB/s
(1009MB/s-1009MB/s), io=3900MiB (4089MB), run=4052-4052msec
  WRITE: bw=321MiB/s (336MB/s), 321MiB/s-321MiB/s
(336MB/s-336MB/s), io=1300MiB (1363MB), run=4052-4052msec


This is without enabling io_uring. I see I can enable it per
VM using the UI by setting the io.policy = io_uring. Will
enable this on a few VMs and see if it works better.

On 7/10/23 15:41, Jorge Luiz Correa wrote:

Hi Granwille! About the READ/WRITE performance, as Levin suggested, 
check
the XML of virtual machines looking at disk/device section. Look for
io='io_uring'.

As stated here:

https://github.com/apache/cloudstack/issues/4883

CloudStack can use io_uring with Qemu >= 5.0 and Libvirt >= 6.3.0.

I tried to do some tests at some points like your environment.

##
VM in NFS Primary Storage (Hybrid NAS)
Default disk offering, thin (no restriction)


 /usr/bin/qemu-system-x86_64
 
   
   
   
   
   fb46fd2c59bd4127851b
   
   
 

fio --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test
--filename=/tmp/ra

Re: Write Speeds


Vivek, yes I have and its in line with my VMs:


  
    
    
  
  
  


On 7/10/23 18:50, Vivek Kumar wrote:

Did you check on network QoS on VR as well if there is any .!


  Vivek Kumar

Sr. Manager - Cloud & DevOps
TechOps | Indiqus Technologies





vivek.ku...@indiqus.com

www.indiqus.com <https://www.indiqus.com/>





On 10-Jul-2023, at 9:34 PM, Granwille Strauss  
wrote:


Hi Levin

I skipped all the VM testing and tested iperf straight from the KVM 
host to determine a base line to the remote USA VM machine:


- KVM to Remote USA VM: 113 Mbits/sec
- USA VM to KVM Host: 35.9 Mbits/sec

I then ran the same test again, this this time the remote host was in 
a DC that's close to our DC  in the same country that we use:


- KVM to remote host: 409 Mbits/sec
- Remote host to KVM: 477 Mbits/sec

So do you think its safe to conclude that somewhere traffic from the 
USA VM to the local KVM host gets throttled? Based on the results 
above, the throttling doesn't seems to be in from ISPs inside our 
country.


So yeah, somewhere some ISP is throttling during the USA routes.

On 7/10/23 15:56, Levin Ng wrote:

Hi Groete,

I’m not sure what is your network setting in ACS, but test between two public 
IP with ~500Mbps  sound like u are saturated by in/out bound traffics in the 
single network path, can you do a test from outside ACS to your VM using an 
same public network segment IP, it will avoid network routing and confusion., 
what is your ACS network driver using? If vxlan, better check with network 
switch multicast performance.

The remote performances clearly shown the ISP put some limit on the line, you 
have to check with them. Unless your line is end-to-end, Metro-Ethernet etc… 
otherwise it is not always have guarantee throughput.


On the disk performance, should you share your fio test command and result 
beforehand. I’m assuming you are doing something like

fio -filename=./testfile.bin -direct=1 -iodepth 8 -thread -rw=randrw 
-rwmixread=50 -ioengine=psync -bs=4k -size=1000M -numjobs=30 -runtime=600 
-group_reporting -name=mytest


Regards,
Levin
On 10 Jul 2023 at 11:33 +0100, Granwille Strauss, wrote:

Hi Guys
Thank you, I have been running more tests now with the feedback you guys gave. 
Firstly, I want to break this up into two sections:
1. Network:
- So I have been running iperf tests between my VMs on their public network, 
and my iperf tests gives me speeds of ~500 Mbps, keep in mind this in between 
two local VMs on the same KVM but on public network.
- I then run iperf tests in and out from my local VMs to remote servers, this 
is where it does funny things. From the remote VM in USA, I run an iperf test 
to my local VM, the speeds show ~50 Mbps. And if I run a test from my local VM 
to a remote USA VM the same ~50 Mbps speeds are accomplished. I ran my iperf 
tests with 1 GB and 2GB flags and the results remain constant
- During all these test I kept an eye on my VR resources, which use default 
service offerings, it never spiked or reach thresholds.
Is it safe to assume that because of the MASSIVE distance between the remote VM 
and my local VMs, the speed dropping to ~50 Mbps is normal? Keep in mind the 
remote VM has 1 Gbps line too and this VM is managed by a big ISP provider in 
the USA. To me its quite a massive drop from 1000 Mbps to 50 Mbps, this kinda 
does not make sense to me. I would understand at least 150 Mbps.
2. Disk Write Speed:
- It seems the only changes that can be made is to implement disk cache 
options. And so far I see write-back seems to be common practise for most cloud 
providers, given that they have the necessary power redundancy and VM backup 
images in place.

But for now, other than write cache types are there anything else that can be 
done to improve disk writing speeds. I checked RedHat guides on optimising VMs 
and I seem to have most in place, but write speeds remain at ~50 Mbps.
On 7/10/23 06:25, Jithin Raju wrote:

Hi  Groete,

The VM virtual NIC network throttling is picked up from its compute offering. 
You may need to create a new compute offering and change the VM’s compute 
offering. If it is not specified in the compute offering, means it is taking 
the values from the global settings: vm.network.throttling.rate .



-Jithin

From: Levin Ng
Date: Sunday, 9 July 2023 at 5:00 AM
To:users@cloudstack.apache.org  , Granwille 
Strauss
Cc:vivek.ku...@indiqus.com  , Nux
Subject: Re: Write Speeds
Dear Groete,

https://github.com/shapeblue/cloudstack/blob/965856057d5147f12b86abe5c9c205cdc5e44615/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/DirectVifDriver.java

https://libvirt.org/formatnetwork.html#quality-of-service

This is in kilobytes/second, u have to divide by 8

1Gbps / 8bit = 128MB = 128000KBps

You can verify by iperf test, and yes, u need to ensure both VR and VM match 
bandwidth settings to get a consistent result, something u

Re: Write Speeds

Hi JorgeHow do you actually enable io_uring via Cloustack?My KVM does have the necessary requirements.I enabled io.policy settings in global settings, local storageand in the VM settings via UI. And my xml dump of VM doesn’t include io_uring under driver for some reason.-- Regards / GroeteGranwille Strauss  //  Senior Systems Administratore: granwi...@namhost.comm: +264 81 323 1260w: www.namhost.comThe content of this message is confidential.For our full privacy policy and disclaimers, please go to https://www.namhost.com/privacy-policyOn 10 Jul 2023, at 5:27 PM, Granwille Strauss  wrote:
  

  
  
Hi Jorge
Thank you so much for this. I used your FIO config and
  surprisingly it seems fine:


  write-test: (g=0): rw=randrw, bs=(R)
1300MiB-1300MiB, (W) 1300MiB-1300MiB, (T) 1300MiB-1300MiB,
ioengine=libaio, iodepth=1
fio-3.19

Run status group 0 (all jobs):
   READ: bw=962MiB/s (1009MB/s), 962MiB/s-962MiB/s
(1009MB/s-1009MB/s), io=3900MiB (4089MB), run=4052-4052msec
  WRITE: bw=321MiB/s (336MB/s), 321MiB/s-321MiB/s
(336MB/s-336MB/s), io=1300MiB (1363MB), run=4052-4052msec
  

This is without enabling io_uring. I see I can enable it per VM
  using the UI by setting the io.policy = io_uring. Will enable this
  on a few VMs and see if it works better. 

On 7/10/23 15:41, Jorge Luiz Correa
  wrote:


  Hi Granwille! About the READ/WRITE performance, as Levin suggested, check
the XML of virtual machines looking at disk/device section. Look for
io='io_uring'.

As stated here:

https://github.com/apache/cloudstack/issues/4883

CloudStack can use io_uring with Qemu >= 5.0 and Libvirt >= 6.3.0.

I tried to do some tests at some points like your environment.

##
VM in NFS Primary Storage (Hybrid NAS)
Default disk offering, thin (no restriction)


/usr/bin/qemu-system-x86_64

  
  
  
  
  fb46fd2c59bd4127851b
  
  


fio --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test
--filename=/tmp/random_read_write.fio --bs=1300m --iodepth=8 --size=6G
--readwrite=randrw

READ: 569MiB/s
WRITE: 195MiB/s

##
VM in Local Primary Storage (local SSD host)
Default disk offering, thin (no restriction)


/usr/bin/qemu-system-x86_64

  
  
  
  
  fb46fd2c59bd4127851b
  
  

fio --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test
--filename=/tmp/random_read_write.fio --bs=1300m --iodepth=8 --size=6G
--readwrite=randrw

First run (a little bit slow if using "thin" because need to allocate space
in qcow2):
READ: bw=796MiB/s
WRITE: bw=265MiB/s

Second run:
READ: bw=952MiB/s
WRITE: bw=317MiB/s

##
Directly in local SSD of host:

fio --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test
--filename=/tmp/random_read_write.fio --bs=1300m --iodepth=8 --size=6G
--readwrite=randrw

READ: bw=931MiB/s
WRITE: bw=310MiB/s

OBS.: parameters of fio test need to be changed to test in your environment
as it depends on the number of cpus, memory, --bs, --iodepth etc.

Host is running 5.15.0-43 kernel, qemu 6.2 and libvirt 8. CloudStack is
4.17.2. So, VM in local SSD of host could have very similar disk
performance from the host.

I hope this could help you!

Thanks.


  

  
  

-- 
  
  

  
Regards / Groete
  

  
  
    
    Granwille
    Strauss  //  Senior Systems Admin
  
  e: granwi...@namhost.com
  m: +264 81 323
  1260
  w: www.namhost.com
  
  
  
  

  
  

  
  
Namhost Internet Services (Pty) Ltd,
  24 Black Eagle Rd, Hermanus, 7210, RSA
  
  

  

  
  

  
The
  content of this message is confidential. If you have
  received it by mistake, please inform us by email reply
  and then delete the message. It is forbidden to copy,
  forward, or in any way reveal the contents of this message
  to anyone without our explicit consent. The integrity and
  security of this email cannot be guaranteed over the
  Internet. Therefore, the sender will not be held liable
  for any damage caused by the message. For our full privacy
  policy and disclaimers, please go to
  https://www.namhost.com/privacy-policy
  

  
   
                                                         
  
  



smime.p7s
Description: S/MIME cryptographic signature

Re: Write Speeds


Hi Levin

I skipped all the VM testing and tested iperf straight from the KVM host 
to determine a base line to the remote USA VM machine:


- KVM to Remote USA VM: 113 Mbits/sec
- USA VM to KVM Host: 35.9 Mbits/sec

I then ran the same test again, this this time the remote host was in a 
DC that's close to our DC  in the same country that we use:


- KVM to remote host: 409 Mbits/sec
- Remote host to KVM: 477 Mbits/sec

So do you think its safe to conclude that somewhere traffic from the USA 
VM to the local KVM host gets throttled? Based on the results above, the 
throttling doesn't seems to be in from ISPs inside our country.


So yeah, somewhere some ISP is throttling during the USA routes.

On 7/10/23 15:56, Levin Ng wrote:

Hi Groete,

I’m not sure what is your network setting in ACS, but test between two public 
IP with ~500Mbps  sound like u are saturated by in/out bound traffics in the 
single network path, can you do a test from outside ACS to your VM using an 
same public network segment IP, it will avoid network routing and confusion., 
what is your ACS network driver using? If vxlan, better check with network 
switch multicast performance.

The remote performances clearly shown the ISP put some limit on the line, you 
have to check with them. Unless your line is end-to-end, Metro-Ethernet etc… 
otherwise it is not always have guarantee throughput.


On the disk performance, should you share your fio test command and result 
beforehand. I’m assuming you are doing something like

fio -filename=./testfile.bin -direct=1 -iodepth 8 -thread -rw=randrw 
-rwmixread=50 -ioengine=psync -bs=4k -size=1000M -numjobs=30 -runtime=600 
-group_reporting -name=mytest


Regards,
Levin
On 10 Jul 2023 at 11:33 +0100, Granwille Strauss, wrote:

Hi Guys
Thank you, I have been running more tests now with the feedback you guys gave. 
Firstly, I want to break this up into two sections:
1. Network:
- So I have been running iperf tests between my VMs on their public network, 
and my iperf tests gives me speeds of ~500 Mbps, keep in mind this in between 
two local VMs on the same KVM but on public network.
- I then run iperf tests in and out from my local VMs to remote servers, this 
is where it does funny things. From the remote VM in USA, I run an iperf test 
to my local VM, the speeds show ~50 Mbps. And if I run a test from my local VM 
to a remote USA VM the same ~50 Mbps speeds are accomplished. I ran my iperf 
tests with 1 GB and 2GB flags and the results remain constant
- During all these test I kept an eye on my VR resources, which use default 
service offerings, it never spiked or reach thresholds.
Is it safe to assume that because of the MASSIVE distance between the remote VM 
and my local VMs, the speed dropping to ~50 Mbps is normal? Keep in mind the 
remote VM has 1 Gbps line too and this VM is managed by a big ISP provider in 
the USA. To me its quite a massive drop from 1000 Mbps to 50 Mbps, this kinda 
does not make sense to me. I would understand at least 150 Mbps.
2. Disk Write Speed:
- It seems the only changes that can be made is to implement disk cache 
options. And so far I see write-back seems to be common practise for most cloud 
providers, given that they have the necessary power redundancy and VM backup 
images in place.

But for now, other than write cache types are there anything else that can be 
done to improve disk writing speeds. I checked RedHat guides on optimising VMs 
and I seem to have most in place, but write speeds remain at ~50 Mbps.
On 7/10/23 06:25, Jithin Raju wrote:

Hi  Groete,

The VM virtual NIC network throttling is picked up from its compute offering. 
You may need to create a new compute offering and change the VM’s compute 
offering. If it is not specified in the compute offering, means it is taking 
the values from the global settings: vm.network.throttling.rate .



-Jithin

From: Levin Ng
Date: Sunday, 9 July 2023 at 5:00 AM
To:users@cloudstack.apache.org  , Granwille 
Strauss
Cc:vivek.ku...@indiqus.com  , Nux
Subject: Re: Write Speeds
Dear Groete,

https://github.com/shapeblue/cloudstack/blob/965856057d5147f12b86abe5c9c205cdc5e44615/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/DirectVifDriver.java

https://libvirt.org/formatnetwork.html#quality-of-service

This is in kilobytes/second, u have to divide by 8

1Gbps / 8bit = 128MB = 128000KBps

You can verify by iperf test, and yes, u need to ensure both VR and VM match 
bandwidth settings to get a consistent result, something u also need to pay 
attention on VR resource, default system router resource offering is quite 
limited, the network speed may throttled if VR running are out of CPU resource.

Regards,
Levin



On 8 Jul 2023 at 09:02 +0100, Granwille Strauss, wrote:

Hi Levin
Thank you very much, I do appreciate your feedback and time replying back to 
me. I believe I have picked up on something. My VMs XML dump ALL show the 
following:

 


Specifically:
   
All my VM has

Re: Write Speeds


Hi Jorge

Thank you so much for this. I used your FIO config and surprisingly it 
seems fine:


write-test: (g=0): rw=randrw, bs=(R) 1300MiB-1300MiB, (W) 
1300MiB-1300MiB, (T) 1300MiB-1300MiB, ioengine=libaio, iodepth=1

fio-3.19

Run status group 0 (all jobs):
   READ: bw=962MiB/s (1009MB/s), 962MiB/s-962MiB/s 
(1009MB/s-1009MB/s), io=3900MiB (4089MB), run=4052-4052msec
  WRITE: bw=321MiB/s (336MB/s), 321MiB/s-321MiB/s (336MB/s-336MB/s), 
io=1300MiB (1363MB), run=4052-4052msec


This is without enabling io_uring. I see I can enable it per VM using 
the UI by setting the io.policy = io_uring. Will enable this on a few 
VMs and see if it works better.


On 7/10/23 15:41, Jorge Luiz Correa wrote:

Hi Granwille! About the READ/WRITE performance, as Levin suggested, check
the XML of virtual machines looking at disk/device section. Look for
io='io_uring'.

As stated here:

https://github.com/apache/cloudstack/issues/4883

CloudStack can use io_uring with Qemu >= 5.0 and Libvirt >= 6.3.0.

I tried to do some tests at some points like your environment.

##
VM in NFS Primary Storage (Hybrid NAS)
Default disk offering, thin (no restriction)


 /usr/bin/qemu-system-x86_64
 
   
   
   
   
   fb46fd2c59bd4127851b
   
   
 

fio --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test
--filename=/tmp/random_read_write.fio --bs=1300m --iodepth=8 --size=6G
--readwrite=randrw

READ: 569MiB/s
WRITE: 195MiB/s

##
VM in Local Primary Storage (local SSD host)
Default disk offering, thin (no restriction)


 /usr/bin/qemu-system-x86_64
 
   
   
   
   
   fb46fd2c59bd4127851b
   
   

fio --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test
--filename=/tmp/random_read_write.fio --bs=1300m --iodepth=8 --size=6G
--readwrite=randrw

First run (a little bit slow if using "thin" because need to allocate space
in qcow2):
READ: bw=796MiB/s
WRITE: bw=265MiB/s

Second run:
READ: bw=952MiB/s
WRITE: bw=317MiB/s

##
Directly in local SSD of host:

fio --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test
--filename=/tmp/random_read_write.fio --bs=1300m --iodepth=8 --size=6G
--readwrite=randrw

READ: bw=931MiB/s
WRITE: bw=310MiB/s

OBS.: parameters of fio test need to be changed to test in your environment
as it depends on the number of cpus, memory, --bs, --iodepth etc.

Host is running 5.15.0-43 kernel, qemu 6.2 and libvirt 8. CloudStack is
4.17.2. So, VM in local SSD of host could have very similar disk
performance from the host.

I hope this could help you!

Thanks.


--
Regards / Groete

<https://www.namhost.com> Granwille Strauss  // Senior Systems Admin

*e:* granwi...@namhost.com
*m:* +264 81 323 1260 
*w:* www.namhost.com <https://www.namhost.com/>

<https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>

<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>

Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA



The content of this message is confidential. If you have received it by 
mistake, please inform us by email reply and then delete the message. It 
is forbidden to copy, forward, or in any way reveal the contents of this 
message to anyone without our explicit consent. The integrity and 
security of this email cannot be guaranteed over the Internet. 
Therefore, the sender will not be held liable for any damage caused by 
the message. For our full privacy policy and disclaimers, please go to 
https://www.namhost.com/privacy-policy


Powered by AdSigner 
<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>

smime.p7s
Description: S/MIME Cryptographic Signature

Re: Async backup

Correct me if I am wrong, all snapshots make a local copy to the 
management server disk before sending it to the secondary storage 
locations, yes. Async or not. But yes, I believe that is the case. 
However, when async is enabled you get to use and work on other tasks 
with the management server, while it does the snapshot process in the 
background. Whereas having async disabled, you are restricted from doing 
other task on the management server until the snapshot process 
completes, that's how I understand it.


On 7/10/23 13:04, Nikolaos Tsinganos wrote:

Thank you, Granwille for your prompt answer.

Can you elaborate on the "locally"  and "storage location" terms?
On a host that has multiple primary storages (e.g. NFS and some other storage 
over iSCSI)  but no local disks are used, what is considered as 'locally'?
Also, by "storage location"  do you mean the secondary storage?

Regards,
Nikolaos

From: Granwille Strauss  
Sent: Monday, July 10, 2023 1:48 PM

To:users@cloudstack.apache.org
Cc: Nikolaos Tsinganos
Subject: Re: Async backup

In short and in layman's terms, it makes the volume snapshot, stores it locally 
and then in the background transfers it to the storage location. This helps 
with server resource usage. But see attached screenshot for a detailed answer.
On 7/10/23 12:38, Nikolaos Tsinganos wrote:
Hi All,

Can somebody explain what the "Async backup" option does while taking volume 
snapshot?

Regards,
Nikolaos


--
Regards / Groete

<https://www.namhost.com> Granwille Strauss  // Senior Systems Admin

*e:* granwi...@namhost.com
*m:* +264 81 323 1260 
*w:* www.namhost.com <https://www.namhost.com/>

<https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>

<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>

Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA



The content of this message is confidential. If you have received it by 
mistake, please inform us by email reply and then delete the message. It 
is forbidden to copy, forward, or in any way reveal the contents of this 
message to anyone without our explicit consent. The integrity and 
security of this email cannot be guaranteed over the Internet. 
Therefore, the sender will not be held liable for any damage caused by 
the message. For our full privacy policy and disclaimers, please go to 
https://www.namhost.com/privacy-policy


Powered by AdSigner 
<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>

smime.p7s
Description: S/MIME Cryptographic Signature

Re: Async backup

In short and in layman's terms, it makes the volume snapshot, stores it 
locally and then in the background transfers it to the storage location. 
This helps with server resource usage. But see attached screenshot for a 
detailed answer.


On 7/10/23 12:38, Nikolaos Tsinganos wrote:

Hi All,

Can somebody explain what the "Async backup" option does while taking volume 
snapshot?

Regards,
Nikolaos


--
Regards / Groete

<https://www.namhost.com>     Granwille Strauss  // Senior Systems Admin

*e:* granwi...@namhost.com
*m:* +264 81 323 1260 
*w:* www.namhost.com <https://www.namhost.com/>

<https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>

<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>

Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA



The content of this message is confidential. If you have received it by 
mistake, please inform us by email reply and then delete the message. It 
is forbidden to copy, forward, or in any way reveal the contents of this 
message to anyone without our explicit consent. The integrity and 
security of this email cannot be guaranteed over the Internet. 
Therefore, the sender will not be held liable for any damage caused by 
the message. For our full privacy policy and disclaimers, please go to 
https://www.namhost.com/privacy-policy


Powered by AdSigner 
<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>

smime.p7s
Description: S/MIME Cryptographic Signature

Re: Write Speeds


Hi Guys

Thank you, I have been running more tests now with the feedback you guys 
gave. Firstly, I want to break this up into two sections:


1. Network:

- So I have been running iperf tests between my VMs on their public 
network, and my iperf tests gives me speeds of ~500 Mbps, keep in mind 
this in between two local VMs on the same KVM but on public network.


- I then run iperf tests in and out from my local VMs to remote servers, 
this is where it does funny things. From the remote VM in USA, I run an 
iperf test to my local VM, the speeds show ~50 Mbps. And if I run a test 
from my local VM to a remote USA VM the same ~50 Mbps speeds are 
accomplished. I ran my iperf tests with 1 GB and 2GB flags and the 
results remain constant


- During all these test I kept an eye on my VR resources, which use 
default service offerings, it never spiked or reach thresholds.


Is it safe to assume that because of the MASSIVE distance between the 
remote VM and my local VMs, the speed dropping to ~50 Mbps is normal? 
Keep in mind the remote VM has 1 Gbps line too and this VM is managed by 
a big ISP provider in the USA. To me its quite a massive drop from 1000 
Mbps to 50 Mbps, this kinda does not make sense to me. I would 
understand at least 150 Mbps.


2. Disk Write Speed:

- It seems the only changes that can be made is to implement disk cache 
options. And so far I see write-back seems to be common practise for 
most cloud providers, given that they have the necessary power 
redundancy and VM backup images in place.


But for now, other than write cache types are there anything else that 
can be done to improve disk writing speeds. I checked RedHat guides on 
optimising VMs and I seem to have most in place, but write speeds remain 
at ~50 Mbps.


On 7/10/23 06:25, Jithin Raju wrote:

Hi  Groete,

The VM virtual NIC network throttling is picked up from its compute offering. 
You may need to create a new compute offering and change the VM’s compute 
offering. If it is not specified in the compute offering, means it is taking 
the values from the global settings: vm.network.throttling.rate .



-Jithin

From: Levin Ng
Date: Sunday, 9 July 2023 at 5:00 AM
To:users@cloudstack.apache.org  , Granwille 
Strauss
Cc:vivek.ku...@indiqus.com  , Nux
Subject: Re: Write Speeds
Dear Groete,

https://github.com/shapeblue/cloudstack/blob/965856057d5147f12b86abe5c9c205cdc5e44615/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/DirectVifDriver.java

https://libvirt.org/formatnetwork.html#quality-of-service

This is in kilobytes/second, u have to divide by 8

1Gbps / 8bit = 128MB = 128000KBps

You can verify by iperf test, and yes, u need to ensure both VR and VM match 
bandwidth settings to get a consistent result, something u also need to pay 
attention on VR resource, default system router resource offering is quite 
limited, the network speed may throttled if VR running are out of CPU resource.

Regards,
Levin



On 8 Jul 2023 at 09:02 +0100, Granwille Strauss, wrote:

Hi Levin
Thank you very much, I do appreciate your feedback and time replying back to 
me. I believe I have picked up on something. My VMs XML dump ALL show the 
following:

 


Specifically:
   
All my VM has this in place, a 128 Mbps limit even the VR, which makes no 
sense. I found this thread posted 10 years ago and Nux says that this value is 
affected by the service 
offerings:https://users.cloudstack.apache.narkive.com/T6Gx7BoV/cloudstack-network-limitation
  But all my service offerings are set to 1000 Mbps. See attached screenshots. 
The 4.18 documentation also confirms if the values are null, which most default 
service offerings are, it takes the values set in global settings 
network.throttling.rate and vm.network.throttling.rate, which I also have set 
as 1000 as you can see in the screenshots.

I then found 
this:https://cwiki.apache.org/confluence/display/CLOUDSTACK/Network+throttling+in+CloudStack
  with no KVM details, seems this part is missing to tell me how KVM throttling 
is applied. So as DuJun said 10 years ago, I feel confused about how Cloudstack 
limit the network rate for guest. And yes, I have stopped my VMs and rebooted 
MANY times it doesn't update the XML at all.
Please also take into account documentation states that in shared networking 
there's supposed to be no limits on incoming traffic (ingress), as far as I 
understand it.
On 7/7/23 23:35, Levin Ng wrote:

Hi Groete,


IMO, You should bypass any ACS provisioning to troubleshoot the performance 
case first, which allow you get more idea on the hardware + kvm performance 
with minimal influent, then you can compare the libvirt xml different between 
plain KVM and ACS. That help you sort out the different where it come from, you 
will see QoS bandwidth setting in the VM xml if you do.

We are trying to tell you, when you diagnose the throughput problem, you should 
first identify the bottleneck where it come first.  Iperf is a tools

Re: Write Speeds

Sorry that I have to ask, can you perhaps be a bit more specific,
please. The only QOS settings I see in service offering are "None",
"Hypervisor" and "Storage", which doesn't really seem network related.
Or am I missing the point? Take note that I use the default offerings
for the VR and VMs but with slight tweaks such as setting local storage
etc and only increased the Network rate from 200 Mbps to 1000 Mbps.

So can you kindly explain by what QOS settings you guys are referring
to, please?

PS, the Write-back disk caching seems to give the VM a slight increase,
I now see writes at 190 Mbps from ~70 Mbps.

On 7/7/23 21:11, Vivek Kumar wrote:

Hello,

IPerf will simply tell you the bandwidth and the open pipe between 2 VMs, so I
don’t think that it’s depends on disk performance, it’s better to check the
network QoS at every layer, VR and VM.

Vivek Kumar
Sr. Manager - Cloud & DevOps
TechOps | Indiqus Technologies

vivek.ku...@indiqus.com <mailto:vivek.ku...@indiqus.com>
www.indiqus.com <https://www.indiqus.com/>

On 07-Jul-2023, at 9:44 PM, Granwille Strauss
wrote:

Hi Levin

Thank you, I am aware of network offering, the first thing I did was make sure
it was set to accommodate the KVM's entire 1 Gbps uplink. But now that I think
if it iperf test prevousily were always stuck on 50 Mbps, but this is because
of the write speeds on the disk at least that's what I believe causes the
network bottle neck. I will double-check this again.

But there is some sort of limit on the VM disk in place. FIO tests show that
write speeds are in the range of 50 - 90 MB/s on the VM, while fio test
confirms on the KVM its over 400 MB/s.

On 7/7/23 18:08, Levin Ng wrote:

Hi Groete,

Forgot to mention, when you are talking about file copies between remote
server, you need to aware there are network QoS option in the offering, make
sure the limits correctness. Do iperf test prove that too, test between server
and via virtual router. Hope you can narrow down the problem soon.

Regards,
Levin

On 7 Jul 2023 at 16:40 +0100, Granwille Strauss
<mailto:granwi...@namhost.com>, wrote:

Hi Levin
Thank you, yes I leave IOPs empty. And the KVM host has SSDs in a hardware RAID
5 configuration, of which I am using local storage pool, yes. I will run fio
test and also playing around with the controller cache settings to see what
happens and provide feedback on this soon.
On 7/7/23 17:23, Levin Ng wrote:

HI Groete,

Should you run a fio test on the VM and the KVM host to get a baseline first.
SSD are tricky device, when it fill up the cache or nearly full, the
performance will drop significantly, especially consumer grade SSD. There are
option to limit IOPs in ACS offering setting, I believe you leave it empty, so
it is no limit. When you talking about KVM uses SSDs, I think you are using
Local Disk Pool right? If you have RAID controller underlying, try toggle the
controller cache, SSD may perform vary on different disk controller cache
setting.

Controller type scsi, or virtio performance are similar, no need to worry about
it. Of coz, in general, using RAW format and thick provisioning could get a
best io performance result, but consume space and lack of snapshot capabliblity
, so most the time it is not prefer go this path.

Please gather more information first

Regards,
Levin
On 7 Jul 2023 at 15:30 +0100, Granwille Strauss
<mailto:granwi...@namhost.com.invalid>, wrote:

Hi Guys
Does Cloudstack have a disk write speed limit somewhere in its setting? We have been transferring
many files from remote servers to VM machines on our Cloudstack instance and we recently noticed
that the VM write speeds are all limited to about 5-8 MB/s. But the underlying hardware of the KVM
uses SSDs capable of write speeds of 300 - 600 MB/s. My disk offering on my current vms are set to
"No Disk Cache" with thin provisioning, could this be the reason? I understand that
"Write Back Disk Cach" has better write speeds. Also I have VMs set as virtio for its
disk controller. What could I be missing in this case?
--
Regards / Groete

Granwille Strauss // Senior Systems Admin

e:granwi...@namhost.com <mailto:granwi...@namhost.com>
m: +264 81 323 1260
w:www.namhost.com <http://www.namhost.com/>

Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus, 7210, RSA

The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It is
forbidden to copy, forward, or in any way reveal the contents of this message
to anyone without our explicit consent. The integrity and security of this
email cannot be guaranteed over the Internet. Therefore, the sender will not be
held liable for any damage caused by the message. For our full privacy policy
and disclaimers, please go tohttps://www.namhost.com/privacy-policy

Re: Write Speeds

Hi Levin

Thank you, I am aware of network offering, the first thing I did was
make sure it was set to accommodate the KVM's entire 1 Gbps uplink. But
now that I think if it iperf test prevousily were always stuck on 50
Mbps, but this is because of the write speeds on the disk at least
that's what I believe causes the network bottle neck. I will
double-check this again.

But there is some sort of limit on the VM disk in place. FIO tests show
that write speeds are in the range of 50 - 90 MB/s on the VM, while fio
test confirms on the KVM its over 400 MB/s.

On 7/7/23 18:08, Levin Ng wrote:

Hi Groete,

Regards,
Levin

On 7 Jul 2023 at 16:40 +0100, Granwille Strauss, wrote:

HI Groete,

Please gather more information first

Regards,
Levin
On 7 Jul 2023 at 15:30 +0100, Granwille Strauss,
wrote:

Granwille Strauss // Senior Systems Admin

e:granwi...@namhost.com
m: +264 81 323 1260
w:www.namhost.com

Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus, 7210, RSA

The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It is
forbidden to copy, forward, or in any way reveal the contents of this message
to anyone without our explicit consent. The integrity and security of this
email cannot be guaranteed over the Internet. Therefore, the sender will not be
held liable for any damage caused by the message. For our full privacy policy
and disclaimers, please go tohttps://www.namhost.com/privacy-policy

--
Regards / Groete

Granwille Strauss // Senior Systems Admin

e:granwi...@namhost.com
m: +264 81 323 1260
w:www.namhost.com

Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus, 7210, RSA

The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It is
forbidden to copy, forward, or in any way reveal the contents of this message
to anyone without our explicit consent. The integrity and security of this
email cannot be guaranteed over the Internet. Therefore, the sender will not be
held liable for any damage caused by the message. For our full privacy policy
and disclaimers, please go tohttps://www.namhost.com/privacy-policy

--
Regards / Groete

<https://www.namhost.com> Granwille Strauss // Senior Systems Admin

*e:* granwi...@namhost.com
*m:* +264 81 323 1260
*w:* www.namhost.com <https://www.namhost.com/>

<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>

Namhost Internet Servic

Re: Write Speeds

Hi Levin

Thank you, yes I leave IOPs empty. And the KVM host has SSDs in a
hardware RAID 5 configuration, of which I am using local storage pool,
yes. I will run fio test and also playing around with the controller
cache settings to see what happens and provide feedback on this soon.

On 7/7/23 17:23, Levin Ng wrote:

HI Groete,

Please gather more information first

Regards,
Levin
On 7 Jul 2023 at 15:30 +0100, Granwille Strauss,
wrote:

Granwille Strauss // Senior Systems Admin

e:granwi...@namhost.com
m: +264 81 323 1260
w:www.namhost.com

Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus, 7210, RSA

The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It is
forbidden to copy, forward, or in any way reveal the contents of this message
to anyone without our explicit consent. The integrity and security of this
email cannot be guaranteed over the Internet. Therefore, the sender will not be
held liable for any damage caused by the message. For our full privacy policy
and disclaimers, please go tohttps://www.namhost.com/privacy-policy

--
Regards / Groete

<https://www.namhost.com> Granwille Strauss // Senior Systems Admin

*e:* granwi...@namhost.com
*m:* +264 81 323 1260
*w:* www.namhost.com <https://www.namhost.com/>

<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>

Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA

The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It
is forbidden to copy, forward, or in any way reveal the contents of this
message to anyone without our explicit consent. The integrity and
security of this email cannot be guaranteed over the Internet.
Therefore, the sender will not be held liable for any damage caused by
the message. For our full privacy policy and disclaimers, please go to
https://www.namhost.com/privacy-policy

smime.p7s
Description: S/MIME Cryptographic Signature

Write Speeds

Hi Guys

Does Cloudstack have a disk write speed limit somewhere in its setting?
We have been transferring many files from remote servers to VM machines
on our Cloudstack instance and we recently noticed that the VM write
speeds are all limited to about 5-8 MB/s. But the underlying hardware of
the KVM uses SSDs capable of write speeds of 300 - 600 MB/s. My disk
offering on my current vms are set to "No Disk Cache" with thin
provisioning, could this be the reason? I understand that "Write Back
Disk Cach" has better write speeds. Also I have VMs set as virtio for
its disk controller. What could I be missing in this case?

--
Regards / Groete

<https://www.namhost.com> Granwille Strauss // Senior Systems Admin

*e:* granwi...@namhost.com
*m:* +264 81 323 1260
*w:* www.namhost.com <https://www.namhost.com/>

<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>

Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA

The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It
is forbidden to copy, forward, or in any way reveal the contents of this
message to anyone without our explicit consent. The integrity and
security of this email cannot be guaranteed over the Internet.
Therefore, the sender will not be held liable for any damage caused by
the message. For our full privacy policy and disclaimers, please go to
https://www.namhost.com/privacy-policy

smime.p7s
Description: S/MIME Cryptographic Signature

Re: Volume Backups Failing whether VM is in running or stopped status


Solved itself - https://github.com/apache/cloudstack/issues/7678

On 6/23/23 09:50, Granwille Strauss wrote:


Hi Wei

I am fairly certain all directories are in root in any case, as that's 
how the entire installation done, with the root user. For what its 
worth, the volume snapshots did work perfectly fine before in 4.17 so 
I do not think permissions is the issue. Today was my first time 
manually trying to make one on 4.18 since the upgrade and now this 
happens. PS VM Snapshots seems to work fine. Its only Volume snaps 
that are failing due to this error.


On 6/23/23 09:33, Granwille Strauss wrote:


Hi Wei

Are you referring to the `/var/lib/libvirt/images` and directory on 
KVM? If yes, its root:root I also check secondary storage directories 
and they are also root:root. Or can you confirm which directories 
you're referring to? Or can you confirm what directories MUST have 
the cloud:cloud owner so that I can verify, please.


On 6/23/23 09:25, Wei ZHOU wrote:

Hi,

Can you check the owner of the directories ? cloudstack uses cloud:cloud
user.

-Wei

On Fri, 23 Jun 2023 at 08:49, Granwille Strauss
  wrote:


Hi Guys

So sorry for the many problems I am bring up. I am now trying to make
volume backups of my VMs in either running or stopped state. This is the
following error thrown in the management log:

2023-06-23 08:32:24,955 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-20:ctx-8f5395ec job-4099/job-4100 ctx-23f04f9f)
(logid:931f0d86) Seq 36-8282682664687772956: Sending  { Cmd , MgmtId:
66988330791812, via: 36(hostname), Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"eec8285e-8715-441b-b418-b71b231f1bab","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"85e49c98-0988-4627-984c-8f4719c401e3","name":"hostname-local-85e49c98","id":"24","poolType":"Filesystem","host":"192.168.50.7","path":"/var/lib/libvirt/images","port":"0","url":"Filesystem://
192.168.50.7/var/lib/libvirt/images/?ROLE=Primary=85e49c98-0988-4627-984c-8f4719c401e3","isManaged":"false"}},"name":"ROOT-120","size":"(60,00
GB)
64424509440","path":"f801a1a4-83aa-4490-a589-c2a474ec7bed","volumeId":"193","vmName":"i-2-120-VM","accountId":"2","format":"QCOW2","provisioningType":"THIN","poolId":"24","id":"193","deviceId":"0","cacheMode":"NONE","hypervisorType":"KVM","directDownload":"false","deployAsIs":"false"},"dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"85e49c98-0988-4627-984c-8f4719c401e3","name":"hostname-local-85e49c98","id":"24","poolType":"Filesystem","host":"192.168.50.7","path":"/var/lib/libvirt/images","port":"0","url":"Filesystem://
192.168.50.7/var/lib/libvirt/images/?ROLE=Primary=85e49c98-0988-4627-984c-8f4719c401e3","isManaged":"false"}},"vmName":"i-2-120-VM","name":"NACDO-Backup-23-June-2023","hypervisorType":"KVM","id":"177","quiescevm":"false","physicalSize":"0"}},"wait":"0","bypassHostMaintenance":"false"}}]
}
2023-06-23 08:32:25,017 DEBUG [c.c.a.t.Request]
(AgentManager-Handler-12:null) (logid:) Seq 36-8282682664687772956:
Processing:  { Ans: , MgmtId: 66988330791812, via: 36, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.Answer":{"result":"false","details":"java.lang.NoClassDefFoundError:
Could not initialize class sun.nio.fs.UnixCopyFile
 at
java.base/sun.nio.fs.UnixFileSystemProvider.copy(UnixFileSystemProvider.java:258)
 at java.base/java.nio.file.Files.copy(Files.java:1295)
 at
com.cloud.hypervisor.kvm.storage.KVMStorageProcessor.copySnapshotToPrimaryStorageDir(KVMStorageProcessor.java:1916)
 at
com.cloud.hypervisor.kvm.storage.KVMStorageProcessor.createSnapshot(KVMStorageProcessor.java:1790)
 at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:137)
 at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:65)

Re: Volume Backups Failing whether VM is in running or stopped status


Hi Wei

I am fairly certain all directories are in root in any case, as that's 
how the entire installation done, with the root user. For what its 
worth, the volume snapshots did work perfectly fine before in 4.17 so I 
do not think permissions is the issue. Today was my first time manually 
trying to make one on 4.18 since the upgrade and now this happens. PS VM 
Snapshots seems to work fine. Its only Volume snaps that are failing due 
to this error.


On 6/23/23 09:33, Granwille Strauss wrote:


Hi Wei

Are you referring to the `/var/lib/libvirt/images` and directory on 
KVM? If yes, its root:root I also check secondary storage directories 
and they are also root:root. Or can you confirm which directories 
you're referring to? Or can you confirm what directories MUST have the 
cloud:cloud owner so that I can verify, please.


On 6/23/23 09:25, Wei ZHOU wrote:

Hi,

Can you check the owner of the directories ? cloudstack uses cloud:cloud
user.

-Wei

On Fri, 23 Jun 2023 at 08:49, Granwille Strauss
  wrote:


Hi Guys

So sorry for the many problems I am bring up. I am now trying to make
volume backups of my VMs in either running or stopped state. This is the
following error thrown in the management log:

2023-06-23 08:32:24,955 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-20:ctx-8f5395ec job-4099/job-4100 ctx-23f04f9f)
(logid:931f0d86) Seq 36-8282682664687772956: Sending  { Cmd , MgmtId:
66988330791812, via: 36(hostname), Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"eec8285e-8715-441b-b418-b71b231f1bab","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"85e49c98-0988-4627-984c-8f4719c401e3","name":"hostname-local-85e49c98","id":"24","poolType":"Filesystem","host":"192.168.50.7","path":"/var/lib/libvirt/images","port":"0","url":"Filesystem://
192.168.50.7/var/lib/libvirt/images/?ROLE=Primary=85e49c98-0988-4627-984c-8f4719c401e3","isManaged":"false"}},"name":"ROOT-120","size":"(60,00
GB)
64424509440","path":"f801a1a4-83aa-4490-a589-c2a474ec7bed","volumeId":"193","vmName":"i-2-120-VM","accountId":"2","format":"QCOW2","provisioningType":"THIN","poolId":"24","id":"193","deviceId":"0","cacheMode":"NONE","hypervisorType":"KVM","directDownload":"false","deployAsIs":"false"},"dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"85e49c98-0988-4627-984c-8f4719c401e3","name":"hostname-local-85e49c98","id":"24","poolType":"Filesystem","host":"192.168.50.7","path":"/var/lib/libvirt/images","port":"0","url":"Filesystem://
192.168.50.7/var/lib/libvirt/images/?ROLE=Primary=85e49c98-0988-4627-984c-8f4719c401e3","isManaged":"false"}},"vmName":"i-2-120-VM","name":"NACDO-Backup-23-June-2023","hypervisorType":"KVM","id":"177","quiescevm":"false","physicalSize":"0"}},"wait":"0","bypassHostMaintenance":"false"}}]
}
2023-06-23 08:32:25,017 DEBUG [c.c.a.t.Request]
(AgentManager-Handler-12:null) (logid:) Seq 36-8282682664687772956:
Processing:  { Ans: , MgmtId: 66988330791812, via: 36, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.Answer":{"result":"false","details":"java.lang.NoClassDefFoundError:
Could not initialize class sun.nio.fs.UnixCopyFile
 at
java.base/sun.nio.fs.UnixFileSystemProvider.copy(UnixFileSystemProvider.java:258)
 at java.base/java.nio.file.Files.copy(Files.java:1295)
 at
com.cloud.hypervisor.kvm.storage.KVMStorageProcessor.copySnapshotToPrimaryStorageDir(KVMStorageProcessor.java:1916)
 at
com.cloud.hypervisor.kvm.storage.KVMStorageProcessor.createSnapshot(KVMStorageProcessor.java:1790)
 at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:137)
 at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:65)
 at
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStorageSubSystemCommandWrapper.execute(LibvirtStor

Re: Volume Backups Failing whether VM is in running or stopped status


Hi Wei

Are you referring to the `/var/lib/libvirt/images` and directory on KVM? 
If yes, its root:root I also check secondary storage directories and 
they are also root:root. Or can you confirm which directories you're 
referring to? Or can you confirm what directories MUST have the 
cloud:cloud owner so that I can verify, please.


On 6/23/23 09:25, Wei ZHOU wrote:

Hi,

Can you check the owner of the directories ? cloudstack uses cloud:cloud
user.

-Wei

On Fri, 23 Jun 2023 at 08:49, Granwille Strauss
  wrote:


Hi Guys

So sorry for the many problems I am bring up. I am now trying to make
volume backups of my VMs in either running or stopped state. This is the
following error thrown in the management log:

2023-06-23 08:32:24,955 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-20:ctx-8f5395ec job-4099/job-4100 ctx-23f04f9f)
(logid:931f0d86) Seq 36-8282682664687772956: Sending  { Cmd , MgmtId:
66988330791812, via: 36(hostname), Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"eec8285e-8715-441b-b418-b71b231f1bab","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"85e49c98-0988-4627-984c-8f4719c401e3","name":"hostname-local-85e49c98","id":"24","poolType":"Filesystem","host":"192.168.50.7","path":"/var/lib/libvirt/images","port":"0","url":"Filesystem://
192.168.50.7/var/lib/libvirt/images/?ROLE=Primary=85e49c98-0988-4627-984c-8f4719c401e3","isManaged":"false"}},"name":"ROOT-120","size":"(60,00
GB)
64424509440","path":"f801a1a4-83aa-4490-a589-c2a474ec7bed","volumeId":"193","vmName":"i-2-120-VM","accountId":"2","format":"QCOW2","provisioningType":"THIN","poolId":"24","id":"193","deviceId":"0","cacheMode":"NONE","hypervisorType":"KVM","directDownload":"false","deployAsIs":"false"},"dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"85e49c98-0988-4627-984c-8f4719c401e3","name":"hostname-local-85e49c98","id":"24","poolType":"Filesystem","host":"192.168.50.7","path":"/var/lib/libvirt/images","port":"0","url":"Filesystem://
192.168.50.7/var/lib/libvirt/images/?ROLE=Primary=85e49c98-0988-4627-984c-8f4719c401e3","isManaged":"false"}},"vmName":"i-2-120-VM","name":"NACDO-Backup-23-June-2023","hypervisorType":"KVM","id":"177","quiescevm":"false","physicalSize":"0"}},"wait":"0","bypassHostMaintenance":"false"}}]
}
2023-06-23 08:32:25,017 DEBUG [c.c.a.t.Request]
(AgentManager-Handler-12:null) (logid:) Seq 36-8282682664687772956:
Processing:  { Ans: , MgmtId: 66988330791812, via: 36, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.Answer":{"result":"false","details":"java.lang.NoClassDefFoundError:
Could not initialize class sun.nio.fs.UnixCopyFile
 at
java.base/sun.nio.fs.UnixFileSystemProvider.copy(UnixFileSystemProvider.java:258)
 at java.base/java.nio.file.Files.copy(Files.java:1295)
 at
com.cloud.hypervisor.kvm.storage.KVMStorageProcessor.copySnapshotToPrimaryStorageDir(KVMStorageProcessor.java:1916)
 at
com.cloud.hypervisor.kvm.storage.KVMStorageProcessor.createSnapshot(KVMStorageProcessor.java:1790)
 at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:137)
 at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:65)
 at
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStorageSubSystemCommandWrapper.execute(LibvirtStorageSubSystemCommandWrapper.java:36)
 at
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStorageSubSystemCommandWrapper.execute(LibvirtStorageSubSystemCommandWrapper.java:30)
 at
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
 at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1847)
 at com.cloud.agent.Agent.processRequest(Agent.java:662)
 at com.cloud.agent.Agent$AgentR

Volume Backups Failing whether VM is in running or stopped status


Hi Guys

So sorry for the many problems I am bring up. I am now trying to make 
volume backups of my VMs in either running or stopped state. This is the 
following error thrown in the management log:


2023-06-23 08:32:24,955 DEBUG [c.c.a.t.Request] 
(Work-Job-Executor-20:ctx-8f5395ec job-4099/job-4100 ctx-23f04f9f) 
(logid:931f0d86) Seq 36-8282682664687772956: Sending  { Cmd , MgmtId: 
66988330791812, via: 36(hostname), Ver: v1, Flags: 100011, 
[{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"eec8285e-8715-441b-b418-b71b231f1bab","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"85e49c98-0988-4627-984c-8f4719c401e3","name":"hostname-local-85e49c98","id":"24","poolType":"Filesystem","host":"192.168.50.7","path":"/var/lib/libvirt/images","port":"0","url":"Filesystem://192.168.50.7/var/lib/libvirt/images/?ROLE=Primary=85e49c98-0988-4627-984c-8f4719c401e3","isManaged":"false"}},"name":"ROOT-120","size":"(60,00 
GB) 
64424509440","path":"f801a1a4-83aa-4490-a589-c2a474ec7bed","volumeId":"193","vmName":"i-2-120-VM","accountId":"2","format":"QCOW2","provisioningType":"THIN","poolId":"24","id":"193","deviceId":"0","cacheMode":"NONE","hypervisorType":"KVM","directDownload":"false","deployAsIs":"false"},"dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"85e49c98-0988-4627-984c-8f4719c401e3","name":"hostname-local-85e49c98","id":"24","poolType":"Filesystem","host":"192.168.50.7","path":"/var/lib/libvirt/images","port":"0","url":"Filesystem://192.168.50.7/var/lib/libvirt/images/?ROLE=Primary=85e49c98-0988-4627-984c-8f4719c401e3","isManaged":"false"}},"vmName":"i-2-120-VM","name":"NACDO-Backup-23-June-2023","hypervisorType":"KVM","id":"177","quiescevm":"false","physicalSize":"0"}},"wait":"0","bypassHostMaintenance":"false"}}] 
}
2023-06-23 08:32:25,017 DEBUG [c.c.a.t.Request] 
(AgentManager-Handler-12:null) (logid:) Seq 36-8282682664687772956: 
Processing:  { Ans: , MgmtId: 66988330791812, via: 36, Ver: v1, Flags: 
10, 
[{"com.cloud.agent.api.Answer":{"result":"false","details":"java.lang.NoClassDefFoundError: 
Could not initialize class sun.nio.fs.UnixCopyFile
    at 
java.base/sun.nio.fs.UnixFileSystemProvider.copy(UnixFileSystemProvider.java:258)

    at java.base/java.nio.file.Files.copy(Files.java:1295)
    at 
com.cloud.hypervisor.kvm.storage.KVMStorageProcessor.copySnapshotToPrimaryStorageDir(KVMStorageProcessor.java:1916)
    at 
com.cloud.hypervisor.kvm.storage.KVMStorageProcessor.createSnapshot(KVMStorageProcessor.java:1790)
    at 
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:137)
    at 
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:65)
    at 
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStorageSubSystemCommandWrapper.execute(LibvirtStorageSubSystemCommandWrapper.java:36)
    at 
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStorageSubSystemCommandWrapper.execute(LibvirtStorageSubSystemCommandWrapper.java:30)
    at 
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
    at 
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1847)

    at com.cloud.agent.Agent.processRequest(Agent.java:662)
    at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1082)
    at com.cloud.utils.nio.Task.call(Task.java:83)
    at com.cloud.utils.nio.Task.call(Task.java:29)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at 
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

    at java.base/java.lang.Thread.run(Thread.java:829)
","wait":"0",&qu

Re: Moving Multiple Shared IP ranges from one Guest network

2023-06-22 Thread Granwille Strauss


Hi Jithin

Yes, I understand that. This is an add-on question/problem if I do 
proceed with making these changes. In short, by creating a shared 
network for each IP range, once that range is fully used, you can no 
longer assign secondary IPs to VMs on the same NIC. So that would mean I 
need to add a new range to an existing shared network, just so that I 
can accomplish adding secondary IPs to a VM. Kinda defeating the 
purpose. Or is there a way around this.


On 6/22/23 10:42, Jithin Raju wrote:


Hi Granwille,

The steps I suggested doesn’t involve using secondary IPs, instead, 
there is another option change IP Address (CS 4.18) .  API: updateVmNicIp


Once the IP ranges are freed up and you have created new networks you 
need to add a new network to the VM ( I believe this is what you are 
looking for).


A screenshot of a computer Description automatically generated with 
low confidence


-Jithin


*From: *Granwille Strauss 
*Date: *Thursday, 22 June 2023 at 1:54 PM
*To: *users@cloudstack.apache.org 
*Cc: *Jithin Raju 
*Subject: *Re: Moving Multiple Shared IP ranges from one Guest network

Jithin,

Awesome thank you. Just one more question, I noticed a flaw or 
probably a lack of my understanding. When I have completed this 
process, and lets assume I have now maxed out all IPs on one shared 
network, how do I assign a secondary IP to a VM, but on the same NIC?


With the current setup I have, because I have more than one range in 
my existing network, I have the option to assign secondary IPs to an 
existing NIC for a VM. It seems by following the advice I was given 
this does not seem to be possible any more when my range is maxed on 
one shared network. Is there a way around this?


I created two new networks with each their own range now. But on my 
existing VMs, I do not see the option to add an IP address as a 
secondary IP to the existing NIC, because its on a different network. 
And for me to use the new network I have to assign an additional NIC, 
which is not really what I would want on a VPS.


On 6/22/23 10:12, Jithin Raju wrote:

Granwille,

Your understanding is correct.

If you want to assign the same IPs back to the original VMs you
need to add the new network as additional NIC to the same VMs then
change the new network to default and remove the old one.

-Jithin


*From: *Granwille Strauss 
<mailto:granwi...@namhost.com>
*Date: *Thursday, 22 June 2023 at 12:28 PM
*To: *users@cloudstack.apache.org 
<mailto:users@cloudstack.apache.org>
*Cc: *Jithin Raju 
<mailto:jithin.r...@shapeblue.com>
*Subject: *Re: Moving Multiple Shared IP ranges from one Guest network

Hi Jithin

Thank you for getting back to me. Sorry, that came out wrong I
make use of Advanced and not basic zone. That said, I am not sure
I understand the following:

Identify the IP range VR uses that needs to be retained in the network

Change IP to the one range that you want to retain.


Since my network has three subnets that are being used by VMs and
one VR, do I understand correctly that I need to leave one range
in place, the range that is tied to the VRs public IP, yes? And
then, for the other two ranges, after I have stopped VMs and
changed their IPs I can remove those two ranges and create two new
networks for each range. Will I then be able to assign the old IPs
back to the VM afterwards?

On 6/22/23 08:46, Jithin Raju wrote:

Hi Granwille,

  


So you are using a basic zone and you have multiple subnets for the 
‘defaultGuestNetwork’ in the same pod?

You want to create new guest networks right, but that is not possible 
in a basic zone. You could add a new pod and add the new subnets there but 
within the same guest network.

  


If you are using an advanced zone with shared network you can follow 
the below steps:

  


Stop VM

Identify the IP range VR uses that needs to be retained in the network

Change IP to the one range that you want to retain.

     Goto VM->NICs -> change IP

  


Once you change the IPs for all the VMs you should be able to delete 
the range from the existing network and create a new one.

  


-Jithin

      


    From: Granwille Strauss  
<mailto:granwi...@namhost.com.INVALID>

Date: Wednesday, 21 June 2023 at 9:47 PM

To:users@cloudstack.apache.org
<mailto:users@cloudstack.apache.org>

Subject: Moving Multiple Shared IP ranges from one Guest network

  


Hi Guys

  


I have basic private cloud setup. I have one guest network that 
contains three shared public IP subnet ranges consisting of /28 and /29 ranges. 
I have been recently advised that it would be better to make a guest network 
for each subnet range instead

Re: Moving Multiple Shared IP ranges from one Guest network

2023-06-22 Thread Granwille Strauss


Jithin,

Awesome thank you. Just one more question, I noticed a flaw or probably 
a lack of my understanding. When I have completed this process, and lets 
assume I have now maxed out all IPs on one shared network, how do I 
assign a secondary IP to a VM, but on the same NIC?


With the current setup I have, because I have more than one range in my 
existing network, I have the option to assign secondary IPs to an 
existing NIC for a VM. It seems by following the advice I was given this 
does not seem to be possible any more when my range is maxed on one 
shared network. Is there a way around this?


I created two new networks with each their own range now. But on my 
existing VMs, I do not see the option to add an IP address as a 
secondary IP to the existing NIC, because its on a different network. 
And for me to use the new network I have to assign an additional NIC, 
which is not really what I would want on a VPS.


On 6/22/23 10:12, Jithin Raju wrote:


Granwille,

Your understanding is correct.

If you want to assign the same IPs back to the original VMs you need 
to add the new network as additional NIC to the same VMs then change 
the new network to default and remove the old one.


-Jithin


*From: *Granwille Strauss 
*Date: *Thursday, 22 June 2023 at 12:28 PM
*To: *users@cloudstack.apache.org 
*Cc: *Jithin Raju 
*Subject: *Re: Moving Multiple Shared IP ranges from one Guest network

Hi Jithin

Thank you for getting back to me. Sorry, that came out wrong I make 
use of Advanced and not basic zone. That said, I am not sure I 
understand the following:


Identify the IP range VR uses that needs to be retained in the network

Change IP to the one range that you want to retain.


Since my network has three subnets that are being used by VMs and one 
VR, do I understand correctly that I need to leave one range in place, 
the range that is tied to the VRs public IP, yes? And then, for the 
other two ranges, after I have stopped VMs and changed their IPs I can 
remove those two ranges and create two new networks for each range. 
Will I then be able to assign the old IPs back to the VM afterwards?


On 6/22/23 08:46, Jithin Raju wrote:

Hi Granwille,

So you are using a basic zone and you have multiple subnets for the 
‘defaultGuestNetwork’ in the same pod?

You want to create new guest networks right, but that is not possible in a 
basic zone. You could add a new pod and add the new subnets there but within 
the same guest network.

If you are using an advanced zone with shared network you can follow the 
below steps:

Stop VM

Identify the IP range VR uses that needs to be retained in the network

Change IP to the one range that you want to retain.

     Goto VM->NICs -> change IP

Once you change the IPs for all the VMs you should be able to delete the 
range from the existing network and create a new one.

-Jithin

From: Granwille Strauss  
<mailto:granwi...@namhost.com.INVALID>

Date: Wednesday, 21 June 2023 at 9:47 PM

To:users@cloudstack.apache.org
<mailto:users@cloudstack.apache.org>

Subject: Moving Multiple Shared IP ranges from one Guest network

Hi Guys

I have basic private cloud setup. I have one guest network that contains 
three shared public IP subnet ranges consisting of /28 and /29 ranges. I have 
been recently advised that it would be better to make a guest network for each 
subnet range instead of having them all under one network.

The issue is, I already have VMs assigned to those public IP ranges and 
they're currently running. What would be the best way to solve this? Can I just 
remove the ranges from the existing network and quickly add a new network with 
those ranges and have a new VR boot up? Would it then continue to run on the 
existing VMs? I reckon I need to have the VMs stopped first in any case, yes? 
Also, I have ZERO API experience so this needs to happen all via UI.

--

Regards / Groete

[Image removed by sender.]<https://www.namhost.com/>  
<https://www.namhost.com/>

Granwille Strauss  //  Senior Systems Admin

e:granwi...@namhost.com<mailto:granwi...@namhost.com>  
<mailto:granwi...@namhost.com>

m: +264 81 323 1260  

w:www.namhost.com  <http://www.namhost.com><https://www.namhost.com/>  
<https://www.namhost.com/>

[Image removed by sender.]<https://www.facebook.com/namhost>  <https://www.facebook.com/namhost>[Image removed by 
sender.]<https://twitter.com/namhost>  <https://twitter.com/namhost>[Image removed by 
sender.]<https://www.instagram.com/namhostinternetservices/>  <https://www.instagram.com/namhostinternetservices/>[Image 
removed by sender.]<https://www.linkedin.com/company/namhos>  <https://www.linkedin.com/company/namhos>[Image removed by 
sender.]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>  <h

Re: Moving Multiple Shared IP ranges from one Guest network

2023-06-22 Thread Granwille Strauss


Hi Jithin

Thank you for getting back to me. Sorry, that came out wrong I make use 
of Advanced and not basic zone. That said, I am not sure I understand 
the following:



Identify the IP range VR uses that needs to be retained in the network
Change IP to the one range that you want to retain.


Since my network has three subnets that are being used by VMs and one 
VR, do I understand correctly that I need to leave one range in place, 
the range that is tied to the VRs public IP, yes? And then, for the 
other two ranges, after I have stopped VMs and changed their IPs I can 
remove those two ranges and create two new networks for each range. Will 
I then be able to assign the old IPs back to the VM afterwards?


On 6/22/23 08:46, Jithin Raju wrote:

Hi Granwille,

So you are using a basic zone and you have multiple subnets for the 
‘defaultGuestNetwork’ in the same pod?
You want to create new guest networks right, but that is not possible in a 
basic zone. You could add a new pod and add the new subnets there but within 
the same guest network.

If you are using an advanced zone with shared network you can follow the below 
steps:

Stop VM
Identify the IP range VR uses that needs to be retained in the network
Change IP to the one range that you want to retain.
 Goto VM->NICs -> change IP

Once you change the IPs for all the VMs you should be able to delete the range 
from the existing network and create a new one.

-Jithin

From: Granwille Strauss
Date: Wednesday, 21 June 2023 at 9:47 PM
To:users@cloudstack.apache.org  
Subject: Moving Multiple Shared IP ranges from one Guest network

Hi Guys

I have basic private cloud setup. I have one guest network that contains three 
shared public IP subnet ranges consisting of /28 and /29 ranges. I have been 
recently advised that it would be better to make a guest network for each 
subnet range instead of having them all under one network.

The issue is, I already have VMs assigned to those public IP ranges and they're 
currently running. What would be the best way to solve this? Can I just remove 
the ranges from the existing network and quickly add a new network with those 
ranges and have a new VR boot up? Would it then continue to run on the existing 
VMs? I reckon I need to have the VMs stopped first in any case, yes? Also, I 
have ZERO API experience so this needs to happen all via UI.
--

Regards / Groete
[Image removed by sender.]<https://www.namhost.com/>
Granwille Strauss  //  Senior Systems Admin

e:granwi...@namhost.com<mailto:granwi...@namhost.com>
m: +264 81 323 1260
w:www.namhost.com<https://www.namhost.com/>

[Image removed by sender.]<https://www.facebook.com/namhost>[Image removed by 
sender.]<https://twitter.com/namhost>[Image removed by 
sender.]<https://www.instagram.com/namhostinternetservices/>[Image removed by 
sender.]<https://www.linkedin.com/company/namhos>[Image removed by 
sender.]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>

[Image removed by 
sender.]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA

The content of this message is confidential. If you have received it by 
mistake, please inform us by email reply and then delete the message. It is 
forbidden to copy, forward, or in any way reveal the contents of this message 
to anyone without our explicit consent. The integrity and security of this 
email cannot be guaranteed over the Internet. Therefore, the sender will not be 
held liable for any damage caused by the message. For our full privacy policy 
and disclaimers, please go tohttps://www.namhost.com/privacy-policy

[Image removed by sender. Powered by 
AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>

  




--
Regards / Groete

<https://www.namhost.com> Granwille Strauss  // Senior Systems Admin

*e:* granwi...@namhost.com
*m:* +264 81 323 1260 
*w:* www.namhost.com <https://www.namhost.com/>

<https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>

<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>

Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA



The content of this message is confidential. If you have received it by 
mistake, please inform us by email reply and then delete the message. It 
is forbidden to copy, forward, or in any way reveal the contents of this 
message to anyone without our explicit consent. The integrity and 
security of this email cannot be guaranteed over the Internet. 
Therefore, the sender will not be held liable for any damage caused by 
the message. For our full

Moving Multiple Shared IP ranges from one Guest network

2023-06-21 Thread Granwille Strauss


Hi Guys

I have basic private cloud setup. I have one guest network that contains 
three shared public IP subnet ranges consisting of /28 and /29 ranges. I 
have been recently advised that it would be better to make a guest 
network for each subnet range instead of having them all under one network.


The issue is, I already have VMs assigned to those public IP ranges and 
they're currently running. What would be the best way to solve this? Can 
I just remove the ranges from the existing network and quickly add a new 
network with those ranges and have a new VR boot up? Would it then 
continue to run on the existing VMs? I reckon I need to have the VMs 
stopped first in any case, yes? Also, I have ZERO API experience so this 
needs to happen all via UI.


--
Regards / Groete

<https://www.namhost.com>     Granwille Strauss  // Senior Systems Admin

*e:* granwi...@namhost.com
*m:* +264 81 323 1260 
*w:* www.namhost.com <https://www.namhost.com/>

<https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>

<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>

Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA



The content of this message is confidential. If you have received it by 
mistake, please inform us by email reply and then delete the message. It 
is forbidden to copy, forward, or in any way reveal the contents of this 
message to anyone without our explicit consent. The integrity and 
security of this email cannot be guaranteed over the Internet. 
Therefore, the sender will not be held liable for any damage caused by 
the message. For our full privacy policy and disclaimers, please go to 
https://www.namhost.com/privacy-policy


Powered by AdSigner 
<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>

smime.p7s
Description: S/MIME Cryptographic Signature

Re: Virtual Router Constantly Fails to Update /etc/hosts file

2023-06-20 Thread Granwille Strauss

Hi Guys

I logged a bug on GitHub for this error:
https://github.com/apache/cloudstack/issues/7653

In the meantime, I am now stuck where I now need to manually put entries
in for any newly created VM. I want to confirm, if I live patch the
router, is that an OS re-install process? As in would this most likely
solve the issue for now? Or can I destroy the virtual router? Will cloud
stack automatically create a new one, similar to how SystemVM and
ConsoleVM recreates itself when destroyed?

On 6/15/23 08:26, Granwille Strauss wrote:

Hi Guys

I have a very basic private cloud setup, we make use of a a virtual
router in Cloudstack Management, for some reason, each time we restart
a VM instance via UI, our router keeps failing a health check by
having missing entries in the /etc/hosts file. To solve this we
manually need to access the virtual router and manually add the
entries for the health checks to pass.

This started to happen when we created a new VM instance but after a
few hours, terminated the VM and expunged it. And then re-created the
VM again, using the exact same name as before. This somehow causes
this behaviour with the virtual router. Is there any way to solve
this? I have rebooted the router and still doesn't work, only manually
entering the values fixes it, until we need to reboot a VM again or
something.

--
Regards / Groete

<https://www.namhost.com> Granwille Strauss // Senior Systems Admin

*e:* granwi...@namhost.com
*m:* +264 81 323 1260
*w:* www.namhost.com <https://www.namhost.com/>

<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>

Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus, 7210, RSA

--
Regards / Groete

<https://www.namhost.com> Granwille Strauss // Senior Systems Admin

*e:* granwi...@namhost.com
*m:* +264 81 323 1260
*w:* www.namhost.com <https://www.namhost.com/>

<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>

Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA

The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It
is forbidden to copy, forward, or in any way reveal the contents of this
message to anyone without our explicit consent. The integrity and
security of this email cannot be guaranteed over the Internet.
Therefore, the sender will not be held liable for any damage caused by
the message. For our full privacy policy and disclaimers, please go to
https://www.namhost.com/privacy-policy

smime.p7s
Description: S/MIME Cryptographic Signature

Virtual Router Constantly Fails to Update /etc/hosts file

2023-06-15 Thread Granwille Strauss

Hi Guys

I have a very basic private cloud setup, we make use of a a virtual
router in Cloudstack Management, for some reason, each time we restart a
VM instance via UI, our router keeps failing a health check by having
missing entries in the /etc/hosts file. To solve this we manually need
to access the virtual router and manually add the entries for the health
checks to pass.

This started to happen when we created a new VM instance but after a few
hours, terminated the VM and expunged it. And then re-created the VM
again, using the exact same name as before. This somehow causes this
behaviour with the virtual router. Is there any way to solve this? I
have rebooted the router and still doesn't work, only manually entering
the values fixes it, until we need to reboot a VM again or something.

--
Regards / Groete

<https://www.namhost.com> Granwille Strauss // Senior Systems Admin

*e:* granwi...@namhost.com
*m:* +264 81 323 1260
*w:* www.namhost.com <https://www.namhost.com/>

<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>

Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA

The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It
is forbidden to copy, forward, or in any way reveal the contents of this
message to anyone without our explicit consent. The integrity and
security of this email cannot be guaranteed over the Internet.
Therefore, the sender will not be held liable for any damage caused by
the message. For our full privacy policy and disclaimers, please go to
https://www.namhost.com/privacy-policy

smime.p7s
Description: S/MIME Cryptographic Signature

Re: Change Default Primary Storage from NFS to Local

2023-05-31 Thread Granwille Strauss


Hi

It seems selecting "Override root disk offering" when creating  a new VM 
solves the issue.


On 5/31/23 15:20, Granwille Strauss wrote:


Hi

I created one now, also had to enable local storage option for my 
zone. I deployed a new VM now, but from an existing template. It seems 
the template still creates the main volume in NFS, but the new disk 
offering I assigned, created a data disk but on local storage. I 
reckon this means I need to deploy a new template one that uses this 
new service offering, yes?


On 5/31/23 14:48, Ruben Bosch wrote:

Did you create new service offerings with storage type = local?

Met vriendelijke groet / Kind regards,

Ruben Bosch
CLDIN


On 31 May 2023, at 14:43, Granwille Strauss  
wrote:

Hi Guys

Just a quick one. I configured my setup initially to use NFS for primary 
storage. This all worked well until we noticed that NFS storage wasn't going to 
last us long because we didn't have a cluster storage setup or SAN configured. 
So, we have since stopped our VM machines and migrated their storage volumes 
from the NFS to local, which we are happy with to continue with. I have about 
two VMs still using the NFS primary, the rest are on local.


Now, when I create a new VM instance, it still deploys to NFS storage, I want 
it to automatically deploy to the local storage of the KVM when its created. 
How do I accomplish this? Because now, after deploying the new VM I must stop 
it again and then migrate it each time, which isn't really nice to do.

--
Regards / Groete

  <https://www.namhost.com/>  Granwille Strauss  //  Senior Systems Admin

e:granwi...@namhost.com  <mailto:granwi...@namhost.com>
m: +264 81 323 1260
w:www.namhost.com  <https://www.namhost.com/>

  <https://www.facebook.com/namhost>   <https://twitter.com/namhost>   
<https://www.instagram.com/namhostinternetservices/>   <https://www.linkedin.com/company/namhos>  
 <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>

  
<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>

Namhost Internet Services (Pty) Ltd, 24 Black Eagle Rd, Hermanus, 7210, RSA

The content of this message is confidential. If you have received it by 
mistake, please inform us by email reply and then delete the message. It is 
forbidden to copy, forward, or in any way reveal the contents of this message 
to anyone without our explicit consent. The integrity and security of this 
email cannot be guaranteed over the Internet. Therefore, the sender will not be 
held liable for any damage caused by the message. For our full privacy policy 
and disclaimers, please go tohttps://www.namhost.com/privacy-policy
  
<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>

--
Regards / Groete

<https://www.namhost.com> Granwille Strauss  // Senior Systems Admin

*e:* granwi...@namhost.com
*m:* +264 81 323 1260 
*w:* www.namhost.com <https://www.namhost.com/>

<https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>

<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>

Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus, 7210, RSA


The content of this message is confidential. If you have received it 
by mistake, please inform us by email reply and then delete the 
message. It is forbidden to copy, forward, or in any way reveal the 
contents of this message to anyone without our explicit consent. The 
integrity and security of this email cannot be guaranteed over the 
Internet. Therefore, the sender will not be held liable for any damage 
caused by the message. For our full privacy policy and disclaimers, 
please go to https://www.namhost.com/privacy-policy


Powered by AdSigner 
<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>

--
Regards / Groete

<https://www.namhost.com> Granwille Strauss  // Senior Systems Admin

*e:* granwi...@namhost.com
*m:* +264 81 323 1260 
*w:* www.namhost.com <https://www.namhost.com/>

<https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>

<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>

Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA



The content of this message is confidential. If you have received it by 
mistake, please inform us by email reply and then delete the message. It 
is forbidden to copy, forward, or in any way reveal the contents of this 
message to anyone without our explicit con

Re: Change Default Primary Storage from NFS to Local

2023-05-31 Thread Granwille Strauss

I created one now, also had to enable local storage option for my zone.
I deployed a new VM now, but from an existing template. It seems the
template still creates the main volume in NFS, but the new disk offering
I assigned, created a data disk but on local storage. I reckon this
means I need to deploy a new template one that uses this new service
offering, yes?

On 5/31/23 14:48, Ruben Bosch wrote:

Did you create new service offerings with storage type = local?

Met vriendelijke groet / Kind regards,

Ruben Bosch
CLDIN

On 31 May 2023, at 14:43, Granwille Strauss
wrote:

Hi Guys

Just a quick one. I configured my setup initially to use NFS for primary
storage. This all worked well until we noticed that NFS storage wasn't going to
last us long because we didn't have a cluster storage setup or SAN configured.
So, we have since stopped our VM machines and migrated their storage volumes
from the NFS to local, which we are happy with to continue with. I have about
two VMs still using the NFS primary, the rest are on local.

Now, when I create a new VM instance, it still deploys to NFS storage, I want
it to automatically deploy to the local storage of the KVM when its created.
How do I accomplish this? Because now, after deploying the new VM I must stop
it again and then migrate it each time, which isn't really nice to do.

--
Regards / Groete

<https://www.namhost.com/> Granwille Strauss // Senior Systems Admin

e:granwi...@namhost.com <mailto:granwi...@namhost.com>
m: +264 81 323 1260
w:www.namhost.com <https://www.namhost.com/>

<https://www.facebook.com/namhost> <https://twitter.com/namhost>
<https://www.instagram.com/namhostinternetservices/> <https://www.linkedin.com/company/namhos>
<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>

<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>

Namhost Internet Services (Pty) Ltd, 24 Black Eagle Rd, Hermanus, 7210, RSA

The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It is
forbidden to copy, forward, or in any way reveal the contents of this message
to anyone without our explicit consent. The integrity and security of this
email cannot be guaranteed over the Internet. Therefore, the sender will not be
held liable for any damage caused by the message. For our full privacy policy
and disclaimers, please go tohttps://www.namhost.com/privacy-policy

<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>

--
Regards / Groete

<https://www.namhost.com> Granwille Strauss // Senior Systems Admin

*e:* granwi...@namhost.com
*m:* +264 81 323 1260
*w:* www.namhost.com <https://www.namhost.com/>

<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>

Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA

The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It
is forbidden to copy, forward, or in any way reveal the contents of this
message to anyone without our explicit consent. The integrity and
security of this email cannot be guaranteed over the Internet.
Therefore, the sender will not be held liable for any damage caused by
the message. For our full privacy policy and disclaimers, please go to
https://www.namhost.com/privacy-policy

smime.p7s
Description: S/MIME Cryptographic Signature

Change Default Primary Storage from NFS to Local

2023-05-31 Thread Granwille Strauss

Hi Guys

Just a quick one. I configured my setup initially to use NFS for primary
storage. This all worked well until we noticed that NFS storage wasn't
going to last us long because we didn't have a cluster storage setup or
SAN configured. So, we have since stopped our VM machines and migrated
their storage volumes from the NFS to local, which we are happy with to
continue with. I have about two VMs still using the NFS primary, the
rest are on local.

Now, when I create a new VM instance, it still deploys to NFS storage, I
want it to automatically deploy to the local storage of the KVM when its
created. How do I accomplish this? Because now, after deploying the new
VM I must stop it again and then migrate it each time, which isn't
really nice to do.

--
Regards / Groete

<https://www.namhost.com> Granwille Strauss // Senior Systems Admin

*e:* granwi...@namhost.com
*m:* +264 81 323 1260
*w:* www.namhost.com <https://www.namhost.com/>

<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>

Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA

The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It
is forbidden to copy, forward, or in any way reveal the contents of this
message to anyone without our explicit consent. The integrity and
security of this email cannot be guaranteed over the Internet.
Therefore, the sender will not be held liable for any damage caused by
the message. For our full privacy policy and disclaimers, please go to
https://www.namhost.com/privacy-policy

smime.p7s
Description: S/MIME Cryptographic Signature

Re: VMs have no network


Hi

Thank you for the feedback, it was our IP subnet range (rented), which 
we have been paying for. They just gave it away and now, the new owners 
are refusing to give back our subnet, so now WE must make changes to our 
VMs networks. Its a very shitty story and they will soon regret this 
mistake they caused.


On 5/25/23 21:07, K B Shiv Kumar wrote:

Hi Granwille,

You need to ensure that IP range associated to CloudStack is not used by
any other setup. If you have already done that and can't undo it but want
to keep the remaining dedicated to CloudStack, you can mark the IPs
allocated outside CloudStack as Allocated in the DB.

Tangential point  I believe you can't ping your VR's link local
IP(169.254.X.Y) from your user VMs. That IP can only be reached by the
underlying host and system VMs and VRs on the same host.

Regards,
Shiv
(Sent from mobile device. Please excuse brevity and typos.)

On Thu, 25 May 2023, 23:23 Granwille Strauss,
wrote:


Hi

Yep my IP subnet range has been allocated to new clients a complete eff up
at the DC. Can I please get the correct steps I can take to ensure the new
range is correctly assigned to my VMs? is it as simple as unassigning an IP
and re-assigning one again? And then obviously update the IP network in the
VM?
On 5/25/23 19:10, Simon Weller wrote:

ok great. Glad you're making progress toward a resolution.

On Thu, May 25, 2023 at 12:01 PM Granwille Strauss
wrote:


Hi

Thank you for the update I believe I found the issue my DC just confirmed
that a certain subnet range is not configured for my host machine. After we
have been using it for MONTHS now. And I tested other VMs on the same HOST
but they use a different subnet and they have full internet access. And the
VMs not working are the ones with the missing subnet from my DC.

I am busy fighting this now with them and will provide feedback if this
was indeed the issue, but this far all fingers seems to point towards it.
On 5/25/23 18:47, Simon Weller wrote:

Can you tell me more about your advanced network setup?

Also, can you provide some info from your VM?

If it's Linux, perhaps the output of "ip a" and the output of "netstat
-r" or "route -s".

If it's windows,a screenshot of the network configuration including IP
addresses and gateway.

-Si

On Thu, May 25, 2023, 11:39 AM Granwille Strauss
wrote:


Yes I have a virtual router and only one. I have a very basic set up
nothing additional or extra. That said, it seems I cannot ping my virtual
router private IP 169.254.127.238 from the VM no. Attached is a
screenshot for you to confirm I am pinging the right IP
On 5/25/23 18:32, Simon Weller wrote:

Ok, so I assume you have Virtual Routers as well. Are you running redundant
VRs, or just one? Can you ping the VR inside IP from your VM (guest network
gateway)?

On Thu, May 25, 2023, 11:27 AM Granwille Strauss  

wrote:


Hi The host can ping and do everything correctly, my SSVM is also on the
same host as the VMs and my SSVM works 100% fine and can also ping outside.
Its only my VMs that's failing to do so. I am using advanced network. And
can you provide me with more details to VIF and how to determine this?
On 5/25/23 18:20, Simon Weller wrote:

I'm assume the underlay network is ok? Can you ping your gateway from the
host?

What type of networking are you using with ACS/KVM?

Are you seeing the VIF for a VM being plugged into the correct bridge?

On Thu, May 25, 2023, 11:10 AM Granwille Strauss  

wrote:


Hi

Thank you I managed to free space now successfully, thank you. However,
VMs still have zero network. May I also add that I manually shutdown VMs in
KVM using the virsh command. And then boot them up again via Cloudstack UI.
Perhaps this has some affect.
On 5/25/23 17:54, Simon Weller wrote:

Granwille,

Use du . -h on your major partitions and try and narrow down the disk usage.

Check the MySQL log and and also for any temp files that could be blowing
out your disk usage .

Are you running any MySQL replication? If so, make sure you have log
rotation and deletion setup so they you only keep a few days of the
replication logs. If you do see lots of binlogs, you can trim them using
the FLUSH LOGS MySQL cli command.

-Si


On Thu, May 25, 2023, 10:45 AM Granwille Strauss
  
  wrote:


Hi Guys

I need any help I can get, please. I have a cloudstack management server
and I configured NFS on it and have a few VMs that use this NFS as its
primary storage. Today my VMs randomly just went down, no one did anything
whatsoever. Upon review we found that the cloudstack management server
drive shows its at 100% usage. But it makes ZERO sense. There is space, the
drive is 872 GB big and only 840 GB is being used.


So now, I managed to to move files and delete logs and so to try and free
space. I deleted over 13 GB and still server shows 100% usage. But now I
can finally log into Cloudstack UI, previously I couldn't. I am now moving
those VMs volumes from the NF

Re: VMs have no network


Hi

Yep my IP subnet range has been allocated to new clients a complete eff 
up at the DC. Can I please get the correct steps I can take to ensure 
the new range is correctly assigned to my VMs? is it as simple as 
unassigning an IP and re-assigning one again? And then obviously update 
the IP network in the VM?


On 5/25/23 19:10, Simon Weller wrote:

ok great. Glad you're making progress toward a resolution.

On Thu, May 25, 2023 at 12:01 PM Granwille Strauss 
 wrote:


Hi

Thank you for the update I believe I found the issue my DC just
confirmed that a certain subnet range is not configured for my
host machine. After we have been using it for MONTHS now. And I
tested other VMs on the same HOST but they use a different subnet
and they have full internet access. And the VMs not working are
the ones with the missing subnet from my DC.

I am busy fighting this now with them and will provide feedback if
this was indeed the issue, but this far all fingers seems to point
towards it.

On 5/25/23 18:47, Simon Weller wrote:

Can you tell me more about your advanced network setup?

Also, can you provide some info from your VM?

If it's Linux, perhaps the output of "ip a" and the output of
"netstat -r" or "route -s".

If it's windows,a screenshot of the network configuration
including IP addresses and gateway.

-Si

On Thu, May 25, 2023, 11:39 AM Granwille Strauss
 wrote:

Yes I have a virtual router and only one. I have a very basic
set up nothing additional or extra. That said, it seems I
cannot ping my virtual router private IP 169.254.127.238 from
the VM no. Attached is a screenshot for you to confirm I am
pinging the right IP

On 5/25/23 18:32, Simon Weller wrote:

Ok, so I assume you have Virtual Routers as well. Are you running 
redundant
VRs, or just one? Can you ping the VR inside IP from your VM (guest 
network
gateway)?

On Thu, May 25, 2023, 11:27 AM Granwille Strauss  
<mailto:granwi...@namhost.com>
wrote:


Hi The host can ping and do everything correctly, my SSVM is also on the
same host as the VMs and my SSVM works 100% fine and can also ping 
outside.
Its only my VMs that's failing to do so. I am using advanced network. 
And
can you provide me with more details to VIF and how to determine this?
On 5/25/23 18:20, Simon Weller wrote:

I'm assume the underlay network is ok? Can you ping your gateway from 
the
host?

What type of networking are you using with ACS/KVM?

Are you seeing the VIF for a VM being plugged into the correct bridge?

    On Thu, May 25, 2023, 11:10 AM Granwille Strauss  
<mailto:granwi...@namhost.com>
<mailto:granwi...@namhost.com>
wrote:


Hi

Thank you I managed to free space now successfully, thank you. However,
VMs still have zero network. May I also add that I manually shutdown 
VMs in
KVM using the virsh command. And then boot them up again via Cloudstack 
UI.
Perhaps this has some affect.
On 5/25/23 17:54, Simon Weller wrote:

Granwille,

Use du . -h on your major partitions and try and narrow down the disk 
usage.

Check the MySQL log and and also for any temp files that could be 
blowing
out your disk usage .

Are you running any MySQL replication? If so, make sure you have log
rotation and deletion setup so they you only keep a few days of the
replication logs. If you do see lots of binlogs, you can trim them using
the FLUSH LOGS MySQL cli command.

    -Si


    On Thu, May 25, 2023, 10:45 AM Granwille Strauss  
<mailto:granwi...@namhost.com.invalid>
<mailto:granwi...@namhost.com.invalid>
<mailto:granwi...@namhost.com.invalid>
<mailto:granwi...@namhost.com.invalid>  wrote:


Hi Guys

I need any help I can get, please. I have a cloudstack management server
and I configured NFS on it and have a few VMs that use this NFS as its
primary storage. Today my VMs randomly just went down, no one did 
anything
whatsoever. Upon review we found that the cloudstack management server
drive shows its at 100% usage. But it makes ZERO sense. There is space, 
the
drive is 872 GB big and only 840 GB is being used.


So now, I managed to to move files and delete logs and so to try and 
free
space. I deleted over 13 GB and still server shows 100% usage. But now I
can finally log into Cloudstack UI, previously I couldn't. I am now 
moving
those VMs volumes from the NFS to the local KVM disk. After successfully
migrating the volume, I now boot the VM  and it runs 100% fine, but 
there
is ZERO internet access to the VM I can

Re: VMs have no network

Thank you for the update I believe I found the issue my DC just
confirmed that a certain subnet range is not configured for my host
machine. After we have been using it for MONTHS now. And I tested other
VMs on the same HOST but they use a different subnet and they have full
internet access. And the VMs not working are the ones with the missing
subnet from my DC.

I am busy fighting this now with them and will provide feedback if this
was indeed the issue, but this far all fingers seems to point towards it.

On 5/25/23 18:47, Simon Weller wrote:

Can you tell me more about your advanced network setup?

Also, can you provide some info from your VM?

If it's Linux, perhaps the output of "ip a" and the output of "netstat
-r" or "route -s".

If it's windows,a screenshot of the network configuration including IP
addresses and gateway.

-Si

On Thu, May 25, 2023, 11:39 AM Granwille Strauss
wrote:

Yes I have a virtual router and only one. I have a very basic set
up nothing additional or extra. That said, it seems I cannot ping
my virtual router private IP 169.254.127.238 from the VM no.
Attached is a screenshot for you to confirm I am pinging the right IP

On 5/25/23 18:32, Simon Weller wrote:

Ok, so I assume you have Virtual Routers as well. Are you running redundant
VRs, or just one? Can you ping the VR inside IP from your VM (guest network
gateway)?

On Thu, May 25, 2023, 11:27 AM Granwille Strauss
<mailto:granwi...@namhost.com>
wrote:

Hi The host can ping and do everything correctly, my SSVM is also on the
same host as the VMs and my SSVM works 100% fine and can also ping outside.
Its only my VMs that's failing to do so. I am using advanced network. And
can you provide me with more details to VIF and how to determine this?
On 5/25/23 18:20, Simon Weller wrote:

I'm assume the underlay network is ok? Can you ping your gateway from the
host?

What type of networking are you using with ACS/KVM?

Are you seeing the VIF for a VM being plugged into the correct bridge?

On Thu, May 25, 2023, 11:10 AM Granwille Strauss
<mailto:granwi...@namhost.com>
<mailto:granwi...@namhost.com>
wrote:

Thank you I managed to free space now successfully, thank you. However,
VMs still have zero network. May I also add that I manually shutdown VMs in
KVM using the virsh command. And then boot them up again via Cloudstack UI.
Perhaps this has some affect.
On 5/25/23 17:54, Simon Weller wrote:

Granwille,

Use du . -h on your major partitions and try and narrow down the disk usage.

Check the MySQL log and and also for any temp files that could be blowing
out your disk usage .

Are you running any MySQL replication? If so, make sure you have log
rotation and deletion setup so they you only keep a few days of the
replication logs. If you do see lots of binlogs, you can trim them using
the FLUSH LOGS MySQL cli command.

-Si

On Thu, May 25, 2023, 10:45 AM Granwille Strauss
<mailto:granwi...@namhost.com.invalid>
<mailto:granwi...@namhost.com.invalid>
<mailto:granwi...@namhost.com.invalid>
<mailto:granwi...@namhost.com.invalid> wrote:

Hi Guys

I need any help I can get, please. I have a cloudstack management server
and I configured NFS on it and have a few VMs that use this NFS as its
primary storage. Today my VMs randomly just went down, no one did anything
whatsoever. Upon review we found that the cloudstack management server
drive shows its at 100% usage. But it makes ZERO sense. There is space, the
drive is 872 GB big and only 840 GB is being used.

So now, I managed to to move files and delete logs and so to try and free
space. I deleted over 13 GB and still server shows 100% usage. But now I
can finally log into Cloudstack UI, previously I couldn't. I am now moving
those VMs volumes from the NFS to the local KVM disk. After successfully
migrating the volume, I now boot the VM and it runs 100% fine, but there
is ZERO internet access to the VM I cannot even pinggoogle.com
<http://google.com> from the
VM. SSVM and the virtual router are working 100% fine, zero failures on
health check and also the SSVM check. The KVM host itself has full internet
access.

Any ideas? There are zero errors being thrown at all.
--
Regards / Groete<https://www.namhost.com> <https://www.namhost.com> <https://www.namhost.com>
<https://www.namhost.com> <https://www.namhost.com> <https://www.namhost.com>
<https://www.namhost.com> <https://www.namhost.com> Granwille Strauss // Senior Systems Admin

*e:*granwi...@namhost.com
*m:* +264 81 323 1260 <+264813231260>
*w:*www.namhost.com <http://www.namhost.com><https://

Re: VMs have no network


Also In my VM XML I have the following:

  bridge='cloudbr0'/>  peak='128000'/>  
   state='up'/>  bus='0x00' slot='0x03' function='0x0'/> 


And checking host via ip linkw show confirms this is the correct VIF:



$ ip link show
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN 
mode DEFAULT group default qlen 1000

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eno1:  mtu 1500 qdisc mq master 
cloudbr0 state UP mode DEFAULT group default qlen 1000

    link/ether 0c:c4:7a:7f:e1:1c brd ff:ff:ff:ff:ff:ff
    altname enp6s0
3: eno2:  mtu 1500 qdisc mq master 
cloudbr1 state UP mode DEFAULT group default qlen 1000

    link/ether 0c:c4:7a:7f:e1:1d brd ff:ff:ff:ff:ff:ff
    altname enp7s0
4: cloudbr1:  mtu 1500 qdisc noqueue 
state UP mode DEFAULT group default qlen 1000

    link/ether 0c:c4:7a:7f:e1:1d brd ff:ff:ff:ff:ff:ff
5: cloudbr0:  mtu 1500 qdisc noqueue 
state UP mode DEFAULT group default qlen 1000

    link/ether 0c:c4:7a:7f:e1:1c brd ff:ff:ff:ff:ff:ff
6: cloud0:  mtu 1500 qdisc noqueue 
state UP mode DEFAULT group default qlen 1000

    link/ether fe:00:a9:fe:59:51 brd ff:ff:ff:ff:ff:ff
142: vnet125:  mtu 1500 qdisc noqueue 
master cloud0 state UNKNOWN mode DEFAULT group default qlen 1000

    link/ether fe:00:a9:fe:59:51 brd ff:ff:ff:ff:ff:ff
143: vnet126:  mtu 1500 qdisc noqueue 
master cloudbr1 state UNKNOWN mode DEFAULT group default qlen 1000

    link/ether fe:00:6a:00:00:10 brd ff:ff:ff:ff:ff:ff
144: vnet127:  mtu 1500 qdisc noqueue 
master cloudbr0 state UNKNOWN mode DEFAULT group default qlen 1000

    link/ether fe:00:93:00:00:1f brd ff:ff:ff:ff:ff:ff
145: vnet128:  mtu 1500 qdisc noqueue 
master cloudbr1 state UNKNOWN mode DEFAULT group default qlen 1000

    link/ether fe:00:4d:00:00:45 brd ff:ff:ff:ff:ff:ff
166: vnet149:  mtu 1500 qdisc htb 
master cloudbr0 state UNKNOWN mode DEFAULT group default qlen 1000

    link/ether fe:00:e9:00:00:4c brd ff:ff:ff:ff:ff:ff
167: vnet150:  mtu 1500 qdisc noqueue 
master cloud0 state UNKNOWN mode DEFAULT group default qlen 1000

    link/ether fe:00:a9:fe:7f:ee brd ff:ff:ff:ff:ff:ff
171: vnet154:  mtu 1500 qdisc htb 
master cloudbr0 state UNKNOWN mode DEFAULT group default qlen 1000

    link/ether fe:00:5e:00:00:58 brd ff:ff:ff:ff:ff:ff



On 5/25/23 18:39, Granwille Strauss wrote:


Yes I have a virtual router and only one. I have a very basic set up 
nothing additional or extra. That said, it seems I cannot ping my 
virtual router private IP 169.254.127.238 from the VM no. Attached is 
a screenshot for you to confirm I am pinging the right IP


On 5/25/23 18:32, Simon Weller wrote:

Ok, so I assume you have Virtual Routers as well. Are you running redundant
VRs, or just one? Can you ping the VR inside IP from your VM (guest network
gateway)?

On Thu, May 25, 2023, 11:27 AM Granwille Strauss
wrote:


Hi The host can ping and do everything correctly, my SSVM is also on the
same host as the VMs and my SSVM works 100% fine and can also ping outside.
Its only my VMs that's failing to do so. I am using advanced network. And
can you provide me with more details to VIF and how to determine this?
On 5/25/23 18:20, Simon Weller wrote:

I'm assume the underlay network is ok? Can you ping your gateway from the
host?

What type of networking are you using with ACS/KVM?

Are you seeing the VIF for a VM being plugged into the correct bridge?

On Thu, May 25, 2023, 11:10 AM Granwille Strauss  

wrote:


Hi

Thank you I managed to free space now successfully, thank you. However,
VMs still have zero network. May I also add that I manually shutdown VMs in
KVM using the virsh command. And then boot them up again via Cloudstack UI.
Perhaps this has some affect.
On 5/25/23 17:54, Simon Weller wrote:

Granwille,

Use du . -h on your major partitions and try and narrow down the disk usage.

Check the MySQL log and and also for any temp files that could be blowing
out your disk usage .

Are you running any MySQL replication? If so, make sure you have log
rotation and deletion setup so they you only keep a few days of the
replication logs. If you do see lots of binlogs, you can trim them using
the FLUSH LOGS MySQL cli command.

-Si


On Thu, May 25, 2023, 10:45 AM Granwille Strauss  

  wrote:


Hi Guys

I need any help I can get, please. I have a cloudstack management server
and I configured NFS on it and have a few VMs that use this NFS as its
primary storage. Today my VMs randomly just went down, no one did anything
whatsoever. Upon review we found that the cloudstack management server
drive shows its at 100% usage. But it makes ZERO sense. There is space, the
drive is 872 GB big and only 840 GB is being used.


So now, I managed to to move files and delete logs and so to try and free
space. I deleted over 13 GB and still server shows 100% usage. But now I
can finally log into Cloudstack UI, previously I couldn't. I am now moving
those VMs volumes from the NFS to the local KVM disk. After successfully
migrating the volume, I now

Re: VMs have no network

Hi The host can ping and do everything correctly, my SSVM is also on the
same host as the VMs and my SSVM works 100% fine and can also ping
outside. Its only my VMs that's failing to do so. I am using advanced
network. And can you provide me with more details to VIF and how to
determine this?

On 5/25/23 18:20, Simon Weller wrote:

I'm assume the underlay network is ok? Can you ping your gateway from the
host?

What type of networking are you using with ACS/KVM?

Are you seeing the VIF for a VM being plugged into the correct bridge?

On Thu, May 25, 2023, 11:10 AM Granwille Strauss
wrote:

Granwille,

Use du . -h on your major partitions and try and narrow down the disk usage.

Check the MySQL log and and also for any temp files that could be blowing
out your disk usage .

-Si

On Thu, May 25, 2023, 10:45 AM Granwille Strauss
wrote:

Hi Guys

So now, I managed to to move files and delete logs and so to try and free
space. I deleted over 13 GB and still server shows 100% usage. But now I
can finally log into Cloudstack UI, previously I couldn't. I am now moving
those VMs volumes from the NFS to the local KVM disk. After successfully
migrating the volume, I now boot the VM and it runs 100% fine, but there
is ZERO internet access to the VM I cannot even ping google.com from the
VM. SSVM and the virtual router are working 100% fine, zero failures on
health check and also the SSVM check. The KVM host itself has full internet
access.

Any ideas? There are zero errors being thrown at all.
--
Regards / Groete
<https://www.namhost.com> <https://www.namhost.com> Granwille Strauss //
Senior Systems Admin

*e:*granwi...@namhost.com
*m:* +264 81 323 1260 <+264813231260>
*w:*www.namhost.com
<https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://twitter.com/namhost>
<https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/>
<https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos>
<https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>

<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>

Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA

The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It is
forbidden to copy, forward, or in any way reveal the contents of this
message to anyone without our explicit consent. The integrity and security
of this email cannot be guaranteed over the Internet. Therefore, the sender
will not be held liable for any damage caused by the message. For our full
privacy policy and disclaimers, please go
tohttps://www.namhost.com/privacy-policy

[image: Powered by
AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>

--
Regards / Groete

<https://www.namhost.com> Granwille Strauss // Senior Systems Admin

*e:*granwi...@namhost.com
*m:* +264 81 323 1260 <+264813231260>
*w:*www.namhost.com

<https://www.facebook.com/namhost> <https://twitter.com/namhost>
<https://www.instagram.com/namhostinternetservices/>
<https://www.linkedin.com/company/namhos>
<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>

<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>

Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA

The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It is
forbidde

Re: VMs have no network

Thank you I managed to free space now successfully, thank you. However,
VMs still have zero network. May I also add that I manually shutdown VMs
in KVM using the virsh command. And then boot them up again via
Cloudstack UI. Perhaps this has some affect.

On 5/25/23 17:54, Simon Weller wrote:

Granwille,

Use du . -h on your major partitions and try and narrow down the disk usage.

Check the MySQL log and and also for any temp files that could be blowing
out your disk usage .

-Si

On Thu, May 25, 2023, 10:45 AM Granwille Strauss
wrote:

Hi Guys

So now, I managed to to move files and delete logs and so to try and free
space. I deleted over 13 GB and still server shows 100% usage. But now I
can finally log into Cloudstack UI, previously I couldn't. I am now moving
those VMs volumes from the NFS to the local KVM disk. After successfully
migrating the volume, I now boot the VM and it runs 100% fine, but there
is ZERO internet access to the VM I cannot even ping google.com from the
VM. SSVM and the virtual router are working 100% fine, zero failures on
health check and also the SSVM check. The KVM host itself has full internet
access.

Any ideas? There are zero errors being thrown at all.
--
Regards / Groete

<https://www.namhost.com> Granwille Strauss // Senior Systems Admin

*e:*granwi...@namhost.com
*m:* +264 81 323 1260 <+264813231260>
*w:*www.namhost.com

<https://www.facebook.com/namhost> <https://twitter.com/namhost>
<https://www.instagram.com/namhostinternetservices/>
<https://www.linkedin.com/company/namhos>
<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>

<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>

Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA

The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It is
forbidden to copy, forward, or in any way reveal the contents of this
message to anyone without our explicit consent. The integrity and security
of this email cannot be guaranteed over the Internet. Therefore, the sender
will not be held liable for any damage caused by the message. For our full
privacy policy and disclaimers, please go to
https://www.namhost.com/privacy-policy

[image: Powered by AdSigner]
<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>

--
Regards / Groete

<https://www.namhost.com> Granwille Strauss // Senior Systems Admin

*e:* granwi...@namhost.com
*m:* +264 81 323 1260
*w:* www.namhost.com <https://www.namhost.com/>

<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>

Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA

The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It
is forbidden to copy, forward, or in any way reveal the contents of this
message to anyone without our explicit consent. The integrity and
security of this email cannot be guaranteed over the Internet.
Therefore, the sender will not be held liable for any damage caused by
the message. For our full privacy policy and disclaimers, please go to
https://www.namhost.com/privacy-policy

smime.p7s
Description: S/MIME Cryptographic Signature

VMs have no network

Hi Guys

I need any help I can get, please. I have a cloudstack management server
and I configured NFS on it and have a few VMs that use this NFS as its
primary storage. Today my VMs randomly just went down, no one did
anything whatsoever. Upon review we found that the cloudstack management
server drive shows its at 100% usage. But it makes ZERO sense. There is
space, the drive is 872 GB big and only 840 GB is being used.

So now, I managed to to move files and delete logs and so to try and
free space. I deleted over 13 GB and still server shows 100% usage. But
now I can finally log into Cloudstack UI, previously I couldn't. I am
now moving those VMs volumes from the NFS to the local KVM disk. After
successfully migrating the volume, I now boot the VM and it runs 100%
fine, but there is ZERO internet access to the VM I cannot even ping
google.com from the VM. SSVM and the virtual router are working 100%
fine, zero failures on health check and also the SSVM check. The KVM
host itself has full internet access.

Any ideas? There are zero errors being thrown at all.

--
Regards / Groete

<https://www.namhost.com> Granwille Strauss // Senior Systems Admin

*e:* granwi...@namhost.com
*m:* +264 81 323 1260
*w:* www.namhost.com <https://www.namhost.com/>

<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>

Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA

The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It
is forbidden to copy, forward, or in any way reveal the contents of this
message to anyone without our explicit consent. The integrity and
security of this email cannot be guaranteed over the Internet.
Therefore, the sender will not be held liable for any damage caused by
the message. For our full privacy policy and disclaimers, please go to
https://www.namhost.com/privacy-policy

smime.p7s
Description: S/MIME Cryptographic Signature

Re: preventing VM Live migration between Pods

2023-05-16 Thread Granwille Strauss

While I understand the concern, using the 'host-model' should solve the 
migration issue. I currently have a Intel Xeon E5-2620 v4 (32) KVM host 
and my VMs shows as Intel (Broadwell, IBRS). A massive difference, but 
migrations work fine and VM performance drop is not that noticeable. You 
could change back to host-passthrough once the migration is completed.


On 5/16/23 17:03, Gary Dixon wrote:


We have tried host-model – however virsh capabilities on the newer 
servers doesn’t even pick up the correct cpu map xml definition – the 
actual CPU is a AMD EPYC 7763 (codename Milan) and libvirt thinks it’s 
a ‘Rome’ cpu – a whole generation earlier !!!


Gary Dixon
Senior Technical Consultant
T:  +44 161 537 4990
E: *v* ms@quadris‑support.com
W: www.quadris.co.uk

The information contained in this e-mail from Quadris may be 
confidential and privileged for the private use of the named 
recipient. The contents of this e-mail may not necessarily represent 
the official views of Quadris.  If you have received this information 
in error you must not copy, distribute or take any action or reliance 
on its contents.  Please destroy any hard copies and delete this message.


*From:*Granwille Strauss 
*Sent:* Tuesday, May 16, 2023 3:51 PM
*To:* users@cloudstack.apache.org
*Cc:* gary.di...@quadris.co.uk.INVALID
*Subject:* Re: preventing VM Live migration between Pods

Hi Gary

I am still fairly new to ACS myself, but as far as I can recall, using 
the 'host-passthrough' option is prone to cause problems during 
migrations, this is also mentioned in the documentation: 
https://docs.cloudstack.apache.org/en/latest/installguide/hypervisor/kvm.html?highlight=host-passthrough#configure-cpu-model-for-kvm-guest-optional 
I suggest changing to 'host-model' instead:


*host-passthrough* may lead to migration failure,if you have this
problem, you should use host-model or custom. guest.cpu.features
will force cpu features as a required policy so make sure to put
only those features that are provided by the host CPU. As your kvm
cluster needs to be made up of homogenous nodes anyway (see System
Requirements), it might make most sense to use
guest.cpu.mode=host-model or guest.cpu.mode=host-passthrough

On 5/16/23 15:13, Gary Dixon wrote:

Hi everyone

Other than disabling a Pod – is there a way to prevent live
migration of VM’s between Pods in ACS ?

We are on version 4.15.2 with Ubuntu 20.04 KVM hosts.  Each Pod
contains a single cluster of Homogenous hosts – however there are
only slight differences between the CPU’s on the physical hosts in
each Cluster. We have the guest.cpu.mode set to host-passthrough
but have noticed serious issues when a VM is live migrated between
specific Pods (usually if a VM is started on a cluster with the
slightly better CPU’s and then live migrated to an older Pod)

We have tried setting the guest.cpu.mode to host-passthrough with
specific CPU features using the “guest.cpu.features=” and then
setting all of the host’s CPU flags shown from the output of the
lscpu command in a space separated list as instructed from ACS
documentation – but we then are unable to even start a VM –
insufficient resources error, - if we remove the
guest.cpu.features from the agent.properties – then we can start a
VM again.

It would be good if we had an option or a setting to just not
allow live migration of VM’s between pods and therefore can only
perform a ‘cold’ migration if we wish to move a VM to another Pod.

Any thoughts on this ?

BR

Gary


*Gary Dixon*****

Senior Technical Consultant

T:  +44 161 537 4990

E: *v* ms@quadris‑support.com

W: www.quadris.co.uk <http://www.quadris.co.uk>

*The information contained in this e-mail from Quadris may be
confidential and privileged for the private use of the named
recipient. The contents of this e-mail may not necessarily
represent the official views of Quadris. If you have received this
information in error you must not copy, distribute or take any
action or reliance on its contents.  Please destroy any hard
copies and delete this message.***

--

Regards / Groete

<https://www.namhost.com/>



*Granwille Strauss** // *Senior Systems Admin

*e:*granwi...@namhost.com <mailto:granwi...@namhost.com>
*m:*+264 81 323 1260 
*w:*www.namhost.com <https://www.namhost.com/>

<https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>


<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>

Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA

The content of this message is confidential. If you have received it 
by mistake,

Re: preventing VM Live migration between Pods

2023-05-16 Thread Granwille Strauss


Hi Gary

I am still fairly new to ACS myself, but as far as I can recall, using 
the 'host-passthrough' option is prone to cause problems during 
migrations, this is also mentioned in the documentation: 
https://docs.cloudstack.apache.org/en/latest/installguide/hypervisor/kvm.html?highlight=host-passthrough#configure-cpu-model-for-kvm-guest-optional 
I suggest changing to 'host-model' instead:


host-passthroughmay lead to migration failure,if you have this 
problem, you should use host-model or custom. guest.cpu.features will 
force cpu features as a required policy so make sure to put only those 
features that are provided by the host CPU. As your kvm cluster needs 
to be made up of homogenous nodes anyway (see System Requirements), it 
might make most sense to use guest.cpu.mode=host-model or 
guest.cpu.mode=host-passthrough


On 5/16/23 15:13, Gary Dixon wrote:


Hi everyone

Other than disabling a Pod – is there a way to prevent live migration 
of VM’s between Pods in ACS ?


We are on version 4.15.2 with Ubuntu 20.04 KVM hosts.  Each Pod 
contains a single cluster of Homogenous hosts – however there are only 
slight differences between the CPU’s on the physical hosts in each 
Cluster. We have the guest.cpu.mode set to host-passthrough but have 
noticed serious issues when a VM is live migrated between specific 
Pods (usually if a VM is started on a cluster with the slightly better 
CPU’s and then live migrated to an older Pod)


We have tried setting the guest.cpu.mode to host-passthrough with 
specific CPU features using the “guest.cpu.features=” and then setting 
all of the host’s CPU flags shown from the output of the lscpu command 
in a space separated list as instructed from ACS documentation – but 
we then are unable to even start a VM – insufficient resources error, 
- if we remove the guest.cpu.features from the agent.properties – then 
we can start a VM again.


It would be good if we had an option or a setting to just not allow 
live migration of VM’s between pods and therefore can only perform a 
‘cold’ migration if we wish to move a VM to another Pod.


Any thoughts on this ?

BR

Gary


Gary Dixon
Senior Technical Consultant
T:  +44 161 537 4990
E: *v* ms@quadris‑support.com
W: www.quadris.co.uk

The information contained in this e-mail from Quadris may be 
confidential and privileged for the private use of the named 
recipient.  The contents of this e-mail may not necessarily represent 
the official views of Quadris.  If you have received this information 
in error you must not copy, distribute or take any action or reliance 
on its contents.  Please destroy any hard copies and delete this message.



--
Regards / Groete

<https://www.namhost.com>     Granwille Strauss  // Senior Systems Admin

*e:* granwi...@namhost.com
*m:* +264 81 323 1260 
*w:* www.namhost.com <https://www.namhost.com/>

<https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>

<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>

Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA



The content of this message is confidential. If you have received it by 
mistake, please inform us by email reply and then delete the message. It 
is forbidden to copy, forward, or in any way reveal the contents of this 
message to anyone without our explicit consent. The integrity and 
security of this email cannot be guaranteed over the Internet. 
Therefore, the sender will not be held liable for any damage caused by 
the message. For our full privacy policy and disclaimers, please go to 
https://www.namhost.com/privacy-policy


Powered by AdSigner 
<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>

smime.p7s
Description: S/MIME Cryptographic Signature

Re: AW: Cloud Usage Question

2023-04-27 Thread Granwille Strauss

Hi Swen

Thank you, yeah documentation seemed to be only API related. Will need
to reach out to a developer then. Thank you.

On 4/27/23 14:30, m...@swen.io wrote:

Hi Granwille,

usage is api only and the documentation is very poor. You can start
with the api docu and the Usage block:
https://cloudstack.apache.org/api/apidocs-4.18/

Do a listUsageTypes call to get all usage types to get a better
understanding what kind of info you are able to get from usage
service. Type 3 and 4 are for Network Usage.

You need to run listUsageRecords api call to get the records. With the
output you should be able to build a microservice to achive your goal.

regards,

Swen

*Von:* Granwille Strauss
*Gesendet:* Donnerstag, 27. April 2023 13:09
*An:* users@cloudstack.apache.org
*Betreff:* Cloud Usage Question

Hi Guys

Are there any online guides, apart from documentation for Cloud usage
or is there a UI for the cloud usage server that I can use? I am
fairly new to Cloudstack, and I have cloudusage installed and running
too but I have no idea how to use it. Based on my findings it seems to
be fully API related, or was there some UI where you can specially
access Cloudusage?

I really want to monitor my VM monthly bandwidth usages to see how
much each uses and to also restrict VMs from going over certain limits
such as 500 GB per month and I am not quite sure how to go about it.

Regards / Groete

Das Bild wurde vom Absender entfernt. <https://www.namhost.com/>

*Granwille Strauss* * // * Senior Systems Admin

*e:*granwi...@namhost.com <mailto:granwi...@namhost.com>
*m:*+264 81 323 1260
*w:*www.namhost.com <https://www.namhost.com/>

Das Bild wurde vom Absender entfernt.
<https://www.facebook.com/namhost>Das Bild wurde vom Absender
entfernt. <https://twitter.com/namhost>Das Bild wurde vom Absender
entfernt. <https://www.instagram.com/namhostinternetservices/>Das Bild
wurde vom Absender entfernt.
<https://www.linkedin.com/company/namhos>Das Bild wurde vom Absender
entfernt. <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>

Das Bild wurde vom Absender entfernt.
<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>

Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA

Das Bild wurde vom Absender entfernt. Powered by AdSigner
<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>

--
Regards / Groete

<https://www.namhost.com> Granwille Strauss // Senior Systems Admin

*e:* granwi...@namhost.com
*m:* +264 81 323 1260
*w:* www.namhost.com <https://www.namhost.com/>

<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>

Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA

The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It
is forbidden to copy, forward, or in any way reveal the contents of this
message to anyone without our explicit consent. The integrity and
security of this email cannot be guaranteed over the Internet.
Therefore, the sender will not be held liable for any damage caused by
the message. For our full privacy policy and disclaimers, please go to
https://www.namhost.com/privacy-policy

smime.p7s
Description: S/MIME Cryptographic Signature

Cloud Usage Question

2023-04-27 Thread Granwille Strauss


Hi Guys

Are there any online guides, apart from documentation for Cloud usage or 
is there a UI for the cloud usage server that I can use? I am fairly new 
to Cloudstack, and I have cloudusage installed and running too but I 
have no idea how to use it. Based on my findings it seems to be fully 
API related, or was there some UI where you can specially access 
Cloudusage?


I really want to monitor my VM monthly bandwidth usages to see how much 
each uses and to also restrict VMs from going over certain limits such 
as 500 GB per month and I am not quite sure how to go about it.


--
Regards / Groete

<https://www.namhost.com>     Granwille Strauss  // Senior Systems Admin

*e:* granwi...@namhost.com
*m:* +264 81 323 1260 
*w:* www.namhost.com <https://www.namhost.com/>

<https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>

<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>

Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA



The content of this message is confidential. If you have received it by 
mistake, please inform us by email reply and then delete the message. It 
is forbidden to copy, forward, or in any way reveal the contents of this 
message to anyone without our explicit consent. The integrity and 
security of this email cannot be guaranteed over the Internet. 
Therefore, the sender will not be held liable for any damage caused by 
the message. For our full privacy policy and disclaimers, please go to 
https://www.namhost.com/privacy-policy


Powered by AdSigner 
<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>

smime.p7s
Description: S/MIME Cryptographic Signature

Re: SSVM routing issue on /23 storage network

2023-02-28 Thread Granwille Strauss

I recently had a similar issue and now that I look at my routing tables, 
storage goes via eth1 and not eth3. Full details on this here: 
https://github.com/apache/cloudstack/issues/7244#issuecomment-1434755523 
This, therefore, also explains why I randomly got this error with my SSVM.


On 2/28/23 11:35, Daan Hoogland wrote:

does sound like a bug Antoine,
I did the network calculation and it seems you are right.
I wonder about the last two routes as well. did you do anything for those?

On Sun, Feb 26, 2023 at 9:47 PM Antoine Boucher
wrote:


Hello,

I'm having a networking issue on SSVMs,  I have the following networks
defined in “Zone 1”.

Management: 10.101.0.0/22
Storage: 10.101.6.0/23

All worked well until we decided to configure new storage devices on
10.101.7.x, the hosts and management server have no issue but the SSVM is
not able to reach it.  Here are the defined interfaces of the SSVM and the
routing table of the SSVM:

1: lo:  mtu 65536 qdisc noqueue state UNKNOWN group
default qlen 1000
 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
 inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: eth0:  mtu 1500 qdisc pfifo_fast state
UP group default qlen 1000
 link/ether 0e:00:a9:fe:72:ec brd ff:ff:ff:ff:ff:ff
 altname enp0s3
 altname ens3
 inet 169.254.114.236/16 brd 169.254.255.255 scope global eth0
valid_lft forever preferred_lft forever
3: eth1:  mtu 1500 qdisc pfifo_fast state
UP group default qlen 1000
 link/ether 1e:00:a1:00:00:06 brd ff:ff:ff:ff:ff:ff
 altname enp0s4
 altname ens4
 inet 10.101.3.205/22 brd 10.101.3.255 scope global eth1
valid_lft forever preferred_lft forever
4: eth2:  mtu 1500 qdisc pfifo_fast state
UP group default qlen 1000
 link/ether 1e:00:6b:00:00:3d brd ff:ff:ff:ff:ff:ff
 altname enp0s5
 altname ens5
 inet 148.59.36.61/28 brd 148.59.36.63 scope global eth2
valid_lft forever preferred_lft forever
5: eth3:  mtu 1500 qdisc pfifo_fast state
UP group default qlen 1000
 link/ether 1e:00:09:00:00:6b brd ff:ff:ff:ff:ff:ff
 altname enp0s6
 altname ens6
 inet 10.101.7.226/23 brd 10.101.7.255 scope global eth3
valid_lft forever preferred_lft forever

default via 148.59.36.49 dev eth2
10.0.0.0/8 via 10.101.0.1 dev eth1
10.91.0.0/23 via 10.101.0.1 dev eth1
10.91.6.0/24 via 10.101.0.1 dev eth1
10.101.0.0/22 dev eth1 proto kernel scope link src 10.101.3.253
10.101.6.0/23 via 10.101.0.1 dev eth1
148.59.36.48/28 dev eth2 proto kernel scope link src 148.59.36.61
169.254.0.0/16 dev eth0 proto kernel scope link src 169.254.232.208
172.16.0.0/12 via 10.101.0.1 dev eth1
192.168.0.0/16 via 10.101.0.1 dev eth1

Why is the routing for 10.101.6.0/23 routing via eth1, shoudn’nt it be
using eth3?  The router seems to be bypassing the routing rules for
10.101.6.x since I see no traffic going through the gateway but I see
traffic going through the gateway when the destination is 10.101.7.x

If I modify the routing for 10.101.6.0/23 to eth3 all is well.

Is this by design?

Regards,
Antoine Boucher




--
Regards / Groete

<https://www.namhost.com>     Granwille Strauss  // Senior Systems Admin

*e:* granwi...@namhost.com
*m:* +264 81 323 1260 
*w:* www.namhost.com <https://www.namhost.com/>

<https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>

<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>

Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA



The content of this message is confidential. If you have received it by 
mistake, please inform us by email reply and then delete the message. It 
is forbidden to copy, forward, or in any way reveal the contents of this 
message to anyone without our explicit consent. The integrity and 
security of this email cannot be guaranteed over the Internet. 
Therefore, the sender will not be held liable for any damage caused by 
the message. For our full privacy policy and disclaimers, please go to 
https://www.namhost.com/privacy-policy


Powered by AdSigner 
<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>

smime.p7s
Description: S/MIME Cryptographic Signature

Re: SSVM Randomly Stopped Connecting


Hi

It seems the SSVM eventually changes to "up" after a few long minutes. 
However, can you confirm when the log shows this entry:


2023-02-15 11:04:15,499 INFO 
[storage.resource.NfsSecondaryStorageResource] (Thread-2:null) Started 
post upload server on port 8210 with 15 workers


Will the status of SSVM remain "connecting" as oppose to being "up"? I 
also made a small tweak in the ip route table by removing an entry in 
SSVM that somehow made it fail to connect to the NFS secondary server. 
It seems SSVM can now fully connect to the NFS server as I no longer see 
failed connections in the cloud.log. But still the agent state remains 
"connecting" until one day its done.


PS the NFS servers do not connect via local network but over the 
internet as its remotely located.


On 2/15/23 12:12, Granwille Strauss wrote:


The value is currently null.

On 2/15/23 12:09, Wei ZHOU wrote:

What is the value of global setting secstorage.nfs.version ?


On Wednesday, 15 February 2023, Granwille Strauss
wrote:


Hi Wei

Here is the cloud.log after restarting cloud service in SSVM:
https://jpst.it/36KHU

I believe these is the error:

2023-02-15 09:35:35,956 INFO  [storage.resource.NfsSecondaryStorageResource]
(agentRequest-Handler-1:null) Determined host NFSServerIP corresponds to IP
NFSServerIP
2023-02-15 09:37:35,372 ERROR [storage.resource.NfsSecondaryStorageResource]
(agentRequest-Handler-1:null) Unable to mount NFSServerIP:/data/secondary
at /mnt/SecStorage/91de6d1c-4c04-359c-82b5-fdcfe4a83da7 due to
2023-02-15 09:37:35,375 ERROR [storage.resource.NfsSecondaryStorageResource]
(agentRequest-Handler-1:null) GetRootDir for nfs://NFSServerIP/data/secondary
failed due to com.cloud.utils.exception.CloudRuntimeException: Unable to
mount NFSServerIP:/data/secondary at 
/mnt/SecStorage/91de6d1c-4c04-359c-82b5-fdcfe4a83da7
due to
com.cloud.utils.exception.CloudRuntimeException: Unable to mount
NFSServerIP:/data/secondary at 
/mnt/SecStorage/91de6d1c-4c04-359c-82b5-fdcfe4a83da7
due to

2023-02-15 09:35:35,936 ERROR [storage.resource.NfsSecondaryStorageResource]
(agentRequest-Handler-2:null) Unable to mount NFSServerIP:/data/secondary
at /mnt/SecStorage/91de6d1c-4c04-359c-82b5-fdcfe4a83da7 due to mount.nfs:
parsing error on 'vers=' option
2023-02-15 09:35:35,936 ERROR [storage.resource.NfsSecondaryStorageResource]
(agentRequest-Handler-2:null) GetRootDir for nfs://NFSServerIP/data/secondary
failed due to com.cloud.utils.exception.CloudRuntimeException: Unable to
mount NFSServerIP:/data/secondary at 
/mnt/SecStorage/91de6d1c-4c04-359c-82b5-fdcfe4a83da7
due to mount.nfs: parsing error on 'vers=' option
com.cloud.utils.exception.CloudRuntimeException: Unable to mount
NFSServerIP:/data/secondary at 
/mnt/SecStorage/91de6d1c-4c04-359c-82b5-fdcfe4a83da7
due to mount.nfs: parsing error on 'vers=' option

But I have no idea what any of it means?
On 2/15/23 11:34, Wei ZHOU wrote:

Hi,

Can you restart "cloud" service in SSVM and upload some logs in
/var/log/cloud.log ?

When the cloud service is started, it will check templates on secondary
storage, which takes time if there are many templates.

-Wei


On Wednesday, 15 February 2023, Granwille Strauss  

wrote:


Hi Wei

I fully opened port 2049 now on the NFS server and ran telnet from SSVM,
after rebooting it:

root@s-141-VM:~# telnet NFSSERVERIP 2049
Trying NFSSERVERIP...
Connected to NFSSERVERIP.
Escape character is '^]'.
^]

However, status remains "connecting".
On 2/15/23 11:16, Wei ZHOU wrote:

Right.

I have seen this issue recently, which is because SSVM cannot connect to
secondary storage.

-Wei

On Wednesday, 15 February 2023, Kiran Chavala  


wrote:


HI

Are you able to ssh to the ssvm, If yes can you please execute health
check script

/usr/local/cloud/systemvm/ssvm-check.sh

Also are you able to connect to the management server on port 8250

from the ssvm

telnet  8250

Also could you please share the /var/log/cloud.out log it will be helpful

Regards
Kiran






From: Granwille Strauss
Sent: Wednesday, February 15, 2023 14:15
To:users@cloudstack.apache.org
Cc: Harikrishna Patnala
Subject: Re: SSVM Randomly Stopped Connecting


Hi Harikrishna

I have done so, status remains showing "connecting". And on the dashboard
I can see my secondary storage usage, which means its supposed to be
connected. However, if I reboot the cloudstack-management, it will show 0
again. See attached.

May I add it most likely was not "random" as I mentioned previously. I did
update my opensense gateway, which resulted in a reboot. However,
connections and routing has been perfect again. And then I did also run yum
update on the kvm host where this SSVM is tied to. And the only error I
experienced was that mysql-connect-python required python 2.7, which I
installed to resolve the dependency issue, but python3.6 is still active on
the host as well.

On 2/

Re: SSVM Randomly Stopped Connecting


The value is currently null.

On 2/15/23 12:09, Wei ZHOU wrote:

What is the value of global setting secstorage.nfs.version ?


On Wednesday, 15 February 2023, Granwille Strauss
wrote:


Hi Wei

Here is the cloud.log after restarting cloud service in SSVM:
https://jpst.it/36KHU

I believe these is the error:

2023-02-15 09:35:35,956 INFO  [storage.resource.NfsSecondaryStorageResource]
(agentRequest-Handler-1:null) Determined host NFSServerIP corresponds to IP
NFSServerIP
2023-02-15 09:37:35,372 ERROR [storage.resource.NfsSecondaryStorageResource]
(agentRequest-Handler-1:null) Unable to mount NFSServerIP:/data/secondary
at /mnt/SecStorage/91de6d1c-4c04-359c-82b5-fdcfe4a83da7 due to
2023-02-15 09:37:35,375 ERROR [storage.resource.NfsSecondaryStorageResource]
(agentRequest-Handler-1:null) GetRootDir for nfs://NFSServerIP/data/secondary
failed due to com.cloud.utils.exception.CloudRuntimeException: Unable to
mount NFSServerIP:/data/secondary at 
/mnt/SecStorage/91de6d1c-4c04-359c-82b5-fdcfe4a83da7
due to
com.cloud.utils.exception.CloudRuntimeException: Unable to mount
NFSServerIP:/data/secondary at 
/mnt/SecStorage/91de6d1c-4c04-359c-82b5-fdcfe4a83da7
due to

2023-02-15 09:35:35,936 ERROR [storage.resource.NfsSecondaryStorageResource]
(agentRequest-Handler-2:null) Unable to mount NFSServerIP:/data/secondary
at /mnt/SecStorage/91de6d1c-4c04-359c-82b5-fdcfe4a83da7 due to mount.nfs:
parsing error on 'vers=' option
2023-02-15 09:35:35,936 ERROR [storage.resource.NfsSecondaryStorageResource]
(agentRequest-Handler-2:null) GetRootDir for nfs://NFSServerIP/data/secondary
failed due to com.cloud.utils.exception.CloudRuntimeException: Unable to
mount NFSServerIP:/data/secondary at 
/mnt/SecStorage/91de6d1c-4c04-359c-82b5-fdcfe4a83da7
due to mount.nfs: parsing error on 'vers=' option
com.cloud.utils.exception.CloudRuntimeException: Unable to mount
NFSServerIP:/data/secondary at 
/mnt/SecStorage/91de6d1c-4c04-359c-82b5-fdcfe4a83da7
due to mount.nfs: parsing error on 'vers=' option

But I have no idea what any of it means?
On 2/15/23 11:34, Wei ZHOU wrote:

Hi,

Can you restart "cloud" service in SSVM and upload some logs in
/var/log/cloud.log ?

When the cloud service is started, it will check templates on secondary
storage, which takes time if there are many templates.

-Wei


On Wednesday, 15 February 2023, Granwille Strauss  

wrote:


Hi Wei

I fully opened port 2049 now on the NFS server and ran telnet from SSVM,
after rebooting it:

root@s-141-VM:~# telnet NFSSERVERIP 2049
Trying NFSSERVERIP...
Connected to NFSSERVERIP.
Escape character is '^]'.
^]

However, status remains "connecting".
On 2/15/23 11:16, Wei ZHOU wrote:

Right.

I have seen this issue recently, which is because SSVM cannot connect to
secondary storage.

-Wei

On Wednesday, 15 February 2023, Kiran Chavala  


wrote:


HI

Are you able to ssh to the ssvm, If yes can you please execute health
check script

/usr/local/cloud/systemvm/ssvm-check.sh

Also are you able to connect to the management server on port 8250

from the ssvm

telnet  8250

Also could you please share the /var/log/cloud.out log it will be helpful

Regards
Kiran


____



From: Granwille Strauss
Sent: Wednesday, February 15, 2023 14:15
To:users@cloudstack.apache.org
Cc: Harikrishna Patnala
Subject: Re: SSVM Randomly Stopped Connecting


Hi Harikrishna

I have done so, status remains showing "connecting". And on the dashboard
I can see my secondary storage usage, which means its supposed to be
connected. However, if I reboot the cloudstack-management, it will show 0
again. See attached.

May I add it most likely was not "random" as I mentioned previously. I did
update my opensense gateway, which resulted in a reboot. However,
connections and routing has been perfect again. And then I did also run yum
update on the kvm host where this SSVM is tied to. And the only error I
experienced was that mysql-connect-python required python 2.7, which I
installed to resolve the dependency issue, but python3.6 is still active on
the host as well.

On 2/15/23 10:30, Harikrishna Patnala wrote:

Hi,

Have you tried destroying the SSVM, it will be recreated automatically.
Please try that, it may solve the issue.

Thanks,
Harikrishna

From: Granwille Strauss


Date: Wednesday, 15 February 2023 at 1:46 PM
To:users@cloudstack.apache.org<mailto:users@cloudstack.apache.org>
  
<mailto:users@cloudstack.apache.org>
  
Subject: SSVM Randomly Stopped Connecting

Hi Guys

As the error suggests, this morning my SSVM just randomly stopped
connecting and agent status remains in "Connecting" mode. ConsolveVM seems
perfectly fine. I did the regular checks in SSVM and it all seems fine, see
attached screenshot. I tried force rebooting the SSVM but still nothing.
The host kvm server is 100% connected as one typically gets the "host
unreachable" error when something like this, but not

Re: SSVM Randomly Stopped Connecting

False alarm, I restarted cloudstack-management service and now its 
"connecting" again I I now see also the following error:


2023-02-15 09:57:21,782 INFO 
[storage.resource.NfsSecondaryStorageResource] 
(agentRequest-Handler-3:null) Determined host 192.168.50.2 corresponds 
to IP 192.168.50.2
2023-02-15 09:57:21,783 WARN  [cloud.agent.Agent] 
(agentRequest-Handler-2:null) Caught:
com.cloud.utils.exception.CloudRuntimeException: GetRootDir for 
nfs://NFSServerIP/data/secondary failed due to 
com.cloud.utils.exception.CloudRuntimeException: Unable to mount 
NFSServerIP:/data/secondary at 
/mnt/SecStorage/91de6d1c-4c04-359c-82b5-fdcfe4a83da7 due to mount.nfs: 
Connection timed out


On 2/15/23 11:54, Granwille Strauss wrote:


U

I have ABSOLUTELY no idea what just happened now, but the status has 
now magically changed to "Up" and I have not touched a single thing on 
the servers since I sent the cloud.log that Wei requested.


On 2/15/23 11:46, Granwille Strauss wrote:


Hi Wei

Here is the cloud.log after restarting cloud service in SSVM: 
https://jpst.it/36KHU


I believe these is the error:

2023-02-15 09:35:35,956 INFO 
[storage.resource.NfsSecondaryStorageResource] 
(agentRequest-Handler-1:null) Determined host NFSServerIP 
corresponds to IP NFSServerIP
2023-02-15 09:37:35,372 ERROR 
[storage.resource.NfsSecondaryStorageResource] 
(agentRequest-Handler-1:null) Unable to mount 
NFSServerIP:/data/secondary at 
/mnt/SecStorage/91de6d1c-4c04-359c-82b5-fdcfe4a83da7 due to
2023-02-15 09:37:35,375 ERROR 
[storage.resource.NfsSecondaryStorageResource] 
(agentRequest-Handler-1:null) GetRootDir for 
nfs://NFSServerIP/data/secondary failed due to 
com.cloud.utils.exception.CloudRuntimeException: Unable to mount 
NFSServerIP:/data/secondary at 
/mnt/SecStorage/91de6d1c-4c04-359c-82b5-fdcfe4a83da7 due to
com.cloud.utils.exception.CloudRuntimeException: Unable to mount 
NFSServerIP:/data/secondary at 
/mnt/SecStorage/91de6d1c-4c04-359c-82b5-fdcfe4a83da7 due to


2023-02-15 09:35:35,936 ERROR 
[storage.resource.NfsSecondaryStorageResource] 
(agentRequest-Handler-2:null) Unable to mount 
NFSServerIP:/data/secondary at 
/mnt/SecStorage/91de6d1c-4c04-359c-82b5-fdcfe4a83da7 due to 
mount.nfs: parsing error on 'vers=' option
2023-02-15 09:35:35,936 ERROR 
[storage.resource.NfsSecondaryStorageResource] 
(agentRequest-Handler-2:null) GetRootDir for 
nfs://NFSServerIP/data/secondary failed due to 
com.cloud.utils.exception.CloudRuntimeException: Unable to mount 
NFSServerIP:/data/secondary at 
/mnt/SecStorage/91de6d1c-4c04-359c-82b5-fdcfe4a83da7 due to 
mount.nfs: parsing error on 'vers=' option
com.cloud.utils.exception.CloudRuntimeException: Unable to mount 
NFSServerIP:/data/secondary at 
/mnt/SecStorage/91de6d1c-4c04-359c-82b5-fdcfe4a83da7 due to 
mount.nfs: parsing error on 'vers=' option


But I have no idea what any of it means?

On 2/15/23 11:34, Wei ZHOU wrote:

Hi,

Can you restart "cloud" service in SSVM and upload some logs in
/var/log/cloud.log ?

When the cloud service is started, it will check templates on secondary
storage, which takes time if there are many templates.

-Wei


On Wednesday, 15 February 2023, Granwille Strauss
wrote:


Hi Wei

I fully opened port 2049 now on the NFS server and ran telnet from SSVM,
after rebooting it:

root@s-141-VM:~# telnet NFSSERVERIP 2049
Trying NFSSERVERIP...
Connected to NFSSERVERIP.
Escape character is '^]'.
^]

However, status remains "connecting".
On 2/15/23 11:16, Wei ZHOU wrote:

Right.

I have seen this issue recently, which is because SSVM cannot connect to
secondary storage.

-Wei

On Wednesday, 15 February 2023, Kiran Chavala  

wrote:


HI

Are you able to ssh to the ssvm, If yes can you please execute health
check script

/usr/local/cloud/systemvm/ssvm-check.sh

Also are you able to connect to the management server on port 8250

from the ssvm

telnet  8250

Also could you please share the /var/log/cloud.out log it will be helpful

Regards
Kiran


________



From: Granwille Strauss
Sent: Wednesday, February 15, 2023 14:15
To:users@cloudstack.apache.org
Cc: Harikrishna Patnala
Subject: Re: SSVM Randomly Stopped Connecting


Hi Harikrishna

I have done so, status remains showing "connecting". And on the dashboard
I can see my secondary storage usage, which means its supposed to be
connected. However, if I reboot the cloudstack-management, it will show 0
again. See attached.

May I add it most likely was not "random" as I mentioned previously. I did
update my opensense gateway, which resulted in a reboot. However,
connections and routing has been perfect again. And then I did also run yum
update on the kvm host where this SSVM is tied to. And the only error I
experienced was that mysql-connect-python required python 2.7, which I
installed to resolve the dependency issue, but python3.6 is still active on
the host as well.

On 2/15/23 10:30, Harikrishna Patnala wrote:

H

Re: SSVM Randomly Stopped Connecting


U

I have ABSOLUTELY no idea what just happened now, but the status has now 
magically changed to "Up" and I have not touched a single thing on the 
servers since I sent the cloud.log that Wei requested.


On 2/15/23 11:46, Granwille Strauss wrote:


Hi Wei

Here is the cloud.log after restarting cloud service in SSVM: 
https://jpst.it/36KHU


I believe these is the error:

2023-02-15 09:35:35,956 INFO 
[storage.resource.NfsSecondaryStorageResource] 
(agentRequest-Handler-1:null) Determined host NFSServerIP corresponds 
to IP NFSServerIP
2023-02-15 09:37:35,372 ERROR 
[storage.resource.NfsSecondaryStorageResource] 
(agentRequest-Handler-1:null) Unable to mount 
NFSServerIP:/data/secondary at 
/mnt/SecStorage/91de6d1c-4c04-359c-82b5-fdcfe4a83da7 due to
2023-02-15 09:37:35,375 ERROR 
[storage.resource.NfsSecondaryStorageResource] 
(agentRequest-Handler-1:null) GetRootDir for 
nfs://NFSServerIP/data/secondary failed due to 
com.cloud.utils.exception.CloudRuntimeException: Unable to mount 
NFSServerIP:/data/secondary at 
/mnt/SecStorage/91de6d1c-4c04-359c-82b5-fdcfe4a83da7 due to
com.cloud.utils.exception.CloudRuntimeException: Unable to mount 
NFSServerIP:/data/secondary at 
/mnt/SecStorage/91de6d1c-4c04-359c-82b5-fdcfe4a83da7 due to


2023-02-15 09:35:35,936 ERROR 
[storage.resource.NfsSecondaryStorageResource] 
(agentRequest-Handler-2:null) Unable to mount 
NFSServerIP:/data/secondary at 
/mnt/SecStorage/91de6d1c-4c04-359c-82b5-fdcfe4a83da7 due to 
mount.nfs: parsing error on 'vers=' option
2023-02-15 09:35:35,936 ERROR 
[storage.resource.NfsSecondaryStorageResource] 
(agentRequest-Handler-2:null) GetRootDir for 
nfs://NFSServerIP/data/secondary failed due to 
com.cloud.utils.exception.CloudRuntimeException: Unable to mount 
NFSServerIP:/data/secondary at 
/mnt/SecStorage/91de6d1c-4c04-359c-82b5-fdcfe4a83da7 due to 
mount.nfs: parsing error on 'vers=' option
com.cloud.utils.exception.CloudRuntimeException: Unable to mount 
NFSServerIP:/data/secondary at 
/mnt/SecStorage/91de6d1c-4c04-359c-82b5-fdcfe4a83da7 due to 
mount.nfs: parsing error on 'vers=' option


But I have no idea what any of it means?

On 2/15/23 11:34, Wei ZHOU wrote:

Hi,

Can you restart "cloud" service in SSVM and upload some logs in
/var/log/cloud.log ?

When the cloud service is started, it will check templates on secondary
storage, which takes time if there are many templates.

-Wei


On Wednesday, 15 February 2023, Granwille Strauss
wrote:


Hi Wei

I fully opened port 2049 now on the NFS server and ran telnet from SSVM,
after rebooting it:

root@s-141-VM:~# telnet NFSSERVERIP 2049
Trying NFSSERVERIP...
Connected to NFSSERVERIP.
Escape character is '^]'.
^]

However, status remains "connecting".
On 2/15/23 11:16, Wei ZHOU wrote:

Right.

I have seen this issue recently, which is because SSVM cannot connect to
secondary storage.

-Wei

On Wednesday, 15 February 2023, Kiran Chavala  

wrote:


HI

Are you able to ssh to the ssvm, If yes can you please execute health
check script

/usr/local/cloud/systemvm/ssvm-check.sh

Also are you able to connect to the management server on port 8250

from the ssvm

telnet  8250

Also could you please share the /var/log/cloud.out log it will be helpful

Regards
Kiran


________



From: Granwille Strauss
Sent: Wednesday, February 15, 2023 14:15
To:users@cloudstack.apache.org
Cc: Harikrishna Patnala
Subject: Re: SSVM Randomly Stopped Connecting


Hi Harikrishna

I have done so, status remains showing "connecting". And on the dashboard
I can see my secondary storage usage, which means its supposed to be
connected. However, if I reboot the cloudstack-management, it will show 0
again. See attached.

May I add it most likely was not "random" as I mentioned previously. I did
update my opensense gateway, which resulted in a reboot. However,
connections and routing has been perfect again. And then I did also run yum
update on the kvm host where this SSVM is tied to. And the only error I
experienced was that mysql-connect-python required python 2.7, which I
installed to resolve the dependency issue, but python3.6 is still active on
the host as well.

On 2/15/23 10:30, Harikrishna Patnala wrote:

Hi,

Have you tried destroying the SSVM, it will be recreated automatically.
Please try that, it may solve the issue.

Thanks,
Harikrishna

From: Granwille Strauss  
  

Date: Wednesday, 15 February 2023 at 1:46 PM
To:users@cloudstack.apache.org<mailto:users@cloudstack.apache.org>   
 <mailto:users@cloudstack.apache.org>  

Subject: SSVM Randomly Stopped Connecting

Hi Guys

As the error suggests, this morning my SSVM just randomly stopped
connecting and agent status remains in "Connecting" mode. ConsolveVM seems
perfectly fine. I did the regular checks in SSVM and it all seems fine, see
attached screenshot. I tried force rebooting the SSVM but still nothing.
The host kvm server is 100% connected as one typically gets

Re: SSVM Randomly Stopped Connecting


Hi Wei

Here is the cloud.log after restarting cloud service in SSVM: 
https://jpst.it/36KHU


I believe these is the error:

2023-02-15 09:35:35,956 INFO 
[storage.resource.NfsSecondaryStorageResource] 
(agentRequest-Handler-1:null) Determined host NFSServerIP corresponds 
to IP NFSServerIP
2023-02-15 09:37:35,372 ERROR 
[storage.resource.NfsSecondaryStorageResource] 
(agentRequest-Handler-1:null) Unable to mount 
NFSServerIP:/data/secondary at 
/mnt/SecStorage/91de6d1c-4c04-359c-82b5-fdcfe4a83da7 due to
2023-02-15 09:37:35,375 ERROR 
[storage.resource.NfsSecondaryStorageResource] 
(agentRequest-Handler-1:null) GetRootDir for 
nfs://NFSServerIP/data/secondary failed due to 
com.cloud.utils.exception.CloudRuntimeException: Unable to mount 
NFSServerIP:/data/secondary at 
/mnt/SecStorage/91de6d1c-4c04-359c-82b5-fdcfe4a83da7 due to
com.cloud.utils.exception.CloudRuntimeException: Unable to mount 
NFSServerIP:/data/secondary at 
/mnt/SecStorage/91de6d1c-4c04-359c-82b5-fdcfe4a83da7 due to


2023-02-15 09:35:35,936 ERROR 
[storage.resource.NfsSecondaryStorageResource] 
(agentRequest-Handler-2:null) Unable to mount 
NFSServerIP:/data/secondary at 
/mnt/SecStorage/91de6d1c-4c04-359c-82b5-fdcfe4a83da7 due to mount.nfs: 
parsing error on 'vers=' option
2023-02-15 09:35:35,936 ERROR 
[storage.resource.NfsSecondaryStorageResource] 
(agentRequest-Handler-2:null) GetRootDir for 
nfs://NFSServerIP/data/secondary failed due to 
com.cloud.utils.exception.CloudRuntimeException: Unable to mount 
NFSServerIP:/data/secondary at 
/mnt/SecStorage/91de6d1c-4c04-359c-82b5-fdcfe4a83da7 due to mount.nfs: 
parsing error on 'vers=' option
com.cloud.utils.exception.CloudRuntimeException: Unable to mount 
NFSServerIP:/data/secondary at 
/mnt/SecStorage/91de6d1c-4c04-359c-82b5-fdcfe4a83da7 due to mount.nfs: 
parsing error on 'vers=' option


But I have no idea what any of it means?

On 2/15/23 11:34, Wei ZHOU wrote:

Hi,

Can you restart "cloud" service in SSVM and upload some logs in
/var/log/cloud.log ?

When the cloud service is started, it will check templates on secondary
storage, which takes time if there are many templates.

-Wei


On Wednesday, 15 February 2023, Granwille Strauss
wrote:


Hi Wei

I fully opened port 2049 now on the NFS server and ran telnet from SSVM,
after rebooting it:

root@s-141-VM:~# telnet NFSSERVERIP 2049
Trying NFSSERVERIP...
Connected to NFSSERVERIP.
Escape character is '^]'.
^]

However, status remains "connecting".
On 2/15/23 11:16, Wei ZHOU wrote:

Right.

I have seen this issue recently, which is because SSVM cannot connect to
secondary storage.

-Wei

On Wednesday, 15 February 2023, Kiran Chavala  

wrote:


HI

Are you able to ssh to the ssvm, If yes can you please execute health
check script

/usr/local/cloud/systemvm/ssvm-check.sh

Also are you able to connect to the management server on port 8250

from the ssvm

telnet  8250

Also could you please share the /var/log/cloud.out log it will be helpful

Regards
Kiran


____



From: Granwille Strauss
Sent: Wednesday, February 15, 2023 14:15
To:users@cloudstack.apache.org
Cc: Harikrishna Patnala
Subject: Re: SSVM Randomly Stopped Connecting


Hi Harikrishna

I have done so, status remains showing "connecting". And on the dashboard
I can see my secondary storage usage, which means its supposed to be
connected. However, if I reboot the cloudstack-management, it will show 0
again. See attached.

May I add it most likely was not "random" as I mentioned previously. I did
update my opensense gateway, which resulted in a reboot. However,
connections and routing has been perfect again. And then I did also run yum
update on the kvm host where this SSVM is tied to. And the only error I
experienced was that mysql-connect-python required python 2.7, which I
installed to resolve the dependency issue, but python3.6 is still active on
the host as well.

On 2/15/23 10:30, Harikrishna Patnala wrote:

Hi,

Have you tried destroying the SSVM, it will be recreated automatically.
Please try that, it may solve the issue.

Thanks,
Harikrishna

From: Granwille Strauss  
  

Date: Wednesday, 15 February 2023 at 1:46 PM
To:users@cloudstack.apache.org<mailto:users@cloudstack.apache.org>   
 <mailto:users@cloudstack.apache.org>  

Subject: SSVM Randomly Stopped Connecting

Hi Guys

As the error suggests, this morning my SSVM just randomly stopped
connecting and agent status remains in "Connecting" mode. ConsolveVM seems
perfectly fine. I did the regular checks in SSVM and it all seems fine, see
attached screenshot. I tried force rebooting the SSVM but still nothing.
The host kvm server is 100% connected as one typically gets the "host
unreachable" error when something like this, but not this time.

I SSH into the SSVM and ran the 'reboot' command and tailed the management
log, this is what it throws:https://jpst.it/36K93

I do not see any errors there either. Any

Re: SSVM Randomly Stopped Connecting


Hi Wei

I fully opened port 2049 now on the NFS server and ran telnet from SSVM, 
after rebooting it:



root@s-141-VM:~# telnet NFSSERVERIP 2049
Trying NFSSERVERIP...
Connected to NFSSERVERIP.
Escape character is '^]'.
^]


However, status remains "connecting".

On 2/15/23 11:16, Wei ZHOU wrote:

Right.

I have seen this issue recently, which is because SSVM cannot connect to
secondary storage.

-Wei

On Wednesday, 15 February 2023, Kiran Chavala
wrote:


HI

Are you able to ssh to the ssvm, If yes can you please execute health
check script

/usr/local/cloud/systemvm/ssvm-check.sh

Also are you able to connect to the management server on port 8250

from the ssvm

telnet  8250

Also could you please share the /var/log/cloud.out log it will be helpful

Regards
Kiran






From: Granwille Strauss
Sent: Wednesday, February 15, 2023 14:15
To:users@cloudstack.apache.org
Cc: Harikrishna Patnala
Subject: Re: SSVM Randomly Stopped Connecting


Hi Harikrishna

I have done so, status remains showing "connecting". And on the dashboard
I can see my secondary storage usage, which means its supposed to be
connected. However, if I reboot the cloudstack-management, it will show 0
again. See attached.

May I add it most likely was not "random" as I mentioned previously. I did
update my opensense gateway, which resulted in a reboot. However,
connections and routing has been perfect again. And then I did also run yum
update on the kvm host where this SSVM is tied to. And the only error I
experienced was that mysql-connect-python required python 2.7, which I
installed to resolve the dependency issue, but python3.6 is still active on
the host as well.

On 2/15/23 10:30, Harikrishna Patnala wrote:

Hi,

Have you tried destroying the SSVM, it will be recreated automatically.
Please try that, it may solve the issue.

Thanks,
Harikrishna

From: Granwille Strauss
Date: Wednesday, 15 February 2023 at 1:46 PM
To:users@cloudstack.apache.org<mailto:users@cloudstack.apache.org>  <
users@cloudstack.apache.org><mailto:users@cloudstack.apache.org>
Subject: SSVM Randomly Stopped Connecting

Hi Guys

As the error suggests, this morning my SSVM just randomly stopped
connecting and agent status remains in "Connecting" mode. ConsolveVM seems
perfectly fine. I did the regular checks in SSVM and it all seems fine, see
attached screenshot. I tried force rebooting the SSVM but still nothing.
The host kvm server is 100% connected as one typically gets the "host
unreachable" error when something like this, but not this time.

I SSH into the SSVM and ran the 'reboot' command and tailed the management
log, this is what it throws:https://jpst.it/36K93

I do not see any errors there either. Any suggestions that can help with
this, please?
--

Regards / Groete
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/
621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/
cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>< 
https://www.namhost.com/>
Granwille Strauss  //  Senior Systems Admin

e:granwi...@namhost.com<mailto:granwi...@namhost.com>ranwi...@namhost.com><mailto:granwi...@namhost.com>

m: +264 81 323 1260
w:www.namhost.com<http://www.namhost.com><https://www.namhost.com/>< 
https://www.namhost.com/>


[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/
621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/
9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[
https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/
621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/
85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://
www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/
social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-
5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/

<https://www.instagram.com/namhostinternetservices/>[http

s://www.adsigner.com/v1/s/631091998d4670001fe43ec2/
621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/
3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos

[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/

621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/
3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]s://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>


[https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/
621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/ 
631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>< 
https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/ 
621c9b76c140bb001ed0f818/banner>

Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA

The content of this message is confidential. If you have received it by
mistake, pl

Re: Updating storage addresses

2023-02-14 Thread Granwille Strauss

Hi Jeremy

The best would be to add both primary and secondary storages, perhaps
look at giving new NFS paths as well, since it seems to sound like the
same storage devices. Once you have added them, you an use the
"Complete" migration policy option for secondary storage to migrate data
between each other. And for primary storage, you can move the volumes of
your instances to the new storage via UI. Keep in mind you may want to
power down the instance before doing so.

Once done, delete your old secondary storage and maybe try rebooting
ssvm and consolevm (May need to destroy but reboot first). You can then
also delete old primary storage once you have confirmed all volumes have
been moved accordingly.

On 2/14/23 11:00, Jeremy Hansen wrote:
I have to update the primary and secondary storage address just to a
new IP for NFS. I can’t seem to find a way to do this through the
interface. Do I have to do an update to the db? Any notes on how to
do that?

Thanks
-jeremy

--
Regards / Groete

<https://www.namhost.com> Granwille Strauss // Senior Systems Admin

*e:* granwi...@namhost.com
*m:* +264 81 323 1260
*w:* www.namhost.com <https://www.namhost.com/>

<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>

Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA

The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It
is forbidden to copy, forward, or in any way reveal the contents of this
message to anyone without our explicit consent. The integrity and
security of this email cannot be guaranteed over the Internet.
Therefore, the sender will not be held liable for any damage caused by
the message. For our full privacy policy and disclaimers, please go to
https://www.namhost.com/privacy-policy

smime.p7s
Description: S/MIME Cryptographic Signature

Re: Compute Offerings

Hi Wei

Nice to hear from you. Awesome stuff. I will test this out before doing
it via production. Thank you again.

On 2/13/23 17:53, Wei ZHOU wrote:

Hi,

There is not an API to update existing service offerings to HA enabled.

You can do it via manual DB changes

update service_offering set ha_enabled=1 where id=
update vm_instance set ha_enabled=1 where service_offering_id=

-Wei

On Mon, 13 Feb 2023 at 16:48, Granwille Strauss
wrote:

Hi Guys

Another noob question, how does one go about updating an existing compute
offering. I have custom unconstrained offering in place which most our VMs
use. However, I have recently discovered HA and tested for my VMs now I
want to either update an existing offering to affect all existing VM or is
it possible to assign a new offering to existing VMs? Which approach is
possible and or recommended?
--
Regards / Groete

<https://www.namhost.com> Granwille Strauss // Senior Systems Admin

*e:*granwi...@namhost.com
*m:* +264 81 323 1260 <+264813231260>
*w:*www.namhost.com

<https://www.facebook.com/namhost> <https://twitter.com/namhost>
<https://www.instagram.com/namhostinternetservices/>
<https://www.linkedin.com/company/namhos>
<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>

<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>

Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA

The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It is
forbidden to copy, forward, or in any way reveal the contents of this
message to anyone without our explicit consent. The integrity and security
of this email cannot be guaranteed over the Internet. Therefore, the sender
will not be held liable for any damage caused by the message. For our full
privacy policy and disclaimers, please go to
https://www.namhost.com/privacy-policy

[image: Powered by AdSigner]
<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>

--
Regards / Groete

<https://www.namhost.com> Granwille Strauss // Senior Systems Admin

*e:* granwi...@namhost.com
*m:* +264 81 323 1260
*w:* www.namhost.com <https://www.namhost.com/>

<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>

Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA

The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It
is forbidden to copy, forward, or in any way reveal the contents of this
message to anyone without our explicit consent. The integrity and
security of this email cannot be guaranteed over the Internet.
Therefore, the sender will not be held liable for any damage caused by
the message. For our full privacy policy and disclaimers, please go to
https://www.namhost.com/privacy-policy

smime.p7s
Description: S/MIME Cryptographic Signature

Compute Offerings


Hi Guys

Another noob question, how does one go about updating an existing 
compute offering. I have custom unconstrained offering in place which 
most our VMs use. However, I have recently discovered HA and tested for 
my VMs now I want to either update an existing offering to affect all 
existing VM or is it possible to assign a new offering to existing VMs? 
Which approach is possible and or recommended?


--
Regards / Groete

<https://www.namhost.com>     Granwille Strauss  // Senior Systems Admin

*e:* granwi...@namhost.com
*m:* +264 81 323 1260 
*w:* www.namhost.com <https://www.namhost.com/>

<https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>

<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>

Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA



The content of this message is confidential. If you have received it by 
mistake, please inform us by email reply and then delete the message. It 
is forbidden to copy, forward, or in any way reveal the contents of this 
message to anyone without our explicit consent. The integrity and 
security of this email cannot be guaranteed over the Internet. 
Therefore, the sender will not be held liable for any damage caused by 
the message. For our full privacy policy and disclaimers, please go to 
https://www.namhost.com/privacy-policy


Powered by AdSigner 
<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>

smime.p7s
Description: S/MIME Cryptographic Signature

Re: Removing a zone


Hi

Thank you, we were able to boil down the issue to volumes that were in a 
"Destroy" state in the volume table. When cleaned out, I managed to 
delete the zone.


However, I am now having a slight an anxiety attack since you said 
removing ISOs this method is risky. I deleted the ISOs "xs-tools.iso" 
and "vmware-tools.iso", as per attached screenshot sent initially, which 
were automatically installed when I installed Cloudstack Management the 
first time. Are they needed in the existing zone? I have not made use of 
them, at least from what I can recall, unless there's some automated 
process that uses them and need them active?


On 2/13/23 14:05, Andrija Panic wrote:

Removing ISOs as such might be risky, assuming you have other Zones which
you don't want to affect.

I would suggest that you check the DB - "storage_pool" table, and ensure
that all pools in that table, that have the zone_id= are
marked as Staus=Maintenance and have the Removed column with a date set.
Similar with the "image_store" table (Secondary Storage)


On Mon, 13 Feb 2023 at 11:36, Granwille Strauss
  wrote:


Thank you,

After removing ISO via DB method you provided, error still exists so you
were right, it was not the ISO causing this. Here's the log entry when I
attempt to delete zone:

2023-02-13 12:31:48,258 ERROR [c.c.a.ApiServer]
(qtp262366552-454:ctx-f498ef20 ctx-6c40cbcd) (logid:dd5b8f8e) unhandled
exception executing api command: [Ljava.lang.String;@38b34249
com.cloud.utils.exception.CloudRuntimeException: The zone cannot be
deleted because there are storage volumes in this zone.
 at
com.cloud.configuration.ConfigurationManagerImpl.checkIfZoneIsDeletable(ConfigurationManagerImpl.java:2234)
 at
com.cloud.configuration.ConfigurationManagerImpl.deleteZone(ConfigurationManagerImpl.java:2380)
 at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
 at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.base/java.lang.reflect.Method.invoke(Method.java:566)
 at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
 at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
 at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
 at
org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
 at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
 at
com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
 at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
 at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
 at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
 at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
 at com.sun.proxy.$Proxy122.deleteZone(Unknown Source)
 at
org.apache.cloudstack.api.command.admin.zone.DeleteZoneCmd.execute(DeleteZoneCmd.java:72)
 at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:163)
 at com.cloud.api.ApiServer.queueCommand(ApiServer.java:776)
 at com.cloud.api.ApiServer.handleRequest(ApiServer.java:600)
 at
com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:327)
 at com.cloud.api.ApiServlet$1.run(ApiServlet.java:145)
 at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
 at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
 at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
 at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:142)
 at com.cloud.api.ApiServlet.doGet(ApiServlet.java:96)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:645)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
 at
org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
 at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
 at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550)
 at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
 at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
 at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
 at
org.eclipse.jetty.server.handl

Re: Removing a zone


Hi Kiran


No, all running instances are on a different zone and none gives me the 
option to delete. However, I checked the volumes table now and found 
three "Destroy" instances, and used your SQL query to clear them out, 
which then successfully allowed me to delete the zone. Thank you very 
much for your speedy assistance in this regard. However, I believe this 
is a known bug, which stems from 2015 
https://issues.apache.org/jira/browse/CLOUDSTACK-9090 May be worth while 
for the dev's to look at.



On 2/13/23 12:44, Kiran Chavala wrote:

The logs are pointing that there are still active volume in the zone

From the UI are you able to destroy the data volumes or root volumes 
attached to any running instances




DB hack

update volumes set removed=now(), state=Destroy where id="";



Regards
Kiran


----
*From:* Granwille Strauss
*Sent:* Monday, February 13, 2023 16:05
*To:* users@cloudstack.apache.org
*Cc:* Kiran Chavala
*Subject:* Re: Removing a zone

Thank you,

After removing ISO via DB method you provided, error still exists so 
you were right, it was not the ISO causing this. Here's the log entry 
when I attempt to delete zone:


2023-02-13 12:31:48,258 ERROR [c.c.a.ApiServer] 
(qtp262366552-454:ctx-f498ef20 ctx-6c40cbcd) (logid:dd5b8f8e) 
unhandled exception executing api command: [Ljava.lang.String;@38b34249
com.cloud.utils.exception.CloudRuntimeException: The zone cannot be 
deleted because there are storage volumes in this zone.
    at 
com.cloud.configuration.ConfigurationManagerImpl.checkIfZoneIsDeletable(ConfigurationManagerImpl.java:2234)
    at 
com.cloud.configuration.ConfigurationManagerImpl.deleteZone(ConfigurationManagerImpl.java:2380)
    at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
    at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at 
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
    at 
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
    at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
    at 
org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
    at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
    at 
com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
    at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
    at 
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
    at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
    at 
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)

    at com.sun.proxy.$Proxy122.deleteZone(Unknown Source)
    at 
org.apache.cloudstack.api.command.admin.zone.DeleteZoneCmd.execute(DeleteZoneCmd.java:72)

    at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:163)
    at com.cloud.api.ApiServer.queueCommand(ApiServer.java:776)
    at com.cloud.api.ApiServer.handleRequest(ApiServer.java:600)
    at 
com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:327)

    at com.cloud.api.ApiServlet$1.run(ApiServlet.java:145)
    at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
    at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
    at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)

    at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:142)
    at com.cloud.api.ApiServlet.doGet(ApiServlet.java:96)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:645)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
    at 
org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
    at 
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
    at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550)
    at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
    at 
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
    at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
    at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
    at 
org.eclipse.jetty.server.session.Ses

Re: Removing a zone