Re: Advice Needed: Connecting VPCs Across Different Account

[Simon Weller]

Hi Palash,

If you are handling your own routing, a Private Gateway will be a lot more
performant.

-Si

On Tue, Feb 27, 2024 at 7:09 AM Palash Biswas  wrote:

> Hi everyone,
>
> I'm seeking advice on the best method to connect VPCs across different
> accounts. Should I use a private gateway or a site-to-site VPN?
>
> I have two accounts and want to establish communication between the VPC in
> Account A and the VPC in Account B.
>
> Thanks,
> Palash Biswas
>

Re: KVM cluster + cloudstack

[Simon Weller]

Hi Francisco,

Which storage platform are you planning on using? Storage will dictate
whether you need any clustering or not. The most widely used
storage platforms CloudStack with KVM don't require any outside clustering.

-Si

On Wed, Dec 27, 2023 at 9:05 AM Francisco Arencibia Quesada <
arencibia.franci...@gmail.com> wrote:

> Good morning guys,
>
> Have you ever implemented KVM+cloudstack? In XCP-ng we create a cluster
> easily, but in KVM it is supposed to be done with pacemaker+corosync, but I
> have read somewhere that cloudstack doesn't get along with corosync and
> pacemaker. Could you confirm this guys?
>
>
> Kind regards
> --
> *Francisco Arencibia Quesada.*
> *DevOps Engineer*
>

Re: CloudStack - TF Integration Status

[Simon Weller]

All,

There are some ongoing talks regarding a potential fork of Tungsten Fabric.
 I suggest we let those play out before we make any decisions on the
plugin.

-Si


On Mon, Oct 30, 2023 at 9:47 AM Rohit Yadav 
wrote:

> Hi Rohit P,
>
> As I've shared in my earlier reply, CloudStack already supports Tungsten
> Fabric so given news on TF's website it's really up our community to
> discuss this further, propose and evaluate options.
>
> Apart from TF, we do have alternative SDN support in CloudStack, first and
> foremost CloudStack's very own native virtual router and support for VPC,
> isolated, L2 and shared networks;  along with that, we've a bunch of
> network plugins for variety of network functions (
> https://github.com/apache/cloudstack/tree/main/plugins/network-elements)
> and an upcoming integration with VMware NSX (there's a talk proposes for
> CCC https://www.cloudstackcollab.org/#schedule and a work-in-progress
> pull request).
>
> Generally speaking, as with any software project and integration
> dependencies, when there is lack of access to software or hardware and
> licenses (for testing, CI/CD) by the community, and lack of support by the
> vendor or users, the original integration may develop stability and
> maintenance issues longer term.
>
>
> Regards.
>
> 
> From: Rohit Palsulkar 
> Sent: Monday, October 30, 2023 16:20
> To: users@cloudstack.apache.org 
> Subject: Re: CloudStack - TF Integration Status
>
> Hi Rohit,
>
> Kindly help me with the query I have asked earlier.
>
> "Could you please confirm if the development towards Tungsten Integration
> will not happen going forward? If yes, do we have any alternatives to
> Tungsten?"
>
>
> On Fri, 27 Oct 2023 at 03:06, Rohit Palsulkar  >
> wrote:
>
> > Thank you Stephan.
> >
> > Yes, Rohit Yadav
> > I was referring to the *Tungsten Fabric* Website only. Wherein the
> > Website's Landing page says -
> >
> > *"Thank you for your interest in Tungsten Fabric. The community has
> > decided to shut down the project and will sunset this website on August
> 1,
> > 2024."*
> >
> > Could you please confirm if the development towards Tungsten Integration
> > will not happen going forward? If yes, do we have any alternatives to
> > Tungsten?
> >
> > On Thu, 26 Oct 2023 at 19:30, Rohit Yadav 
> > wrote:
> >
> >> Thanks Stephan, Rohit P. for the thread. I couldn't see the attached
> >> screenshot which possibly got stripped from the mail (mailing lists
> doesn't
> >> allow such attachments).
> >>
> >> Support for Tungsten Fabric already exists in CloudStack, but given the
> >> new update on TF's website I suppose all of us in the community needs to
> >> review this change and evaluate options.
> >>
> >>
> >> Regards.
> >>
> >> 
> >> From: Stephan Bienek 
> >> Sent: Thursday, October 26, 2023 14:45
> >> To: users@cloudstack.apache.org 
> >> Subject: Re: CloudStack - TF Integration Status
> >>
> >> Rohit P. might be referring to Tungsten Fabric instead of Terraform.
> >>
> >> On the project website of Tungsten Fabric the project team states:
> >>
> >> "Thank you for your interest in Tungsten Fabric. The community has
> >> decided to shut down the project and will sunset this website on August
> 1,
> >> 2024."
> >>
> >> which would give the right relation to Rohit`s question of how this will
> >> impact the integration with Cloudstack
> >>
> >> Best regards,
> >> Stephan
> >>
> >> > Rohit Yadav  hat am 26.10.2023 10:59 CEST
> >> geschrieben:
> >> >
> >> >
> >> > Hi Rohit,
> >> >
> >> > Some triaging work was done last month, we hope to work towards the
> >> next TF release v0.5.0 after ACS 4.19 release -
> >> https://github.com/apache/cloudstack-terraform-provider/milestone/2
> >> >
> >> > In the meanwhile, any user efforts in testing and submitting
> >> bugs/improvement requests on Github are welcome.
> >> >
> >> >
> >> > Regards.
> >> >
> >> > 
> >> > From: Rohit Palsulkar 
> >> > Sent: Thursday, October 26, 2023 14:25
> >> > To: users@cloudstack.apache.org 
> >> > Subject: CloudStack - TF Integration Status
> >> >
> >> > Hi All,
> >> >
> >> > I just wanted to know the status of the CloudStack - TF Integration.
> Is
> >> it still happening?
> >> >
> >> > I came across the TF Website and saw below message -
> >> >
> >> > [Screenshot 2023-10-26 at 2.07.48 PM.png]
> >> >
> >> > Kindly let us know if this will affect the ongoing development towards
> >> the Integration with Cloudstack?
> >> >
> >> > --
> >> > Regards,
> >> > Rohit P
> >> > [
> >>
> https://ci3.googleusercontent.com/mail-sig/AIorK4yvCmOZqibK4B_l3XOxCFDHWF9Qo_rdu2JLbrcXZf9-hhM9s8z-Lg2J-zCpyobbFnVUAAzHtRg
> >> ]
> >> >
> >> > This message is intended only for the use of the individual or entity
> >> to which it is addressed and may contain confidential and/or privileged
> >> information. If you are not the intended recipient, please delete the
> >> original message and any copy of it from your 

Re: Live Instance Backup

[Simon Weller]

Sven,

That's great! I should have said "any storage that supports QCOW2, should
support live VM Snapshots".



On Mon, Oct 30, 2023, 4:05 AM  wrote:

> Hi Fariborz,
>
> I want to add to Simon's answer that also linstor storage plugin supports
> live snapshots and not only NFS storage.
>
> regards,
> Swen
>
> -Ursprüngliche Nachricht-
> Von: Jithin Raju 
> Gesendet: Montag, 30. Oktober 2023 09:32
> An: users@cloudstack.apache.org
> Betreff: Re: Live Instance Backup
>
> Hi Fariborz,
>
> The setting must be : kvm.vmstoragesnapshot.enabled. In case you are
> looking for volume snapshots: kvm.snapshot.enabled .
>
> -Jithin
>
> From: Fariborz Navidan 
> Date: Monday, 30 October 2023 at 12:49 PM
> To: users@cloudstack.apache.org 
> Subject: Re: Live Instance Backup
> Hello,
>
> Thanks a lot for your reply. Could you please let me know what is the
> setting nams in CloudStack?
>
> TIA
>
> On Mon, 30 Oct 2023, 04:37 Simon Weller,  wrote:
>
> > Fariborz,
> >
> > KVM does support live (running) snapshots, but it's dependent on the
> > underlying storage you're using. Currently, only NFS supports live
> > snaps, as KVM stores the memory in a QCOW2 image.
> >
> > There's a global option to enable this feature in CloudStack.
> >
> > -Si
> >
> > On Sun, Oct 29, 2023, 6:02 AM Fariborz Navidan 
> > wrote:
> >
> > > Hi CS community,
> > >
> > > We're running CS 4.15.0.0 and KVM hypervisor. It doesn't support
> > > live instance backups. Does the latest version of CS and KVM/QEMU
> > > have this feature in place?
> > >
> > > TIA
> > > Cheers.
> > >
> >
>
>
>
>
>
>

Re: Live Instance Backup

[Simon Weller]

Fariborz,

KVM does support live (running) snapshots, but it's dependent on the
underlying storage you're using. Currently, only NFS supports live snaps,
as KVM stores the memory in a QCOW2 image.

There's a global option to enable this feature in CloudStack.

-Si

On Sun, Oct 29, 2023, 6:02 AM Fariborz Navidan 
wrote:

> Hi CS community,
>
> We're running CS 4.15.0.0 and KVM hypervisor. It doesn't support live
> instance backups. Does the latest version of CS and KVM/QEMU have this
> feature in place?
>
> TIA
> Cheers.
>

Re: CloudStack agent can't connect to upgraded CEPH Cluster

[Simon Weller]

Mosharaf,

Did you upgrade the Ceph client on your hosts as well?

What does "ceph -s" report? Is your cluster healthy?

Do you have any logs that indicate OSDs are disconnecting?

I'm not very familiar with the new read balancer feature in Reef. Can you
disable it and see if your performance improves?

-Si











On Tue, Sep 19, 2023 at 1:25 AM Mosharaf Hossain <
mosharaf.hoss...@bol-online.com> wrote:

> Hello Andrija
>
>  Presently, CloudStack's host lists exhibited stability prior to the
> disaster, but their statuses are currently fluctuating continuously. Some
> hosts are initially marked as disconnected, but after a period, they
> transition to a connected state."
>
>
>
>
> [image: image.png]
>
> *Using virsh we are getting VM status on cshost1 as below*
> root@cshost1:~# virsh list
>  IdName   State
> ---
>  10i-14-597-VMrunning
>  61r-757-VM   running
>  69i-24-767-VMrunning
>  76r-71-VMrunning
>  82i-24-797-VMrunning
>  113   r-335-VM   running
>  128   r-577-VM   running
>  148   i-14-1151-VM   running
>  164   i-2-1253-VMrunning
>
>
> Regards
> Mosharaf Hossain
> Manager, Product Development
> IT Division
>
> Bangladesh Export Import Company Ltd.
>
> Level-8, SAM Tower, Plot #4, Road #22, Gulshan-1, Dhaka-1212,Bangladesh
>
> Tel: +880 9609 000 999, +880 2 5881 5559, Ext: 14191, Fax: +880 2 9895757
>
> Cell: +8801787680828, Email: mosharaf.hoss...@bol-online.com, Web:
> www.bol-online.com
>
> 
>
>
>
> On Mon, Sep 18, 2023 at 12:43 PM Andrija Panic 
> wrote:
>
>> Hi,
>>
>> the message " Agent-Handler-1:null) (logid:) Connection with libvirtd is
>> broken: invalid connection pointer in virConnectGetVersion " - is a false
>> alarm and does NOT means any errors actually.
>>
>> I can see that ACS agent sees different storage pools - namely
>> "daab90ad-42d3-3c48-a9e4-b4c3c7fcdc84" and
>> "a2d455c6-68cb-303f-a7fa-287e62a5be9c" - and I don't see any explicit error
>> message about these 2 pools (both RBD/Ceph) pools.
>>
>> Also I can see that the cloudstack agent says it's connected to the mgmt
>> host - which means that all pools are in place (otherwise the agent would
>> not connect)
>>
>> 1. Are you KVM hosts all green when checking in CloudStack UI
>> (Connected/Up)?
>> 2. You can always use virsh to list pools and see if they are there
>>
>> Best,
>>
>> On Wed, 13 Sept 2023 at 13:54, Mosharaf Hossain <
>> mosharaf.hoss...@bol-online.com> wrote:
>>
>>> Hello Folks
>>> We've recently performed an upgrade on our Cephadm cluster, transitioning
>>> from Ceph Quiency to Reef. However, following the manual implementation
>>> of
>>> a read balancer in the Reef cluster, we've experienced a significant
>>> slowdown in client I/O operations within the Ceph cluster, affecting both
>>> client bandwidth and overall cluster performance.
>>>
>>> This slowdown has resulted in unresponsiveness across all virtual
>>> machines
>>> within the cluster, despite the fact that the cluster exclusively
>>> utilizes
>>> SSD storage."
>>>
>>> In the CloudStack agent, we are getting libvirrt can't connect to CEPH
>>> pool
>>> and generating an error message.
>>>
>>> 2023-09-13 16:57:51,660 INFO  [cloud.agent.Agent] (Agent-Handler-4:null)
>>> (logid:) Lost connection to host: 10.10.11.61. Attempting reconnection
>>> while we still have 1 command in progress.
>>> 2023-09-13 16:57:51,661 INFO  [utils.nio.NioClient]
>>> (Agent-Handler-4:null)
>>> (logid:) NioClient connection closed
>>> 2023-09-13 16:57:51,662 INFO  [cloud.agent.Agent] (Agent-Handler-4:null)
>>> (logid:) Reconnecting to host:10.10.11.62
>>> 2023-09-13 16:57:51,662 INFO  [utils.nio.NioClient]
>>> (Agent-Handler-4:null)
>>> (logid:) Connecting to 10.10.11.62:8250
>>> 2023-09-13 16:57:51,663 INFO  [utils.nio.Link] (Agent-Handler-4:null)
>>> (logid:) Conf file found: /etc/cloudstack/agent/agent.properties
>>> 2023-09-13 16:57:51,779 INFO  [utils.nio.NioClient]
>>> (Agent-Handler-4:null)
>>> (logid:) SSL: Handshake done
>>> 2023-09-13 16:57:51,779 INFO  [utils.nio.NioClient]
>>> (Agent-Handler-4:null)
>>> (logid:) Connected to 10.10.11.62:8250
>>> 2023-09-13 16:57:51,815 INFO  [utils.linux.KVMHostInfo]
>>> (Agent-Handler-1:null) (logid:) Fetching CPU speed from command "lscpu".
>>> 2023-09-13 16:57:51,836 INFO  [utils.linux.KVMHostInfo]
>>> (Agent-Handler-1:null) (logid:) Command [lscpu | grep -i 'Model name' |
>>> head -n 1 | egrep -o '[[:digit:]].[[:digit:]]+GHz' | sed 's/GHz//g']
>>> resulted in the value [2100] for CPU speed.
>>> 2023-09-13 16:57:51,900 INFO  [kvm.storage.LibvirtStorageAdaptor]
>>> (Agent-Handler-1:null) (logid:) Attempting to create storage pool
>>> e205cf5f-ea32-46c7-ba18-d18f62772b80 (Filesystem) in libvirt
>>> 2023-09-13 16:57:51,901 ERROR [kvm.resource.LibvirtConnection]
>>> (Agent-Handler-1:null) (logid:) Connection with 

Re: Query to Cloudstack API is degraded

[Simon Weller]

Kuasar,

It looks like you have the innodb_buffer_pool_size set to the default. Can
you try and increase this to 70% of your available memory? I think you
mentioned you have 4GB assigned. If it's a dedicated server/VM for MySQL,
can you edit your my.cnf and change the innodb_buffer_pool_size = 2.8G and
do a restart on MySQL?

-Si

On Fri, Aug 25, 2023 at 11:34 AM Kuasar 
wrote:

> Hey guys
>
> Rohit that sounds great about the performance improvement. Regarding the
> questions
>
> root admin, domain admin and/or user accounts
>
> Yes, for all users/projects/accounts etc
>
>  multiple management servers
>
> 2 management servers, but our front is pointing to only one
>
> +-+---+
> | Variable_name | Value |
> +-+---+
> | innodb_buffer_pool_size | 134217728 |
> +-+---+
>
>
> BR,
>
> Ricardo Pertuz
>
>
> On 25 Aug 2023, 10:56 AM -0500, Rohit Yadav ,
> wrote:
> > Hi Ricardo,
> >
> > Thanks for sharing, in addition to what Simon has asked; can you also
> share if this behaviour is consistent across different types of account
> (root admin, domain admin and/or user accounts) and domains, was there any
> change in the infrastructure after which this was observed?
> >
> > We've introduced a small optimisation with
> https://github.com/apache/cloudstack/pull/7911 that you've tested, this
> is coming in 4.18.1.0 and later which calls the listVirtualMachinesMetrics
> API with a limited details view, this is used to render the instance list
> page in the UI. I think most users could see 5-10x improvements in the
> instance page load times.
> >
> >
> > Regards.
> >
> > 
> > From: Simon Weller 
> > Sent: Friday, August 25, 2023 19:18
> > To: users@cloudstack.apache.org 
> > Subject: Re: Query to Cloudstack API is degraded
> >
> > Kuasar,
> >
> > How big is your installation, and what do you have your MySQL
> > innodb_buffer_pool_size set to?
> >
> > Are you running a single, or multiple management servers?
> >
> > -Si
> >
> >
> > On Fri, Aug 25, 2023 at 8:43 AM Kuasar  .invalid>
> > wrote:
> >
> > > Hi Rohit,
> > >
> > > Thanks for you prompt reply.
> > >
> > > Could you say try with a MySQL 8.x?
> > >
> > > We are having this problem in production which is bigger and changing
> the
> > > db engine right now will take us weeks
> > >
> > > you could try Ubuntu 22.04/20.04
> > >
> > > Yes, we are using Ubuntu 20.04
> > >
> > > Can you measure (say using dev tools of your browser)
> > >
> > > List-vms always takes around 5.6s checking on the timing tab of the
> > > browser and
> > >
> > > http:// IP>:8080/client/#/vm?details=min
> > > http:// > > IP>:8080/client/#/vm?details=servoff,tmpl,nics
> > >
> > > Only 1 second, much better
> > >
> > >
> > > BR,
> > >
> > > Ricardo Pertuz
> > >
> > >
> > > On 25 Aug 2023, 6:41 AM -0500, Rohit Yadav  >,
> > > wrote:
> > > > Hi Ricardo,
> > > >
> > > > Thanks for sharing. I can't say without more details on how to
> reproduce
> > > the env/condition, or look at the DB server. It might be statistics or
> > > something else, you could try to log mysql queries and see if they're
> > > slow(ing down).
> > > >
> > > > Could you say try with a MySQL 8.x server and use a different host
> > > server (for example, there is an issue reportedly affecting EL9 -
> > > https://github.com/apache/cloudstack/issues/7910; you could try Ubuntu
> > > 22.04/20.04 or EL8 to rule out the DB server).
> > > >
> > > > Can you measure (say using dev tools of your browser) the time it
> takes
> > > to render the list VM API call (specifically the API would be called
> > > listVirtualMachinesMetrics API), share the outcomes with us.
> > > >
> > > > Can you try the following URLs for the list instances page in the UI,
> > > see if any makes difference in loading times for you:
> > > >
> > > >
> > > > http:// IP>:8080/client/#/vm?details=min
> > > > http:// > > IP>:8080/client/#/vm?details=servoff,tmpl,nics
> > > >
> > > > Compared to the following that you've by default:
> > > >
> > > >
> > > > http://8080/cli

Re: Query to Cloudstack API is degraded

[Simon Weller]

Kuasar,

How big is your installation, and what do you have your MySQL
innodb_buffer_pool_size set to?

Are you running a single, or multiple management servers?

-Si


On Fri, Aug 25, 2023 at 8:43 AM Kuasar 
wrote:

> Hi Rohit,
>
> Thanks for you prompt reply.
>
> Could you say try with a MySQL 8.x?
>
> We are having this problem in production which is bigger and changing the
> db engine right now will take us weeks
>
> you could try Ubuntu 22.04/20.04
>
> Yes, we are using Ubuntu 20.04
>
> Can you measure (say using dev tools of your browser)
>
> List-vms always takes around 5.6s checking on the timing tab of the
> browser and
>
> http://:8080/client/#/vm?details=min
> http:// IP>:8080/client/#/vm?details=servoff,tmpl,nics
>
> Only 1 second, much better
>
>
> BR,
>
> Ricardo Pertuz
>
>
> On 25 Aug 2023, 6:41 AM -0500, Rohit Yadav ,
> wrote:
> > Hi Ricardo,
> >
> > Thanks for sharing. I can't say without more details on how to reproduce
> the env/condition, or look at the DB server. It might be statistics or
> something else, you could try to log mysql queries and see if they're
> slow(ing down).
> >
> > Could you say try with a MySQL 8.x server and use a different host
> server (for example, there is an issue reportedly affecting EL9 -
> https://github.com/apache/cloudstack/issues/7910; you could try Ubuntu
> 22.04/20.04 or EL8 to rule out the DB server).
> >
> > Can you measure (say using dev tools of your browser) the time it takes
> to render the list VM API call (specifically the API would be called
> listVirtualMachinesMetrics API), share the outcomes with us.
> >
> > Can you try the following URLs for the list instances page in the UI,
> see if any makes difference in loading times for you:
> >
> >
> > http://:8080/client/#/vm?details=min
> > http:// IP>:8080/client/#/vm?details=servoff,tmpl,nics
> >
> > Compared to the following that you've by default:
> >
> >
> > http://8080/client/#/vm
> >
> > I've raised a PR to explore an idea further:
> > https://github.com/apache/cloudstack/pull/7911
> >
> >
> > Regards.
> >
> > 
> > From: Kuasar 
> > Sent: Thursday, August 24, 2023 19:21
> > To: users@cloudstack.apache.org ;
> users@cloudstack.apache.org 
> > Subject: Re: Query to Cloudstack API is degraded
> >
> > Hi Rohit,
> >
> > Thanks for asking, my reply below
> >
> > - which APIs you think have degraded ->
> > listing instances
> >
> > - how have you concluded that they are degrading? ->
> > we have an upper app that has a configured timeout and it wasn’t firing,
> we had to change the value, and it is visible as well, it takes 5.6s
> bringing 5 vms.
> >
> > - are those list API or other async/action oriented APIs? ->
> > yes, it it async
> >
> > - which version of CloudStack and MySQL server is this, other details
> about your env (hypervisor, scale/size of resources etc).
> > ACS 4.18.0. —> 4GB 2Cores
> > Mariadb 10.5.x -> 4GB 2Cores
> > Hyp KVM
> >
> > - does restarting mgmt server help? any other information on how to
> reproduce this?
> > no change
> >
> > We are guessing it is related to database performance/index as it not
> happening in QA
> >
> >
> > Atte,
> >
> > Ricardo Pertuz
> >
> >
> > On 24 Aug 2023, 6:33 AM -0500, Rohit Yadav ,
> wrote:
> > > Hi Ricardo,
> > >
> > > Can you share more details, such as;
> > >
> > > - which APIs you think have degraded
> > > - how have you concluded that they are degrading?
> > > - are those list API or other async/action oriented APIs?
> > > - which version of CloudStack and MySQL server is this, other details
> about your env (hypervisor, scale/size of resources etc)
> > > - does restarting mgmt server help? any other information on how to
> reproduce this?
> > >
> > >
> > > Regards.
> > >
> > > 
> > > From: Kuasar 
> > > Sent: Wednesday, August 23, 2023 22:00
> > > To: users@cloudstack.apache.org 
> > > Subject: Query to Cloudstack API is degraded
> > >
> > > During the last weeks we have been experiencing a degrading
> performance in API queries to Cloudstack, is there a procedure you
> recommend to check the possible bottleneck?, we are checking database
> indices.
> > >
> > >
> > > BR,
> > >
> > > Ricardo Pertuz
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
>

Re: SNMP Traps Not Working

[Simon Weller]

Will,

Is your mgmt server using an RFC 1918 address, or is it using localhost?

-Si

On Thu, Aug 17, 2023 at 2:13 PM Will Conrad 
wrote:

> H K,
>
> Thanks for responding.
>
> Not a communication issue. The traps are not being sent. I've turned up
> tcpdump on the mgmt host and monitored for traffic to our trap receiver. No
> trap ever leaves the server.
>
> Regards,
>
> Will
>
>
>
> On Thu, Aug 17, 2023 at 3:06 PM K B Shiv Kumar 
> wrote:
>
> > Sorry didn't go through it fully. Most likely an SNMP communication
> issue.
> > I hope your SNMP server is enabled to support version 2c and it is not
> > disabled. Nowadays v3 is the default I believe.
> >
> > Regards,
> > Shiv
> > (Sent from mobile device. Please excuse brevity and typos.)
> >
> > On Fri, 18 Aug 2023, 00:32 K B Shiv Kumar,  wrote:
> >
> > > Hi Will
> > >
> > > Is your community "public"? Any info in the logs? Did you try Syslog?
> Why
> > > I'm asking is to zero in on SNMP traps vs Syslog or the alerting
> > mechanism
> > > itself.
> > >
> > > Regards,
> > > Shiv
> > > (Sent from mobile device. Please excuse brevity and typos.)
> > >
> > > On Thu, 17 Aug 2023, 23:17 Will Conrad,  .invalid>
> > > wrote:
> > >
> > >> Hi All,
> > >>
> > >> Bump
> > >>
> > >> Any word on this? Does anyone know how to get SNMP working in
> > cloudstack?
> > >>
> > >> On Tue, Aug 8, 2023 at 7:37 AM Will Conrad 
> > >> wrote:
> > >>
> > >> > I followed the cloudstack documentation here:
> > >> >
> > >> >
> > >> >
> > >>
> >
> https://docs.cloudstack.apache.org/en/4.18.0.0/adminguide/management.html?highlight=snmp#configuring-snmp-and-syslog-managers
> > >> >
> > >> > to get SNMP traps working, however, no traps are being sent. I don't
> > see
> > >> > any related errors logged. I see no instances of failed library
> loads,
> > >> but
> > >> > I do not see. Do I need to manually install the libsnmp4j library
> > >> myself?
> > >> >
> > >> > I know it's reading the log4j config as I get this notification in
> the
> > >> > management server log:
> > >> >
> > >> > 2023-08-07 18:20:42,842 INFO  [c.c.u.LogUtils] (main:null) (logid:)
> > >> log4j
> > >> > configuration found at /etc/cloudstack/management/log4j-cloud.xml
> > >> >
> > >> > Here is the log4j config for the SNMP appender (note that I have
> > >> > configured syslog as well, and that is working).
> > >> >
> > >> >   > >> > class="org.apache.cloudstack.alert.snmp.SnmpTrapAppender">
> > >> >
> > >> >   
> > >> >
> > >> >   
> > >> >
> > >> >   
> > >> >
> > >> >   
> > >> >
> > >> >> >> > class="org.apache.cloudstack.alert.snmp.SnmpEnhancedPatternLayout">
> > >> >
> > >> >  
> > >> >
> > >> >  
> > >> >
> > >> >   
> > >> >
> > >> >
> > >> >
> > >> > Any insight will be greatly appreciated.
> > >> >
> > >> > Regards,
> > >> >
> > >> > Willard
> > >> >
> > >>
> > >
> >
> > --
> > This message is intended only for the use of the individual or entity to
> > which it is addressed and may contain confidential and/or privileged
> > information. If you are not the intended recipient, please delete the
> > original message and any copy of it from your computer system. You are
> > hereby notified that any dissemination, distribution or copying of this
> > communication is strictly prohibited unless proper authorization has been
> > obtained for such action. If you have received this communication in
> > error,
> > please notify the sender immediately. Although IndiQus attempts to sweep
> > e-mail and attachments for viruses, it does not guarantee that both are
> > virus-free and accepts no liability for any damage sustained as a result
> > of
> > viruses.
> >
>

Re: Cloudstack DB HA, do you use db.ha.enabled?

[Simon Weller]

Lucian,

Check to see whether the mysql-ha jar is being loaded. There's a separate
mysql-ha package that needs to be installed.

Is this Ubuntu or rpm? I'm not sure whether the default Ubuntu builds
include the extra package. I believe the shapeblue build does though.

-Si


On Thu, Jul 20, 2023, 3:03 PM Nux  wrote:

> Cheers Daniel,
>
> Can you share any other db.ha parameters you may have tuned?
> For me it didn't work out of the box as you described.
>
> Thanks
>
> On 2023-07-20 14:04, Daniel Salvador wrote:
> > Hello Nux,
> >
> > Normally I set up three nodes with MariaDB and Galera cluster; then, in
> > the
> > "db.properties" file I mark "db.ha.enabled" as true, and I define one
> > of
> > the nodes as main and the other as replicas. When the main node goes
> > down,
> > one of the replicas takes over, and so on.
> >
> > The current properties we have on "db.properties" regarding DB HA are
> > hard
> > coded and only address some MySQL properties; which is not the perfect
> > scenario for MariaDB HA. However, it provides a minimum DB HA setup. Me
> > and
> > other contributors are already working on a flexible solution to
> > address
> > other MySQL properties, and MariaDB properties as well.
> >
> > Best regards,
> > Daniel Salvador (gutoveronezi)
> >
> > On Thu, Jul 20, 2023 at 7:46 AM Nux  wrote:
> >
> >> Hello,
> >>
> >> As per the subject, how do you make your DB layer HA and do you use
> >> the
> >> db.ha.enabled feature/setting in the Cloudstack management server
> >> db.properties file?
> >>
> >> Cheers
> >>
>

Re: Database HA in Cloudstack 4.15.2

[Simon Weller]

Hi Frederic,

First of all, have you installed the cloudstack-mysql-ha package? If so,
can you look through the logs on cloudstack-management startup, and check
to see whether the jar is being loaded?

-Si

On Fri, May 26, 2023 at 3:27 AM Frederic Larcher
 wrote:

> Hi,
>
> I am trying to configure HA with 2 Mysql servers in Cloudstack 4.15.2.
> Both DB servers are configured as master-master, and the db.properties file
> has been updated:
>
> --
> # High Availability And Cluster Properties
> db.ha.enabled=true
> db.ha.loadBalanceStrategy=com.cloud.utils.db.StaticStrategy
> # cloud stack Database
> db.cloud.slaves=10.10.10.20
> db.cloud.autoReconnect=true
> db.cloud.failOverReadOnly=false
> db.cloud.reconnectAtTxEnd=true
> db.cloud.autoReconnectForPools=true
> db.cloud.secondsBeforeRetryMaster=3600
> db.cloud.queriesBeforeRetryMaster=5000
> db.cloud.initialTimeout=3600
>
> #usage Database
> db.usage.slaves=10.10.10.20
> db.usage.autoReconnect=true
> db.usage.failOverReadOnly=false
> db.usage.reconnectAtTxEnd=true
> db.usage.autoReconnectForPools=true
> db.usage.secondsBeforeRetryMaster=3600
> db.usage.queriesBeforeRetryMaster=5000
> db.usage.initialTimeout=3600
>
> 
>
> When I stop the mysql service or disable the NIC to stop connectivity,
> Cloudstack does not use the second mysql server. The UI keeps loading, and
> the management service cannot be restarted. I am getting these logs on the
> management server:
>
> --
> Unable to keep the db connection for LockMaster1
> java[358521]: java.sql.SQLNonTransientConnectionException: Server shutdown
> in progress
> java[358521]: at
> com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:110)
> java[358521]: at
> com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
> java[358521]: at
> com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
> java[358521]: at
> com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:953)
> java[358521]: at
> com.mysql.cj.jdbc.ClientPreparedStatement.executeQuery(ClientPreparedStatement.java:1003)
> java[358521]: at
> jdk.internal.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
> java[358521]: at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> java[358521]: at
> java.base/java.lang.reflect.Method.invoke(Method.java:566)
> java[358521]: at
> com.mysql.cj.jdbc.ha.MultiHostConnectionProxy$JdbcInterfaceProxy.invoke(MultiHostConnectionProxy.java:108)
> java[358521]: at
> com.mysql.cj.jdbc.ha.FailoverConnectionProxy$FailoverJdbcInterfaceProxy.invoke(FailoverConnectionProxy.java:98)
> java[358521]: at com.sun.proxy.$Proxy25.executeQuery(Unknown
> Source)
> java[358521]: at
> org.apache.commons.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:122)
> java[358521]: at
> org.apache.commons.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:122)
> java[358521]: at
> com.cloud.utils.db.ConnectionConcierge$ConnectionConciergeManager.testValidity(ConnectionConcierge.java:148)
> java[358521]: at
> com.cloud.utils.db.ConnectionConcierge$ConnectionConciergeManager$1.runInContext(ConnectionConcierge.java:203)
> java[358521]: at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> java[358521]: at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> java[358521]: at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> java[358521]: at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> java[358521]: at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> java[358521]: at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> java[358521]: at
> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
> java[358521]: at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> java[358521]: at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> java[358521]: at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> java[358521]: at java.base/java.lang.Thread.run(Thread.java:829)
>
> 
> We are using Mysql version 8.0.33 on 

Re: VMs have no network

[Simon Weller]

ok, it is possible via the GUI, I just needed to figure it out.

Firstly, add the new public range to the zone.

Then go to each VM via the Computer/Instances and click on the NICS tab.
Click the + Icon next to the NIC you want to change the IP on.
Click the little wavy icon above the "ID".
Manually enter the new IP address.



On Thu, May 25, 2023 at 1:44 PM Simon Weller  wrote:

> So firstly, since they screwed up, it would be much easier if they pulled
> that block back from the other customer and gave it back to you.
>
> If that's not possible, then you'll add another public subnet to your
> zone. In terms of changing the allocated IPs to created VMs, I'm not sure
> that is possible without some fairly major tweaking in the database, or
> assigning a new default NIC with the new range.
>
>
>
>
> On Thu, May 25, 2023 at 12:51 PM Granwille Strauss 
> wrote:
>
>> Hi
>>
>> Yep my IP subnet range has been allocated to new clients a complete eff
>> up at the DC. Can I please get the correct steps I can take to ensure the
>> new range is correctly assigned to my VMs? is it as simple as unassigning
>> an IP and re-assigning one again? And then obviously update the IP network
>> in the VM?
>> On 5/25/23 19:10, Simon Weller wrote:
>>
>> ok great. Glad you're making progress toward a resolution.
>>
>> On Thu, May 25, 2023 at 12:01 PM Granwille Strauss 
>> wrote:
>>
>>> Hi
>>>
>>> Thank you for the update I believe I found the issue my DC just
>>> confirmed that a certain subnet range is not configured for my host
>>> machine. After we have been using it for MONTHS now. And I tested other VMs
>>> on the same HOST but they use a different subnet and they have full
>>> internet access. And the VMs not working are the ones with the missing
>>> subnet from my DC.
>>>
>>> I am busy fighting this now with them and will provide feedback if this
>>> was indeed the issue, but this far all fingers seems to point towards it.
>>> On 5/25/23 18:47, Simon Weller wrote:
>>>
>>> Can you tell me more about your advanced network setup?
>>>
>>> Also, can you provide some info from your VM?
>>>
>>> If it's Linux, perhaps the output of "ip a" and the output of "netstat
>>> -r" or "route -s".
>>>
>>> If it's windows,a screenshot of the network configuration including IP
>>> addresses and gateway.
>>>
>>> -Si
>>>
>>> On Thu, May 25, 2023, 11:39 AM Granwille Strauss 
>>> wrote:
>>>
>>>> Yes I have a virtual router and only one. I have a very basic set up
>>>> nothing additional or extra. That said, it seems I cannot ping my virtual
>>>> router private IP 169.254.127.238 from the VM no. Attached is a
>>>> screenshot for you to confirm I am pinging the right IP
>>>> On 5/25/23 18:32, Simon Weller wrote:
>>>>
>>>> Ok, so I assume you have Virtual Routers as well. Are you running redundant
>>>> VRs, or just one? Can you ping the VR inside IP from your VM (guest network
>>>> gateway)?
>>>>
>>>> On Thu, May 25, 2023, 11:27 AM Granwille Strauss  
>>>> 
>>>> wrote:
>>>>
>>>>
>>>> Hi The host can ping and do everything correctly, my SSVM is also on the
>>>> same host as the VMs and my SSVM works 100% fine and can also ping outside.
>>>> Its only my VMs that's failing to do so. I am using advanced network. And
>>>> can you provide me with more details to VIF and how to determine this?
>>>> On 5/25/23 18:20, Simon Weller wrote:
>>>>
>>>> I'm assume the underlay network is ok? Can you ping your gateway from the
>>>> host?
>>>>
>>>> What type of networking are you using with ACS/KVM?
>>>>
>>>> Are you seeing the VIF for a VM being plugged into the correct bridge?
>>>>
>>>> On Thu, May 25, 2023, 11:10 AM Granwille Strauss  
>>>>   
>>>> wrote:
>>>>
>>>>
>>>> Hi
>>>>
>>>> Thank you I managed to free space now successfully, thank you. However,
>>>> VMs still have zero network. May I also add that I manually shutdown VMs in
>>>> KVM using the virsh command. And then boot them up again via Cloudstack UI.
>>>> Perhaps this has some affect.
>>>> On 5/25/23 17:54, Simon Weller wrote:
>>>>
>>>> Granwille,
>>>>
>>>> Use d

Re: VMs have no network

[Simon Weller]

Can you tell me more about your advanced network setup?

Also, can you provide some info from your VM?

If it's Linux, perhaps the output of "ip a" and the output of "netstat -r"
or "route -s".

If it's windows,a screenshot of the network configuration including IP
addresses and gateway.

-Si

On Thu, May 25, 2023, 11:39 AM Granwille Strauss 
wrote:

> Yes I have a virtual router and only one. I have a very basic set up
> nothing additional or extra. That said, it seems I cannot ping my virtual
> router private IP 169.254.127.238 from the VM no. Attached is a
> screenshot for you to confirm I am pinging the right IP
> On 5/25/23 18:32, Simon Weller wrote:
>
> Ok, so I assume you have Virtual Routers as well. Are you running redundant
> VRs, or just one? Can you ping the VR inside IP from your VM (guest network
> gateway)?
>
> On Thu, May 25, 2023, 11:27 AM Granwille Strauss  
> 
> wrote:
>
>
> Hi The host can ping and do everything correctly, my SSVM is also on the
> same host as the VMs and my SSVM works 100% fine and can also ping outside.
> Its only my VMs that's failing to do so. I am using advanced network. And
> can you provide me with more details to VIF and how to determine this?
> On 5/25/23 18:20, Simon Weller wrote:
>
> I'm assume the underlay network is ok? Can you ping your gateway from the
> host?
>
> What type of networking are you using with ACS/KVM?
>
> Are you seeing the VIF for a VM being plugged into the correct bridge?
>
> On Thu, May 25, 2023, 11:10 AM Granwille Strauss  
>   
> wrote:
>
>
> Hi
>
> Thank you I managed to free space now successfully, thank you. However,
> VMs still have zero network. May I also add that I manually shutdown VMs in
> KVM using the virsh command. And then boot them up again via Cloudstack UI.
> Perhaps this has some affect.
> On 5/25/23 17:54, Simon Weller wrote:
>
> Granwille,
>
> Use du . -h on your major partitions and try and narrow down the disk usage.
>
> Check the MySQL log and and also for any temp files that could be blowing
> out your disk usage .
>
> Are you running any MySQL replication? If so, make sure you have log
> rotation and deletion setup so they you only keep a few days of the
> replication logs. If you do see lots of binlogs, you can trim them using
> the FLUSH LOGS MySQL cli command.
>
> -Si
>
>
> On Thu, May 25, 2023, 10:45 AM Granwille 
> Strauss  
>   
>   
>   wrote:
>
>
> Hi Guys
>
> I need any help I can get, please. I have a cloudstack management server
> and I configured NFS on it and have a few VMs that use this NFS as its
> primary storage. Today my VMs randomly just went down, no one did anything
> whatsoever. Upon review we found that the cloudstack management server
> drive shows its at 100% usage. But it makes ZERO sense. There is space, the
> drive is 872 GB big and only 840 GB is being used.
>
>
> So now, I managed to to move files and delete logs and so to try and free
> space. I deleted over 13 GB and still server shows 100% usage. But now I
> can finally log into Cloudstack UI, previously I couldn't. I am now moving
> those VMs volumes from the NFS to the local KVM disk. After successfully
> migrating the volume, I now boot the VM  and it runs 100% fine, but there
> is ZERO internet access to the VM I cannot even ping google.com from the
> VM. SSVM and the virtual router are working 100% fine, zero failures on
> health check and also the SSVM check. The KVM host itself has full internet
> access.
>
> Any ideas? There are zero errors being thrown at all.
> --
> Regards / Groete<https://www.namhost.com> <https://www.namhost.com> 
> <https://www.namhost.com> <https://www.namhost.com> <https://www.namhost.com> 
> <https://www.namhost.com> <https://www.namhost.com> <https://www.namhost.com> 
> Granwille Strauss  //  Senior Systems Admin
>
> *e:* granwi...@namhost.com
> *m:* +264 81 323 1260 <+264813231260>
> *w:* www.namhost.com<https://www.facebook.com/namhost> 
> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost> 
> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost> 
> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost> 
> <https://www.facebook.com/namhost> <https://twitter.com/namhost> 
> <https://twitter.com/namhost> <https://twitter.com/namhost> 
> <https://twitter.com/namhost> <https://twitter.com/namhost> 
> <https://twitter.com/namhost> <https://twitter.com/namhost> 
> <https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/>
>  <https://www.ins

Re: VMs have no network

[Simon Weller]

Ok, so I assume you have Virtual Routers as well. Are you running redundant
VRs, or just one? Can you ping the VR inside IP from your VM (guest network
gateway)?

On Thu, May 25, 2023, 11:27 AM Granwille Strauss 
wrote:

> Hi The host can ping and do everything correctly, my SSVM is also on the
> same host as the VMs and my SSVM works 100% fine and can also ping outside.
> Its only my VMs that's failing to do so. I am using advanced network. And
> can you provide me with more details to VIF and how to determine this?
> On 5/25/23 18:20, Simon Weller wrote:
>
> I'm assume the underlay network is ok? Can you ping your gateway from the
> host?
>
> What type of networking are you using with ACS/KVM?
>
> Are you seeing the VIF for a VM being plugged into the correct bridge?
>
> On Thu, May 25, 2023, 11:10 AM Granwille Strauss  
> 
> wrote:
>
>
> Hi
>
> Thank you I managed to free space now successfully, thank you. However,
> VMs still have zero network. May I also add that I manually shutdown VMs in
> KVM using the virsh command. And then boot them up again via Cloudstack UI.
> Perhaps this has some affect.
> On 5/25/23 17:54, Simon Weller wrote:
>
> Granwille,
>
> Use du . -h on your major partitions and try and narrow down the disk usage.
>
> Check the MySQL log and and also for any temp files that could be blowing
> out your disk usage .
>
> Are you running any MySQL replication? If so, make sure you have log
> rotation and deletion setup so they you only keep a few days of the
> replication logs. If you do see lots of binlogs, you can trim them using
> the FLUSH LOGS MySQL cli command.
>
> -Si
>
>
> On Thu, May 25, 2023, 10:45 AM Granwille 
> Strauss  
>   wrote:
>
>
> Hi Guys
>
> I need any help I can get, please. I have a cloudstack management server
> and I configured NFS on it and have a few VMs that use this NFS as its
> primary storage. Today my VMs randomly just went down, no one did anything
> whatsoever. Upon review we found that the cloudstack management server
> drive shows its at 100% usage. But it makes ZERO sense. There is space, the
> drive is 872 GB big and only 840 GB is being used.
>
>
> So now, I managed to to move files and delete logs and so to try and free
> space. I deleted over 13 GB and still server shows 100% usage. But now I
> can finally log into Cloudstack UI, previously I couldn't. I am now moving
> those VMs volumes from the NFS to the local KVM disk. After successfully
> migrating the volume, I now boot the VM  and it runs 100% fine, but there
> is ZERO internet access to the VM I cannot even ping google.com from the
> VM. SSVM and the virtual router are working 100% fine, zero failures on
> health check and also the SSVM check. The KVM host itself has full internet
> access.
>
> Any ideas? There are zero errors being thrown at all.
> --
> Regards / Groete<https://www.namhost.com> <https://www.namhost.com> 
> <https://www.namhost.com> <https://www.namhost.com> Granwille Strauss  //  
> Senior Systems Admin
>
> *e:* granwi...@namhost.com
> *m:* +264 81 323 1260 <+264813231260>
> *w:* www.namhost.com<https://www.facebook.com/namhost> 
> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost> 
> <https://www.facebook.com/namhost> <https://twitter.com/namhost> 
> <https://twitter.com/namhost> <https://twitter.com/namhost> 
> <https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/>
>  <https://www.instagram.com/namhostinternetservices/> 
> <https://www.instagram.com/namhostinternetservices/> 
> <https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos>
>  <https://www.linkedin.com/company/namhos> 
> <https://www.linkedin.com/company/namhos> 
> <https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>  <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> 
> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> 
> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>  
> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>  
> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>  
> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>
> Namhost Internet Services (Pty) Ltd,
>
> 24 Black Eagle Rd, Hermanus, 7210, RSA
>
>
>
> The content of this message is confidential. If you

Re: VMs have no network

[Simon Weller]

I'm assume the underlay network is ok? Can you ping your gateway from the
host?

What type of networking are you using with ACS/KVM?

Are you seeing the VIF for a VM being plugged into the correct bridge?

On Thu, May 25, 2023, 11:10 AM Granwille Strauss 
wrote:

> Hi
>
> Thank you I managed to free space now successfully, thank you. However,
> VMs still have zero network. May I also add that I manually shutdown VMs in
> KVM using the virsh command. And then boot them up again via Cloudstack UI.
> Perhaps this has some affect.
> On 5/25/23 17:54, Simon Weller wrote:
>
> Granwille,
>
> Use du . -h on your major partitions and try and narrow down the disk usage.
>
> Check the MySQL log and and also for any temp files that could be blowing
> out your disk usage .
>
> Are you running any MySQL replication? If so, make sure you have log
> rotation and deletion setup so they you only keep a few days of the
> replication logs. If you do see lots of binlogs, you can trim them using
> the FLUSH LOGS MySQL cli command.
>
> -Si
>
>
> On Thu, May 25, 2023, 10:45 AM Granwille 
> Strauss  wrote:
>
>
> Hi Guys
>
> I need any help I can get, please. I have a cloudstack management server
> and I configured NFS on it and have a few VMs that use this NFS as its
> primary storage. Today my VMs randomly just went down, no one did anything
> whatsoever. Upon review we found that the cloudstack management server
> drive shows its at 100% usage. But it makes ZERO sense. There is space, the
> drive is 872 GB big and only 840 GB is being used.
>
>
> So now, I managed to to move files and delete logs and so to try and free
> space. I deleted over 13 GB and still server shows 100% usage. But now I
> can finally log into Cloudstack UI, previously I couldn't. I am now moving
> those VMs volumes from the NFS to the local KVM disk. After successfully
> migrating the volume, I now boot the VM  and it runs 100% fine, but there
> is ZERO internet access to the VM I cannot even ping google.com from the
> VM. SSVM and the virtual router are working 100% fine, zero failures on
> health check and also the SSVM check. The KVM host itself has full internet
> access.
>
> Any ideas? There are zero errors being thrown at all.
> --
> Regards / Groete
> <https://www.namhost.com> <https://www.namhost.com> Granwille Strauss  //  
> Senior Systems Admin
>
> *e:* granwi...@namhost.com
> *m:* +264 81 323 1260 <+264813231260>
> *w:* www.namhost.com
> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost> 
> <https://twitter.com/namhost> 
> <https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/>
>  
> <https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos>
>  
> <https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>  <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>
> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>  
> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>
> Namhost Internet Services (Pty) Ltd,
>
> 24 Black Eagle Rd, Hermanus, 7210, RSA
>
>
>
> The content of this message is confidential. If you have received it by
> mistake, please inform us by email reply and then delete the message. It is
> forbidden to copy, forward, or in any way reveal the contents of this
> message to anyone without our explicit consent. The integrity and security
> of this email cannot be guaranteed over the Internet. Therefore, the sender
> will not be held liable for any damage caused by the message. For our full
> privacy policy and disclaimers, please go 
> tohttps://www.namhost.com/privacy-policy
>
> [image: Powered by 
> AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>  
> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>
> --
> Regards / Groete
>
> <https://www.namhost.com> Granwille Strauss  //  Senior Systems Admin
>
> *e:* granwi...@namhost.com
> *m:* +264 81 323 1260 <+264813231260>
> *w:* www.namhost.com
>
> <https://www.facebook.com/namhost> <https://twitter.com/namhost>
> <https://www.instagram.com/namhostinternetservices/>
> <https://www.linkedin.com/company/namhos>
> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>
>
> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>
> Namhost Internet Services (Pty) Ltd,
>
> 24 Black Eagle Rd, Herm

Re: VMs have no network

[Simon Weller]

Granwille,

Use du . -h on your major partitions and try and narrow down the disk usage.

Check the MySQL log and and also for any temp files that could be blowing
out your disk usage .

Are you running any MySQL replication? If so, make sure you have log
rotation and deletion setup so they you only keep a few days of the
replication logs. If you do see lots of binlogs, you can trim them using
the FLUSH LOGS MySQL cli command.

-Si


On Thu, May 25, 2023, 10:45 AM Granwille Strauss
 wrote:

> Hi Guys
>
> I need any help I can get, please. I have a cloudstack management server
> and I configured NFS on it and have a few VMs that use this NFS as its
> primary storage. Today my VMs randomly just went down, no one did anything
> whatsoever. Upon review we found that the cloudstack management server
> drive shows its at 100% usage. But it makes ZERO sense. There is space, the
> drive is 872 GB big and only 840 GB is being used.
>
>
> So now, I managed to to move files and delete logs and so to try and free
> space. I deleted over 13 GB and still server shows 100% usage. But now I
> can finally log into Cloudstack UI, previously I couldn't. I am now moving
> those VMs volumes from the NFS to the local KVM disk. After successfully
> migrating the volume, I now boot the VM  and it runs 100% fine, but there
> is ZERO internet access to the VM I cannot even ping google.com from the
> VM. SSVM and the virtual router are working 100% fine, zero failures on
> health check and also the SSVM check. The KVM host itself has full internet
> access.
>
> Any ideas? There are zero errors being thrown at all.
> --
> Regards / Groete
>
>  Granwille Strauss  //  Senior Systems Admin
>
> *e:* granwi...@namhost.com
> *m:* +264 81 323 1260 <+264813231260>
> *w:* www.namhost.com
>
>  
> 
> 
> 
>
>
> 
>
> Namhost Internet Services (Pty) Ltd,
>
> 24 Black Eagle Rd, Hermanus, 7210, RSA
>
>
>
> The content of this message is confidential. If you have received it by
> mistake, please inform us by email reply and then delete the message. It is
> forbidden to copy, forward, or in any way reveal the contents of this
> message to anyone without our explicit consent. The integrity and security
> of this email cannot be guaranteed over the Internet. Therefore, the sender
> will not be held liable for any damage caused by the message. For our full
> privacy policy and disclaimers, please go to
> https://www.namhost.com/privacy-policy
>
> [image: Powered by AdSigner]
> 
>

Re: preventing VM Live migration between Pods

[Simon Weller]

Gary,

There are some global settings you can enable/disable to prevent certain VM
(and storage) movements.

migrate.vm.across.clusters - indicates whether the VM can be migrated to
different cluster if no host is found in same cluster
enable.storage.migration - Enable/disable storage migration across primary
storage
enable.ha.storage.migration - Enable/disable storage migration across
primary storage during HA

-Si

On Tue, May 16, 2023 at 8:13 AM Gary Dixon 
wrote:

> Hi everyone
>
>
>
> Other than disabling a Pod – is there a way to prevent live migration of
> VM’s between Pods in ACS ?
>
>
>
> We are on version 4.15.2 with Ubuntu 20.04 KVM hosts.  Each Pod contains a
> single cluster of Homogenous hosts – however there are only slight
> differences between the CPU’s on the physical hosts in each Cluster. We
> have the guest.cpu.mode set to host-passthrough but have noticed serious
> issues when a VM is live migrated between specific Pods (usually if a VM is
> started on a cluster with the slightly better CPU’s and then live migrated
> to an older Pod)
>
>
>
> We have tried setting the guest.cpu.mode to host-passthrough with specific
> CPU features using the “guest.cpu.features=” and then setting all of the
> host’s CPU flags shown from the output of the lscpu command in a space
> separated list as instructed from ACS documentation – but we then are
> unable to even start a VM – insufficient resources error, - if we remove
> the guest.cpu.features from the agent.properties – then we can start a VM
> again.
>
>
>
> It would be good if we had an option or a setting to just not allow live
> migration of VM’s between pods and therefore can only perform a ‘cold’
> migration if we wish to move a VM to another Pod.
>
>
>
> Any thoughts on this ?
>
>
>
> BR
>
>
>
> Gary
>
>
>
>
> Gary Dixon​
> Senior Technical Consultant
> T:  +44 161 537 4990
> E:  *v* <+44%207989717661>ms@quadris‑support.com
> W: www.quadris.co.uk
> The information contained in this e-mail from Quadris may be confidential
> and privileged for the private use of the named recipient.  The contents of
> this e-mail may not necessarily represent the official views of Quadris.
> If you have received this information in error you must not copy,
> distribute or take any action or reliance on its contents.  Please destroy
> any hard copies and delete this message.
>

[ANNOUNCE] New VP of Apache CloudStack - Rohit Yadav

[Simon Weller]

All,

I'm very pleased to announce that the ASF board has accepted the nomination
of Rohit Yadav to be the new VP of the Apache CloudStack project.

It has been my pleasure to serve as the VP over the past year, and I'd like
to thank the community for all of the support.

Rohit, congratulations and I wish you the best as you take on this new role.

-Simon

Re: Daan Hoogland - New ASF Member

[Simon Weller]

Wow, that's fantastic. Congratulations Daan!

-Si

On Fri, Mar 24, 2023 at 4:29 AM Paul Angus  wrote:

>
>
> It is my pleasure to announce that Daan Hoogland as been elected to become
> a
> member of the ASF.
>
> The ASF would like to recognize both his practical involvement and the way
> in which he has interacted with others in and around the ASF.
>
>
>
> Congratulations  Daan.
>
>
>
>
>
>
>
>
>
> Kind regards
>
>
>
> Paul Angus
>
>
>
>

Re: Getting Usage Response from deleted Root-Disk: Cloudstack 4.15

[Simon Weller]

Joan,

Can you give us some logs and perhaps some sample records where you see the
issue?

Are all of your systems successfully syncing NTP?

-Si

On Wed, Mar 22, 2023 at 9:17 AM Joan g  wrote:

> Hi,
>
> I am using cloudstack 4.16 and usage records are generated and wrong hours
> are available for cloudstack deleted root disks in usage response.
>
> Can someone explain to me how to stop those invalid responses ?
>

Re: Lifecycle hooks

[Simon Weller]

Hi Stuart,

The easiest way to implement some lifecycle hooks is by using the event bus
feature. CloudStack supports In-memory, RabbitMQ and Kafta for publishing
events.
You can filter these down via the queuing mechanism of your choice and
trigger based on the event.

See here for more info -
https://docs.cloudstack.apache.org/en/latest/adminguide/events.html?highlight=eventing

-Si


On Thu, Mar 16, 2023 at 3:46 PM Stuart Whitman 
wrote:

> Hello,
>
> How would you implement a lifecycle hook in Cloudstack? For example, when
> a VM for a particular purpose is created, I want to register it with Red
> Hat Identify Management and unregister it when it is destroyed.
>
> Thanks,
> -Stu
> _
> The information contained in this e-mail and any attachments from Group W
> may contain confidential and/or proprietary information and is intended
> only for the named recipient to whom it was originally addressed. If you
> are not the intended recipient, be aware that any disclosure, distribution,
> or copying of this e-mail or its attachments is strictly prohibited. If you
> have received this e-mail in error, please notify the sender immediately of
> that fact by return e-mail and permanently delete the e-mail and any
> attachments to it.
>

Re: [4.18][RC3][VOTE] new release candidate 4.18.0.0-RC20230311T0935

[Simon Weller]

Installed RC3 from packages on AlmaLinux 8.

Tested -
KVM advanced zone
Isolated network and VPC network
VM Lifecycle Management

+1 (binding)

On Sat, Mar 11, 2023 at 2:46 AM Daan Hoogland  wrote:

> Hi All,
>
> I've created a new 4.18 release candidate, with the following
> artifacts up for a vote:
>
> Git Branch and Commit
> SH:
> https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;a=shortlog;h=refs/heads/4.18.0.0-RC20230311T0935
> Commit: 0574087284f8b646ebc41617cfd70b3a31e3ae94
>
> Source release (checksums and signatures are available at the same
> location): https://dist.apache.org/repos/dist/dev/cloudstack/4.18.0.0
>
> PGP release keys (signed using
> 256ABDFB8D89EDE07540BE6ACEF9E802B86FEED4):
> https://dist.apache.org/repos/dist/release/cloudstack/KEYS
>
> Vote will be open for 72 hours. (aiming for wednesday late)
>
> For sanity in tallying the vote, can PMC members please be sure to
> indicate "(binding)" with their vote?
>
> [ ] +1  approve
> [ ] +0  no opinion
> [ ] -1  disapprove (and reason why)
>
> As usual lately, I will work on getting some convenience packages
> ready and will follow up on this thread to link you to those.
>
> enjoy
>

Re: Console Proxy VM TLS version and cipher suites

[Simon Weller]

Gary,

Can you provide more information as to which CloudStack version you're
running and also where you made modifications? Was it to the Tomcat config?
As Kiran indicated, you should not see any old TLS versions offered in
modern versions of CloudStack. So, if you are, we want to get to the bottom
of it quickly.

-Si

On Wed, Mar 8, 2023 at 3:48 AM Gary Dixon 
wrote:

>
> The PEN test had picked up that a JBoss Enterprise Application was
> allowing TLS v1.0 and TLS v1.1- we have managed to disable this now but
> obviously we would need to build this in to a new System VM template to
> make the change persist a Console Proxy VM rebuild
> Gary Dixon​
> Senior Technical Consultant
> T:  +44 161 537 4990
> E:  *v* <+44%207989717661>ms@quadris‑support.com
> W: www.quadris.co.uk
> The information contained in this e-mail from Quadris may be confidential
> and privileged for the private use of the named recipient.  The contents of
> this e-mail may not necessarily represent the official views of Quadris.
> If you have received this information in error you must not copy,
> distribute or take any action or reliance on its contents.  Please destroy
> any hard copies and delete this message.
>
> From: Kiran Chavala 
> Sent: Tuesday, March 7, 2023 12:59 PM
> To: users@cloudstack.apache.org
> Subject: Re: Console Proxy VM TLS version and cipher suites
>
> Hi Gary
>
> AFAIK, I think cloudstack has disabled anything below TLS v1.2 from 4.11.0
> release
>
>
>
>
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fcloudstack%2Fpull%2F2480=05%7C01%7CGary.Dixon%40quadris.co.uk%7C8bc43b9aac7341c924db08db1f0bee5d%7Cf1d6abf3d3b44894ae16db0fb93a96a2%7C0%7C0%7C638137908353696323%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=Ezd9nXe6wavgsWaZntbfm6s3fj%2FdaWRle%2BNQbZYcaKg%3D=0
>
>
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCLOUDSTACK-10319=05%7C01%7CGary.Dixon%40quadris.co.uk%7C8bc43b9aac7341c924db08db1f0bee5d%7Cf1d6abf3d3b44894ae16db0fb93a96a2%7C0%7C0%7C638137908353696323%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=5DMAQJ38va8zfrqiNml2l6xp8KNEiQWjFVc8DQDjePQ%3D=0
>
> [
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fopengraph.githubassets.com%2F2b9813d128412ed49741e9c7523f4d3fb466d19b3c3b290539fb876ba1bcf0a9%2Fapache%2Fcloudstack%2Fpull%2F2480=05%7C01%7CGary.Dixon%40quadris.co.uk%7C8bc43b9aac7341c924db08db1f0bee5d%7Cf1d6abf3d3b44894ae16db0fb93a96a2%7C0%7C0%7C638137908353696323%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=9lCnFXXAzx6fkhd1mm4ICMFgA1wqQwXAr%2BM4gQfOgFw%3D=0
> ]<
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fcloudstack%2Fpull%2F2480=05%7C01%7CGary.Dixon%40quadris.co.uk%7C8bc43b9aac7341c924db08db1f0bee5d%7Cf1d6abf3d3b44894ae16db0fb93a96a2%7C0%7C0%7C638137908353696323%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=Ezd9nXe6wavgsWaZntbfm6s3fj%2FdaWRle%2BNQbZYcaKg%3D=0
> >
> CLOUDSTACK-10319: Prefer TLSv1.2, deprecate TLSv1.0,1.1 by rohityadavcloud
> · Pull Request #2480 · apache/cloudstack<
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fcloudstack%2Fpull%2F2480=05%7C01%7CGary.Dixon%40quadris.co.uk%7C8bc43b9aac7341c924db08db1f0bee5d%7Cf1d6abf3d3b44894ae16db0fb93a96a2%7C0%7C0%7C638137908353696323%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=Ezd9nXe6wavgsWaZntbfm6s3fj%2FdaWRle%2BNQbZYcaKg%3D=0
> >
> This deprecates and remove TLS 1.0 and 1.1 from preferred list of
> protocols and keeps only TLSv1.2. @blueorangutan package github.com 
>
>
> Regards
> Kiran
> 
> From: Gary Dixon 
> Sent: 07 March 2023 17:35
> To: users@cloudstack.apache.org 
> Subject: Console Proxy VM TLS version and cipher suites
>
>
>
>
>
>
> Hi all
>
>
>
> Is there a way of limiting the console proxy to allow nothing below TLS
> v1.2, 1.3 and only allow strong cipher suites – we are failing a PEN test
> currently and need to strengthen the CPVM security ?
>
>
>
> TIA
>
>
>
> Gary
>
> Gary Dixon​
> Senior Technical Consultant
> T: +44 161 537 4990
> E: vms@quadris‑support.com
> W:
> https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.quadris.co.uk%2F=05%7C01%7CGary.Dixon%40quadris.co.uk%7C8bc43b9aac7341c924db08db1f0bee5d%7Cf1d6abf3d3b44894ae16db0fb93a96a2%7C0%7C0%7C638137908353696323%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=ELMyfDyavuFHOtvcyf7PvqWUFkMwhmWHJPADH6nd%2FnE%3D=0
> [cid:image056775.png@576B6FF7.488A06BD]
> The information contained in this e-mail from Quadris may be confidential
> and privileged for the private use of the 

Re: User-Data for Windows Instance

[Simon Weller]

Ranjit,

As Stephan has suggested, Cloudinit for Windows, supported by Cloudbase is
the way to go. It works very well and is fully compatible with CloudStack
user data.

-Si

On Wed, Mar 8, 2023 at 2:29 PM Stephan Bienek 
wrote:

> Hi Ranjit,
>
> while searching for the same some time ago i stumbled across
> https://cloudbase.it/cloudbase-init/
> which looks like the ready-to-use windows-enabled cloud-init i was looking
> for.
>
> Not tested yet, but the description sounds promising.
>
> If you prefer to self-develop something similar, independent of the OS you
> can always query
> http://data-server./latest/user-data
> and to retrieve parse the provided user-data via scripts and take actions.
> (see
> https://docs.cloudstack.apache.org/en/latest/adminguide/virtual_machines/user-data.html
> )
>
> Best regards,
> Stephan
>
> > Ranjit Jadhav  hat am 08.03.2023 20:04 CET
> geschrieben:
> >
> >
> > Hello Folks.
> >
> > On the KVM hypervisor, we need to pass user-data i.e. bat script while
> > creating a windows instance. Is it possible?  Which guest-tools should we
> > use?
> >
> > Thank you,
> > Ranjit
>

Re: SSVM unable to connect to SecStorage via Storage Network

[Simon Weller]

Antoine,

When you defined the  zone, did you place storage on its own traffic label?

Could you share your zone network config, along with your traffic labels
and the associated services on each label?

-Si

On Mon, Feb 27, 2023, 9:03 PM Antoine Boucher  wrote:

> Following to my previous message I have decided to simplify my
> observations.
>
> Looking at the SSVM network and routing.  My storage ip is defined on eth3
> but the routing of the storage subnet is configured via eth1, the
> Management Network.
>
> Is this an oversight or a desired behaviour?  The only way to solve the
> issue is for me to use the Management Network for SecStorage or change the
> routing through eth3.
>
> I’m assuming that this issue has been reported a number of times?
>
> Regards,
> Antoine
>
>

Re: Cloudstack Software Defined Storage

[Simon Weller]

Adding a couple of others, I missed.

On Fri, Feb 24, 2023 at 9:55 AM Simon Weller  wrote:

> Hi Mariusz,
>
> CloudStack supports a wide range of storage options that are suitable for
> hyperconverged deployments (in no particular order) -
>
> Non-clustered Local storage
> LinStor
> StorPool
> Ceph
> Dell VxFlex
> NFS
> PowerFlex/ScaleIO
>
Solidfire (NetApp)
Nexenta
Datera
Cloudbyte

>
> If you plan to start small, some of these are too overkill for what you're
> trying to do.
> The other consideration is the hypervisor you are planning to use, as not
> all storage solutions support all hypervisors.
>
> Could you share your use case and the workloads you are planning to run?
>
> -Si
>
>
> On Fri, Feb 24, 2023 at 5:29 AM Mariusz Wojtarek
>  wrote:
>
>> I would like to build at start 2 or 3 node HCI with shared software
>> defined storage, what solution you recommend ?
>>
>>
>>
>>
>>
>>
>>
>> [image: Support Online]
>>
>> *Mariusz Wojtarek*
>>
>> Administrator iT
>>
>> *P: *22 335 28 00
>>
>> *E: *mariusz.wojta...@support-online.pl
>>
>> www.support-online.pl
>>
>> Poleczki 23 | 02-822 Warszawa
>>
>> [image: cidimage005.png@01D77E41.D4CA7110]
>> <https://www.facebook.com/Support-Online-147141642016372>   [image:
>> cidimage007.png@01D77E41.D4CA7110]
>> <https://linkedin.com/company/support-online>
>>
>>
>>
>>
>> --
>> Support OnLine Sp. z o.o., ul. Poleczki 23, 02-822 Warszawa, NIP:
>> 951-20-32-692, Regon: 017431965, KRS: 078497,
>> XIII Wydział Gospodarczy Krajowego Rejestru Sądowego w Warszawie, Kapitał
>> zakładowy: 50 000 PLN - opłacony w pełnej wysokości.
>> W przypadku podania w ramach niniejszej korespondencji Państwa danych
>> osobowych, prosimy zapoznać się z następującą informacją
>> https://www.support-online.pl/dane-osobowe
>> In case any of your private data was included within this conversation,
>> please consult this website
>> https://www.support-online.pl/en/personal-data
>>
>

Re: Cloudstack Software Defined Storage

[Simon Weller]

Hi Mariusz,

CloudStack supports a wide range of storage options that are suitable for
hyperconverged deployments (in no particular order) -

Non-clustered Local storage
LinStor
StorPool
Ceph
Dell VxFlex
NFS
PowerFlex/ScaleIO

If you plan to start small, some of these are too overkill for what you're
trying to do.
The other consideration is the hypervisor you are planning to use, as not
all storage solutions support all hypervisors.

Could you share your use case and the workloads you are planning to run?

-Si


On Fri, Feb 24, 2023 at 5:29 AM Mariusz Wojtarek
 wrote:

> I would like to build at start 2 or 3 node HCI with shared software
> defined storage, what solution you recommend ?
>
>
>
>
>
>
>
> [image: Support Online]
>
> *Mariusz Wojtarek*
>
> Administrator iT
>
> *P: *22 335 28 00
>
> *E: *mariusz.wojta...@support-online.pl
>
> www.support-online.pl
>
> Poleczki 23 | 02-822 Warszawa
>
> [image: cidimage005.png@01D77E41.D4CA7110]
>    [image:
> cidimage007.png@01D77E41.D4CA7110]
> 
>
>
>
>
> --
> Support OnLine Sp. z o.o., ul. Poleczki 23, 02-822 Warszawa, NIP:
> 951-20-32-692, Regon: 017431965, KRS: 078497,
> XIII Wydział Gospodarczy Krajowego Rejestru Sądowego w Warszawie, Kapitał
> zakładowy: 50 000 PLN - opłacony w pełnej wysokości.
> W przypadku podania w ramach niniejszej korespondencji Państwa danych
> osobowych, prosimy zapoznać się z następującą informacją
> https://www.support-online.pl/dane-osobowe
> In case any of your private data was included within this conversation,
> please consult this website https://www.support-online.pl/en/personal-data
>

Re: Stuck in Preparing for maintenance on primary storage

[Simon Weller]

The VM filesystem is putting itself in read-only mode, so something odd
appears to be going on.

On Wed, Feb 22, 2023, 7:07 AM Simon Weller  wrote:

> Jeremy,
>
> Any chance you have a write permission problem on your new NFS server?
> Those errors indicate an underlying storage issue.
>
> -Si
>
> On Tue, Feb 21, 2023, 11:46 PM Jeremy Hansen 
> wrote:
>
>> Oh and the system vm’s continue to stay in Starting state.
>>
>> -jeremy
>>
>>
>>
>> On Tuesday, Feb 21, 2023 at 9:44 PM, Me  wrote:
>> The vm’s finally stopped and restarted.  This is what I’m seeing in dmesg
>> on the secondary storage vm:
>>
>> root@s-60-VM:~# dmesg  | grep -i error
>> [3.861852] blk_update_request: I/O error, dev vda, sector 6787872 op
>> 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
>> [3.865833] blk_update_request: I/O error, dev vda, sector 6787872 op
>> 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
>> [3.869553] systemd[1]: Failed to read configured hostname:
>> Input/output error
>> [4.560419] EXT4-fs (vda6): re-mounted. Opts: errors=remount-ro
>> [4.646460] blk_update_request: I/O error, dev vda, sector 6787160 op
>> 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
>> [4.650710] blk_update_request: I/O error, dev vda, sector 6787160 op
>> 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
>> [4.975915] blk_update_request: I/O error, dev vda, sector 6787856 op
>> 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
>> [4.980318] blk_update_request: I/O error, dev vda, sector 6787856 op
>> 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
>> [5.018828] blk_update_request: I/O error, dev vda, sector 6787136 op
>> 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
>> [5.022976] blk_update_request: I/O error, dev vda, sector 6787136 op
>> 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
>> [5.026750] blk_update_request: I/O error, dev vda, sector 6787136 op
>> 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
>> [5.460315] blk_update_request: I/O error, dev vda, sector 6787856 op
>> 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
>> [   10.415215] print_req_error: 16 callbacks suppressed
>> [   10.415219] blk_update_request: I/O error, dev vda, sector 6787864 op
>> 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
>> [   13.362595] blk_update_request: I/O error, dev vda, sector 6787136 op
>> 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
>> [   13.388990] blk_update_request: I/O error, dev vda, sector 6787136 op
>> 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
>> [   13.787276] blk_update_request: I/O error, dev vda, sector 6399408 op
>> 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
>> [   13.791575] blk_update_request: I/O error, dev vda, sector 6399408 op
>> 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
>> [   14.632299] blk_update_request: I/O error, dev vda, sector 6787136 op
>> 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
>> [   14.658283] blk_update_request: I/O error, dev vda, sector 6787136 op
>> 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
>>
>> -jeremy
>>
>>
>>
>> On Tuesday, Feb 21, 2023 at 8:57 PM, Me  wrote:
>> The node cloudstack is claiming the system vm’s is starting on shows no
>> signs of any vm’s running.  virsh list is black.
>>
>> Thanks
>> -jeremy
>>
>>
>>
>> On Tuesday, Feb 21, 2023 at 8:23 PM, Me  wrote:
>> Also, just to note, I’m not sure how much made it in to the logs.  The
>> system vm’s are stuck in starting state and trying to kill through the
>> interface doesn’t seem to do anything.
>>
>> -jeremy
>>
>>
>>
>>
>> On Tuesday, Feb 21, 2023 at 8:20 PM, Me  wrote:
>> Is there something else I can use to submit logs?  Too much for pastebin.
>>
>> Thanks
>> -jeremy
>>
>>
>>
>> On Tuesday, Feb 21, 2023 at 7:07 PM, Simon Weller 
>> wrote:
>> Can you pull some management server logs and also put the CloudStack KVM
>> agent into debug mode before destroying the ssvm and share the logs?
>>
>>
>> https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=30147350#content/view/30147350
>>
>> On Tue, Feb 21, 2023, 8:33 PM Jeremy Hansen 
>> wrote:
>>
>> Yes. It’s just a different partition on the same nfs server.
>>
>>
>>
>> On Tuesday, Feb 21, 2023 at 6:02 PM, Simon Weller 
>> wrote:
>> The new and old primary storage is in the same zone, correct?
>> Did you also change out the secondary storage?
>>
>> On Tue, Feb 21, 2023, 7:59 PM Jeremy Hansen 
>

Re: Stuck in Preparing for maintenance on primary storage

[Simon Weller]

Jeremy,

Any chance you have a write permission problem on your new NFS server?
Those errors indicate an underlying storage issue.

-Si

On Tue, Feb 21, 2023, 11:46 PM Jeremy Hansen 
wrote:

> Oh and the system vm’s continue to stay in Starting state.
>
> -jeremy
>
>
>
> On Tuesday, Feb 21, 2023 at 9:44 PM, Me  wrote:
> The vm’s finally stopped and restarted.  This is what I’m seeing in dmesg
> on the secondary storage vm:
>
> root@s-60-VM:~# dmesg  | grep -i error
> [3.861852] blk_update_request: I/O error, dev vda, sector 6787872 op
> 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
> [3.865833] blk_update_request: I/O error, dev vda, sector 6787872 op
> 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
> [3.869553] systemd[1]: Failed to read configured hostname:
> Input/output error
> [4.560419] EXT4-fs (vda6): re-mounted. Opts: errors=remount-ro
> [4.646460] blk_update_request: I/O error, dev vda, sector 6787160 op
> 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
> [4.650710] blk_update_request: I/O error, dev vda, sector 6787160 op
> 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
> [4.975915] blk_update_request: I/O error, dev vda, sector 6787856 op
> 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
> [4.980318] blk_update_request: I/O error, dev vda, sector 6787856 op
> 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
> [5.018828] blk_update_request: I/O error, dev vda, sector 6787136 op
> 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
> [5.022976] blk_update_request: I/O error, dev vda, sector 6787136 op
> 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
> [5.026750] blk_update_request: I/O error, dev vda, sector 6787136 op
> 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
> [5.460315] blk_update_request: I/O error, dev vda, sector 6787856 op
> 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
> [   10.415215] print_req_error: 16 callbacks suppressed
> [   10.415219] blk_update_request: I/O error, dev vda, sector 6787864 op
> 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
> [   13.362595] blk_update_request: I/O error, dev vda, sector 6787136 op
> 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
> [   13.388990] blk_update_request: I/O error, dev vda, sector 6787136 op
> 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
> [   13.787276] blk_update_request: I/O error, dev vda, sector 6399408 op
> 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
> [   13.791575] blk_update_request: I/O error, dev vda, sector 6399408 op
> 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
> [   14.632299] blk_update_request: I/O error, dev vda, sector 6787136 op
> 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
> [   14.658283] blk_update_request: I/O error, dev vda, sector 6787136 op
> 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
>
> -jeremy
>
>
>
> On Tuesday, Feb 21, 2023 at 8:57 PM, Me  wrote:
> The node cloudstack is claiming the system vm’s is starting on shows no
> signs of any vm’s running.  virsh list is black.
>
> Thanks
> -jeremy
>
>
>
> On Tuesday, Feb 21, 2023 at 8:23 PM, Me  wrote:
> Also, just to note, I’m not sure how much made it in to the logs.  The
> system vm’s are stuck in starting state and trying to kill through the
> interface doesn’t seem to do anything.
>
> -jeremy
>
>
>
>
> On Tuesday, Feb 21, 2023 at 8:20 PM, Me  wrote:
> Is there something else I can use to submit logs?  Too much for pastebin.
>
> Thanks
> -jeremy
>
>
>
> On Tuesday, Feb 21, 2023 at 7:07 PM, Simon Weller 
> wrote:
> Can you pull some management server logs and also put the CloudStack KVM
> agent into debug mode before destroying the ssvm and share the logs?
>
>
> https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=30147350#content/view/30147350
>
> On Tue, Feb 21, 2023, 8:33 PM Jeremy Hansen 
> wrote:
>
> Yes. It’s just a different partition on the same nfs server.
>
>
>
> On Tuesday, Feb 21, 2023 at 6:02 PM, Simon Weller 
> wrote:
> The new and old primary storage is in the same zone, correct?
> Did you also change out the secondary storage?
>
> On Tue, Feb 21, 2023, 7:59 PM Jeremy Hansen 
> wrote:
>
> Yes. On Kvm. I’ve been trying to destroy them from the interface and it
> just keep churning. I did a destroy with virsh, but no status changed in
> the interface. Also, the newly created ones don’t seem to bring up their
> agent and never fully start.
>
> Thanks
>
>
>
> On Tuesday, Feb 21, 2023 at 4:37 PM, Simon Weller 
> wrote:
> Just destroy the old system VMs and they will be recreated on available
> storage.
>
> Are you on KVM?
>
>
>
> On Tue, Feb 21, 2023, 6:14 PM Jeremy Hansen 
> wrote:
>
> How do I completely recreate 

Re: Stuck in Preparing for maintenance on primary storage

[Simon Weller]

Can you pull some management server logs and also put the CloudStack KVM
agent into debug mode before destroying the ssvm and share the logs?

https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=30147350#content/view/30147350

On Tue, Feb 21, 2023, 8:33 PM Jeremy Hansen 
wrote:

> Yes. It’s just a different partition on the same nfs server.
>
>
>
> On Tuesday, Feb 21, 2023 at 6:02 PM, Simon Weller 
> wrote:
> The new and old primary storage is in the same zone, correct?
> Did you also change out the secondary storage?
>
> On Tue, Feb 21, 2023, 7:59 PM Jeremy Hansen 
> wrote:
>
> Yes. On Kvm. I’ve been trying to destroy them from the interface and it
> just keep churning. I did a destroy with virsh, but no status changed in
> the interface. Also, the newly created ones don’t seem to bring up their
> agent and never fully start.
>
> Thanks
>
>
>
> On Tuesday, Feb 21, 2023 at 4:37 PM, Simon Weller 
> wrote:
> Just destroy the old system VMs and they will be recreated on available
> storage.
>
> Are you on KVM?
>
>
>
> On Tue, Feb 21, 2023, 6:14 PM Jeremy Hansen 
> wrote:
>
> How do I completely recreate the system vm?
>
> I was able to get the old storage in to full maintenance and deleted it,
> so maybe the system vm are still using the old storage? Is there a way to
> tell the system vm’s to use the new storage? Db change?
>
> Thanks!
>
>
>
> On Tuesday, Feb 21, 2023 at 1:36 PM, Simon Weller 
> wrote:
> Hey Jeremy,
>
> Is there anything in the management logs that indicate why it's not
> completing the maintenance action?
> Usually, this state is triggered by some stuck VMs that haven't migrated
> off of the primary storage.
>
> You mentioned the system VMs. Are they still on the old storage? Could
> this
> be due to some storage tags?
>
> -Si
>
> On Tue, Feb 21, 2023 at 2:35 PM Jeremy Hansen 
> wrote:
>
> Any ideas on this? I’m completely stuck. Can’t bring up my system vm’s
> and I can’t remove the old primary storage.
>
> -jeremy
>
>
>
> On Tuesday, Feb 21, 2023 at 2:35 AM, Me  wrote:
> I tried to put one of my primary storage definitions in to maintenance
> mode. Now it’s stuck in preparing for maintenance and I’m not sure how to
> remedy this situation:
>
> Cancel maintenance mode
> (NFS Primary) Resource [StoragePool:1] is unreachable: Primary storage
> with id 1 is not ready to complete migration, as the status
> is:PrepareForMaintenance
>
> Restarted manager, agents, libvirtd. My secondarystoragevm can’t start…
>
> 4.17.2.0. Using NFS for primary and secondary storage. I was attempting
> to migrate to a new volume. All volumes were moved to the new storage. I
> was simply trying to delete the old storage definition.
>
> Thanks
> -jeremy
>
>
>
>
>
>
>

Re: Stuck in Preparing for maintenance on primary storage

[Simon Weller]

The new and old primary storage is in the same zone, correct?
Did you also change out the secondary storage?

On Tue, Feb 21, 2023, 7:59 PM Jeremy Hansen 
wrote:

> Yes. On Kvm. I’ve been trying to destroy them from the interface and it
> just keep churning. I did a destroy with virsh, but no status changed in
> the interface. Also, the newly created ones don’t seem to bring up their
> agent and never fully start.
>
> Thanks
>
>
>
> On Tuesday, Feb 21, 2023 at 4:37 PM, Simon Weller 
> wrote:
> Just destroy the old system VMs and they will be recreated on available
> storage.
>
> Are you on KVM?
>
>
>
> On Tue, Feb 21, 2023, 6:14 PM Jeremy Hansen 
> wrote:
>
> How do I completely recreate the system vm?
>
> I was able to get the old storage in to full maintenance and deleted it,
> so maybe the system vm are still using the old storage? Is there a way to
> tell the system vm’s to use the new storage? Db change?
>
> Thanks!
>
>
>
> On Tuesday, Feb 21, 2023 at 1:36 PM, Simon Weller 
> wrote:
> Hey Jeremy,
>
> Is there anything in the management logs that indicate why it's not
> completing the maintenance action?
> Usually, this state is triggered by some stuck VMs that haven't migrated
> off of the primary storage.
>
> You mentioned the system VMs. Are they still on the old storage? Could
> this
> be due to some storage tags?
>
> -Si
>
> On Tue, Feb 21, 2023 at 2:35 PM Jeremy Hansen 
> wrote:
>
> Any ideas on this? I’m completely stuck. Can’t bring up my system vm’s
> and I can’t remove the old primary storage.
>
> -jeremy
>
>
>
> On Tuesday, Feb 21, 2023 at 2:35 AM, Me  wrote:
> I tried to put one of my primary storage definitions in to maintenance
> mode. Now it’s stuck in preparing for maintenance and I’m not sure how to
> remedy this situation:
>
> Cancel maintenance mode
> (NFS Primary) Resource [StoragePool:1] is unreachable: Primary storage
> with id 1 is not ready to complete migration, as the status
> is:PrepareForMaintenance
>
> Restarted manager, agents, libvirtd. My secondarystoragevm can’t start…
>
> 4.17.2.0. Using NFS for primary and secondary storage. I was attempting
> to migrate to a new volume. All volumes were moved to the new storage. I
> was simply trying to delete the old storage definition.
>
> Thanks
> -jeremy
>
>
>
>
>
>

Re: Stuck in Preparing for maintenance on primary storage

[Simon Weller]

Just destroy the old system VMs and they will be recreated on available
storage.

Are you on KVM?



On Tue, Feb 21, 2023, 6:14 PM Jeremy Hansen 
wrote:

> How do I completely recreate the system vm?
>
> I was able to get the old storage in to full maintenance and deleted it,
> so maybe the system vm are still using the old storage?  Is there a way to
> tell the system vm’s to use the new storage?  Db change?
>
> Thanks!
>
>
>
> On Tuesday, Feb 21, 2023 at 1:36 PM, Simon Weller 
> wrote:
> Hey Jeremy,
>
> Is there anything in the management logs that indicate why it's not
> completing the maintenance action?
> Usually, this state is triggered by some stuck VMs that haven't migrated
> off of the primary storage.
>
> You mentioned the system VMs. Are they still on the old storage? Could
> this
> be due to some storage tags?
>
> -Si
>
> On Tue, Feb 21, 2023 at 2:35 PM Jeremy Hansen 
> wrote:
>
> Any ideas on this? I’m completely stuck. Can’t bring up my system vm’s
> and I can’t remove the old primary storage.
>
> -jeremy
>
>
>
> On Tuesday, Feb 21, 2023 at 2:35 AM, Me  wrote:
> I tried to put one of my primary storage definitions in to maintenance
> mode. Now it’s stuck in preparing for maintenance and I’m not sure how to
> remedy this situation:
>
> Cancel maintenance mode
> (NFS Primary) Resource [StoragePool:1] is unreachable: Primary storage
> with id 1 is not ready to complete migration, as the status
> is:PrepareForMaintenance
>
> Restarted manager, agents, libvirtd. My secondarystoragevm can’t start…
>
> 4.17.2.0. Using NFS for primary and secondary storage. I was attempting
> to migrate to a new volume. All volumes were moved to the new storage. I
> was simply trying to delete the old storage definition.
>
> Thanks
> -jeremy
>
>
>
>
>

Re: Stuck in Preparing for maintenance on primary storage

[Simon Weller]

Hey Jeremy,

Is there anything in the management logs that indicate why it's not
completing the maintenance action?
Usually, this state is triggered by some stuck VMs that haven't migrated
off of the primary storage.

You mentioned the system VMs. Are they still on the old storage? Could this
be due to some storage tags?

-Si

On Tue, Feb 21, 2023 at 2:35 PM Jeremy Hansen 
wrote:

> Any ideas on this?  I’m completely stuck.  Can’t bring up my system vm’s
> and I can’t remove the old primary storage.
>
> -jeremy
>
>
>
> On Tuesday, Feb 21, 2023 at 2:35 AM, Me  wrote:
> I tried to put one of my primary storage definitions in to maintenance
> mode.  Now it’s stuck in preparing for maintenance and I’m not sure how to
> remedy this situation:
>
> Cancel maintenance mode
> (NFS Primary) Resource [StoragePool:1] is unreachable: Primary storage
> with id 1 is not ready to complete migration, as the status
> is:PrepareForMaintenance
>
> Restarted manager, agents, libvirtd.  My secondarystoragevm can’t start…
>
> 4.17.2.0.  Using NFS for primary and secondary storage.  I was attempting
> to migrate to a new volume.  All volumes were moved to the new storage.  I
> was simply trying to delete the old storage definition.
>
> Thanks
> -jeremy
>
>
>
>

[ANNOUNCE] Ivet Petrova has joined the PMC

[Simon Weller]

Hi everyone,

It gives me great pleasure to announce that Ivet has been invited to join
the
CloudStack PMC and she has accepted.

Please join me in congratulating Ivet!

-Simon (on behalf of the CloudStack PMC)

Re: Cluster HA - KVM Hosts

[Simon Weller]

Hey Jafar,

For HA to work on KVM, you need to configure each host with out-of-band
management using ipmitool or Redfish. This is so Cloudstack can fence your
nodes.

This covers the concept -
https://docs.cloudstack.apache.org/en/latest/adminguide/reliability.html?highlight=fencing#ha-enabled-hosts
Out-of-band documentation is here -
https://docs.cloudstack.apache.org/en/latest/adminguide/hosts.html#out-of-band-management
For details on how the feature works, check out the original design
document - https://cwiki.apache.org/confluence/display/CLOUDSTACK/Host+HA

-Si




On Fri, Feb 10, 2023 at 12:32 PM Jafar Aghabalayev <
jafar.aghabala...@pasha-technology.com> wrote:

> Hello Community,
>
> I have configured cluster with 2 KVM Hosts.
> HA enabled for hosts, ha.tag attribute on global configuration is the same
> as for ha.tag attribute for hosts. VMs are running with HA enabled
> offering. NFS used as primary storage.
> I tried to shutdown one of the hosts, however HA not performed. State and
> power state indicated as down, but resource state show as up and VM located
> at the failed host show in running state (in real it is unaccessible). I
> tried to stop libvirtd service and the result same.
>
> Can anyone help me with this issue?
>
>
>

Re: noob question about Templates

[Simon Weller]

Yeah, I don't think you need to worry about it. Without the modprobe flag
configured on the KVM host, it won't allow a nested VM to be deployed.

-Si

On Fri, Feb 10, 2023 at 10:51 AM Gary Dixon
 wrote:

>
> Hi Simon
>
> Thanks for the info - its quite interesting !!
>
>
> We are running KVM hypervisor on Ubuntu 20.04 on AMD EPYC cpu's
>
> We haven't been enabling HVM on templates as we don't want users to be
> able to nest hypervisors within their VM's
>
> Gary Dixon​
> Senior Technical Consultant
> T:  +44 161 537 4990
> E:  *v* <+44%207989717661>ms@quadris‑support.com
> W: www.quadris.co.uk
> The information contained in this e-mail from Quadris may be confidential
> and privileged for the private use of the named recipient.  The contents of
> this e-mail may not necessarily represent the official views of Quadris.
> If you have received this information in error you must not copy,
> distribute or take any action or reliance on its contents.  Please destroy
> any hard copies and delete this message.
> -Original Message-
> From: Simon Weller 
> Sent: 10 February 2023 16:23
> To: users@cloudstack.apache.org
> Subject: Re: noob question about Templates
>
> Hey Gary,
>
> Welcome to the list.
>
> Firstly, which hypervisor are you using?
>
> HVM used to mean use hardware accelerated virtualization back in the day
> when it was still fairly new. It was required for some hypervisors, but not
> all.
> These days everything is hardware accelerated.
>
> If you are running KVM, you can enable the ability to run nested
> virilization by doing this -
>
> Intel -
> modprobe -r kvm_intel
> modprobe kvm_intel nested=1
>
> Make it permanent by adding the following line to /etc/modprobe.d/kvm.conf:
> options kvm_intel nested=1
>
> AMD -
> modprobe -r kvm_amd
> modprobe kvm_amd nested=1
>
> Make it permanent by adding the following line to /etc/modprobe.d/kvm.conf:
> options kvm_amd nested=1
>
> Please note that CloudStack will not manage any nested virtualization
> unless Cloudstack itself is nested (e.g. a lab environment).
>
> -Si
>
>
> On Fri, Feb 10, 2023 at 8:28 AM Gary Dixon
> 
> wrote:
>
> > HI
> >
> >
> >
> > Sorry for the newbie question but …..
> >
> >
> >
> > When creating a template – does enabling the HVM option allow the
> > virtual machine to also be a hypervisor so that nested virtualization
> > can be achieved?
> > Gary Dixon​
> > Senior Technical Consultant
> > T: +44 161 537 4990
> > E: *v* <+44%207989717661>ms@quadris‑support.com
> > W:
> > https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.q
> > uadris.co.uk%2F=05%7C01%7CGary.Dixon%40quadris.co.uk%7C425ae13b53
> > 394cabd18808db0b831363%7Cf1d6abf3d3b44894ae16db0fb93a96a2%7C0%7C0%7C63
> > 8116429828439052%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV
> > 2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=2GprAUWMFJ
> > cOjv2PGTIr6gZdnFIDOFAZQd61NGvMGBc%3D=0
> > The information contained in this e-mail from Quadris may be
> > confidential and privileged for the private use of the named
> > recipient. The contents of this e-mail may not necessarily represent the
> official views of Quadris.
> > If you have received this information in error you must not copy,
> > distribute or take any action or reliance on its contents. Please
> > destroy any hard copies and delete this message.
> >
>

Re: noob question about Templates

[Simon Weller]

Hey Gary,

Welcome to the list.

Firstly, which hypervisor are you using?

HVM used to mean use hardware accelerated virtualization back in the day
when it was still fairly new. It was required for some hypervisors, but not
all.
These days everything is hardware accelerated.

If you are running KVM, you can enable the ability to run nested
virilization by doing this -

Intel -
modprobe -r kvm_intel
modprobe kvm_intel nested=1

Make it permanent by adding the following line to /etc/modprobe.d/kvm.conf:
options kvm_intel nested=1

AMD -
modprobe -r kvm_amd
modprobe kvm_amd nested=1

Make it permanent by adding the following line to /etc/modprobe.d/kvm.conf:
options kvm_amd nested=1

Please note that CloudStack will not manage any nested virtualization
unless Cloudstack itself is nested (e.g. a lab environment).

-Si


On Fri, Feb 10, 2023 at 8:28 AM Gary Dixon 
wrote:

> HI
>
>
>
> Sorry for the newbie question but …..
>
>
>
> When creating a template – does enabling the HVM option allow the virtual
> machine to also be a hypervisor so that nested virtualization can be
> achieved?
> Gary Dixon​
> Senior Technical Consultant
> T:  +44 161 537 4990
> E:  *v* <+44%207989717661>ms@quadris‑support.com
> W: www.quadris.co.uk
> The information contained in this e-mail from Quadris may be confidential
> and privileged for the private use of the named recipient.  The contents of
> this e-mail may not necessarily represent the official views of Quadris.
> If you have received this information in error you must not copy,
> distribute or take any action or reliance on its contents.  Please destroy
> any hard copies and delete this message.
>

Re: Rocky 9 and CS 4.17.1.0

[Simon Weller]

Jeremy and Robert,

Out of interest, which cpu profile are you passing through to the guest?

I've seen a lot of reports with 9.x and kernel panics when certain CPU
flags aren't being passed through, in particular x16 lahf popcnt sse4_1
sse4_2 ssse3.
Do you have the ability to set your guest.cpu.mode to "host-passthrough"
for the purposes of testing?

You'd do that by editing /etc/cloudstack/agent/agent.properties on your
host.
Look for guest.cpu.mode and replace it with =
guest.cpu.mode=host-passthrough
Then restart the agent.

-Si


On Thu, Feb 2, 2023 at 9:32 AM Robert Ward  wrote:

> Hi Rohit,
>
> I can report that installing Oracle 9.1 on a VM results in a kernel panic
> right off the bat. I hear there were some major changes between EL8 and 9.
> My SaSS customers are holding off on upgrades as well because of potential
> concerns.
>
> Thanks
>
> On 2/2/23, 10:17 AM, "Rohit Yadav"  wrote:
>
> Jeremy,
>
> For EL9, can you try Alma Linux 9 or Oracle Linux 9 for guest VM? I
> have had bad experiences with Rocky Linux 9 in terms of stability and
> mirror/repo accessibility even though Rocky Linux 9 claims to be EL9
> compatible.
>
> Regards.
>
> 
> From: Jeremy Hansen 
> Sent: Saturday, December 10, 2022 7:53 PM
> To: Vivek Kumar via users 
> Subject: Rocky 9 and CS 4.17.1.0
>
> I’m running Cloudstack 4.17.1.0 and for unknown reasons, I’m having
> issues running Rocky 9. Kernel begins to boot and then it looks like it
> fails on loading initrd and I get a kernel oops.  Just curious if this is a
> known issue or if there’s a work around. I tried using the qcow2 image from
> Rocky as well and just using the install iso to create a new image. Same
> result.
>
> Rocky 8 works fine.
>
> Anyone running Rocky 9?
>
> Thanks
> -jeremy
>
>
>
>
>
>
>
>
>

-- 
Simon Weller
si...@potelweller.com

Re: Rocky 9 and CS 4.17.1.0

[Simon Weller]

How many cores and how much ram are you using for this VM?

On Thu, Feb 2, 2023 at 2:53 PM Robert Ward  wrote:

> Hi Simon,
>
> I can confirm that the cpu mode is host-passthrough
>
> FYI - I run KVM for my Hypervisor
>
> If this is any help I took a screenshot of the console.
>
> 6.1522631 Call Trace:
> 6.1525591
> 6.1528651
> dump_stack_lvl+0x34/0x48
> panic+0x102/0x2d4
> 6.153139]
> 6.153438]
> do_exit.cold+0x14/0x9f
> do_group_exit+0x33/Oxa0
> 6.1537371
> 6.1540621
> 6.154393]
> ? exc_page_fault+0x62/0x150
> 6.1547091
> entry_SYSCALL_64_after_hwframe+0x63/Oxcd
> 6.1550791 RIP: 0033:0x78 acobb?c2d
> 6.1554101 Code: c3 Of 11 84 00 00 00 00 00 f3 Of 1e fa be e7 00 00 00 ba
> 3cl
> 00
> 00 00 eb od 89 do or 05 48 3d 00 fo ff ff 27 1c f4 89 fo of 05 <48› 3d 00
> fo
> ff
> ff
> 76 e7 f7 d8 89 05 ff fe 00 00 eb dd of 1f
> 44 00
> 6.1565621 RSP: 002h: 7ffcf88f6428 EFLAGS: 0246 ORIG RAX:
> 000
> 000e7
> 6.1570511 RAX: fida RBX: 7 dc0bb76130 RCX: 7fdc0bb7c2d1
> 6.1575451 RDX: 003c RSI: 00€7 RDI: 007f
> 6.1580151 RB: 7ffcf88f6eb0 R08: 7ffcf8816899 R09: 
> 6.1586231 R10:  R11: 0246 R12: 7f
> dcObb55000
> 6.1591971 R13: 00230007 R14: 0007 R15: 7ffcf88f6ec0
> 6.1617161 Kernel Offset: 0x2fc0 from Ox8100 (relocation ran
> ge: Ox8000-Oxbfff)
> 6.1626471
> -[ end Kernel panic
> - not syncing: Attempted to kill init? exit
> code=0x7f 00
>
> Thanks
>
> On 2/2/23, 3:11 PM, "Simon Weller"  wrote:
>
> Jeremy and Robert,
>
> Out of interest, which cpu profile are you passing through to the
> guest?
>
> I've seen a lot of reports with 9.x and kernel panics when certain CPU
> flags aren't being passed through, in particular x16 lahf popcnt sse4_1
> sse4_2 ssse3.
> Do you have the ability to set your guest.cpu.mode to
> "host-passthrough"
> for the purposes of testing?
>
> You'd do that by editing /etc/cloudstack/agent/agent.properties on your
> host.
> Look for guest.cpu.mode and replace it with =
> guest.cpu.mode=host-passthrough
> Then restart the agent.
>
> -Si
>
> On Thu, Feb 2, 2023 at 9:32 AM Robert Ward  wrote:
>
> > Hi Rohit,
> >
> > I can report that installing Oracle 9.1 on a VM results in a kernel
> panic
> > right off the bat. I hear there were some major changes between EL8
> and 9.
> > My SaSS customers are holding off on upgrades as well because of
> potential
> > concerns.
> >
> > Thanks
> >
> > On 2/2/23, 10:17 AM, "Rohit Yadav" 
> wrote:
> >
> > Jeremy,
> >
> > For EL9, can you try Alma Linux 9 or Oracle Linux 9 for guest
> VM? I
> > have had bad experiences with Rocky Linux 9 in terms of stability and
> > mirror/repo accessibility even though Rocky Linux 9 claims to be EL9
> > compatible.
> >
> > Regards.
> >
> > 
> > From: Jeremy Hansen 
> > Sent: Saturday, December 10, 2022 7:53 PM
> > To: Vivek Kumar via users 
> > Subject: Rocky 9 and CS 4.17.1.0
> >
> > I’m running Cloudstack 4.17.1.0 and for unknown reasons, I’m
> having
> > issues running Rocky 9. Kernel begins to boot and then it looks like
> it
> > fails on loading initrd and I get a kernel oops.  Just curious if
> this is a
> > known issue or if there’s a work around. I tried using the qcow2
> image from
> > Rocky as well and just using the install iso to create a new image.
> Same
> > result.
> >
> > Rocky 8 works fine.
> >
> > Anyone running Rocky 9?
> >
> > Thanks
> > -jeremy
> >
> >
> >
> >
> >
> >
> >
> >
> >
>
>
>

-- 
Simon Weller
si...@potelweller.com

Re: Rocky 9 and CS 4.17.1.0

[Simon Weller]

Jeremy and Robert,

Out of interest, which cpu profile are you passing through to the guest?

I've seen a lot of reports with 9.x and kernel panics when certain CPU
flags aren't being passed through, in particular x16 lahf popcnt sse4_1
sse4_2 ssse3.
Do you have the ability to set your guest.cpu.mode to "host-passthrough"
for the purposes of testing?

You'd do that by editing /etc/cloudstack/agent/agent.properties on your
host.
Look for guest.cpu.mode and replace it with =
guest.cpu.mode=host-passthrough
Then restart the agent.

-Si

On Thu, Feb 2, 2023 at 9:32 AM Robert Ward  wrote:

> Hi Rohit,
>
> I can report that installing Oracle 9.1 on a VM results in a kernel panic
> right off the bat. I hear there were some major changes between EL8 and 9.
> My SaSS customers are holding off on upgrades as well because of potential
> concerns.
>
> Thanks
>
> On 2/2/23, 10:17 AM, "Rohit Yadav"  wrote:
>
> Jeremy,
>
> For EL9, can you try Alma Linux 9 or Oracle Linux 9 for guest VM? I
> have had bad experiences with Rocky Linux 9 in terms of stability and
> mirror/repo accessibility even though Rocky Linux 9 claims to be EL9
> compatible.
>
> Regards.
>
> 
> From: Jeremy Hansen 
> Sent: Saturday, December 10, 2022 7:53 PM
> To: Vivek Kumar via users 
> Subject: Rocky 9 and CS 4.17.1.0
>
> I’m running Cloudstack 4.17.1.0 and for unknown reasons, I’m having
> issues running Rocky 9. Kernel begins to boot and then it looks like it
> fails on loading initrd and I get a kernel oops.  Just curious if this is a
> known issue or if there’s a work around. I tried using the qcow2 image from
> Rocky as well and just using the install iso to create a new image. Same
> result.
>
> Rocky 8 works fine.
>
> Anyone running Rocky 9?
>
> Thanks
> -jeremy
>
>
>
>
>
>
>
>
>

Re: Volunteers Needed - CCC

[Simon Weller]

Hey Ivet,

I'll be onsite and I'd be happy to help out.

-Si

From: Ivet Petrova 
Sent: Monday, October 3, 2022 11:39 AM
To: Apache CloudStack Marketing ; 
users@cloudstack.apache.org ; 
d...@cloudstack.apache.org 
Subject: Volunteers Needed - CCC

EXTERNAL EMAIL: This message originated outside of ENA. Use caution when 
clicking links, opening attachments, or complying with requests. Click the 
"Phish Alert Report" button above the email, or contact MIS, regarding any 
suspicious message.



Hi all,

As you know we organise the CloudStack Collab oration Conference in November in 
Sofia, Bulgaria as a hybrid event.
As you can imagine, this requires enormous efforts - dealing with speakers, 
sponsors, facility management, branding, attracting attendees, online ads, 
website and much more.
The work on place during the event will also be a significant amount and I need 
help.
At this stage I am searching for a volunteer/s to handle the registration on 
place.
More details:
The registration will happen only on the first event day and will take approx. 
1 hour. During this period, the task is to find the badge for the people 
arriving and to give it to them. We will have printed badges, which everyone 
can wear.
I think we will need 2 people for this. And will be very happy if I can find 
some volunteers :)

Kind regards,

Re: [DISCUSS] release and release manager 4.18

[Simon Weller]

+1 to Daan :-)

From: Rohit Yadav 
Sent: Tuesday, September 20, 2022 9:43 AM
To: users@cloudstack.apache.org ; dev 

Subject: Re: [DISCUSS] release and release manager 4.18

EXTERNAL EMAIL: This message originated outside of ENA. Use caution when 
clicking links, opening attachments, or complying with requests. Click the 
"Phish Alert Report" button above the email, or contact MIS, regarding any 
suspicious message.



+1 thanks for volunteering as 4.18 RM Daan.

Regards.

Regards.

From: Daan Hoogland 
Sent: Tuesday, September 20, 2022 7:53:53 PM
To: dev ; users 
Subject: [DISCUSS] release and release manager 4.18

Hello everybody,
As Abhishek is wrapping up the 4.17.1 release. I´d like to start the
discussion about 4.18. Nobody has volunteered as the next release manager
yet and I'd like to give the community the opportunity to prevent it
becoming me. My idea is that we will have set a feature freeze date and
then a target date for the first RC. Iḿ not proposing one now as mine would
probably be too tight a deadline for new features and there are some big
ones out there that we'd want in.
Please let us all know what you have in the pipeline and what you want in
and what time you need, so we can discuss if it is feasible for 4.18.
Also please come forward if you want this task of triaging issues and PRs
and tagging them to go in the release.
thanks

--
Daan

Re: Cloudstack as cloudprovider in Rancher

[Simon Weller]

Ricardo,

Check out this blog article on Rancher, Terraform and Cloudstack - 
https://fe.ax/rancher-with-terraform-on-cloudstack/

-Si

From: Ricardo Pertuz 
Sent: Thursday, September 1, 2022 6:44 AM
To: users@cloudstack.apache.org 
Subject: Cloudstack as cloudprovider in Rancher

EXTERNAL EMAIL: This message originated outside of ENA. Use caution when 
clicking links, opening attachments, or complying with requests. Click the 
"Phish Alert Report" button above the email, or contact MIS, regarding any 
suspicious message.



Hi community,

Has anyone configured ACS as cloudprovider in Rancher?

BR,

Ricardo

Re: Virtual Router filesystem corruption

[Simon Weller]

sarily represent the official views of Quadris.  If you 
have received this information in error you must not copy, distribute or take 
any action or reliance on its contents.  Please destroy any hard copies and 
delete this message.

-Original Message-----
From: Simon Weller mailto:swel...@ena.com.INVALID>>
Sent: 20 July 2022 22:10
To: users@cloudstack.apache.org<mailto:users@cloudstack.apache.org>
Subject: Re: Virtual Router filesystem corruption

Gary,

No prob with the info, thanks for providing it.

Since you're using Storpool, I'd suggest you reach out to them on this directly 
and see whether they have any information that could be helpful.

There was an issue a while ago (Storpool actually reported it) where a kernel 
commit introduced a bug that caused file corruption. That was back in about 
2018 - 
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstorpool.com%2Fblog%2Fbeware-silent-data-corruption-discovered-in-linux-kernels-4-10-4-17%2Fdata=05%7C01%7CGary.Dixon%40quadris.co.uk%7Cece461d8f261463ed2b408da6a9433b7%7Cf1d6abf3d3b44894ae16db0fb93a96a2%7C0%7C0%7C637939481997555790%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7Csdata=yajiRaYix9u2mhCLVnG9%2FDQHcS9tSPNuhzORVgTEBQ4%3Dreserved=0
I believe ACS 4.15.x uses Debian 10.5 (Buster) for the VR images (dates to 
August 2020), That release is based on kernel 4.19.0-10.

-Si







From: Gary Dixon 
mailto:gary.di...@quadris.co.uk.INVALID>>
Sent: Wednesday, July 20, 2022 3:00 PM
To: users@cloudstack.apache.org<mailto:users@cloudstack.apache.org> 
mailto:users@cloudstack.apache.org>>
Subject: Re: Virtual Router filesystem corruption

EXTERNAL EMAIL: This message originated outside of ENA. Use caution when 
clicking links, opening attachments, or complying with requests. Click the 
"Phish Alert Report" button above the email, or contact MIS, regarding any 
suspicious message.


Hi SI

Sure. Sorry for the lack of info. First time posting on the forum.
We are using KVM hyper visor on Ubuntu 20.04 hosts. Primary storage is 
Storpool. Let me know if you need more info Best regards Gary

Get Outlook for 
iOS<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2Fo0ukefdata=05%7C01%7CGary.Dixon%40quadris.co.uk%7Cece461d8f261463ed2b408da6a9433b7%7Cf1d6abf3d3b44894ae16db0fb93a96a2%7C0%7C0%7C637939481997555790%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7Csdata=mTRmXAFdzzfnILhmSaX%2BYityquQqipKEofQN5XSsLUs%3Dreserved=0>
Gary Dixon​
Technical Consultant
T: 0161 537 4980
W: 
https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.quadris.co.uk%2Fdata=05%7C01%7CGary.Dixon%40quadris.co.uk%7Cece461d8f261463ed2b408da6a9433b7%7Cf1d6abf3d3b44894ae16db0fb93a96a2%7C0%7C0%7C637939481997555790%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7Csdata=7akNVPG9bwo%2FYtqh1NH0DBJxW81kL2EuF%2Br9dqiWoAU%3Dreserved=0
[cid:image937435.png@813B89FF.AD6D2675]
The information contained in this e-mail from Quadris may be confidential and 
privileged for the private use of the named recipient. The contents of this 
e-mail may not necessarily represent the official views of Quadris. If you have 
received this information in error you must not copy, distribute or take any 
action or reliance on its contents. Please destroy any hard copies and delete 
this message.

From: Simon Weller mailto:swel...@ena.com.INVALID>>
Sent: Wednesday, July 20, 2022 8:55:22 PM
To: users@cloudstack.apache.org<mailto:users@cloudstack.apache.org> 
mailto:users@cloudstack.apache.org>>
Subject: Re: Virtual Router filesystem corruption

Gary,

Can you provide some information about the OS, underlying hypervisor and 
primary storage in use?

-Si

From: Gary Dixon 
mailto:gary.di...@quadris.co.uk.INVALID>>
Sent: Wednesday, July 20, 2022 11:15 AM
To: users@cloudstack.apache.org<mailto:users@cloudstack.apache.org> 
mailto:users@cloudstack.apache.org>>
Subject: Virtual Router filesystem corruption

EXTERNAL EMAIL: This message originated outside of ENA. Use caution when 
clicking links, opening attachments, or complying with requests. Click the 
"Phish Alert Report" button above the email, or contact MIS, regarding any 
suspicious message.





Hi All



We are seeing ext4 filesystem corruption on a number of virtual routers 
recently and manually running fsck doesn’t appear to help at all in fixing the 
issue (Corrupt inode bitmap)

We end up having to restart the associated VPC with cleanup enable to rebuild a 
new VR. Is this a common issue with ACS 4.15.1 ? Or are there specific 
circumstances that are causing the VR fs corruption that we could perhaps 
mitigate ?



Kind regards



Gary

Gary Dixon​​

Re: Virtual Router filesystem corruption

[Simon Weller]

Gary,

No prob with the info, thanks for providing it.

Since you're using Storpool, I'd suggest you reach out to them on this directly 
and see whether they have any information that could be helpful.

There was an issue a while ago (Storpool actually reported it) where a kernel 
commit introduced a bug that caused file corruption. That was back in about 
2018  - 
https://storpool.com/blog/beware-silent-data-corruption-discovered-in-linux-kernels-4-10-4-17/
I believe ACS 4.15.x uses Debian 10.5 (Buster) for the VR images (dates to 
August 2020), That release is based on kernel 4.19.0-10.

-Si







From: Gary Dixon 
Sent: Wednesday, July 20, 2022 3:00 PM
To: users@cloudstack.apache.org 
Subject: Re: Virtual Router filesystem corruption

EXTERNAL EMAIL: This message originated outside of ENA. Use caution when 
clicking links, opening attachments, or complying with requests. Click the 
"Phish Alert Report" button above the email, or contact MIS, regarding any 
suspicious message.


Hi SI

Sure. Sorry for the lack of info. First time posting on the forum.
We are using KVM hyper visor on Ubuntu 20.04 hosts. Primary storage is 
Storpool. Let me know if you need more info
Best regards
Gary

Get Outlook for iOS<https://aka.ms/o0ukef>
Gary Dixon​
Technical Consultant
T:  0161 537 4980
W: www.quadris.co.uk
[cid:image937435.png@813B89FF.AD6D2675]
The information contained in this e-mail from Quadris may be confidential and 
privileged for the private use of the named recipient.  The contents of this 
e-mail may not necessarily represent the official views of Quadris.  If you 
have received this information in error you must not copy, distribute or take 
any action or reliance on its contents.  Please destroy any hard copies and 
delete this message.
________
From: Simon Weller 
Sent: Wednesday, July 20, 2022 8:55:22 PM
To: users@cloudstack.apache.org 
Subject: Re: Virtual Router filesystem corruption

Gary,

Can you provide some information about the OS, underlying hypervisor and 
primary storage in use?

-Si

From: Gary Dixon 
Sent: Wednesday, July 20, 2022 11:15 AM
To: users@cloudstack.apache.org 
Subject: Virtual Router filesystem corruption

EXTERNAL EMAIL: This message originated outside of ENA. Use caution when 
clicking links, opening attachments, or complying with requests. Click the 
"Phish Alert Report" button above the email, or contact MIS, regarding any 
suspicious message.





Hi All



We are seeing ext4 filesystem corruption on a number of virtual routers 
recently and manually running fsck doesn’t appear to help at all in fixing the 
issue (Corrupt inode bitmap)

We end up having to restart the associated VPC with cleanup enable to rebuild a 
new VR. Is this a common issue with ACS 4.15.1 ? Or are there specific 
circumstances that are causing the VR fs corruption that we could perhaps 
mitigate ?



Kind regards



Gary

Gary Dixon​​

Technical Consultant

T:  0161 537 4980

W: 
https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.quadris.co.uk%2Fdata=05%7C01%7CGary.Dixon%40quadris.co.uk%7C6494994721954e76957d08da6a89d5be%7Cf1d6abf3d3b44894ae16db0fb93a96a2%7C0%7C0%7C637939437472349569%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7Csdata=qmSsPmr6P17hETukglH7IHduYr0TFdKtmjIk61MF1w8%3Dreserved=0

[cid:image001.png@01D89C5B.5685DCC0]

The information contained in this e-mail from Quadris may be confidential and 
privileged for the private use of the named recipient.  The contents of this 
e-mail may not necessarily represent the official views of Quadris.  If you 
have received this information in error you must not copy, distribute or take 
any action or reliance on its contents.  Please destroy any hard copies and 
delete this message.

Re: Virtual Router filesystem corruption

[Simon Weller]

Gary,

Can you provide some information about the OS, underlying hypervisor and 
primary storage in use?

-Si

From: Gary Dixon 
Sent: Wednesday, July 20, 2022 11:15 AM
To: users@cloudstack.apache.org 
Subject: Virtual Router filesystem corruption

EXTERNAL EMAIL: This message originated outside of ENA. Use caution when 
clicking links, opening attachments, or complying with requests. Click the 
"Phish Alert Report" button above the email, or contact MIS, regarding any 
suspicious message.





Hi All



We are seeing ext4 filesystem corruption on a number of virtual routers 
recently and manually running fsck doesn’t appear to help at all in fixing the 
issue (Corrupt inode bitmap)

We end up having to restart the associated VPC with cleanup enable to rebuild a 
new VR. Is this a common issue with ACS 4.15.1 ? Or are there specific 
circumstances that are causing the VR fs corruption that we could perhaps 
mitigate ?



Kind regards



Gary

Gary Dixon​​

Technical Consultant

T:  0161 537 4980

W: www.quadris.co.uk

[cid:image001.png@01D89C5B.5685DCC0]

The information contained in this e-mail from Quadris may be confidential and 
privileged for the private use of the named recipient.  The contents of this 
e-mail may not necessarily represent the official views of Quadris.  If you 
have received this information in error you must not copy, distribute or take 
any action or reliance on its contents.  Please destroy any hard copies and 
delete this message.

Re: Asymmetric traffic issues?

[Simon Weller]

Good to hear. Thanks for providing a follow up Steve.

-Si

From: S.Fuller 
Sent: Friday, July 15, 2022 1:08 PM
To: users@cloudstack.apache.org 
Subject: Re: Asymmetric traffic issues?

EXTERNAL EMAIL: This message originated outside of ENA. Use caution when 
clicking links, opening attachments, or complying with requests. Click the 
"Phish Alert Report" button above the email, or contact MIS, regarding any 
suspicious message.



Update on this, we were able to enable the same performance changes by
using ethtool to disable gro on the private and public interfaces on the
virtual routers.

Steve

On Fri, Jul 8, 2022 at 2:14 PM Simon Weller  wrote:

> That's great news! Glad it all worked out.
>
> -Si
> 
> From: S.Fuller 
> Sent: Friday, July 8, 2022 12:49 PM
> To: users@cloudstack.apache.org 
> Subject: Re: Asymmetric traffic issues?
>
> EXTERNAL EMAIL: This message originated outside of ENA. Use caution when
> clicking links, opening attachments, or complying with requests. Click the
> "Phish Alert Report" button above the email, or contact MIS, regarding any
> suspicious message.
>
>
>
> Holy cow Simon!!! That did the trick!!!
>
> Thank you so much! I feel like I should send you coffee or beer or
> something :)
>
> Steve
>
> On Fri, Jul 8, 2022 at 11:48 AM Simon Weller 
> wrote:
>
> > I recall an issue a while back where the Windows Virtio drivers didn't
> > support generic support offload (GRO) and it created some performance
> > problems with Cisco NICs.
> > This was particular to Windows and didn't occur for Linux VMs.
> >
> > Can you try disabling GRO on your NICs and retesting?
> > ethtool -K  gro off
> >
> > -Si
> > 
> > From: S.Fuller 
> > Sent: Thursday, July 7, 2022 9:00 AM
> > To: users@cloudstack.apache.org 
> > Subject: Re: Asymmetric traffic issues?
> >
> > EXTERNAL EMAIL: This message originated outside of ENA. Use caution when
> > clicking links, opening attachments, or complying with requests. Click
> the
> > "Phish Alert Report" button above the email, or contact MIS, regarding
> any
> > suspicious message.
> >
> >
> >
> > The servers in question are in a Cisco UCS blade chassis. The physical
> NIC
> > card is a Cisco UCS VIC 1240, it's a multiport 10gb card that presents
> > itself multiple virtual NICs to the server. Those virtual NICs use the
> > Cisco enic driver.
> >
> > Steve
> >
> > On Fri, Jul 1, 2022 at 4:05 PM Simon Weller 
> > wrote:
> >
> > > ok, I managed to access your xml file via the mailing list archives.
> > >
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > >  > function="0x0"/>
> > > 
> > >
> > > So, it does look correct in terms of virtio-net. One thing of note is
> > that
> > > kvm guests can't transmit or recieve packets in parallel, since
> > virtio-net
> > > only has one TX and RX queue. This tends to reduce performance on
> Windows
> > > guests.
> > > This can be addressed by enabling multi-queue (you assign a queue for
> > > vCPU). There was talk of this back in 2021 on the list, but I'm not
> sure
> > > whether it was implemented or not.
> > >
> > > What brand and chipset are your NICs on the servers?
> > >
> > >
> > >
> > >
> > >
> > >
> > > 
> > > From: Simon Weller 
> > > Sent: Friday, July 1, 2022 2:34 PM
> > > To: users@cloudstack.apache.org 
> > > Subject: Re: Asymmetric traffic issues?
> > >
> > > EXTERNAL EMAIL: This message originated outside of ENA. Use caution
> when
> > > clicking links, opening attachments, or complying with requests. Click
> > the
> > > "Phish Alert Report" button above the email, or contact MIS, regarding
> > any
> > > suspicious message.
> > >
> > >
> > >
> > > I'm assuming the para-virtualized network driver is probably not being
> > > used either based on the template.
> > > If you take a close look at the xml for the interface, you should be
> able
> > > to confirm that or not.
> > >
> > > The templates tend to package all the para-virtualized drivers
> together,
> > > so they're normally in use, or they're not.
> > >
> > >
> > > 
> > > From: S.

Re: Asymmetric traffic issues?

[Simon Weller]

That's great news! Glad it all worked out.

-Si

From: S.Fuller 
Sent: Friday, July 8, 2022 12:49 PM
To: users@cloudstack.apache.org 
Subject: Re: Asymmetric traffic issues?

EXTERNAL EMAIL: This message originated outside of ENA. Use caution when 
clicking links, opening attachments, or complying with requests. Click the 
"Phish Alert Report" button above the email, or contact MIS, regarding any 
suspicious message.



Holy cow Simon!!! That did the trick!!!

Thank you so much! I feel like I should send you coffee or beer or
something :)

Steve

On Fri, Jul 8, 2022 at 11:48 AM Simon Weller 
wrote:

> I recall an issue a while back where the Windows Virtio drivers didn't
> support generic support offload (GRO) and it created some performance
> problems with Cisco NICs.
> This was particular to Windows and didn't occur for Linux VMs.
>
> Can you try disabling GRO on your NICs and retesting?
> ethtool -K  gro off
>
> -Si
> 
> From: S.Fuller 
> Sent: Thursday, July 7, 2022 9:00 AM
> To: users@cloudstack.apache.org 
> Subject: Re: Asymmetric traffic issues?
>
> EXTERNAL EMAIL: This message originated outside of ENA. Use caution when
> clicking links, opening attachments, or complying with requests. Click the
> "Phish Alert Report" button above the email, or contact MIS, regarding any
> suspicious message.
>
>
>
> The servers in question are in a Cisco UCS blade chassis. The physical NIC
> card is a Cisco UCS VIC 1240, it's a multiport 10gb card that presents
> itself multiple virtual NICs to the server. Those virtual NICs use the
> Cisco enic driver.
>
> Steve
>
> On Fri, Jul 1, 2022 at 4:05 PM Simon Weller 
> wrote:
>
> > ok, I managed to access your xml file via the mailing list archives.
> >
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> >  function="0x0"/>
> > 
> >
> > So, it does look correct in terms of virtio-net. One thing of note is
> that
> > kvm guests can't transmit or recieve packets in parallel, since
> virtio-net
> > only has one TX and RX queue. This tends to reduce performance on Windows
> > guests.
> > This can be addressed by enabling multi-queue (you assign a queue for
> > vCPU). There was talk of this back in 2021 on the list, but I'm not sure
> > whether it was implemented or not.
> >
> > What brand and chipset are your NICs on the servers?
> >
> >
> >
> >
> >
> >
> > 
> > From: Simon Weller 
> > Sent: Friday, July 1, 2022 2:34 PM
> > To: users@cloudstack.apache.org 
> > Subject: Re: Asymmetric traffic issues?
> >
> > EXTERNAL EMAIL: This message originated outside of ENA. Use caution when
> > clicking links, opening attachments, or complying with requests. Click
> the
> > "Phish Alert Report" button above the email, or contact MIS, regarding
> any
> > suspicious message.
> >
> >
> >
> > I'm assuming the para-virtualized network driver is probably not being
> > used either based on the template.
> > If you take a close look at the xml for the interface, you should be able
> > to confirm that or not.
> >
> > The templates tend to package all the para-virtualized drivers together,
> > so they're normally in use, or they're not.
> >
> >
> > 
> > From: S.Fuller 
> > Sent: Friday, July 1, 2022 1:52 PM
> > To: users@cloudstack.apache.org 
> > Subject: Re: Asymmetric traffic issues?
> >
> > EXTERNAL EMAIL: This message originated outside of ENA. Use caution when
> > clicking links, opening attachments, or complying with requests. Click
> the
> > "Phish Alert Report" button above the email, or contact MIS, regarding
> any
> > suspicious message.
> >
> >
> >
> > Will give that a try. I'm curious why you think changing the disk drivers
> > over to virtio will have an effect on the network throughput.
> >
> > - Steve
> >
> > On Fri, Jul 1, 2022 at 10:55 AM Simon Weller 
> > wrote:
> >
> > > Unfortunately, I can't open the xml via the mailing list, but based on
> > the
> > > template you're using, it's not setup for Virtio-blk or virtio-scsi.
> > Change
> > > the template to Virtio-SCSI 64bit and the correct interface drive type
> > will
> > > be specified in the XML.
> > >
> > > 
> > > From: S.Fuller 
> > > Sent: Thursday, June 30, 2022 12:43 PM
> > > To: users@cl

Re: Asymmetric traffic issues?

[Simon Weller]

I recall an issue a while back where the Windows Virtio drivers didn't support 
generic support offload (GRO) and it created some performance problems with 
Cisco NICs.
This was particular to Windows and didn't occur for Linux VMs.

Can you try disabling GRO on your NICs and retesting?
ethtool -K  gro off

-Si

From: S.Fuller 
Sent: Thursday, July 7, 2022 9:00 AM
To: users@cloudstack.apache.org 
Subject: Re: Asymmetric traffic issues?

EXTERNAL EMAIL: This message originated outside of ENA. Use caution when 
clicking links, opening attachments, or complying with requests. Click the 
"Phish Alert Report" button above the email, or contact MIS, regarding any 
suspicious message.



The servers in question are in a Cisco UCS blade chassis. The physical NIC
card is a Cisco UCS VIC 1240, it's a multiport 10gb card that presents
itself multiple virtual NICs to the server. Those virtual NICs use the
Cisco enic driver.

Steve

On Fri, Jul 1, 2022 at 4:05 PM Simon Weller  wrote:

> ok, I managed to access your xml file via the mailing list archives.
>
> 
> 
> 
> 
> 
> 
> 
> 
> 
>
> So, it does look correct in terms of virtio-net. One thing of note is that
> kvm guests can't transmit or recieve packets in parallel, since virtio-net
> only has one TX and RX queue. This tends to reduce performance on Windows
> guests.
> This can be addressed by enabling multi-queue (you assign a queue for
> vCPU). There was talk of this back in 2021 on the list, but I'm not sure
> whether it was implemented or not.
>
> What brand and chipset are your NICs on the servers?
>
>
>
>
>
>
> 
> From: Simon Weller 
> Sent: Friday, July 1, 2022 2:34 PM
> To: users@cloudstack.apache.org 
> Subject: Re: Asymmetric traffic issues?
>
> EXTERNAL EMAIL: This message originated outside of ENA. Use caution when
> clicking links, opening attachments, or complying with requests. Click the
> "Phish Alert Report" button above the email, or contact MIS, regarding any
> suspicious message.
>
>
>
> I'm assuming the para-virtualized network driver is probably not being
> used either based on the template.
> If you take a close look at the xml for the interface, you should be able
> to confirm that or not.
>
> The templates tend to package all the para-virtualized drivers together,
> so they're normally in use, or they're not.
>
>
> 
> From: S.Fuller 
> Sent: Friday, July 1, 2022 1:52 PM
> To: users@cloudstack.apache.org 
> Subject: Re: Asymmetric traffic issues?
>
> EXTERNAL EMAIL: This message originated outside of ENA. Use caution when
> clicking links, opening attachments, or complying with requests. Click the
> "Phish Alert Report" button above the email, or contact MIS, regarding any
> suspicious message.
>
>
>
> Will give that a try. I'm curious why you think changing the disk drivers
> over to virtio will have an effect on the network throughput.
>
> - Steve
>
> On Fri, Jul 1, 2022 at 10:55 AM Simon Weller 
> wrote:
>
> > Unfortunately, I can't open the xml via the mailing list, but based on
> the
> > template you're using, it's not setup for Virtio-blk or virtio-scsi.
> Change
> > the template to Virtio-SCSI 64bit and the correct interface drive type
> will
> > be specified in the XML.
> >
> > 
> > From: S.Fuller 
> > Sent: Thursday, June 30, 2022 12:43 PM
> > To: users@cloudstack.apache.org 
> > Subject: Re: Asymmetric traffic issues?
> >
> >
> > I'm using the Windows Server 2016 profile for the template. After the VM
> > came up, I installed the virtio drivers from the latest ISO, and then
> added
> > a new NIC so that it used the virtio network drivers vs the default E1000
> > drivers.
> >
> > XML dump is attached
> >
> > On Thu, Jun 30, 2022 at 12:03 PM Simon Weller 
> > wrote:
> > Can you provide a redacted dump of your libvirt xml file for the VM?
> > Also, which OS profile are you using in Cloudstack for this? The Windows
> > specific ones don't use virtio if I remember correctly, so you will need
> to
> > select Virtio-SCSI 64bit in order for the xml to be built correctly.
> >
> > -Si
> > 
> > From: S.Fuller mailto:steveful...@gmail.com>>
> > Sent: Thursday, June 30, 2022 11:38 AM
> > To: users@cloudstack.apache.org<mailto:users@cloudstack.apache.org> <
> > users@cloudstack.apache.org<mailto:users@cloudstack.apache.org>>
> > Subject: Re: Asymmetric traffic issues?
> >
> > EXTERNAL EMAIL: This message ori

Re: Asymmetric traffic issues?

[Simon Weller]

ok, I managed to access your xml file via the mailing list archives.











So, it does look correct in terms of virtio-net. One thing of note is that kvm 
guests can't transmit or recieve packets in parallel, since virtio-net only has 
one TX and RX queue. This tends to reduce performance on Windows guests.
This can be addressed by enabling multi-queue (you assign a queue for vCPU). 
There was talk of this back in 2021 on the list, but I'm not sure whether it 
was implemented or not.

What brand and chipset are your NICs on the servers?







From: Simon Weller 
Sent: Friday, July 1, 2022 2:34 PM
To: users@cloudstack.apache.org 
Subject: Re: Asymmetric traffic issues?

EXTERNAL EMAIL: This message originated outside of ENA. Use caution when 
clicking links, opening attachments, or complying with requests. Click the 
"Phish Alert Report" button above the email, or contact MIS, regarding any 
suspicious message.



I'm assuming the para-virtualized network driver is probably not being used 
either based on the template.
If you take a close look at the xml for the interface, you should be able to 
confirm that or not.

The templates tend to package all the para-virtualized drivers together, so 
they're normally in use, or they're not.



From: S.Fuller 
Sent: Friday, July 1, 2022 1:52 PM
To: users@cloudstack.apache.org 
Subject: Re: Asymmetric traffic issues?

EXTERNAL EMAIL: This message originated outside of ENA. Use caution when 
clicking links, opening attachments, or complying with requests. Click the 
"Phish Alert Report" button above the email, or contact MIS, regarding any 
suspicious message.



Will give that a try. I'm curious why you think changing the disk drivers
over to virtio will have an effect on the network throughput.

- Steve

On Fri, Jul 1, 2022 at 10:55 AM Simon Weller 
wrote:

> Unfortunately, I can't open the xml via the mailing list, but based on the
> template you're using, it's not setup for Virtio-blk or virtio-scsi. Change
> the template to Virtio-SCSI 64bit and the correct interface drive type will
> be specified in the XML.
>
> 
> From: S.Fuller 
> Sent: Thursday, June 30, 2022 12:43 PM
> To: users@cloudstack.apache.org 
> Subject: Re: Asymmetric traffic issues?
>
>
> I'm using the Windows Server 2016 profile for the template. After the VM
> came up, I installed the virtio drivers from the latest ISO, and then added
> a new NIC so that it used the virtio network drivers vs the default E1000
> drivers.
>
> XML dump is attached
>
> On Thu, Jun 30, 2022 at 12:03 PM Simon Weller 
> wrote:
> Can you provide a redacted dump of your libvirt xml file for the VM?
> Also, which OS profile are you using in Cloudstack for this? The Windows
> specific ones don't use virtio if I remember correctly, so you will need to
> select Virtio-SCSI 64bit in order for the xml to be built correctly.
>
> -Si
> 
> From: S.Fuller mailto:steveful...@gmail.com>>
> Sent: Thursday, June 30, 2022 11:38 AM
> To: users@cloudstack.apache.org<mailto:users@cloudstack.apache.org> <
> users@cloudstack.apache.org<mailto:users@cloudstack.apache.org>>
> Subject: Re: Asymmetric traffic issues?
>
> EXTERNAL EMAIL: This message originated outside of ENA. Use caution when
> clicking links, opening attachments, or complying with requests. Click the
> "Phish Alert Report" button above the email, or contact MIS, regarding any
> suspicious message.
>
>
>
> Neither the block nor the scsi drivers appear to be running as far as I can
> tell.
>
> - Steve
>
> On Thu, Jun 30, 2022 at 11:07 AM Simon Weller 
> wrote:
>
> > Steve,
> >
> > Are you running the virtio-block or virtio-scsi drivers?
> >
> > -Si
> >
> >
> > 
> > From: S.Fuller mailto:steveful...@gmail.com>>
> > Sent: Thursday, June 30, 2022 8:52 AM
> > To: users@cloudstack.apache.org<mailto:users@cloudstack.apache.org> <
> users@cloudstack.apache.org<mailto:users@cloudstack.apache.org>>
> > Subject: Re: Asymmetric traffic issues?
> >
> > EXTERNAL EMAIL: This message originated outside of ENA. Use caution when
> > clicking links, opening attachments, or complying with requests. Click
> the
> > "Phish Alert Report" button above the email, or contact MIS, regarding
> any
> > suspicious message.
> >
> >
> >
> > Well, I checked everything that I could for any QOS settings and there is
> > nothing configured there. What is curious is that Idon't see this
> behavior
> > when the transmitting host is running Linux, or if I'm using the E10

Re: Asymmetric traffic issues?

[Simon Weller]

I'm assuming the para-virtualized network driver is probably not being used 
either based on the template.
If you take a close look at the xml for the interface, you should be able to 
confirm that or not.

The templates tend to package all the para-virtualized drivers together, so 
they're normally in use, or they're not.



From: S.Fuller 
Sent: Friday, July 1, 2022 1:52 PM
To: users@cloudstack.apache.org 
Subject: Re: Asymmetric traffic issues?

EXTERNAL EMAIL: This message originated outside of ENA. Use caution when 
clicking links, opening attachments, or complying with requests. Click the 
"Phish Alert Report" button above the email, or contact MIS, regarding any 
suspicious message.



Will give that a try. I'm curious why you think changing the disk drivers
over to virtio will have an effect on the network throughput.

- Steve

On Fri, Jul 1, 2022 at 10:55 AM Simon Weller 
wrote:

> Unfortunately, I can't open the xml via the mailing list, but based on the
> template you're using, it's not setup for Virtio-blk or virtio-scsi. Change
> the template to Virtio-SCSI 64bit and the correct interface drive type will
> be specified in the XML.
>
> 
> From: S.Fuller 
> Sent: Thursday, June 30, 2022 12:43 PM
> To: users@cloudstack.apache.org 
> Subject: Re: Asymmetric traffic issues?
>
>
> I'm using the Windows Server 2016 profile for the template. After the VM
> came up, I installed the virtio drivers from the latest ISO, and then added
> a new NIC so that it used the virtio network drivers vs the default E1000
> drivers.
>
> XML dump is attached
>
> On Thu, Jun 30, 2022 at 12:03 PM Simon Weller 
> wrote:
> Can you provide a redacted dump of your libvirt xml file for the VM?
> Also, which OS profile are you using in Cloudstack for this? The Windows
> specific ones don't use virtio if I remember correctly, so you will need to
> select Virtio-SCSI 64bit in order for the xml to be built correctly.
>
> -Si
> 
> From: S.Fuller mailto:steveful...@gmail.com>>
> Sent: Thursday, June 30, 2022 11:38 AM
> To: users@cloudstack.apache.org<mailto:users@cloudstack.apache.org> <
> users@cloudstack.apache.org<mailto:users@cloudstack.apache.org>>
> Subject: Re: Asymmetric traffic issues?
>
> EXTERNAL EMAIL: This message originated outside of ENA. Use caution when
> clicking links, opening attachments, or complying with requests. Click the
> "Phish Alert Report" button above the email, or contact MIS, regarding any
> suspicious message.
>
>
>
> Neither the block nor the scsi drivers appear to be running as far as I can
> tell.
>
> - Steve
>
> On Thu, Jun 30, 2022 at 11:07 AM Simon Weller 
> wrote:
>
> > Steve,
> >
> > Are you running the virtio-block or virtio-scsi drivers?
> >
> > -Si
> >
> >
> > 
> > From: S.Fuller mailto:steveful...@gmail.com>>
> > Sent: Thursday, June 30, 2022 8:52 AM
> > To: users@cloudstack.apache.org<mailto:users@cloudstack.apache.org> <
> users@cloudstack.apache.org<mailto:users@cloudstack.apache.org>>
> > Subject: Re: Asymmetric traffic issues?
> >
> > EXTERNAL EMAIL: This message originated outside of ENA. Use caution when
> > clicking links, opening attachments, or complying with requests. Click
> the
> > "Phish Alert Report" button above the email, or contact MIS, regarding
> any
> > suspicious message.
> >
> >
> >
> > Well, I checked everything that I could for any QOS settings and there is
> > nothing configured there. What is curious is that Idon't see this
> behavior
> > when the transmitting host is running Linux, or if I'm using the E1000
> > drivers (although with the E1000 driver, the overall throughput is
> lower).
> > It really feels like I'mrunning into some weird issue with the virtio
> > drivers on Windows. My Windows hosts are (to my knowledge) using the
> latest
> > version of the virtio drivers - 100.90.104.21700 dated 2/23/2022.
> >
> > Steve Fuller
> > steveful...@gmail.com<mailto:steveful...@gmail.com>
> >
> > On Wed, Jun 29, 2022 at 3:04 PM S.Fuller  steveful...@gmail.com>> wrote:
> >
> > > Vivek,
> > >
> > > Thanks for the reply. I am using the KVM hypervisor. I'll will review
> the
> > > QoS on the hypervisor for both of the nodes.
> > >
> > > I'm checking throughput between two different VMs that running on two
> > > different hosts within the same cluster. As of right now, I'm receiving
> > > similar results using b

Re: Asymmetric traffic issues?

[Simon Weller]

Unfortunately, I can't open the xml via the mailing list, but based on the 
template you're using, it's not setup for Virtio-blk or virtio-scsi. Change the 
template to Virtio-SCSI 64bit and the correct interface drive type will be 
specified in the XML.


From: S.Fuller 
Sent: Thursday, June 30, 2022 12:43 PM
To: users@cloudstack.apache.org 
Subject: Re: Asymmetric traffic issues?


I'm using the Windows Server 2016 profile for the template. After the VM came 
up, I installed the virtio drivers from the latest ISO, and then added a new 
NIC so that it used the virtio network drivers vs the default E1000 drivers.

XML dump is attached

On Thu, Jun 30, 2022 at 12:03 PM Simon Weller  wrote:
Can you provide a redacted dump of your libvirt xml file for the VM?
Also, which OS profile are you using in Cloudstack for this? The Windows 
specific ones don't use virtio if I remember correctly, so you will need to 
select Virtio-SCSI 64bit in order for the xml to be built correctly.

-Si

From: S.Fuller mailto:steveful...@gmail.com>>
Sent: Thursday, June 30, 2022 11:38 AM
To: users@cloudstack.apache.org<mailto:users@cloudstack.apache.org> 
mailto:users@cloudstack.apache.org>>
Subject: Re: Asymmetric traffic issues?

EXTERNAL EMAIL: This message originated outside of ENA. Use caution when 
clicking links, opening attachments, or complying with requests. Click the 
"Phish Alert Report" button above the email, or contact MIS, regarding any 
suspicious message.



Neither the block nor the scsi drivers appear to be running as far as I can
tell.

- Steve

On Thu, Jun 30, 2022 at 11:07 AM Simon Weller 
wrote:

> Steve,
>
> Are you running the virtio-block or virtio-scsi drivers?
>
> -Si
>
>
> 
> From: S.Fuller mailto:steveful...@gmail.com>>
> Sent: Thursday, June 30, 2022 8:52 AM
> To: users@cloudstack.apache.org<mailto:users@cloudstack.apache.org> 
> mailto:users@cloudstack.apache.org>>
> Subject: Re: Asymmetric traffic issues?
>
> EXTERNAL EMAIL: This message originated outside of ENA. Use caution when
> clicking links, opening attachments, or complying with requests. Click the
> "Phish Alert Report" button above the email, or contact MIS, regarding any
> suspicious message.
>
>
>
> Well, I checked everything that I could for any QOS settings and there is
> nothing configured there. What is curious is that Idon't see this behavior
> when the transmitting host is running Linux, or if I'm using the E1000
> drivers (although with the E1000 driver, the overall throughput is lower).
> It really feels like I'mrunning into some weird issue with the virtio
> drivers on Windows. My Windows hosts are (to my knowledge) using the latest
> version of the virtio drivers - 100.90.104.21700 dated 2/23/2022.
>
> Steve Fuller
> steveful...@gmail.com<mailto:steveful...@gmail.com>
>
> On Wed, Jun 29, 2022 at 3:04 PM S.Fuller 
> mailto:steveful...@gmail.com>> wrote:
>
> > Vivek,
> >
> > Thanks for the reply. I am using the KVM hypervisor. I'll will review the
> > QoS on the hypervisor for both of the nodes.
> >
> > I'm checking throughput between two different VMs that running on two
> > different hosts within the same cluster. As of right now, I'm receiving
> > similar results using both iperf3 and nuttcp as the testing tools. We are
> > only seeing this issue when the VM is not on the same host as the vrouter
> > for its isolated network.
> >
> > On Tue, Jun 28, 2022 at 6:40 AM Vivek Kumar
> > mailto:vivek.ku...@indiqus.com>.invalid> wrote:
> >
> >> Hey Fuller,
> >>
> >> What hypervisor are you using ? I know you have checked all bandwidth
> >> limit on templates and global settings, but it’s worth to check the QoS
> on
> >> the hypervisor level, because at the end it’s the hypervisor which
> manages
> >> all.  And from where are you trying to check the network throughout,
> >> between client and server ?
> >>
> >>
> >> Vivek Kumar
> >> Sr. Manager - Cloud & DevOps
> >> TechOps | Indiqus Technologies
> >>
> >> + 91 7503460090 
> >> vivek.ku...@indiqus.com<mailto:vivek.ku...@indiqus.com> 
> >> <mailto:vivek.ku...@indiqus.com<mailto:vivek.ku...@indiqus.com>>
> >> www.indiqus.com<http://www.indiqus.com><http://www.indiqus.com> <
> https://www.indiqus.com/>
> >>
> >>
> >>
> >> > On 28-Jun-2022, at 1:58 AM, S.Fuller 
> >> > mailto:steveful...@gmail.com>> wrote:
> >> >
> >> > Environment:
> >

Re: Asymmetric traffic issues?

[Simon Weller]

Can you provide a redacted dump of your libvirt xml file for the VM?
Also, which OS profile are you using in Cloudstack for this? The Windows 
specific ones don't use virtio if I remember correctly, so you will need to 
select Virtio-SCSI 64bit in order for the xml to be built correctly.

-Si

From: S.Fuller 
Sent: Thursday, June 30, 2022 11:38 AM
To: users@cloudstack.apache.org 
Subject: Re: Asymmetric traffic issues?

EXTERNAL EMAIL: This message originated outside of ENA. Use caution when 
clicking links, opening attachments, or complying with requests. Click the 
"Phish Alert Report" button above the email, or contact MIS, regarding any 
suspicious message.



Neither the block nor the scsi drivers appear to be running as far as I can
tell.

- Steve

On Thu, Jun 30, 2022 at 11:07 AM Simon Weller 
wrote:

> Steve,
>
> Are you running the virtio-block or virtio-scsi drivers?
>
> -Si
>
>
> 
> From: S.Fuller 
> Sent: Thursday, June 30, 2022 8:52 AM
> To: users@cloudstack.apache.org 
> Subject: Re: Asymmetric traffic issues?
>
> EXTERNAL EMAIL: This message originated outside of ENA. Use caution when
> clicking links, opening attachments, or complying with requests. Click the
> "Phish Alert Report" button above the email, or contact MIS, regarding any
> suspicious message.
>
>
>
> Well, I checked everything that I could for any QOS settings and there is
> nothing configured there. What is curious is that Idon't see this behavior
> when the transmitting host is running Linux, or if I'm using the E1000
> drivers (although with the E1000 driver, the overall throughput is lower).
> It really feels like I'mrunning into some weird issue with the virtio
> drivers on Windows. My Windows hosts are (to my knowledge) using the latest
> version of the virtio drivers - 100.90.104.21700 dated 2/23/2022.
>
> Steve Fuller
> steveful...@gmail.com
>
> On Wed, Jun 29, 2022 at 3:04 PM S.Fuller  wrote:
>
> > Vivek,
> >
> > Thanks for the reply. I am using the KVM hypervisor. I'll will review the
> > QoS on the hypervisor for both of the nodes.
> >
> > I'm checking throughput between two different VMs that running on two
> > different hosts within the same cluster. As of right now, I'm receiving
> > similar results using both iperf3 and nuttcp as the testing tools. We are
> > only seeing this issue when the VM is not on the same host as the vrouter
> > for its isolated network.
> >
> > On Tue, Jun 28, 2022 at 6:40 AM Vivek Kumar
> >  wrote:
> >
> >> Hey Fuller,
> >>
> >> What hypervisor are you using ? I know you have checked all bandwidth
> >> limit on templates and global settings, but it’s worth to check the QoS
> on
> >> the hypervisor level, because at the end it’s the hypervisor which
> manages
> >> all.  And from where are you trying to check the network throughout,
> >> between client and server ?
> >>
> >>
> >> Vivek Kumar
> >> Sr. Manager - Cloud & DevOps
> >> TechOps | Indiqus Technologies
> >>
> >> + 91 7503460090 
> >> vivek.ku...@indiqus.com <mailto:vivek.ku...@indiqus.com>
> >> www.indiqus.com<http://www.indiqus.com> <
> https://www.indiqus.com/>
> >>
> >>
> >>
> >> > On 28-Jun-2022, at 1:58 AM, S.Fuller  wrote:
> >> >
> >> > Environment:
> >> >
> >> > Two physical hosts
> >> > - Cloudstack 4.11.3
> >> > - Verified that there are no bandwidth limits in place on any of the
> >> > templates or in global settings.
> >> >
> >> > Two isolated networks ("Client" and  "Server")
> >> > - Each has a vrouter with a public and private address
> >> > - One Windows 2016 VM on each network (running the latest virtio
> >> drivers)
> >> > - each node running latest version of Iperf3 to test throughput
> >> >
> >> > Testing/Observation:
> >> >
> >> > If the Client VM and the vrouter for the isolated Client network are
> on
> >> the
> >> > same physical host, we see symmetrical throughput in the 2 Gbps range,
> >> > whether we run iperf in regular mode or in reverse mode (iperf -R).
> >> >
> >> > If the Client VM and the vrouter for the isolated Client network are
> on
> >> > different physical hosts, we are seeing 25% of the throughput running
> >> iperf
> >> > in regular mode vs running it in reverse mode.
> >> >
> 

Re: Asymmetric traffic issues?

[Simon Weller]

Steve,

Are you running the virtio-block or virtio-scsi drivers?

-Si



From: S.Fuller 
Sent: Thursday, June 30, 2022 8:52 AM
To: users@cloudstack.apache.org 
Subject: Re: Asymmetric traffic issues?

EXTERNAL EMAIL: This message originated outside of ENA. Use caution when 
clicking links, opening attachments, or complying with requests. Click the 
"Phish Alert Report" button above the email, or contact MIS, regarding any 
suspicious message.



Well, I checked everything that I could for any QOS settings and there is
nothing configured there. What is curious is that Idon't see this behavior
when the transmitting host is running Linux, or if I'm using the E1000
drivers (although with the E1000 driver, the overall throughput is lower).
It really feels like I'mrunning into some weird issue with the virtio
drivers on Windows. My Windows hosts are (to my knowledge) using the latest
version of the virtio drivers - 100.90.104.21700 dated 2/23/2022.

Steve Fuller
steveful...@gmail.com

On Wed, Jun 29, 2022 at 3:04 PM S.Fuller  wrote:

> Vivek,
>
> Thanks for the reply. I am using the KVM hypervisor. I'll will review the
> QoS on the hypervisor for both of the nodes.
>
> I'm checking throughput between two different VMs that running on two
> different hosts within the same cluster. As of right now, I'm receiving
> similar results using both iperf3 and nuttcp as the testing tools. We are
> only seeing this issue when the VM is not on the same host as the vrouter
> for its isolated network.
>
> On Tue, Jun 28, 2022 at 6:40 AM Vivek Kumar
>  wrote:
>
>> Hey Fuller,
>>
>> What hypervisor are you using ? I know you have checked all bandwidth
>> limit on templates and global settings, but it’s worth to check the QoS on
>> the hypervisor level, because at the end it’s the hypervisor which manages
>> all.  And from where are you trying to check the network throughout,
>> between client and server ?
>>
>>
>> Vivek Kumar
>> Sr. Manager - Cloud & DevOps
>> TechOps | Indiqus Technologies
>>
>> + 91 7503460090 
>> vivek.ku...@indiqus.com 
>> www.indiqus.com 
>>
>>
>>
>> > On 28-Jun-2022, at 1:58 AM, S.Fuller  wrote:
>> >
>> > Environment:
>> >
>> > Two physical hosts
>> > - Cloudstack 4.11.3
>> > - Verified that there are no bandwidth limits in place on any of the
>> > templates or in global settings.
>> >
>> > Two isolated networks ("Client" and  "Server")
>> > - Each has a vrouter with a public and private address
>> > - One Windows 2016 VM on each network (running the latest virtio
>> drivers)
>> > - each node running latest version of Iperf3 to test throughput
>> >
>> > Testing/Observation:
>> >
>> > If the Client VM and the vrouter for the isolated Client network are on
>> the
>> > same physical host, we see symmetrical throughput in the 2 Gbps range,
>> > whether we run iperf in regular mode or in reverse mode (iperf -R).
>> >
>> > If the Client VM and the vrouter for the isolated Client network are on
>> > different physical hosts, we are seeing 25% of the throughput running
>> iperf
>> > in regular mode vs running it in reverse mode.
>> >
>> > Has anyone encountered this issue before? If we change the Client VM to
>> > Linux (either CentOS 7 or Ubuntu) OR we use the E1000 driver, we see
>> > symmetrical throughput in our tests, no matter where the vrouter is in
>> > relation to the Client VM.
>> >
>> > --
>> > Steve Fuller
>> > steveful...@gmail.com
>>
>>
>> --
>> This message is intended only for the use of the individual or entity to
>> which it is addressed and may contain confidential and/or privileged
>> information. If you are not the intended recipient, please delete the
>> original message and any copy of it from your computer system. You are
>> hereby notified that any dissemination, distribution or copying of this
>> communication is strictly prohibited unless proper authorization has been
>> obtained for such action. If you have received this communication in
>> error,
>> please notify the sender immediately. Although IndiQus attempts to sweep
>> e-mail and attachments for viruses, it does not guarantee that both are
>> virus-free and accepts no liability for any damage sustained as a result
>> of
>> viruses.
>>
>
>
> --
> Steve Fuller
> steveful...@gmail.com
>


--
Steve Fuller
steveful...@gmail.com

Re: Is there any issue with Ceph RBD as Primary Storage to achieve HA-Enabled VM?

[Simon Weller]

Hi Nazmul,

Currently, RBD backed VMs will auto-recover as long as the host doesn't fail. 
If the underlying host fails, intervention will be required to take the host 
out of production to force the VMs to be restarted on another host.
There is a PR currently under review that will add full HA with heartbeat for 
RBD and KVM - https://github.com/apache/cloudstack/pull/5862

-Si

From: Nazmul Parvej 
Sent: Tuesday, June 28, 2022 6:05 AM
To: users@cloudstack.apache.org 
Cc: Product Development | BEXIMCO IT 
Subject: Is there any issue with Ceph RBD as Primary Storage to achieve 
HA-Enabled VM?

EXTERNAL EMAIL: This message originated outside of ENA. Use caution when 
clicking links, opening attachments, or complying with requests. Click the 
"Phish Alert Report" button above the email, or contact MIS, regarding any 
suspicious message.



Hello Team,

Is there any issue with Ceph RBD as Primary Storage to achieve HA-Enabled
VM?

I am using KVM Hypervisors and my CloudStack version is 4.17

Yours sincerely,


Nazmul Parvej
Deputy Manager, Product Development
IT Division

Bangladesh Export Import Company Ltd.

Level-9, SAM Tower, Plot #4, Road #22, Gulshan-1, Dhaka-1212,Bangladesh

Tel: +880 9609 000 999, +880 2 5881 5559, Ext: 14193, Fax:  +880 2 
95757

Cell: +8801787680841, Email: nazmul.par...@bol-online.com, Web:
www.bol-online.com

Re: Clarity on when to use "Non-OSS" ACS

[Simon Weller]

Hey Mike,

Did you see my response to your first post around this? It sounds like you may 
have missed it.

-Simon

From: Corey, Mike 
Sent: Thursday, January 13, 2022 6:45 AM
To: users@cloudstack.apache.org 
Subject: RE: Clarity on when to use "Non-OSS" ACS


Second attempt for clarity…



From: Corey, Mike
Sent: Tuesday, January 11, 2022 2:53 PM
To: users@cloudstack.apache.org
Subject: Clarity on when to use "Non-OSS" ACS



Hi,



I’ve been working with ACS 4.15 deploying to VMware hypervisors for over a year 
now.  I’m catching up on 4.16 docs for an upgrade of our landscape.  For 
whatever reason I’m just seeing the below regarding Non-OSS package creation if 
using VMware.



Could I get some clarity on this please?  I’m not sure what it is meant by 
VMware, NetApp, or other non-oss plug-ins.  Up to this point I’ve been using 
the “normal” CloudStack CentOS package repo for the ACS installation.  Have I 
done it all wrong?





### ACS 4.16 Guide:



If you need support for the VMware, NetApp, F5, NetScaler, SRX, or any other 
non-Open Source Software (nonoss) plugins, you’ll need to download a few 
components on your own and follow a slightly different procedure to build from 
source.



Some of the plugins supported by CloudStack cannot be distributed with 
CloudStack for licensing reasons. In some cases, some of the required 
libraries/JARs are under a proprietary license. In other cases, the required 
libraries may be under a license that’s not compatible with Apache’s licensing 
guidelines for third-party products.



###





Mike Corey

Technology Senior Consultant, IT CS CTW Operation & Virtualization Service US

SAP AMERICA, INC. 3999 West Chester Pike, Newtown Square, 19073 United States

T +1 610 661 0905, M +1 484 274 2658, E 
mike.co...@sap.com


[cid:image002.png@01D80851.80040570]

Re: Clarity on when to use "Non-OSS" ACS

[Simon Weller]

Mike,

Cloudstack can be compiled with or without the non APL2 licensed JARS. These 
JARS are typically API interfaces for various commercial products and typically 
are closed source or have incompatible licenses. If you're using VMWare, you 
are probably using a package source that has compiled Cloudstack with the 
included JARS.
The only reason two options are offered is that Cloudstack can't distribute the 
JARS as part of the source code, but a third party can (and does) include them 
as part of a compiled release for convenience to the community.

Hope this helps.

-Si


From: Corey, Mike 
Sent: Tuesday, January 11, 2022 1:53 PM
To: users@cloudstack.apache.org 
Subject: Clarity on when to use "Non-OSS" ACS


Hi,



I’ve been working with ACS 4.15 deploying to VMware hypervisors for over a year 
now.  I’m catching up on 4.16 docs for an upgrade of our landscape.  For 
whatever reason I’m just seeing the below regarding Non-OSS package creation if 
using VMware.



Could I get some clarity on this please?  I’m not sure what it is meant by 
VMware, NetApp, or other non-oss plug-ins.  Up to this point I’ve been using 
the “normal” CloudStack CentOS package repo for the ACS installation.  Have I 
done it all wrong?





### ACS 4.16 Guide:



If you need support for the VMware, NetApp, F5, NetScaler, SRX, or any other 
non-Open Source Software (nonoss) plugins, you’ll need to download a few 
components on your own and follow a slightly different procedure to build from 
source.



Some of the plugins supported by CloudStack cannot be distributed with 
CloudStack for licensing reasons. In some cases, some of the required 
libraries/JARs are under a proprietary license. In other cases, the required 
libraries may be under a license that’s not compatible with Apache’s licensing 
guidelines for third-party products.



###





Mike Corey

Technology Senior Consultant, IT CS CTW Operation & Virtualization Service US

SAP AMERICA, INC. 3999 West Chester Pike, Newtown Square, 19073 United States

T +1 610 661 0905, M +1 484 274 2658, E mike.co...@sap.com



[cid:image002.png@01D806FA.EE5ECAD0]

Re: DDoS protection

[Simon Weller]

As Hean suggests, the best form of DDOS protection is from your upstream 
internet provider(s), as hopefully they have a lot more capacity to be able to 
absorb a large attack.
If you're looking at protecting web properties, then you might want to take a 
look at services from companies like Akamai or Cloudflare.

From: Hean Seng 
Sent: Saturday, October 30, 2021 1:42 PM
To: users@cloudstack.apache.org 
Cc: dev 
Subject: Re: DDoS protection

Hi

I suppose this is not related to Cloudstack. You need to Network Provider
that support DDOS protection

On Sun, Oct 31, 2021 at 2:40 AM Ranjit Jadhav 
wrote:

> Hello,
>
> What are options available for DDoS protection which we can integrate with
> Cloudstack?
>
> Thank you,
> Ranjit
>


--
Regards,
Hean Seng

Re: 2FA

[Simon Weller]

Rakesh,

ACS does support SAML2 and in order to deploy 2FA/MFA, you could integrate it 
with an Identity and Access Management System such as Keycloak 
(https://www.keycloak.org/).

-Si


From: Rakesh Venkatesh 
Sent: Tuesday, August 10, 2021 4:34 AM
To: users ; dev 
Subject: 2FA

Hello

Has anyone thought about 2FA or about how to implement it in cloudstack?
Looks like this will be good addition to enhance the security. I have some
idea about implementing in the backend but dont have much idea on how to
display the QR code in ui or other functionalities which is needed for
frontend part.

--
Thanks and regards
Rakesh

Re: [PROPOSE] RM for CloudStack Kubernetes Provider v1.0

[Simon Weller]

+1



From: David Jumani 
Sent: Thursday, July 15, 2021 1:31 AM
To: users ; d...@cloudstack.apache.org 

Subject: [PROPOSE] RM for CloudStack Kubernetes Provider v1.0

Hi,

I'd like to put myself forward as the release manager for CloudStack Kubernetes 
Provider v1.0.

This will be the first release of CloudStack Kubernetes Provider which 
facilitates Kubernetes deployments on Cloudstack.
It allows Kubernetes to dynamically allocate IP addresses and the respective 
networking rules on CloudStack to ensure seamless TCP, UDP and TCP-Proxy 
LoadBalancer deployments on Kubernetes.

It was initially the Cloudstack provider in Kubernetes which was later 
extracted to allow for pluggable providers.
A lot of work and effort has gone into developing it, and we are looking 
forward to its grand debut.

The list of open issues triaged for the v1.0 milestone can be found at 
https://github.com/apache/cloudstack-kubernetes-provider/milestone/1
If you encounter any issues, please do report them at 
https://github.com/apache/cloudstack-kubernetes-provider/issues

Looking forward to your support

Thanks,
David

Re: [DISCUSS] Using qemu-kvm vs qemu-kvm-ev with CloudStack

[Simon Weller]

Hi Rohit,

We've been using ev exclusively for a few years now. Our main reason was in 
order to support features we upstreamed around KVM iop limits a couple of years 
back.
Short of one challenge that was addressed on the ACS side a while ago related 
to the patchviasocket integration, it has worked very well and has been very 
stable.

-Si


From: Rohit Yadav 
Sent: Friday, April 9, 2021 2:26 AM
To: d...@cloudstack.apache.org ; 
users@cloudstack.apache.org 
Subject: [DISCUSS] Using qemu-kvm vs qemu-kvm-ev with CloudStack

All,

We've recently seen some tests around live VM with storage failing on CentOS7 
which is addressed in this PR:
https://github.com/apache/cloudstack/pull/4801

Some users have added on the original issue ticket that it works with 
qemu-kvm-ev on CentOS:
https://github.com/apache/cloudstack/issues/4757#issuecomment-812595973

I also see many other IaaS platforms notably oVirt using qemu-kvm-ev, is there 
any interest and argument in saying we test and update our docs to advise users 
to use qemu-kvm-ev on CentOS? Are there any CloudStack users who want to share 
their experience with it who may be using it already?

The installation steps don't require configuring any 3rd party repository 
manually and usually done with:

yum install centos-release-qemu-ev
yum install qemu-kvm-ev

Additional references:
https://lists.centos.org/pipermail/centos-virt/2015-October/004717.html (what 
is qemu-kvm vs qemu-kvm-ev)
https://wiki.centos.org/SpecialInterestGroup/Virtualization (the SIG that is 
behind the qemu-kvm-ev repository)

Thanks and regards.

rohit.ya...@shapeblue.com
www.shapeblue.com
3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
@shapeblue

Re: Volume could not be found after KVM host update

[Simon Weller]

Do you have the qcow2 file on the NFS host?


From: Fariborz Navidan 
Sent: Friday, April 2, 2021 11:08 AM
To: users@cloudstack.apache.org 
Subject: Volume could not be found after KVM host update

Hello,

One of my primary storages is a local directory on KVM host exposed via
NFS. Just after updating system packages, one of instances which was
running crashed and now cannot be started. I tried creating a template from
it's root volume and re-deploy the VM but I get following exception:
Couldn't find volume id [volume-id]

The primary storage is built on top of Linux Soft Raid and it's filesystem
is XFS.

Is the VM totally lost? What can I do?

Re: Congratulations to Sven - Apache Software Foundation Member

[Simon Weller]

Congrats Sven!

From: Paul Angus 
Sent: Wednesday, March 17, 2021 4:13 PM
To: d...@cloudstack.apache.org ; 
users@cloudstack.apache.org 
Subject: Congratulations to Sven - Apache Software Foundation Member

Hi All,



More great news.



Please join me in congratulating Sven,  for being made a Member of the
Apache Software Foundation.



Congratulations Sven, keep up the good work!



Kind regards



Paul Angus

Re: Congratulations to Gabriel - CloudStack PMC Chair

[Simon Weller]

Congrats Gabriel!

From: Paul Angus 
Sent: Wednesday, March 17, 2021 4:10 PM
To: d...@cloudstack.apache.org ; 
users@cloudstack.apache.org 
Cc: priv...@cloudstack.apache.org 
Subject: Congratulations to Gabriel - CloudStack PMC Chair

Hi All CloudStack enthusiasts!



Please join me in congratulating Gabriel for becoming the next CloudStack
PMC Chair.

Congratulations Gabriel, very well deserved!



I would also like to thank Sven for his great work of the past year!







Kind regards



Paul Angus

Re: Cloudstack developer training

[Simon Weller]

Fantastic contribution!  Thanks to the ShapeBlue team for making this happen.

-Si

From: Giles Sirett 
Sent: Friday, February 26, 2021 9:42 AM
To: d...@cloudstack.apache.org ; 
users@cloudstack.apache.org ; Apache CloudStack 
Marketing 
Subject: Cloudstack developer training

Hi all

One of the biggest challenges with Cloudstack is learning its architecture and 
codebase  - its big and its complicated. Onboarding new software engineers can 
be a daunting process.
For the last 2 years, we at ShapeBlue have built up a set of resources to help 
us with onboarding on new engineers who will be working on Cloudstack.

This has evolved into a self-study course that we call "hackerbook"- the logic 
being that it's a training course that gets engineers hands-on hacking in the 
code ASAP.  It's a mix of videos, exercises and other resources.

Today, we've opensourced this resource in order to make it available to anybody 
who may want to learn to develop on Cloudstack.

Feedback and improvement PRs will be warmly accepted

Its currently sitting in a shapeblue repo, happy to move under ASF if anybody 
thinks that's important

https://github.com/shapeblue/hackerbook

Happy Hacking

Kind regards
Giles


giles.sir...@shapeblue.com
www.shapeblue.com
3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
@shapeblue

Re: manager does not start

[Simon Weller]

Piotr,

When you restored your DB, did you drop the old one first just to make sure a 
clean backup was restored? Are your positive this backup was taken before the 
attempted change? Do you have an older backup available?
The log seems to indicate it rolled back a failed DB transaction related to 
createSystemServiceOfferings. How did your colleague attempt to make the cert 
change?

-Si


From: Piotr Pisz 
Sent: Thursday, January 21, 2021 6:42 AM
To: users@cloudstack.apache.org 
Subject: manager does not start

Hello,

We have a strange problem, after restart of mgmt serwer it no running properly:

2021-01-21 13:28:31,769 INFO  [c.c.c.ConsoleProxyManagerImpl] (main:null) 
(logid:) Start configuring console proxy manager : ConsoleProxyManagerImpl
2021-01-21 13:28:31,769 INFO  [c.c.c.ConsoleProxyManagerImpl] (main:null) 
(logid:) Console proxy max session soft limit : 50
2021-01-21 13:28:31,769 INFO  [c.c.c.ConsoleProxyManagerImpl] (main:null) 
(logid:) Console proxy standby capacity : 10
2021-01-21 13:28:31,771 DEBUG [c.c.a.m.AgentManagerImpl] (main:null) (logid:) 
Registering listener ConsoleProxyListener with id 3
2021-01-21 13:28:31,785 DEBUG [c.c.u.d.T.Transaction] (main:null) (logid:) 
Rolling back the transaction: Time = 3 Name =  createSystemServiceOfferings; 
called by 
-TransactionLegacy.rollback:890-TransactionLegacy.removeUpTo:833-TransactionLegacy.close:657-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:175-ExposeInvocationInterceptor.invoke:95-ReflectiveMethodInvocation.proceed:186-JdkDynamicAopProxy.invoke:212-$Proxy102.createSystemServiceOfferings:-1-ConsoleProxyManagerImpl.configure:1328-CloudStackExtendedLifeCycle$3.with:115-CloudStackExtendedLifeCycle.with:154
2021-01-21 13:28:31,786 WARN  [o.a.c.s.m.c.ResourceApplicationContext] 
(main:null) (logid:) Exception encountered during context initialization - 
cancelling refresh attempt: 
org.springframework.context.ApplicationContextException: Failed to start bean 
'cloudStackLifeCycle'; nested exception is java.lang.NullPointerException
2021-01-21 13:28:31,788 INFO  [o.a.c.s.NfsMountManager] (main:null) (logid:) 
Clean up mounted NFS mount points used in current session.
2021-01-21 13:28:31,791 WARN  [o.e.j.w.WebAppContext] (main:null) (logid:) 
Failed startup of context 
o.e.j.w.WebAppContext@2d0399f4{/client,file:///usr/share/cloudstack-management/webapp/,UNAVAILABLE}{/usr/share/cloudstack-management/webapp}
org.springframework.context.ApplicationContextException: Failed to start bean 
'cloudStackLifeCycle'; nested exception is java.lang.NullPointerException
at 
org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:185)
at 
org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:53)
at 
org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:360)
at 
org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:158)
at 
org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:122)
at 
org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:894)
at 
org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:553)
at 
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContext(DefaultModuleDefinitionSet.java:145)
at 
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet$2.with(DefaultModuleDefinitionSet.java:122)
at 
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:245)
at 
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:250)
at 
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:250)
at 
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:233)
at 
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContexts(DefaultModuleDefinitionSet.java:117)
at 
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.load(DefaultModuleDefinitionSet.java:79)
at 
org.apache.cloudstack.spring.module.factory.ModuleBasedContextFactory.loadModules(ModuleBasedContextFactory.java:37)
at 
org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.init(CloudStackSpringContext.java:71)
at 
org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.(CloudStackSpringContext.java:58)
at 

Re: Experiences with KVM, iSCSI and OCFS2 (SharedMountPoint)

[Simon Weller]

We used to use CLVM a while ago before we shifted to Ceph. Cluster 
suite/corosync was a bit of a nightmare, and fencing events caused all sorts of 
locking (DLM) problems.
I helped a CloudStack user out a couple of month ago, after they upgraded and 
CLVM broke, so I know it's still out there in limited places.
I wouldn't recommend using it today unless you're very brave and have the 
capability of troubleshooting the code yourself.

-Si



From: Wido den Hollander 
Sent: Thursday, January 21, 2021 11:26 AM
To: d...@cloudstack.apache.org ; n...@li.nux.ro 

Cc: users@cloudstack.apache.org 
Subject: Re: Experiences with KVM, iSCSI and OCFS2 (SharedMountPoint)



On 21/01/2021 11:34, n...@li.nux.ro wrote:
> Hi,
>
> I used SharedMountPoint a very long time ago with GlusterFS before
> Cloudstack had native integration.
> Don't remember the details, but overall my impression was that it worked
> surprisingly well, of course back then there weren't as many feature, so
> less stuff to test. I would give it a go.
>
> As a side note, I did also use iSCSI with CLVM with success, it was
> quite fast. I ended up doing it because it was difficult to get OCFS
> running on EL6 and GFS2 had a reputation for being very slow. Marcus has
> a lot of experience with this, might want to get in touch with him:
> https://www.slideshare.net/MarcusLSorensen/cloud-stack-clvm

I assume you used CLVM with Corosync?

My concern with LVM is:

- No thin provisioning (when used with CloudStack)
- No snapshots (Right?)
- Not very much used

OCFS2 doesn't have my preference either, but otherwise you have to use
corosync.

Anybody else otherwise using CLVM?

Wido

>
> HTH,
> Lucian
>
> On 2021-01-21 09:32, Wido den Hollander wrote:
>> Hi,
>>
>> For a specific use-case I'm looking into the possibility to use iSCSI
>> in combination with KVM.
>>
>> Use-case: Low-latenc I/O with 4k blocks and QD=1
>>
>> KVM with CloudStack doesn't support iSCSI natively and the docs and
>> other blogs refer to using 'SharedMountPoint' with OCFS2 or GFS2:
>>
>> -
>> http://docs.cloudstack.apache.org/en/latest/adminguide/storage.html#hypervisor-support-for-primary-storage
>>
>> -
>> https://www.shapeblue.com/installing-and-configuring-an-ocfs2-clustered-file-system/
>>
>>
>> It has been a really long time since I've used OCFS2 and I wanted to
>> see what experiences from other people are.
>>
>> How is the stability and performance of OFCS2? It seems that
>> performance should be rather good as lock/s is a problem with
>> clustered filesystems, but since we only lock the QCOW2 file on boot
>> of the VM that shouldn't be an issue.
>>
>> In addition to OCFS2, how mature is 'SharedMountPoint' as a storage
>> pool with KVM. Does is support all the features NFS supports?
>>
>> Thanks,
>>
>> Wido

Re: Cloudstack 4.15 Release Candidate

[Simon Weller]

Hi Bill,

Once the release artifacts go up for a vote, you can grab them and start 
testing.

-Si

From: William Hankard 
Sent: Tuesday, December 1, 2020 2:37 PM
To: users@cloudstack.apache.org 
Subject: Cloudstack 4.15 Release Candidate


Hello,

How can I participate in the testing of the latest 4.15 Apache Cloudstack
release.   Thank you.

Bill
William D. Hankard
Senior Enterprise Virtualization Architect / Backend Developer
IBM Security
X-Force Threat Intelligence and Integration Lab
william_hank...@us.ibm.com
Phone: 617-910-8562

Re: [ANNOUNCE] new committer: Nguyen Mai Hoang

[Simon Weller]

Congrats Hoang!!

From: Suresh Anaparti 
Sent: Wednesday, November 18, 2020 3:37 AM
To: hoang.ngu...@ewerk.com ; d...@cloudstack.apache.org 
; users@cloudstack.apache.org 

Subject: Re: [ANNOUNCE] new committer: Nguyen Mai Hoang

Congrats Hoang!

Regards,
Suresh

On 18/11/20, 4:19 AM, "Sven Vogel"  wrote:

Hi everyone,



 The Project Management Committee (PMC) for Apache CloudStack
has invited Nguyen Mai Hoang to become a committer and we are pleased
to announce that he has accepted.

Please join me in congratulating Hoang on this accomplishment.


Being a committer enables easier contribution to the
project since there is no need to go via the patch
submission process. This should enable better productivity.


Thanks and Cheers,



Sven Vogel
Apache CloudStack PMC member


suresh.anapa...@shapeblue.com
www.shapeblue.com
3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
@shapeblue

Re: [ANNOUNCE] new committer: Rakesh Venkatesh

[Simon Weller]

Congrats Rakesh!!

From: Suresh Anaparti 
Sent: Wednesday, November 18, 2020 3:36 AM
To: r.venkat...@global.leaseweb.com ; 
d...@cloudstack.apache.org ; 
users@cloudstack.apache.org 
Subject: Re: [ANNOUNCE] new committer: Rakesh Venkatesh

Congrats Rakesh!

Regards,
Suresh

On 18/11/20, 4:18 AM, "Sven Vogel"  wrote:

Hi everyone,



 The Project Management Committee (PMC) for Apache CloudStack
has invited Rakesh Venkatesh to become a committer and we are pleased

to announce that he has accepted.

Please join me in congratulating Rakesh on this accomplishment.


Being a committer enables easier contribution to the
project since there is no need to go via the patch
submission process. This should enable better productivity.


Thanks and Cheers,



Sven Vogel
Apache CloudStack PMC member


suresh.anapa...@shapeblue.com
www.shapeblue.com
3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
@shapeblue

Re: [ANNOUNCE] new committer: Abhishek Kumar

[Simon Weller]

Congrats Abhishek!!

From: Suresh Anaparti 
Sent: Wednesday, November 18, 2020 3:35 AM
To: Abhishek Kumar ; users@cloudstack.apache.org 
; d...@cloudstack.apache.org 

Subject: Re: [ANNOUNCE] new committer: Abhishek Kumar

Congrats Abhishek...

Regards,
Suresh

On 18/11/20, 4:18 AM, "Sven Vogel"  wrote:

Hi everyone,


The Project Management Committee (PMC) for Apache CloudStack
has invited Abhishek Kumar to become a committer and we are pleased
to announce that he has accepted.

Please join me in congratulating Abhishek on this accomplishment.


Being a committer enables easier contribution to the
project since there is no need to go via the patch
submission process. This should enable better productivity.




Thanks and Cheers,



Sven Vogel
Apache CloudStack PMC member


suresh.anapa...@shapeblue.com
www.shapeblue.com
3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
@shapeblue

Re: [ANNOUNCE] new committer: Suresh Anaparti

[Simon Weller]

Congrats Suresh!!

From: Rohit Yadav 
Sent: Wednesday, November 18, 2020 3:13 AM
To: d...@cloudstack.apache.org ; 
users@cloudstack.apache.org 
Cc: Suresh Anaparti 
Subject: Re: [ANNOUNCE] new committer: Suresh Anaparti

Congratulations Suresh!


Regards.


From: Sven Vogel 
Sent: Wednesday, November 18, 2020 04:17
To: d...@cloudstack.apache.org ; 
users@cloudstack.apache.org 
Cc: Suresh Anaparti 
Subject: [ANNOUNCE] new committer: Suresh Anaparti

Hi everyone,


 The Project Management Committee (PMC) for Apache CloudStack
has invited Suresh Anaparti to become a committer and we are pleased
to announce that he has accepted.

Please join me in congratulating Suresh on this accomplishment.


Being a committer enables easier contribution to the
project since there is no need to go via the patch
submission process. This should enable better productivity.


Thanks and Cheers,



Sven Vogel
Apache CloudStack PMC member

rohit.ya...@shapeblue.com
www.shapeblue.com
3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
@shapeblue

Re: libvirtd fails to start after installing OVMF package

[Simon Weller]

Can you post the libvirtd log? Any other logs of interest?

From: Fariborz Navidan 
Sent: Monday, October 26, 2020 9:34 AM
To: users@cloudstack.apache.org 
Subject: libvirtd fails to start after installing OVMF package

Hi All,

I have a VM with 5TB GPT formatted disk which needs UEFI support. I just
installed OVMF package on the host to enable UEFI support. However
afterward libvirtd fails to start, Removing the package did not restore the
libvirtd. Host OS is CentOS 7

Thanks for helps in advance.

Re: Adding Regions - In use environments

[Simon Weller]

Hi Cristian,

Each region is its own entity and they're loosely coupled together. I'm not 
aware of any limits at install that would prevent you from adding regions.

-Si


From: cristian.c@istream.today 
Sent: Monday, September 21, 2020 6:23 AM
To: users@cloudstack.apache.org 
Subject: Adding Regions - In use environments

Hello,



 It is not possible to have multiple regions in Cloudstack Environments
if are in use (not configured with multiple regions at deploy) ?  I'm asking
this because it is not very clear from this article :
http://docs.cloudstack.apache.org/projects/archived-cloudstack-installation/
en/4.11/configuration.html



   If is like this then makes no sense, you can not see in the future how
many regions do you expect to have.





Cristian

Re: KVM Snapshot with Ceph

[Simon Weller]

Vincent,

VM Snapshots aren't supported with Ceph as the primary storage. As you note, 
when using QCOW2 files, the memory payload for the VM snapshot is included 
within the QCOW2 file. Since Ceph is using raw storage, there is no where to 
store that memory (at least today).
Storage based snapshots are supported on Ceph though and you can snap any disk 
instance using the native Ceph snapshot functionality.

-Si



From: Vincent Hermes 
Sent: Wednesday, June 10, 2020 9:51 AM
To: users@cloudstack.apache.org 
Subject: KVM Snapshot with Ceph

Hi Guys,

so we have built a Cluster with a few CentOS7 CS Nodes with KVM as the 
Hypervisor AND Ceph as Primary Storage in a hyperconverged way and we got 
something interesting, maybe some of you have an idea whats up with that. As 
KVM works with QCOW2 Image Format for its VMs, all of the Storage Volumes and 
Templates are QCOW2 Files. However trying to take a Snapshot including RAM ends 
with the error "We only support create vm snapshots from vm with QCOW2 image". 
Trying a "virsh snapshot-create [vm-id]" brings us "error: unsupported 
configuration: internal snapshot for disk vda unsupported for storage type 
raw". The qemu.log states "-drive [...] format=raw [...]" and the according XML 
file of the VM of course says the same.

Do you have experience in getting Snapshots to work with KVM and Ceph in CS?

Many thanks in advance and best regards from Germany

Vincent

Re: Hello and first doubts!

[Simon Weller]

Welcome Augusto!

I'll do my best to answer your questions and I'm sure others will chime in as 
well.


  1.  CloudStack is designed to scale to massive levels and supports various 
levels of availability. Typically, if you are going to be building out services 
in different geographic areas, you'd separate those into CloudStack Regions.  A 
region is designed to be completely self-contained, with no dependencies on 
other regions. Please see this URL for more detail on this - 
https://docs.cloudstack.apache.org/en/latest/conceptsandterminology/concepts.html#cloudstack-terminology
  2.  AWS and other vendors do offer some bare metal services. As long as you 
have direct access to the underlying server hardware, you could definitely 
utilize them as CloudStack Hosts. CloudStack does support a number of 
hypervisors, either directly (as in the case with KVM), or via other management 
systems (such as VMWare with vSphere). Think of CloudStack as an orchestrator 
or orchestrators.
  3.  CloudStack networking can be deploying in a number of ways. At the most 
simplistic level, you can deploy hosts and use host based security groups for 
separation with little to no complexity on the underlay network.  You can also 
utilize advanced networking and separate Virtual Private Clouds (VPCs) via 
VLANs, VXLANs or other tunnel protocols depending on what the underlining 
hypervisor supports.

CloudStack is in use by large cloud providers and huge corporations world wide 
and it built to be extremely robust, while still providing the flexibility that 
different entities require. We have a large community and a lot of vendor 
integrations allowing the cloud operator to mix and match technologies that 
better suit their needs.

-Si




From: i...@defendhosting.com 
Sent: Friday, May 15, 2020 7:05 AM
To: users@cloudstack.apache.org 
Subject: Hello and first doubts!


Hello,

My name is Augusto and I am new here. Hello to all. Glad to be here.

I am new in cloudstack and I have some doubts that may be can be
answered here. I already provide server and vps service, but I would
like to provide cloud service with cloudstack in several locations.

First of all, I am sorry because my cloudstack knowledge is near to
null. I plan to start now, and for that reason I have these doubs:

1).- The master and the hosts must be in same datacenter?. I mean, if I
install a master in europe, can I manage the hosts in Europe, USA,
Canada and other locations?. For example using a hetzner server for
master and ovh or sys for hosts.

2).- I saw some videos about vmware, indicating that it is possible to
use AWS infraestructure to deploy hosts and use their infraestructure to
growup creating several "virtual datacenters". That can be done with
cloudstack? (or with other major clouds)

3).- If hosts are deployed in same datacenter, for example OVH, Is there
additional tasks to acomplish regarding the network? I mean, creating
for example an vRack or it is not necessary and cloudstack will manage all?

Thank you for your

--

�� DEFEND HOSTING - www.defendhosting.com - A 
Company You Can Trust
�� Shared Hosting | VPS | Dedicated Servers | SEO | PBX, CRM and SEO Servers
�� USA/EU Locations | Fast Network/Server Backbone Connection
�� Best Value For Your Money | Outstanding Uptime | Customer Tailored
Support | Money-Back Guarantee

--
El software de antivirus Avast ha analizado este correo electrónico en busca de 
virus.
https://www.avast.com/antivirus

Re: I/O speed in local NFS vs local Ceph

[Simon Weller]

Ceph uses data replicas, so even if you only use 2 replicas (3 is recommend), 
you'd basically have best case of the IO of a single drive.  You also need to 
have a minimum of 3 management nodes for Ceph, so personally, I'd stick with 
local storage if you're focused on speed.
You're also running quite a risk by using RAID-0. A single drive failure and 
you'll lose all of your data. Is there a reason you are using NFS and not just 
direct local storage?



From: Fariborz Navidan 
Sent: Tuesday, April 7, 2020 10:36 AM
To: users@cloudstack.apache.org 
Subject: I/O speed in local NFS vs local Ceph

Hello,

I have a single physical host running  CloudStack. Primary storage is
currently mounted as a NFS share. The underlying filesystem is XFS
running on top of Linux Soft RAID-0. The underlying hardware consists of 2
SSD-NVMe drives.

The question is that, could I reach faster I/O on VMs if I would use Ceph
adding 2 physical devices directly to the cluster and expose it via RBD?
How much could it make the I/O faster?

Thanks

Re: Instances lose internet

[Simon Weller]

Is the version current? I remember a bug related to this in virtio a couple of 
years ago.

From: Adam Witwicki 
Sent: Tuesday, April 7, 2020 10:22 AM
To: users 
Subject: Re: Instances lose internet

Yes using virtio for nics and storage

Thanks
Adam


From: Simon Weller 
Sent: Tuesday, April 7, 2020 4:19:59 PM
To: users@cloudstack.apache.org 
Subject: Re: Instances lose internet

** This mail originated from OUTSIDE the Oakford corporate network. Treat 
hyperlinks and attachments in this email with caution. **

Are you using the virtio drivers?

From: Adam Witwicki 
Sent: Tuesday, April 7, 2020 1:33 AM
To: users@cloudstack.apache.org 
Subject: Instances lose internet

Hello,

I wonder if anyone else has seen this, problerly not cloudstack but a user on 
our systems says that their instances randomly lose connection to the internet.
The windows instance says the nic is connected but can not ping any other 
server. There is no Virtual router as it's a L2 network
I found a post that mentioned an issue with bonding - but I can not see any 
bonding errors in the syslog

Thanks

Adam


Cloudstack Version 4.11
Ubuntu 16 KVM
Bonded Nics

/usr/bin/qemu-system-x86_64 --version
QEMU emulator version 2.5.0 (Debian 1:2.5+dfsg-5ubuntu10.31), Copyright (c) 
2003-2008 Fabrice Bellard

virsh version
Compiled against library: libvirt 1.3.1
Using library: libvirt 1.3.1
Using API: QEMU 1.3.1
Running hypervisor: QEMU 2.5.0

Distributor ID: Ubuntu
Description:Ubuntu 16.04.4 LTS
Release:16.04
Codename:   xenial

Disclaimer Notice:
This email has been sent by Oakford Technology Limited, while we have checked 
this e-mail and any attachments for viruses, we can not guarantee that they are 
virus-free. You must therefore take full responsibility for virus checking.
This message and any attachments are confidential and should only be read by 
those to whom they are addressed. If you are not the intended recipient, please 
contact us, delete the message from your computer and destroy any copies. Any 
distribution or copying without our prior permission is prohibited.
Internet communications are not always secure and therefore Oakford Technology 
Limited does not accept legal responsibility for this message. The recipient is 
responsible for verifying its authenticity before acting on the contents. Any 
views or opinions presented are solely those of the author and do not 
necessarily represent those of Oakford Technology Limited.
Registered address: Oakford Technology Limited, The Manor House, Potterne, 
Wiltshire. SN10 5PN.
Registered in England and Wales No. 5971519


Disclaimer Notice:
This email has been sent by Oakford Technology Limited, while we have checked 
this e-mail and any attachments for viruses, we can not guarantee that they are 
virus-free. You must therefore take full responsibility for virus checking.
This message and any attachments are confidential and should only be read by 
those to whom they are addressed. If you are not the intended recipient, please 
contact us, delete the message from your computer and destroy any copies. Any 
distribution or copying without our prior permission is prohibited.
Internet communications are not always secure and therefore Oakford Technology 
Limited does not accept legal responsibility for this message. The recipient is 
responsible for verifying its authenticity before acting on the contents. Any 
views or opinions presented are solely those of the author and do not 
necessarily represent those of Oakford Technology Limited.
Registered address: Oakford Technology Limited, The Manor House, Potterne, 
Wiltshire. SN10 5PN.
Registered in England and Wales No. 5971519

Re: Instances lose internet

[Simon Weller]

Are you using the virtio drivers?

From: Adam Witwicki 
Sent: Tuesday, April 7, 2020 1:33 AM
To: users@cloudstack.apache.org 
Subject: Instances lose internet

Hello,

I wonder if anyone else has seen this, problerly not cloudstack but a user on 
our systems says that their instances randomly lose connection to the internet.
The windows instance says the nic is connected but can not ping any other 
server. There is no Virtual router as it's a L2 network
I found a post that mentioned an issue with bonding - but I can not see any 
bonding errors in the syslog

Thanks

Adam


Cloudstack Version 4.11
Ubuntu 16 KVM
Bonded Nics

/usr/bin/qemu-system-x86_64 --version
QEMU emulator version 2.5.0 (Debian 1:2.5+dfsg-5ubuntu10.31), Copyright (c) 
2003-2008 Fabrice Bellard

virsh version
Compiled against library: libvirt 1.3.1
Using library: libvirt 1.3.1
Using API: QEMU 1.3.1
Running hypervisor: QEMU 2.5.0

Distributor ID: Ubuntu
Description:Ubuntu 16.04.4 LTS
Release:16.04
Codename:   xenial

Disclaimer Notice:
This email has been sent by Oakford Technology Limited, while we have checked 
this e-mail and any attachments for viruses, we can not guarantee that they are 
virus-free. You must therefore take full responsibility for virus checking.
This message and any attachments are confidential and should only be read by 
those to whom they are addressed. If you are not the intended recipient, please 
contact us, delete the message from your computer and destroy any copies. Any 
distribution or copying without our prior permission is prohibited.
Internet communications are not always secure and therefore Oakford Technology 
Limited does not accept legal responsibility for this message. The recipient is 
responsible for verifying its authenticity before acting on the contents. Any 
views or opinions presented are solely those of the author and do not 
necessarily represent those of Oakford Technology Limited.
Registered address: Oakford Technology Limited, The Manor House, Potterne, 
Wiltshire. SN10 5PN.
Registered in England and Wales No. 5971519

Re: Agent staying in alert mode

[Simon Weller]

No, it will not. All VMs will stay up.

From: Adam Witwicki 
Sent: Wednesday, March 18, 2020 9:27 AM
To: users@cloudstack.apache.org 
Subject: RE: Agent staying in alert mode

Will that stop all the Instances?

Thanks

Adam

-Original Message-
From: Simon Weller 
Sent: 18 March 2020 14:26
To: users@cloudstack.apache.org
Subject: Re: Agent staying in alert mode

** This mail originated from OUTSIDE the Oakford corporate network. Treat 
hyperlinks and attachments in this email with caution. **

Have you tried restarting libvirtd and then restarting the cloudstack agent?


From: Adam Witwicki 
Sent: Wednesday, March 18, 2020 3:06 AM
To: users@cloudstack.apache.org 
Subject: RE: Agent staying in alert mode

Hi Andrija,

Setting the host to management IP, and not our LB VIP. The agent process seems 
to stall

2020-03-18 07:44:01,969 DEBUG [kvm.resource.LibvirtComputingResource] 
(Agent-Handler-1:null) (logid:) cpus=32, speed=3800, ram=269385596928, 
_dom0MinMem=1073741824, _dom0OvercommitMem=0, cpu sockets=2
2020-03-18 07:44:01,972 DEBUG [cloud.resource.ServerResourceBase] 
(Agent-Handler-1:null) (logid:) Parameters for private nic: 10.10.240.126 - 
a0:36:9f:24:32:6a-255.255.255.0
2020-03-18 07:44:01,973 DEBUG [cloud.resource.ServerResourceBase] 
(Agent-Handler-1:null) (logid:) Parameters for storage nic: 10.10.240.126 - 
a0:36:9f:24:32:6a-255.255.255.0
2020-03-18 07:44:01,973 DEBUG [cloud.resource.ServerResourceBase] 
(Agent-Handler-1:null) (logid:) Parameters for public nic: 10.10.240.126 - 
a0:36:9f:24:32:6a-255.255.255.0
2020-03-18 07:44:01,973 DEBUG [kvm.resource.LibvirtComputingResource] 
(Agent-Handler-1:null) (logid:) Executing: 
/usr/share/cloudstack-common/scripts/vm/hypervisor/versions.sh
2020-03-18 07:44:01,984 DEBUG [kvm.resource.LibvirtComputingResource] 
(Agent-Handler-1:null) (logid:) Execution is successful.
2020-03-18 07:44:01,985 DEBUG [kvm.resource.LibvirtComputingResource] 
(Agent-Handler-1:null) (logid:) Executing: sudo grep InitiatorName= 
/etc/iscsi/initiatorname.iscsi
2020-03-18 07:44:01,992 DEBUG [kvm.resource.LibvirtComputingResource] 
(Agent-Handler-1:null) (logid:) Execution is successful.
2020-03-18 07:44:01,993 INFO  [kvm.storage.LibvirtStorageAdaptor] 
(Agent-Handler-1:null) (logid:) Attempting to create storage pool 
257c9140-3863-4be2-85e2-9835650f1b45 (Filesystem) in libvirt
2020-03-18 07:44:01,993 DEBUG [kvm.resource.LibvirtConnection] 
(Agent-Handler-1:null) (logid:) Looking for libvirtd connection at: 
qemu:///system HALTS HERE AND DOES NOT CONTINUE

I waited 10 mins for a timeout then stopped the agent

Thanks

Adam


-Original Message-
From: Andrija Panic 
Sent: 17 March 2020 18:52
To: users 
Subject: Re: Agent staying in alert mode

** This mail originated from OUTSIDE the Oakford corporate network. Treat 
hyperlinks and attachments in this email with caution. **

Can you change that to the IP of the mgmt server and then restart the agent? it 
should always point to the IP not hostname (what is that value there, "who put 
it there" etc - perhaps  a valid
question?)

Best,


On Tue, 17 Mar 2020 at 15:38, Adam Witwicki  wrote:

> Hi Suresh
>
> /etc/cloudstack/agent/agent.properties
> host=cs.cloudstack
>
> telnet
> user@ois-mh-p1-c1-h2-b:~$ telnet cs.cloudstack 8250 Trying
> 10.10.240.10...
> Connected to cs.cloudstack.
> Escape character is '^]'.
>
>
> Same for all my other KVM nodes. 7 of 8 connected
>
> Thanks
>
> Adam
>
> -Original Message-
> From: Suresh Kumar Anaparti 
> Sent: 17 March 2020 13:11
> To: users@cloudstack.apache.org
> Subject: Re: Agent staying in alert mode
>
> ** This mail originated from OUTSIDE the Oakford corporate network.
> Treat hyperlinks and attachments in this email with caution. **
>
> Hi Adam,
>
> What is the value of the host parameter in
> "/etc/cloudstack/agent/agent.properties" file of the KVM host with
> agent connectivity issue. Can you telnet to that management server on
> port 8250 from the KVM host.
>
> Are other KVM agents connected to same management server?
>
> Thanks,
> Suresh
>
> On Tue, Mar 17, 2020 at 3:05 PM Adam Witwicki
> 
> wrote:
>
> > Hi Suresh,
> >
> > Thanks for looking
> >
> > All KVM hosts can connect to management server - via ping
> >
> > The mgmt_server_id is Null for the alert host
> >
> > Host
> >
> > name;uuid;status;type;private_ip_address;private_netmask;private_mac
> > _a
> > ddress;storage_ip_address;storage_netmask;storage_mac_address;storag
> > e_
> > ip_address_2;storage_mac_address_2;storage_netmask_2;cluster_id;publ
> > ic
> > _ip_address;public_netmask;public_mac_address;proxy_port;data_center
> > _i
> > d;pod_id;cpu_sockets;cpus;speed;url;fs_type

Re: Agent staying in alert mode

[Simon Weller]

Have you tried restarting libvirtd and then restarting the cloudstack agent?


From: Adam Witwicki 
Sent: Wednesday, March 18, 2020 3:06 AM
To: users@cloudstack.apache.org 
Subject: RE: Agent staying in alert mode

Hi Andrija,

Setting the host to management IP, and not our LB VIP. The agent process seems 
to stall

2020-03-18 07:44:01,969 DEBUG [kvm.resource.LibvirtComputingResource] 
(Agent-Handler-1:null) (logid:) cpus=32, speed=3800, ram=269385596928, 
_dom0MinMem=1073741824, _dom0OvercommitMem=0, cpu sockets=2
2020-03-18 07:44:01,972 DEBUG [cloud.resource.ServerResourceBase] 
(Agent-Handler-1:null) (logid:) Parameters for private nic: 10.10.240.126 - 
a0:36:9f:24:32:6a-255.255.255.0
2020-03-18 07:44:01,973 DEBUG [cloud.resource.ServerResourceBase] 
(Agent-Handler-1:null) (logid:) Parameters for storage nic: 10.10.240.126 - 
a0:36:9f:24:32:6a-255.255.255.0
2020-03-18 07:44:01,973 DEBUG [cloud.resource.ServerResourceBase] 
(Agent-Handler-1:null) (logid:) Parameters for public nic: 10.10.240.126 - 
a0:36:9f:24:32:6a-255.255.255.0
2020-03-18 07:44:01,973 DEBUG [kvm.resource.LibvirtComputingResource] 
(Agent-Handler-1:null) (logid:) Executing: 
/usr/share/cloudstack-common/scripts/vm/hypervisor/versions.sh
2020-03-18 07:44:01,984 DEBUG [kvm.resource.LibvirtComputingResource] 
(Agent-Handler-1:null) (logid:) Execution is successful.
2020-03-18 07:44:01,985 DEBUG [kvm.resource.LibvirtComputingResource] 
(Agent-Handler-1:null) (logid:) Executing: sudo grep InitiatorName= 
/etc/iscsi/initiatorname.iscsi
2020-03-18 07:44:01,992 DEBUG [kvm.resource.LibvirtComputingResource] 
(Agent-Handler-1:null) (logid:) Execution is successful.
2020-03-18 07:44:01,993 INFO  [kvm.storage.LibvirtStorageAdaptor] 
(Agent-Handler-1:null) (logid:) Attempting to create storage pool 
257c9140-3863-4be2-85e2-9835650f1b45 (Filesystem) in libvirt
2020-03-18 07:44:01,993 DEBUG [kvm.resource.LibvirtConnection] 
(Agent-Handler-1:null) (logid:) Looking for libvirtd connection at: 
qemu:///system
HALTS HERE AND DOES NOT CONTINUE

I waited 10 mins for a timeout then stopped the agent

Thanks

Adam


-Original Message-
From: Andrija Panic 
Sent: 17 March 2020 18:52
To: users 
Subject: Re: Agent staying in alert mode

** This mail originated from OUTSIDE the Oakford corporate network. Treat 
hyperlinks and attachments in this email with caution. **

Can you change that to the IP of the mgmt server and then restart the agent? it 
should always point to the IP not hostname (what is that value there, "who put 
it there" etc - perhaps  a valid
question?)

Best,


On Tue, 17 Mar 2020 at 15:38, Adam Witwicki  wrote:

> Hi Suresh
>
> /etc/cloudstack/agent/agent.properties
> host=cs.cloudstack
>
> telnet
> user@ois-mh-p1-c1-h2-b:~$ telnet cs.cloudstack 8250 Trying
> 10.10.240.10...
> Connected to cs.cloudstack.
> Escape character is '^]'.
>
>
> Same for all my other KVM nodes. 7 of 8 connected
>
> Thanks
>
> Adam
>
> -Original Message-
> From: Suresh Kumar Anaparti 
> Sent: 17 March 2020 13:11
> To: users@cloudstack.apache.org
> Subject: Re: Agent staying in alert mode
>
> ** This mail originated from OUTSIDE the Oakford corporate network.
> Treat hyperlinks and attachments in this email with caution. **
>
> Hi Adam,
>
> What is the value of the host parameter in
> "/etc/cloudstack/agent/agent.properties" file of the KVM host with
> agent connectivity issue. Can you telnet to that management server on
> port 8250 from the KVM host.
>
> Are other KVM agents connected to same management server?
>
> Thanks,
> Suresh
>
> On Tue, Mar 17, 2020 at 3:05 PM Adam Witwicki
> 
> wrote:
>
> > Hi Suresh,
> >
> > Thanks for looking
> >
> > All KVM hosts can connect to management server - via ping
> >
> > The mgmt_server_id is Null for the alert host
> >
> > Host
> >
> > name;uuid;status;type;private_ip_address;private_netmask;private_mac
> > _a
> > ddress;storage_ip_address;storage_netmask;storage_mac_address;storag
> > e_
> > ip_address_2;storage_mac_address_2;storage_netmask_2;cluster_id;publ
> > ic
> > _ip_address;public_netmask;public_mac_address;proxy_port;data_center
> > _i
> > d;pod_id;cpu_sockets;cpus;speed;url;fs_type;hypervisor_type;hypervis
> > or
> > _version;ram;resource;version;parent;total_size;capabilities;guid;av
> > ai
> > lable;setup;dom0_memory;last_ping;mgmt_server_id;disconnected;create
> > d;
> > removed;update_count;resource_state;owner;lastUpdated;engine_state
> > ois-mh-p1-c1-h2-b;3d5e4c35-7e10-4560-8a2e-5ebf770e837d;Alert;Routing
> > ;1
> > 0.10.240.126;255.255.255.0;a0:36:9f:24:32:6a;10.10.240.126;255.255.2
> > 55
> > .0;a0:36:9f:24:32:6a;\N;\N;\N;5;10.10.240.126;255.255.255.0;a0:36:9f
> > :2
> > 4:32:6a;\N;5;5;2;32;3800;iqn.1993-08.org.debian:01:7c8e527ccba4;\N;K
> > VM
> > ;\N;269385596928;\N;4.11.0.0;\N;\N;hvm,snapshot;44190139-5b1b-3642-b
> > 78
> > 6-41ca577081e9-LibvirtComputingResource;1;0;0;1546450279;\N;2019-12-
> > 24
> > 09:25:53;2019-06-11 21:59:43;\N;2516;Enabled;\N;\N;Disabled

Re: VXLAN Connectivity

[Simon Weller]

I assume you're using the native linux VXLAN feature. If so, it uses multicast, 
so make sure you have a routable ip address on the interface being used for 
VXLAN, or it won't pass any traffic. Also make sure iptables is allowing 
multicast traffic to pass.

-Soi

From: Mr Jazze 
Sent: Tuesday, March 17, 2020 5:31 PM
To: CloudStack Mailing-List 
Subject: VXLAN Connectivity

Hello Again,

I've reconfigured my test environment to use VXLAN instead of OVS which
went no where. I've of course deployed Advance Mode and put all the pieces
in place which yielded a somewhat functional cloud. I was able to deploy
Windows Server 2016 virtual machine. Initially, this VM didn't acquire it's
DHCP address from VPC router. I noticed VM was running on 2nd host and
router was running on 1st host, so I migrated VM to the same host as
router; then it was able to acquire DHCP address and ping 1.1.1.1. Then,
while trying to troubleshoot why there was no connectivity across hosts the
router took a dump and I had to destroy it to get another router deployed,
now VM is unable to get IP address regardless of which host.

Does anyone have any experience with similar issue with VXLAN connectivity
and/or advice on how to resolve?

--

==

My Search to Build a Private Cloud!

Re: Add private network to a VM

[Simon Weller]

VPC networks provide a bunch of additional features, including network tiers 
(different subnets), private gateways for third party interconnection and IPSEC 
lan-to-lan tunnels. Typically you wouldn't use a VPC for a separate private 
network, it would the main network.


From: Fariborz Navidan 
Sent: Tuesday, October 15, 2019 1:39 AM
To: users@cloudstack.apache.org 
Subject: Re: Add private network to a VM

What difference between isolated guest network and VPC network?

On Wed, Oct 16, 2019 at 1:10 AM Vivek Kumar  wrote:

> Hii,
>
> So in advance zone you can create either isolated network or VPC style
> isolated network.!! Just go the network tab add an isolated network with
> your private subnet or you can select VPC in the drop down option.
>
> ⁣Sent from BlueMail ​
>
> On 16 Oct 2019, 03:06, at 03:06, Fariborz Navidan 
> wrote:
> >Hello,
> >
> >How do I add a private network to a VM in an advanced zone?
> >
> >Thanks
>

Re: Add private network to a VM

[Simon Weller]

Fariborz,

Create a new isolated network under networks and then add an additional network 
interface into the VM that's connected to that new isolated network.

-Si



From: Fariborz Navidan 
Sent: Tuesday, October 15, 2019 1:14 AM
To: users@cloudstack.apache.org 
Subject: Add private network to a VM

Hello,

How do I add a private network to a VM in an advanced zone?

Thanks

Re: [VOTE] Primate as modern UI for CloudStack

[Simon Weller]

+1 (binding)

From: Rohit Yadav 
Sent: Monday, October 7, 2019 6:31 AM
To: d...@cloudstack.apache.org ; 
users@cloudstack.apache.org ; 
priv...@cloudstack.apache.org 
Subject: [VOTE] Primate as modern UI for CloudStack

All,

The feedback and response has been positive on the proposal to use Primate as 
the modern UI for CloudStack [1] [2]. Thank you all.

I'm starting this vote (to):

  *   Accept Primate codebase [3] as a project under Apache CloudStack project
  *   Create and host a new repository (cloudstack-primate) and follow Github 
based development workflow (issues, pull requests etc) as we do with CloudStack
  *   Given this is a new project, to encourage cadence until its feature 
completeness the merge criteria is proposed as:
 *   Manual testing against each PR and/or with screenshots from the author 
or testing contributor, integration with Travis is possible once we get JS/UI 
tests
 *   At least 1 LGTM from any of the active contributors, we'll move this 
to 2 LGTMs when the codebase reaches feature parity wrt the existing/old 
CloudStack UI
 *   Squash and merge PRs
  *   Accept the proposed timeline [1][2] (subject to achievement of goals wrt 
Primate technical release and GA)
 *   the first technical preview targetted with the winter 2019 LTS release 
(~Q1 2020) and release to serve a deprecation notice wrt the older UI
 *   define a release approach before winter LTS
 *   stop taking feature FRs for old/existing UI after winter 2019 LTS 
release, work on upgrade path/documentation from old UI to Primate
 *   the first Primate GA targetted wrt summer LTS 2020 (~H2 2019), but 
still ship old UI with a final deprecation notice
 *   old UI codebase removed from codebase in winter 2020 LTS release

The vote will be up for the next two weeks to give enough time for PMC and the 
community to gather consensus and still have room for questions, feedback and 
discussions. The results to be shared on/after 21th October 2019.

For sanity in tallying the vote, can PMC members please be sure to indicate 
"(binding)" with their vote?

[ ] +1  approve
[ ] +0  no opinion
[ ] -1  disapprove (and reason why)

[1] Primate Proposal:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Proposal%3A+CloudStack+Primate+UI

[2] Email thread reference:
https://markmail.org/message/z6fuvw4regig7aqb

[3] Primate repo current location: https://github.com/shapeblue/primate


Regards,

Rohit Yadav

Software Architect, ShapeBlue

https://www.shapeblue.com

rohit.ya...@shapeblue.com
www.shapeblue.com
Amadeus House, Floral Street, London  WC2E 9DPUK
@shapeblue

Re: [DISCUSS] Primate - new UI for CloudStack?

[Simon Weller]

I like the idea of separating the UI from the main code base. I think that will 
provide a lot more flexibility moving forward and the project is well overdue 
for a new look and feel.
I think the time frame proposed to sunset the old UI is doable and we'll need 
some feedback from those using it today (we have our own UI, so this doesn't 
affect us).

One of our challenges over the last few years is the added work of getting UI 
features into the release and it has added around 30% additional work load due 
to the older style code of the current UI. Having it in VUE is great and I 
think it will also encourage others to contribute.

+1.


From: Anurag A 
Sent: Friday, September 20, 2019 11:26 AM
To: d...@cloudstack.apache.org 
Cc: users@cloudstack.apache.org 
Subject: Re: [DISCUSS] Primate - new UI for CloudStack?

+1 to the new UI as it supports:
1. Faster development of features
2. Better experience as a user
3. Easy customisation declaratively

Regards,
Anurag

> On 20-Sep-2019, at 8:17 AM, Siddhartha Kattoju  wrote:
>
> +1 from me as well.
>
> Just a side note: I feel like there is a high risk of tldr here. May be its
> just me. It may be would be good to put most of the details in a wiki page
> and just post a summarized version on the list ?
>
> *Sid Kattoju*
>
> Cloud Software Architect | Professional Services
>
> c 514.466.0951
>
>
> * *
>
>
>
>
> On Fri, Sep 20, 2019 at 8:10 AM Rohit Yadav 
> wrote:
>
>> All,
>>
>>
>>
>> == Summary ==
>>
>>
>> I have been working on a new, modern role-based UI for Cloudstack (project
>> Primate: https://github.com/shapeblue/primate) I demoed this for the
>> first time at CCCNA19 last week and it was very well received. It was
>> discussed, at length, as an item in the hackathon and the general consensus
>> there was that this could become Cloudstacks new UI. We discussed a plan to
>> achieve that and now I’m bringing that plan to the list for discussion.
>>
>>
>>
>> == Background ==
>>
>>
>> The current CloudStack UI has grown technical debt over time and it has
>> become harder to extend, develop, maintain in the long run, it is also
>> difficult for new contributors to learn and get started. Since late 2018, I
>> started working on a side-project with the aim to create a modern
>> progressive and role-based declaratively-programmed UI for CloudStack,
>> called Primate. Before creating Primate, I set out to create a list of core
>> requirements of what would give us an extensible, modern UI that was easy
>> to develop now and in the future. These are the requirements I came up with:
>>
>>  *   designed from ground up to be  a complete replacement for our
>> combined user/admin UI
>>  *   to respect all entities in cloudstack and not make assumptions on
>> use-cases of cloudstack
>>  *   data-driven and auto-generation of UI widgets and to be easy to
>> learn, develop, extend, customise and maintain.
>>  *   declarative programming
>>  *   support for API discovery and parameter completion like CloudMonkey
>>  *   support for custom roles
>>
>>
>>
>> I looked at existing Cloudstack UI projects but none of them fully
>> satisfied all these requirements and started Primate.
>>
>>
>>
>> == Project Primate ==
>>
>>
>> For the implementation, I compared a couple of opensource JS and UI
>> frameworks and decided to use VueJS (https://vuejs.org)
>> which is a JavaScript framework and AntD (https://ant.design<
>> https://ant.design/>) which is a UI design language with a well-defined
>> spec, styling guide, and an implementation-specific to VueJS. VueJS was
>> selected because among a few other JS frameworks I surveyed it was the
>> easiest (for me) to learn and get started. I also surveyed a few UI
>> frameworks and selected AntD because it came with a well-defined spec,
>> styling guide and VueJS specific implementation which gives several
>> re-usable components out of the box.
>>
>>
>>
>> During the development of Primate, I used my previous experience from
>> CloudMonkey and another PoC angular-based UI ProjectX, and it currently
>> supports:
>>
>>  *   role-based UI based on API autodiscovery
>>  *   auto-generated action/API forms with parameter completion
>>  *   declarative component-driven views
>>  *   modern programming methodologies (hot reloading, npm based
>> build/run/compile etc.)
>>  *   decoupled from core Cloudstack code
>>  *   dynamic translation (most/many of old translation files ported)
>>  *   includes dashboards, async job/API polling, all list views/tables
>> per the old UI
>>  *   browser history and url/route driven navigation
>>  *   support for mobiles/tables/desktop screens
>>  *   configuration driven UI customisation (of navigation, icons, APIs
>> etc)
>>
>>
>>
>> To get to this point, I’ve had some valuable help from Anurag and Sven et
>> al at EWerk.
>> The development strategy to support all APIs out of the box in a
>> data-driven way gives a functioning UI 

Re: [DISCUSS] Change of the official CHAT channel

[Simon Weller]

Sven,

I just sent you an invite.

-Si


From: Sven Vogel 
Sent: Monday, August 26, 2019 10:19 AM
To: users 
Cc: d...@cloudstack.apache.org 
Subject: Re: [DISCUSS] Change of the official CHAT channel

that means,



this is the official URL?

https://apachecloudstack.slack.com/



how can i added to this channel?


On Sunday, 08/25/2019 at 22:19 Andrija Panic wrote:


Right to the bone Nicolas... i.e. exactly my thinking.

On Sun, 25 Aug 2019 at 19:48, Nicolas Vazquez
wrote:

> Hi all,
>
> My 2 cents: I fully agree on promoting the mailing lists as the main
> communication channel. However, I think it will be good to drop IRC
and
> promote the current Slack channel as it is active and with a
considerable
> amount of users (+150 members vs ~30 in the IRC channels).
Personally, I
> have joined the Slack channel a few years ago and have seen new
users
> joining the channel and asking questions. My point is, if we already
have
> this channel working good without "official" promotion, then why not
use it
> as the official 'chat' channel?
>
>
> Regards,
>
> Nicolas Vazquez
>
> 
> From: Gabriel Beims Bräscher
> Sent: Friday, August 23, 2019 3:21 PM
> To: users
> Cc: dev
> Subject: Re: [DISCUSS] Change of the official CHAT channel
>
> I am +1 on opening the VOTE for removal of IRC references.
>
> Em sex, 23 de ago de 2019 às 14:33, Andrija Panic  >
> escreveu:
>
> > I'm inline with that Paul - let's then remove the IRC being
mentioned on
> > main website and any future events. Makes sense?
> >
> > We can keep Slack as it is now, and then definitely (in my
opinion) NOT
> > move to ASF Slack, since as someone mentioned it's a pain to move
all
> > users.
> >
> > Does this sounds good to majority here?
> >
> > If so, I would cast a vote thread for removal of IRC from any
public
> > pages/website/future events marketing.
> >
> >
> > On Fri, Aug 23, 2019, 17:45 Paul Angus  wrote:
> >
> > > I don't think that we should be 'marketing' anything other than
the
> > > mailing lists.That is where most people can be found and
where
> > > previously asked questions can be searched for.
> > > Having a ready to roll 'chat' tool has many advantages, so I'm
cool
> with
> > > having one in the back pocket.
> > > But personally I'd like to see as little fragmentation of the
community
> > as
> > > possible.
> > >
> > > + a new visitor turning up to a slack channel with 4 people on
it is
> not
> > > going to give a positive impression, and that person is
definitely is
> far
> > > less likely to get an answer to any question that they have.
> > >
> > > Paul.
> > >
> > > paul.an...@shapeblue.com
> > > www.shapeblue.com
> > > Amadeus House, Floral Street, London  WC2E 9DPUK
> > > @shapeblue
> > >
> > >
> > >
> > >
> > > -Original Message-
> > > From: Gabriel Beims Bräscher
> > > Sent: 23 August 2019 14:56
> > > To: dev
> > > Cc: Nux!
> > > Subject: Re: [DISCUSS] Change of the official CHAT channel
> > >
> > > Hello folks,
> > >
> > > I see no problem with using chat platforms. The mail should stay
as the
> > > default communication tool indeed, but several times I used
Slack when
> > > helping foes around, raising questions, and pinging folks on
private
> > chat.
> > > As far as we have tools and they are being used by the community
I see
> no
> > > problem with keeping them around.
> > >
> > > However, I think that the main point raised by Andrija is
regarding the
> > > "marketing" of the channels. We normally promote IRC (pointing
to
> > > irc.freenode.net), but CloudStack IRC channels look pretty dead,
> > > especially considering that it is the "official" ACS chat tool.
> > >
> > > Does promoting all channels (IRC + Slack) look a good idea?
Should we
> > keep
> > > only one option? In one hand, IRC currently is not as active as
Slack,
> > but
> > > on the other hand, Slack requires an invitation e-mail.
> > >
> > > Regards,
> > > Gabriel.
> > >
> > >
> > >
> > > Em sex, 23 de ago de 2019 às 08:45, Rohit Yadav <
> > rohit.ya...@shapeblue.com
> > > >
> > > escreveu:
> > >
> > > > All,
> > > >
> > > > I think email is the more persistent form of communication and
it
> must
> > > > continue to be the default communication mechanism for the
project
> > > > dev+user+etc.
> > > >
> > > > For a more real-time/transient communication, IRC or slack are
good
> > > tools.
> > > > I don't have any preference on either, as I'm not an active
user on
> > > > either of them. It just happened that someone asked me to join
> the-asf
> > > > slack for some other group, and I saw many projects having
their
> > > > channels there and I made a comment about it on our current
> > > > apachecloudstack community slack which is outside of the-asf
group.
> > > > I've got both of them setup on my desktop slack now, if it's
too
> > > > difficult to migrate everyone from the old slack group to
the-asf
> > > > slack group, let's continue as is; or perhaps add an

Re: Changing cpu.overprovisioning

[Simon Weller]

Hi Adam,

You shouldn't have to restart any instances. At worst, after the change, 
restart your management server(s) if it doesn't take effect.


From: Adam Witwicki 
Sent: Wednesday, August 21, 2019 7:52 AM
To: users@cloudstack.apache.org 
Subject: Changing cpu.overprovisioning

Hi Guys


How can I change cpu.overprovisioning without having to restart all the 
instances?

It would be extremely inconvenient to do so, I am at 85% of allocated capacity, 
and by hosts are not breaking a sweat





Ver 4.11.0.0





Thanks



Adam







Disclaimer Notice:
This email has been sent by Oakford Technology Limited, while we have checked 
this e-mail and any attachments for viruses, we can not guarantee that they are 
virus-free. You must therefore take full responsibility for virus checking.
This message and any attachments are confidential and should only be read by 
those to whom they are addressed. If you are not the intended recipient, please 
contact us, delete the message from your computer and destroy any copies. Any 
distribution or copying without our prior permission is prohibited.
Internet communications are not always secure and therefore Oakford Technology 
Limited does not accept legal responsibility for this message. The recipient is 
responsible for verifying its authenticity before acting on the contents. Any 
views or opinions presented are solely those of the author and do not 
necessarily represent those of Oakford Technology Limited.
Registered address: Oakford Technology Limited, The Manor House, Potterne, 
Wiltshire. SN10 5PN.
Registered in England and Wales No. 5971519

Re: [ANNOUNCE] Apache CloudStack - Newsweek, Best Business Tools 2019 in 'Cloud Services'

[Simon Weller]

Fantastic!


From: Paul Angus 
Sent: Thursday, August 1, 2019 10:54 AM
To: users@cloudstack.apache.org ; 
d...@cloudstack.apache.org 
Subject: [ANNOUNCE] Apache CloudStack - Newsweek, Best Business Tools 2019 in 
'Cloud Services'


Hi Everyone,

I'm delighted to say that 'we' have been recognised by Newsweek in their 
inaugural list of Best Business Tools in the 'Cloud Services' category.
https://www.newsweek.com/best-business-tools-2019/cloud-services

A quick screen grab of the certificate can be found here:
https://imgur.com/a/7epEqeX

The recognition was for:
- Willingness to recommend the brand to colleagues
- Trustworthiness
- Reliability
- Fulfilment of service promise
- Continuous improvement
- Security

Well done everyone, this is a great community achievement and shows that not 
only is CloudStack great, but the world is starting to take notice of it too!


We'll upload the a good copy of the certificate to the ACS website and a bit of 
marketing stuff around it soon.


Paul Angus
CloudStack PMC
#CloudStackWorks


paul.an...@shapeblue.com
www.shapeblue.com
Amadeus House, Floral Street, London  WC2E 9DPUK
@shapeblue

Re: [ANNOUNCE] Andrija Panic has joined the PMC

[Simon Weller]

Congrats Andrija!!


From: Paul Angus 
Sent: Saturday, July 13, 2019 10:02 AM
To: users@cloudstack.apache.org; d...@cloudstack.apache.org; 
priv...@cloudstack.apache.org
Subject: [ANNOUNCE] Andrija Panic has joined the PMC

Fellow CloudStackers,



It gives me great pleasure to say that Adrija has been invited to join the
PMC and has gracefully accepted.


Please joining me in congratulating Andrija!




Kind regards,



Paul Angus

CloudStack PMC

Re: [ANNOUNCE] Sven Vogel has joined the PMC

[Simon Weller]

Congrats Sven!


From: Boris Stoyanov 
Sent: Tuesday, July 16, 2019 2:08 AM
To: users@cloudstack.apache.org; priv...@cloudstack.apache.org; 
d...@cloudstack.apache.org
Subject: Re: [ANNOUNCE] Sven Vogel has joined the PMC

Congrats Sven!

On 13.07.19, 18:45, "Paul Angus"  wrote:

Fellow CloudStackers,



It gives me great pleasure to say that Sven has been invited to join the
PMC and has gracefully accepted.


Please joining me in congratulating Sven!




Kind regards,



Paul Angus

CloudStack PMC



boris.stoya...@shapeblue.com
www.shapeblue.com
Amadeus House, Floral Street, London  WC2E 9DPUK
@shapeblue

Re: [ANNOUNCE] Gabriel Beims Bräscher has joined the PMC

[Simon Weller]

Congrats Gabriel!


From: Paul Angus 
Sent: Saturday, July 13, 2019 11:00 AM
To: users@cloudstack.apache.org; d...@cloudstack.apache.org; 
priv...@cloudstack.apache.org
Subject: [ANNOUNCE] Gabriel Beims Bräscher has joined the PMC

Fellow CloudStackers,


Its non-stop today!



It gives me great pleasure to say that Gabriel has been invited to join the
PMC and has gracefully accepted.


Please joining me in congratulating Sven!




Kind regards,



Paul Angus

CloudStack PMC

Re: [ANNOUNCE] Bobby (Boris Stoyanov) has joined the PMC

[Simon Weller]

Congrats Bobby!!



From: Paul Angus 
Sent: Tuesday, July 16, 2019 4:12 AM
To: priv...@cloudstack.apache.org; d...@cloudstack.apache.org; 
users@cloudstack.apache.org
Subject: [ANNOUNCE] Bobby (Boris Stoyanov) has joined the PMC

Fellow CloudStackers,



It gives me great pleasure to say that Bobby has been invited to join the
PMC and has gracefully accepted.



Please join me in congratulating  Bobby!





Kind regards,





Paul Angus

CloudStack PMC

Re: VM HA enabled when on ceph

[Simon Weller]

Andrija's right on the money. Sorry, I should have mentioned that.

From: Andrija Panic 
Sent: Thursday, June 27, 2019 3:25 AM
To: swel...@ena.com.invalid
Cc: users@cloudstack.apache.org
Subject: Re: VM HA enabled when on ceph

Jess,

Simon shared hack for existing VMs.
But you might want to make your Compute Offering HA enabled, then all VMs
inherit that (new VMs obviously).

Cheers

On Thu, 27 Jun 2019 at 00:10, Simon Weller  wrote:

> Jess,
>
> Yes, that works.
>
> -Si
>
> 
> From: jesse.wat...@gmail.com 
> Sent: Wednesday, June 26, 2019 3:55 PM
> To: users@cloudstack.apache.org
> Subject: VM HA enabled when on ceph
>
> Hi all,
>
> Have a ACS 4.12.0.0 running with 2 kvm hosts and ceph for primary storage.
> How do I enable HA for my VM(s)? The manual hints "HA features work with
> iSCSI or NFS primary storage", leaves out ceph.
>
> Can I just go in db and enable set vm to "HA" yes?
> mysql> update vm_instance set ha_enabled = 1 where id = ;
>
>
> TIA,
>   Jesse
>


--

Andrija Panić

Re: VM HA enabled when on ceph

[Simon Weller]

Jess,

Yes, that works.

-Si


From: jesse.wat...@gmail.com 
Sent: Wednesday, June 26, 2019 3:55 PM
To: users@cloudstack.apache.org
Subject: VM HA enabled when on ceph

Hi all,

Have a ACS 4.12.0.0 running with 2 kvm hosts and ceph for primary storage.
How do I enable HA for my VM(s)? The manual hints "HA features work with
iSCSI or NFS primary storage", leaves out ceph.

Can I just go in db and enable set vm to "HA" yes?
mysql> update vm_instance set ha_enabled = 1 where id = ;


TIA,
  Jesse

Re: CloudStack and SDN

[Simon Weller]

Juniper contributed a plugin for Contrail quite a number of years ago. With 
that project now under the umbrella of the Linux Foundation (along with Open 
Daylight), it would be nice to see some of us start to explore the open options 
and see if we could move some better integrations forward. With Nokia pulling 
the Nuage plugin, there are no full SDN options available any longer and I 
think that's unfortunate for the project at large.

-Si


From: Sergey Levitskiy 
Sent: Wednesday, April 17, 2019 3:40 PM
To: d...@cloudstack.apache.org
Cc: users
Subject: Re: CloudStack and SDN

Nicira NVP/NSX is very outdated and doesn’t support latest NSX.

On 4/17/19, 1:22 AM, "Andrija Panic"  wrote:

Simplest one that actually works (call it SDN if you like) - VXLAN.

A few others with various implementation states and all hypervisor
dependent:
Nicira NVP / Vmware NSX
Midonet
Various tunneling protocols


On Wed, 17 Apr 2019 at 09:46, Haijiao <18602198...@163.com> wrote:

>
>
> Noted Nuage's decision on ceasing SDN plugin for CloudStack. Just
> wondering is there any other SDN solutoin can be integrated with 
CloudStack
> ?
>
>
> Thanks !
>
>
> Regards,
>
>

--

Andrija Panić

Re: [DISCUSS] Remove support for el6 packaging in 4.13/4.14

[Simon Weller]

+1.



From: Nux! 
Sent: Monday, April 15, 2019 8:20 AM
To: dev
Cc: users
Subject: Re: [DISCUSS] Remove support for el6 packaging in 4.13/4.14

+1, EL6 is in its last phase of support.

--
Sent from the Delta quadrant using Borg technology!

Nux!
www.nux.ro

- Original Message -
> From: "Rohit Yadav" 
> To: "dev" , "users" 
> Sent: Monday, 15 April, 2019 08:44:58
> Subject: [DISCUSS] Remove support for el6 packaging in 4.13/4.14

> All,
>
>
> With CentOS8 around the corner to be released sometime around the summer, I
> would like to propose to deprecate CentOS6 as support management server host
> distro and KVM host distro. Non-systemd enabled Ubuntu releases have been
> already deprecated [1].
>
>
> The older CentOS6 version would hold us back as we try to adapt, use and 
> support
> newer JRE version, kvm/libvirt version, the Linux kernel, and several other
> older dependencies. Both CentOS6 and RHEL6 have reached EOL on May 10th, 2017
> wrt full updates [1].
>
>
> If we don't have any disagreements, I propose we remove el6 packaging support 
> in
> the next major release - 4.13. But, if there are users and organisations that
> will be badly impacted, let 4.13 be the last of releases to support el6 and we
> definitely remove el6 support in 4.14.
>
> What are your thoughts?
>
>
> [1] EOL date wiki reference:
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Hypervisor+and+Management+Server+OS+EOL+Dates
>
>
>
> Regards,
>
> Rohit Yadav
>
> Software Architect, ShapeBlue
>
> https://www.shapeblue.com
>
> rohit.ya...@shapeblue.com
> www.shapeblue.com
> Amadeus House, Floral Street, London  WC2E 9DPUK
> @shapeblue

Latest Qemu KVM EV appears to be broken with ACS

[Simon Weller]

All,

After troubleshooting a strange issue with a new lab environment yesterday, it 
appears that the patchviasocket functionality we rely on for key and ip 
injection into our router/SSVM/CPVM images is broken with 
qemu-kvm-ev-2.12.0-18.el7 (January 2019 release). This was tested on Centos 7.6.
No data is injected and this was confirmed using socat on /dev/vport0p1. 
qemu-kvm-ev-2.10.0-21.el7_5.7.1 works, so hopefully this will save someone some 
pain and suffering trying to figure out why the deployed seems broken.

We're going to dig in and see if can figure out the patches responsible for it 
breaking.

-Si

Re: Windows on KVM - Windows profile with PV drivers vs Other PV?

[Simon Weller]

Lucian,

Windows PV also enables the KVM HyperV Enlightenment features for Windows 
guests on 4.12.

https://www.lfasiallc.com/wp-content/uploads/2017/11/Use-Hyper-V-Enlightenments-to-Increase-KVM-VM-Performance_Density_Chao-Peng.pdf

You can also change the disk controller when creating a template -

[cid:1f21ed36-1c9c-401a-bbf2-34bc7b4ce61a]

-Si


From: Nux! 
Sent: Tuesday, April 9, 2019 8:53 AM
To: dev
Cc: users
Subject: Re: Windows on KVM - Windows profile with PV drivers vs Other PV?

Ok, turns out there's a "Windows PV" profile that sorts out my problem. Didn't 
see that initially.

--
Sent from the Delta quadrant using Borg technology!

Nux!
www.nux.ro

- Original Message -
> From: "Nux!" 
> To: "dev" 
> Cc: "users" 
> Sent: Tuesday, 9 April, 2019 14:26:40
> Subject: Windows on KVM - Windows profile with PV drivers vs Other PV?

> Hi,
>
> I'm trying to add some Windows templates to my test cloud and by using the
> "Windows Server 2016" I end up with Intel 1000 NICs instead of virtio ones;
> this is a bit suboptimal.
>
> When I add the template initially I can see I can specify a type of storage -
> virtio-scsi is what I need, however I can't do the same for the network
> devices.
>
> What does the community recommend here? Any way to force virtio devices on the
> Windows Server profile or should I go for "Other PV" profiles for Windows
> guests?
>
> --
> Sent from the Delta quadrant using Borg technology!
>
> Nux!
> www.nux.ro

Re: NullPointerException when deleting pod

[Simon Weller]

Hi Kevin,

Can you fill this as a bug please - https://github.com/apache/cloudstack/issues

If you could provide the steps to reproduce this, that would be really helpful 
as well.

-Si


From: Kevin Heseler 
Sent: Wednesday, April 3, 2019 8:59 AM
To: users@cloudstack.apache.org
Subject: NullPointerException when deleting pod

Hi guys,

I have an issue when deleting one of my pods. The UI is loading forever
and after a reload it is still there, when clicking delete this
NullPointerException appears in the management.log.

Currently running 4.12 on Bionic.

2019-04-03 15:25:49,829 DEBUG [c.c.a.ApiServlet]
(qtp858242339-18:ctx-37b10f3b) (logid:a21a8737) ===START===  XX
-- GET
command=deletePod=9ffb9311-e401-41ca-b0ed-9293d7$0eac7=json&_=1554297933677
2019-04-03 15:25:49,910 ERROR [c.c.a.ApiServer]
(qtp858242339-18:ctx-37b10f3b ctx-6dd8759a) (logid:a21a8737) unhandled
exception executing api command: [Ljava.lang.String;@1bd0618a
java.lang.NullPointerException
 at
com.cloud.dc.dao.VlanDaoImpl.listVlansForPodByType(VlanDaoImpl.java:182)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
 at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
 at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
 at
com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
 at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
 at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
 at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
 at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
 at com.sun.proxy.$Proxy98.listVlansForPodByType(Unknown Source)
 at
com.cloud.network.NetworkModelImpl.listPodVlans(NetworkModelImpl.java:804)
 at
com.cloud.configuration.ConfigurationManagerImpl$1.doInTransactionWithoutResult(ConfigurationManagerImpl.java:1106)
 at
com.cloud.utils.db.TransactionCallbackNoReturn.doInTransaction(TransactionCallbackNoReturn.java:25)
 at
com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:50)
 at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
 at com.cloud.utils.db.Transaction.execute(Transaction.java:47)
 at
com.cloud.configuration.ConfigurationManagerImpl.deletePod(ConfigurationManagerImpl.java:1086)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
 at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
 at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
 at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
 at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
 at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
 at com.sun.proxy.$Proxy124.deletePod(Unknown Source)
 at
org.apache.cloudstack.api.command.admin.pod.DeletePodCmd.execute(DeletePodCmd.java:70)
 at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:156)
 at com.cloud.api.ApiServer.queueCommand(ApiServer.java:758)
 at com.cloud.api.ApiServer.handleRequest(ApiServer.java:582)
 at
com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:310)
 at com.cloud.api.ApiServlet$1.run(ApiServlet.java:130)
 at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
 at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
 at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
 at