Re: Caching modes

[Andrija Panic]

pls merge also https://github.com/apache/cloudstack-docs-admin/pull/48

just correct code block syntax (to display code properly)

On 20 February 2018 at 21:02, Rafael Weingärtner <
rafaelweingart...@gmail.com> wrote:

> Thanks, we will proceed reviweing
>
> On Tue, Feb 20, 2018 at 3:12 PM, Andrija Panic 
> wrote:
>
> > Here it is:
> >
> > https://github.com/apache/cloudstack-docs-admin/pull/47
> >
> > Added KVM online storage migration (atm only CEPH/NFS to SolidFire, new
> in
> > 4.11 release)
> > Added KVM cache mode setup and limitations.
> >
> >
> > Cheers
> >
> > On 20 February 2018 at 16:49, Rafael Weingärtner <
> > rafaelweingart...@gmail.com> wrote:
> >
> > > If you are willing to write it down, please do so, and open a PR. We
> will
> > > review and merged it afterwards.
> > >
> > > On Tue, Feb 20, 2018 at 12:41 PM, Andrija Panic <
> andrija.pa...@gmail.com
> > >
> > > wrote:
> > >
> > > > I advise (or not...depends on point of view) to stay that way -
> because
> > > > when you activate write-back cache - live migrations will stop, and
> > this
> > > > makes *Enable maintenance mode (put host into maintenance)*
> impossible.
> > > >
> > > > I would perhaps suggest that there is documentation for "advanced
> > users"
> > > or
> > > > similar, that will say "it's possible to enable this and this way via
> > DB
> > > > hack, but be warned on live migration consequences etc..." since this
> > > will
> > > > render more problems if people start using it.
> > > >
> > > > If you choose to do, let me know, I can write that (documentation)
> > > briefly.
> > > >
> > > > Not to mention it can be unsafe (power failure - less possible I
> guess,
> > > but
> > > > rare kernel panic etc might have it's consequences I assume)
> > > >
> > > > It does indeed increase performance on NFS much, but not necessarily
> on
> > > > CEPH (if you are using lirbd cache on client side as explained above)
> > > >
> > > > On 20 February 2018 at 15:48, Rafael Weingärtner <
> > > > rafaelweingart...@gmail.com> wrote:
> > > >
> > > > > Yes. Weird enough, the code is using the value in the database if
> it
> > is
> > > > > provided there, but there is no easy way for users to change that
> > > > > configuration in the database. ¯\_(ツ)_/¯
> > > > >
> > > > > On Tue, Feb 20, 2018 at 11:45 AM, Andrija Panic <
> > > andrija.pa...@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > So it seems that just passing the cachemode value to API is not
> > > there,
> > > > or
> > > > > > somehow messedup, but deployVM process does read DB values from
> > > > > > disk_offering table for sure, and applies it to XML file for KVM.
> > > > > > This is above ACS 4.8.x.
> > > > > >
> > > > > >
> > > > > > On 20 February 2018 at 15:44, Andrija Panic <
> > andrija.pa...@gmail.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > I have edited the disk_offering table, in the cache_mode just
> > enter
> > > > > > > "writeback". Stop and start VM, and it will pickup/inherit the
> > > > > cache_mode
> > > > > > > from it's parrent offering
> > > > > > > This also applies to Compute/Service offering, again inside
> > > > > disk_offering
> > > > > > > table - just tested both
> > > > > > >
> > > > > > > i.e.
> > > > > > >
> > > > > > > UPDATE `cloud`.`disk_offering` SET `cache_mode`='writeback'
> WHERE
> > > > > > > `id`=102; # Compute Offering (Service offering)
> > > > > > > UPDATE `cloud`.`disk_offering` SET `cache_mode`='writeback'
> WHERE
> > > > > > > `id`=114; #data disk offering
> > > > > > >
> > > > > > > Before SQL:
> > > > > > >
> > > > > > > root@ix1-c7-4:~# virsh dumpxml i-2-10-VM | grep cache -A2
> > > > > > >   
> > > > > > >   
> > > > > > >   
> > > > > > > --
> > > > > > >   
> > > > > > >   
> > > > > > >   
> > > > > > > --
> > > > > > >
> > > > > > > STOP and START VM = after SQL
> > > > > > >
> > > > > > > root@ix1-c7-4:~# virsh dumpxml i-2-10-VM | grep cache -A2
> > > > > > >   
> > > > > > >   
> > > > > > >   
> > > > > > > --
> > > > > > >   
> > > > > > >   
> > > > > > >   
> > > > > > > --
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On 20 February 2018 at 14:03, Rafael Weingärtner <
> > > > > > > rafaelweingart...@gmail.com> wrote:
> > > > > > >
> > > > > > >> I have no idea how it can change the performance. If you look
> at
> > > the
> > > > > > >> content of the commit you provided, it is only the commit that
> > > > enabled
> > > > > > the
> > > > > > >> use of getCacheMode from disk offerings. However, it is not
> > > exposing
> > > > > any
> > > > > > >> way to users to change that value/configuration in the
> > database. I
> > > > > might
> > > > > > >> have missed it; do you see any API methods that receive the
> > > > parameter
> > > > > > >> "cacheMode" and then pass this parameter to a "diskOffering"
> > > object,
> > > > > and
> > > > > > >> then persist/update this object in the database?
> > > > > > >>
> > > > > > >> May 

Re: HA issues

[Andrija Panic]

That is good to hear ( no NFS issues causing Agent Disconnect).

I assume you are using "normal" NFS solution with proper HA and no ZFS
(kernel panic etc), but anyway be aware of this one

https://github.com/apache/cloudstack/blob/e532b574ddb186a117da638fb6059356fe7c266c/scripts/vm/hypervisor/kvm/kvmheartbeat.sh#L161



we used to comment this line, because we did have some issues with
communication link, and this commented line saved our a$$ few times :)

CHeers

On 20 February 2018 at 20:50, Sean Lair  wrote:

> Hi Andrija
>
> We are currently running XenServer in production.  We are working on
> moving to KVM and have it deployed in a development environment.
>
> The team is putting CloudStack + KVM through its paces and that is when it
> was discovered how broken VM HA is in 4.9.3.  Initially our patches fixed
> VM HA, but just caused VMs to get started on two hosts during failure
> testing.  The libvirt lockd has solved that issue thus far.
>
> Short answer to you question is :-), we were not having problems with
> Agent Disconnects in a production environment.  It was our testing/QA that
> revealed the issues.  Our NFS has been stable so far, no issues with the
> agent crashing/stopping that wasn't initiated by the team's testing.
>
> Thanks
> Sean
>
>
> -Original Message-
> From: Andrija Panic [mailto:andrija.pa...@gmail.com]
> Sent: Saturday, February 17, 2018 1:49 PM
> To: dev 
> Subject: Re: HA issues
>
> Hi Sean,
>
> (we have 2 threads interleaving on the libvirt lockd..) - so, did you
> manage to understand what can cause the Agent Disconnect in most cases, for
> you specifically? Is there any software (CloudStack) root cause
> (disregarding i.e. networking issues etc)
>
> Just our examples, which you should probably not have:
>
> We had CEPH cluster running (with ACS), and there any exception in librbd
> would crash JVM and the agent, but this has been fixed mostly - Now get
> i.e. agent disconnect when ACS try to delete volume on CEPH (and for some
> reason not succeed withing 30 minutes, volume deletion fails) - then
> libvirt get's completety stuck (virsh list even dont work)...so  agent
> get's disconnect eventually.
>
> It would be good to get rid of agent disconnections in general, obviously
> :) so that is why I'm asking (you are on NFS, so would like to see your
> experience here).
>
> Thanks
>
> On 16 February 2018 at 21:52, Sean Lair  wrote:
>
> > We were in the same situation as Nux.
> >
> > In our test environment we hit the issue with VMs not getting fenced and
> > coming up on two hosts because of VM HA.   However, we updated some of
> the
> > logic for VM HA and turned on libvirtd's locking mechanism.  Now we
> > are working great w/o IPMI.  The locking stops the VMs from starting
> > elsewhere, and everything recovers very nicely when the host starts
> responding again.
> >
> > We are on 4.9.3 and haven't started testing with 4.11 yet, but it may
> > work along-side IPMI just fine - it would just have affect the fencing.
> > However, we *currently* prefer how we are doing it now, because if the
> > agent stops responding, but the host is still up, the VMs continue
> > running and no actual downtime is incurred.  Even when VM HA attempts
> > to power on the VMs on another host, it just fails the power-up and
> > the VMs continue to run on the "agent disconnected" host. The host
> > goes into alarm state and our NOC can look into what is wrong the
> > agent on the host.  If IPMI was enabled, it sounds like it would power
> > off the host (fence) and force downtime for us even if the VMs were
> > actually running OK - and just the agent is unreachable.
> >
> > I plan on submitting our updates via a pull request at some point.
> > But I can also send the updated code to anyone that wants to do some
> > testing before then.
> >
> > -Original Message-
> > From: Marcus [mailto:shadow...@gmail.com]
> > Sent: Friday, February 16, 2018 11:27 AM
> > To: dev@cloudstack.apache.org
> > Subject: Re: HA issues
> >
> > From your other emails it sounds as though you do not have IPMI
> > configured, nor host HA enabled, correct? In this case, the correct
> > thing to do is nothing. If CloudStack cannot guarantee the VM state
> > (as is the case with an unreachable hypervisor), it should do nothing,
> > for fear of causing a split brain and corrupting the VM disk (VM running
> on two hosts).
> >
> > Clustering and fencing is a tricky proposition. When CloudStack (or
> > any other cluster manager) is not configured to or cannot guarantee
> > state then things will simply lock up, in this case your HA VM on your
> > broken hypervisor will not run elsewhere. This has been the case for a
> > long time with CloudStack, HA would only start a VM after the original
> > hypervisor agent came back and reported no VM is running.
> >
> > The new feature, from what I gather, simply adds the possibility of
> > CloudStack 

Re: Caching modes

[Rafael Weingärtner]

Thanks, we will proceed reviweing

On Tue, Feb 20, 2018 at 3:12 PM, Andrija Panic 
wrote:

> Here it is:
>
> https://github.com/apache/cloudstack-docs-admin/pull/47
>
> Added KVM online storage migration (atm only CEPH/NFS to SolidFire, new in
> 4.11 release)
> Added KVM cache mode setup and limitations.
>
>
> Cheers
>
> On 20 February 2018 at 16:49, Rafael Weingärtner <
> rafaelweingart...@gmail.com> wrote:
>
> > If you are willing to write it down, please do so, and open a PR. We will
> > review and merged it afterwards.
> >
> > On Tue, Feb 20, 2018 at 12:41 PM, Andrija Panic  >
> > wrote:
> >
> > > I advise (or not...depends on point of view) to stay that way - because
> > > when you activate write-back cache - live migrations will stop, and
> this
> > > makes *Enable maintenance mode (put host into maintenance)* impossible.
> > >
> > > I would perhaps suggest that there is documentation for "advanced
> users"
> > or
> > > similar, that will say "it's possible to enable this and this way via
> DB
> > > hack, but be warned on live migration consequences etc..." since this
> > will
> > > render more problems if people start using it.
> > >
> > > If you choose to do, let me know, I can write that (documentation)
> > briefly.
> > >
> > > Not to mention it can be unsafe (power failure - less possible I guess,
> > but
> > > rare kernel panic etc might have it's consequences I assume)
> > >
> > > It does indeed increase performance on NFS much, but not necessarily on
> > > CEPH (if you are using lirbd cache on client side as explained above)
> > >
> > > On 20 February 2018 at 15:48, Rafael Weingärtner <
> > > rafaelweingart...@gmail.com> wrote:
> > >
> > > > Yes. Weird enough, the code is using the value in the database if it
> is
> > > > provided there, but there is no easy way for users to change that
> > > > configuration in the database. ¯\_(ツ)_/¯
> > > >
> > > > On Tue, Feb 20, 2018 at 11:45 AM, Andrija Panic <
> > andrija.pa...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > So it seems that just passing the cachemode value to API is not
> > there,
> > > or
> > > > > somehow messedup, but deployVM process does read DB values from
> > > > > disk_offering table for sure, and applies it to XML file for KVM.
> > > > > This is above ACS 4.8.x.
> > > > >
> > > > >
> > > > > On 20 February 2018 at 15:44, Andrija Panic <
> andrija.pa...@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > I have edited the disk_offering table, in the cache_mode just
> enter
> > > > > > "writeback". Stop and start VM, and it will pickup/inherit the
> > > > cache_mode
> > > > > > from it's parrent offering
> > > > > > This also applies to Compute/Service offering, again inside
> > > > disk_offering
> > > > > > table - just tested both
> > > > > >
> > > > > > i.e.
> > > > > >
> > > > > > UPDATE `cloud`.`disk_offering` SET `cache_mode`='writeback' WHERE
> > > > > > `id`=102; # Compute Offering (Service offering)
> > > > > > UPDATE `cloud`.`disk_offering` SET `cache_mode`='writeback' WHERE
> > > > > > `id`=114; #data disk offering
> > > > > >
> > > > > > Before SQL:
> > > > > >
> > > > > > root@ix1-c7-4:~# virsh dumpxml i-2-10-VM | grep cache -A2
> > > > > >   
> > > > > >   
> > > > > >   
> > > > > > --
> > > > > >   
> > > > > >   
> > > > > >   
> > > > > > --
> > > > > >
> > > > > > STOP and START VM = after SQL
> > > > > >
> > > > > > root@ix1-c7-4:~# virsh dumpxml i-2-10-VM | grep cache -A2
> > > > > >   
> > > > > >   
> > > > > >   
> > > > > > --
> > > > > >   
> > > > > >   
> > > > > >   
> > > > > > --
> > > > > >
> > > > > >
> > > > > >
> > > > > > On 20 February 2018 at 14:03, Rafael Weingärtner <
> > > > > > rafaelweingart...@gmail.com> wrote:
> > > > > >
> > > > > >> I have no idea how it can change the performance. If you look at
> > the
> > > > > >> content of the commit you provided, it is only the commit that
> > > enabled
> > > > > the
> > > > > >> use of getCacheMode from disk offerings. However, it is not
> > exposing
> > > > any
> > > > > >> way to users to change that value/configuration in the
> database. I
> > > > might
> > > > > >> have missed it; do you see any API methods that receive the
> > > parameter
> > > > > >> "cacheMode" and then pass this parameter to a "diskOffering"
> > object,
> > > > and
> > > > > >> then persist/update this object in the database?
> > > > > >>
> > > > > >> May I ask how are you guys changing the cacheMode configuration?
> > > > > >>
> > > > > >> On Tue, Feb 20, 2018 at 9:56 AM, Paul Angus <
> > > paul.an...@shapeblue.com
> > > > >
> > > > > >> wrote:
> > > > > >>
> > > > > >> > I'm working with some guys who are experimenting with the
> > setting
> > > as
> > > > > if
> > > > > >> > definitely seems to change the performance of data disks.  It
> > also
> > > > > >> changes
> > > > > >> > the XML of the VM which is created.
> > > > > >> >
> > > > > >> > p.s.
> > > > > >> 

Re: VM HA starting VMs that were powered off within Guest

[Rafael Weingärtner]

Yes. That is the expected behavior

On Tue, Feb 20, 2018 at 4:59 PM, Sean Lair  wrote:

> We have some Windows VMs we have VM HA enabled for.  When a user does a
> shutdown of the VM from within Windows, VM HA reports the following and
> powers the VM back up.  Is this expected behavior?
>
> Log Snip-it:
>
> 2018-02-20 19:51:58,898 INFO  [c.c.v.VirtualMachineManagerImpl]
> (AgentManager-Handler-3:null) (logid:) VM i-26-122-VM is at Running and we
> received a power-off report while there is no pending jobs on it
> 2018-02-20 19:51:58,898 INFO  [c.c.v.VirtualMachineManagerImpl]
> (AgentManager-Handler-3:null) (logid:) Detected out-of-band stop of a HA
> enabled VM i-26-122-VM, will schedule restart
> 2018-02-20 19:51:58,919 INFO  [c.c.h.HighAvailabilityManagerImpl]
> (AgentManager-Handler-3:null) (logid:) Schedule vm for HA:
> VM[User|i-26-122-VM]
>
> Thanks
> Sean
>



-- 
Rafael Weingärtner

VM HA starting VMs that were powered off within Guest

[Sean Lair]

We have some Windows VMs we have VM HA enabled for.  When a user does a 
shutdown of the VM from within Windows, VM HA reports the following and powers 
the VM back up.  Is this expected behavior?

Log Snip-it:

2018-02-20 19:51:58,898 INFO  [c.c.v.VirtualMachineManagerImpl] 
(AgentManager-Handler-3:null) (logid:) VM i-26-122-VM is at Running and we 
received a power-off report while there is no pending jobs on it
2018-02-20 19:51:58,898 INFO  [c.c.v.VirtualMachineManagerImpl] 
(AgentManager-Handler-3:null) (logid:) Detected out-of-band stop of a HA 
enabled VM i-26-122-VM, will schedule restart
2018-02-20 19:51:58,919 INFO  [c.c.h.HighAvailabilityManagerImpl] 
(AgentManager-Handler-3:null) (logid:) Schedule vm for HA:  VM[User|i-26-122-VM]

Thanks
Sean

RE: HA issues

[Sean Lair]

Hi Andrija

We are currently running XenServer in production.  We are working on moving to 
KVM and have it deployed in a development environment.

The team is putting CloudStack + KVM through its paces and that is when it was 
discovered how broken VM HA is in 4.9.3.  Initially our patches fixed VM HA, 
but just caused VMs to get started on two hosts during failure testing.  The 
libvirt lockd has solved that issue thus far.

Short answer to you question is :-), we were not having problems with Agent 
Disconnects in a production environment.  It was our testing/QA that revealed 
the issues.  Our NFS has been stable so far, no issues with the agent 
crashing/stopping that wasn't initiated by the team's testing.

Thanks
Sean


-Original Message-
From: Andrija Panic [mailto:andrija.pa...@gmail.com] 
Sent: Saturday, February 17, 2018 1:49 PM
To: dev 
Subject: Re: HA issues

Hi Sean,

(we have 2 threads interleaving on the libvirt lockd..) - so, did you manage to 
understand what can cause the Agent Disconnect in most cases, for you 
specifically? Is there any software (CloudStack) root cause (disregarding i.e. 
networking issues etc)

Just our examples, which you should probably not have:

We had CEPH cluster running (with ACS), and there any exception in librbd would 
crash JVM and the agent, but this has been fixed mostly - Now get i.e. agent 
disconnect when ACS try to delete volume on CEPH (and for some reason not 
succeed withing 30 minutes, volume deletion fails) - then libvirt get's 
completety stuck (virsh list even dont work)...so  agent get's disconnect 
eventually.

It would be good to get rid of agent disconnections in general, obviously
:) so that is why I'm asking (you are on NFS, so would like to see your 
experience here).

Thanks

On 16 February 2018 at 21:52, Sean Lair  wrote:

> We were in the same situation as Nux.
>
> In our test environment we hit the issue with VMs not getting fenced and
> coming up on two hosts because of VM HA.   However, we updated some of the
> logic for VM HA and turned on libvirtd's locking mechanism.  Now we 
> are working great w/o IPMI.  The locking stops the VMs from starting 
> elsewhere, and everything recovers very nicely when the host starts 
> responding again.
>
> We are on 4.9.3 and haven't started testing with 4.11 yet, but it may 
> work along-side IPMI just fine - it would just have affect the fencing.
> However, we *currently* prefer how we are doing it now, because if the 
> agent stops responding, but the host is still up, the VMs continue 
> running and no actual downtime is incurred.  Even when VM HA attempts 
> to power on the VMs on another host, it just fails the power-up and 
> the VMs continue to run on the "agent disconnected" host. The host 
> goes into alarm state and our NOC can look into what is wrong the 
> agent on the host.  If IPMI was enabled, it sounds like it would power 
> off the host (fence) and force downtime for us even if the VMs were 
> actually running OK - and just the agent is unreachable.
>
> I plan on submitting our updates via a pull request at some point.  
> But I can also send the updated code to anyone that wants to do some 
> testing before then.
>
> -Original Message-
> From: Marcus [mailto:shadow...@gmail.com]
> Sent: Friday, February 16, 2018 11:27 AM
> To: dev@cloudstack.apache.org
> Subject: Re: HA issues
>
> From your other emails it sounds as though you do not have IPMI 
> configured, nor host HA enabled, correct? In this case, the correct 
> thing to do is nothing. If CloudStack cannot guarantee the VM state 
> (as is the case with an unreachable hypervisor), it should do nothing, 
> for fear of causing a split brain and corrupting the VM disk (VM running on 
> two hosts).
>
> Clustering and fencing is a tricky proposition. When CloudStack (or 
> any other cluster manager) is not configured to or cannot guarantee 
> state then things will simply lock up, in this case your HA VM on your 
> broken hypervisor will not run elsewhere. This has been the case for a 
> long time with CloudStack, HA would only start a VM after the original 
> hypervisor agent came back and reported no VM is running.
>
> The new feature, from what I gather, simply adds the possibility of 
> CloudStack being able to reach out and shut down the hypervisor to 
> guarantee state. At that point it can start the VM elsewhere. If 
> something fails in that process (IPMI unreachable, for example, or bad 
> credentials), you're still going to be stuck with a VM not coming back.
>
> It's the nature of the thing. I'd be wary of any HA solution that does 
> not reach out and guarantee state via host or storage fencing before 
> starting a VM elsewhere, as it will be making assumptions. Its 
> entirely possible a VM might be unreachable or unable to access it 
> storage for a short while, a new instance of the VM is started elsewhere, and 
> the original VM comes back.
>
> On 

Re: [4.11] Management to VR connection issues

[Rohit Yadav]

Hi Rene,


Thanks for sharing - I've not seen this in test/production environment yet. 
Does it help to destroy the VR and check if the issue persists? Also, is this 
behaviour system-wide for every VR, or VRs of specific networks or topologies 
such as VPCs? Are these VRs redundant in nature?


4.11+ VRs are systemd enabled and don't reboot after patching which is a major 
difference between 4.9 and 4.11 systemvms/VRs; to make this work for VMware 
when the nics come up we use a hack (that has been followed since at least 
4.6+) to ping the interfaces/gateways:

https://github.com/apache/cloudstack/blob/4.11/systemvm/debian/opt/cloud/bin/setup/common.sh#L335

After nic/mac-addresses change/configure, 4.9 and previous VRs used to reboot 
(i.e. 4.9 and previous VRs on vmware used to reboot twice, once after patching 
and once more to reconfigure nic-mac assignments). 4.11+ VRs don't do reboots 
at all but uses udevadm for nic/mac/interface configurations:

https://github.com/apache/cloudstack/blob/4.11/systemvm/debian/opt/cloud/bin/setup/router.sh#L62

So you may try two tests and see if it makes any difference wrt above mentioned 
code -- (a) one to increase timeout/ping retries and (b) another to reboot 
after udev/mac-address configurations (which would only require re-building the 
systemvm.iso file and scp-ing on the secondary storage in your test 
environment).

Finally, if you can share logs or other details about the test setup and 
environment, I can help you with some investigations.


- Rohit






From: Rene Moser 
Sent: Tuesday, February 20, 2018 1:46:02 PM
To: us...@cloudstack.apache.org; dev@cloudstack.apache.org
Subject: [4.11] Management to VR connection issues

Hi

We upgraded from 4.9 to 4.11. VMware 6.5.0. (Testing environment).

VR upgrade went through. But we noticed that the communication between
the management server and the VR are not working properly.

We do not yet fully understand the issue, one thing we noted is that the
networks configs seems not be bound to the same interfaces after every
reboot. As a result, after a reboot you may can connect to the VR by
SSH, after another reboot you can't anymore.

The Network name eth0 switched from the NIC id 3 to 4 after reboot.

The VR is kept in "starting" state, of course as a consequence we get
many issues related to this, no VM deployments (kept in starting state),
VM expunging failure (cleanup fails), a.s.o.

Have anyone experienced similar issues?

Regards
René

rohit.ya...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 

Re: Caching modes

[Andrija Panic]

Here it is:

https://github.com/apache/cloudstack-docs-admin/pull/47

Added KVM online storage migration (atm only CEPH/NFS to SolidFire, new in
4.11 release)
Added KVM cache mode setup and limitations.


Cheers

On 20 February 2018 at 16:49, Rafael Weingärtner <
rafaelweingart...@gmail.com> wrote:

> If you are willing to write it down, please do so, and open a PR. We will
> review and merged it afterwards.
>
> On Tue, Feb 20, 2018 at 12:41 PM, Andrija Panic 
> wrote:
>
> > I advise (or not...depends on point of view) to stay that way - because
> > when you activate write-back cache - live migrations will stop, and this
> > makes *Enable maintenance mode (put host into maintenance)* impossible.
> >
> > I would perhaps suggest that there is documentation for "advanced users"
> or
> > similar, that will say "it's possible to enable this and this way via DB
> > hack, but be warned on live migration consequences etc..." since this
> will
> > render more problems if people start using it.
> >
> > If you choose to do, let me know, I can write that (documentation)
> briefly.
> >
> > Not to mention it can be unsafe (power failure - less possible I guess,
> but
> > rare kernel panic etc might have it's consequences I assume)
> >
> > It does indeed increase performance on NFS much, but not necessarily on
> > CEPH (if you are using lirbd cache on client side as explained above)
> >
> > On 20 February 2018 at 15:48, Rafael Weingärtner <
> > rafaelweingart...@gmail.com> wrote:
> >
> > > Yes. Weird enough, the code is using the value in the database if it is
> > > provided there, but there is no easy way for users to change that
> > > configuration in the database. ¯\_(ツ)_/¯
> > >
> > > On Tue, Feb 20, 2018 at 11:45 AM, Andrija Panic <
> andrija.pa...@gmail.com
> > >
> > > wrote:
> > >
> > > > So it seems that just passing the cachemode value to API is not
> there,
> > or
> > > > somehow messedup, but deployVM process does read DB values from
> > > > disk_offering table for sure, and applies it to XML file for KVM.
> > > > This is above ACS 4.8.x.
> > > >
> > > >
> > > > On 20 February 2018 at 15:44, Andrija Panic  >
> > > > wrote:
> > > >
> > > > > I have edited the disk_offering table, in the cache_mode just enter
> > > > > "writeback". Stop and start VM, and it will pickup/inherit the
> > > cache_mode
> > > > > from it's parrent offering
> > > > > This also applies to Compute/Service offering, again inside
> > > disk_offering
> > > > > table - just tested both
> > > > >
> > > > > i.e.
> > > > >
> > > > > UPDATE `cloud`.`disk_offering` SET `cache_mode`='writeback' WHERE
> > > > > `id`=102; # Compute Offering (Service offering)
> > > > > UPDATE `cloud`.`disk_offering` SET `cache_mode`='writeback' WHERE
> > > > > `id`=114; #data disk offering
> > > > >
> > > > > Before SQL:
> > > > >
> > > > > root@ix1-c7-4:~# virsh dumpxml i-2-10-VM | grep cache -A2
> > > > >   
> > > > >   
> > > > >   
> > > > > --
> > > > >   
> > > > >   
> > > > >   
> > > > > --
> > > > >
> > > > > STOP and START VM = after SQL
> > > > >
> > > > > root@ix1-c7-4:~# virsh dumpxml i-2-10-VM | grep cache -A2
> > > > >   
> > > > >   
> > > > >   
> > > > > --
> > > > >   
> > > > >   
> > > > >   
> > > > > --
> > > > >
> > > > >
> > > > >
> > > > > On 20 February 2018 at 14:03, Rafael Weingärtner <
> > > > > rafaelweingart...@gmail.com> wrote:
> > > > >
> > > > >> I have no idea how it can change the performance. If you look at
> the
> > > > >> content of the commit you provided, it is only the commit that
> > enabled
> > > > the
> > > > >> use of getCacheMode from disk offerings. However, it is not
> exposing
> > > any
> > > > >> way to users to change that value/configuration in the database. I
> > > might
> > > > >> have missed it; do you see any API methods that receive the
> > parameter
> > > > >> "cacheMode" and then pass this parameter to a "diskOffering"
> object,
> > > and
> > > > >> then persist/update this object in the database?
> > > > >>
> > > > >> May I ask how are you guys changing the cacheMode configuration?
> > > > >>
> > > > >> On Tue, Feb 20, 2018 at 9:56 AM, Paul Angus <
> > paul.an...@shapeblue.com
> > > >
> > > > >> wrote:
> > > > >>
> > > > >> > I'm working with some guys who are experimenting with the
> setting
> > as
> > > > if
> > > > >> > definitely seems to change the performance of data disks.  It
> also
> > > > >> changes
> > > > >> > the XML of the VM which is created.
> > > > >> >
> > > > >> > p.s.
> > > > >> > I've found this commit;
> > > > >> >
> > > > >> > https://github.com/apache/cloudstack/commit/1edaa36cc68e845a
> > > > >> 42339d5f267d49
> > > > >> > c82343aefb
> > > > >> >
> > > > >> > so I've got something to investigate now, but API documentation
> > must
> > > > >> > definitely be askew.
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > paul.an...@shapeblue.com
> > > > >> > www.shapeblue.com
> > > > >> 

Re: Caching modes

[Rafael Weingärtner]

If you are willing to write it down, please do so, and open a PR. We will
review and merged it afterwards.

On Tue, Feb 20, 2018 at 12:41 PM, Andrija Panic 
wrote:

> I advise (or not...depends on point of view) to stay that way - because
> when you activate write-back cache - live migrations will stop, and this
> makes *Enable maintenance mode (put host into maintenance)* impossible.
>
> I would perhaps suggest that there is documentation for "advanced users" or
> similar, that will say "it's possible to enable this and this way via DB
> hack, but be warned on live migration consequences etc..." since this will
> render more problems if people start using it.
>
> If you choose to do, let me know, I can write that (documentation) briefly.
>
> Not to mention it can be unsafe (power failure - less possible I guess, but
> rare kernel panic etc might have it's consequences I assume)
>
> It does indeed increase performance on NFS much, but not necessarily on
> CEPH (if you are using lirbd cache on client side as explained above)
>
> On 20 February 2018 at 15:48, Rafael Weingärtner <
> rafaelweingart...@gmail.com> wrote:
>
> > Yes. Weird enough, the code is using the value in the database if it is
> > provided there, but there is no easy way for users to change that
> > configuration in the database. ¯\_(ツ)_/¯
> >
> > On Tue, Feb 20, 2018 at 11:45 AM, Andrija Panic  >
> > wrote:
> >
> > > So it seems that just passing the cachemode value to API is not there,
> or
> > > somehow messedup, but deployVM process does read DB values from
> > > disk_offering table for sure, and applies it to XML file for KVM.
> > > This is above ACS 4.8.x.
> > >
> > >
> > > On 20 February 2018 at 15:44, Andrija Panic 
> > > wrote:
> > >
> > > > I have edited the disk_offering table, in the cache_mode just enter
> > > > "writeback". Stop and start VM, and it will pickup/inherit the
> > cache_mode
> > > > from it's parrent offering
> > > > This also applies to Compute/Service offering, again inside
> > disk_offering
> > > > table - just tested both
> > > >
> > > > i.e.
> > > >
> > > > UPDATE `cloud`.`disk_offering` SET `cache_mode`='writeback' WHERE
> > > > `id`=102; # Compute Offering (Service offering)
> > > > UPDATE `cloud`.`disk_offering` SET `cache_mode`='writeback' WHERE
> > > > `id`=114; #data disk offering
> > > >
> > > > Before SQL:
> > > >
> > > > root@ix1-c7-4:~# virsh dumpxml i-2-10-VM | grep cache -A2
> > > >   
> > > >   
> > > >   
> > > > --
> > > >   
> > > >   
> > > >   
> > > > --
> > > >
> > > > STOP and START VM = after SQL
> > > >
> > > > root@ix1-c7-4:~# virsh dumpxml i-2-10-VM | grep cache -A2
> > > >   
> > > >   
> > > >   
> > > > --
> > > >   
> > > >   
> > > >   
> > > > --
> > > >
> > > >
> > > >
> > > > On 20 February 2018 at 14:03, Rafael Weingärtner <
> > > > rafaelweingart...@gmail.com> wrote:
> > > >
> > > >> I have no idea how it can change the performance. If you look at the
> > > >> content of the commit you provided, it is only the commit that
> enabled
> > > the
> > > >> use of getCacheMode from disk offerings. However, it is not exposing
> > any
> > > >> way to users to change that value/configuration in the database. I
> > might
> > > >> have missed it; do you see any API methods that receive the
> parameter
> > > >> "cacheMode" and then pass this parameter to a "diskOffering" object,
> > and
> > > >> then persist/update this object in the database?
> > > >>
> > > >> May I ask how are you guys changing the cacheMode configuration?
> > > >>
> > > >> On Tue, Feb 20, 2018 at 9:56 AM, Paul Angus <
> paul.an...@shapeblue.com
> > >
> > > >> wrote:
> > > >>
> > > >> > I'm working with some guys who are experimenting with the setting
> as
> > > if
> > > >> > definitely seems to change the performance of data disks.  It also
> > > >> changes
> > > >> > the XML of the VM which is created.
> > > >> >
> > > >> > p.s.
> > > >> > I've found this commit;
> > > >> >
> > > >> > https://github.com/apache/cloudstack/commit/1edaa36cc68e845a
> > > >> 42339d5f267d49
> > > >> > c82343aefb
> > > >> >
> > > >> > so I've got something to investigate now, but API documentation
> must
> > > >> > definitely be askew.
> > > >> >
> > > >> >
> > > >> >
> > > >> > paul.an...@shapeblue.com
> > > >> > www.shapeblue.com
> > > >> > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > >> > @shapeblue
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> > -Original Message-
> > > >> > From: Rafael Weingärtner [mailto:rafaelweingart...@gmail.com]
> > > >> > Sent: 20 February 2018 12:31
> > > >> > To: dev 
> > > >> > Subject: Re: Caching modes
> > > >> >
> > > >> > This cache mode parameter does not exist in
> "CreateDiskOfferingCmd"
> > > >> > command. I also checked some commits from 2, 3, 4 and 5 years ago,
> > and
> > > >> > this parameter was never there. If you check 

Re: Caching modes

[Andrija Panic]

I advise (or not...depends on point of view) to stay that way - because
when you activate write-back cache - live migrations will stop, and this
makes *Enable maintenance mode (put host into maintenance)* impossible.

I would perhaps suggest that there is documentation for "advanced users" or
similar, that will say "it's possible to enable this and this way via DB
hack, but be warned on live migration consequences etc..." since this will
render more problems if people start using it.

If you choose to do, let me know, I can write that (documentation) briefly.

Not to mention it can be unsafe (power failure - less possible I guess, but
rare kernel panic etc might have it's consequences I assume)

It does indeed increase performance on NFS much, but not necessarily on
CEPH (if you are using lirbd cache on client side as explained above)

On 20 February 2018 at 15:48, Rafael Weingärtner <
rafaelweingart...@gmail.com> wrote:

> Yes. Weird enough, the code is using the value in the database if it is
> provided there, but there is no easy way for users to change that
> configuration in the database. ¯\_(ツ)_/¯
>
> On Tue, Feb 20, 2018 at 11:45 AM, Andrija Panic 
> wrote:
>
> > So it seems that just passing the cachemode value to API is not there, or
> > somehow messedup, but deployVM process does read DB values from
> > disk_offering table for sure, and applies it to XML file for KVM.
> > This is above ACS 4.8.x.
> >
> >
> > On 20 February 2018 at 15:44, Andrija Panic 
> > wrote:
> >
> > > I have edited the disk_offering table, in the cache_mode just enter
> > > "writeback". Stop and start VM, and it will pickup/inherit the
> cache_mode
> > > from it's parrent offering
> > > This also applies to Compute/Service offering, again inside
> disk_offering
> > > table - just tested both
> > >
> > > i.e.
> > >
> > > UPDATE `cloud`.`disk_offering` SET `cache_mode`='writeback' WHERE
> > > `id`=102; # Compute Offering (Service offering)
> > > UPDATE `cloud`.`disk_offering` SET `cache_mode`='writeback' WHERE
> > > `id`=114; #data disk offering
> > >
> > > Before SQL:
> > >
> > > root@ix1-c7-4:~# virsh dumpxml i-2-10-VM | grep cache -A2
> > >   
> > >   
> > >   
> > > --
> > >   
> > >   
> > >   
> > > --
> > >
> > > STOP and START VM = after SQL
> > >
> > > root@ix1-c7-4:~# virsh dumpxml i-2-10-VM | grep cache -A2
> > >   
> > >   
> > >   
> > > --
> > >   
> > >   
> > >   
> > > --
> > >
> > >
> > >
> > > On 20 February 2018 at 14:03, Rafael Weingärtner <
> > > rafaelweingart...@gmail.com> wrote:
> > >
> > >> I have no idea how it can change the performance. If you look at the
> > >> content of the commit you provided, it is only the commit that enabled
> > the
> > >> use of getCacheMode from disk offerings. However, it is not exposing
> any
> > >> way to users to change that value/configuration in the database. I
> might
> > >> have missed it; do you see any API methods that receive the parameter
> > >> "cacheMode" and then pass this parameter to a "diskOffering" object,
> and
> > >> then persist/update this object in the database?
> > >>
> > >> May I ask how are you guys changing the cacheMode configuration?
> > >>
> > >> On Tue, Feb 20, 2018 at 9:56 AM, Paul Angus  >
> > >> wrote:
> > >>
> > >> > I'm working with some guys who are experimenting with the setting as
> > if
> > >> > definitely seems to change the performance of data disks.  It also
> > >> changes
> > >> > the XML of the VM which is created.
> > >> >
> > >> > p.s.
> > >> > I've found this commit;
> > >> >
> > >> > https://github.com/apache/cloudstack/commit/1edaa36cc68e845a
> > >> 42339d5f267d49
> > >> > c82343aefb
> > >> >
> > >> > so I've got something to investigate now, but API documentation must
> > >> > definitely be askew.
> > >> >
> > >> >
> > >> >
> > >> > paul.an...@shapeblue.com
> > >> > www.shapeblue.com
> > >> > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > >> > @shapeblue
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > -Original Message-
> > >> > From: Rafael Weingärtner [mailto:rafaelweingart...@gmail.com]
> > >> > Sent: 20 February 2018 12:31
> > >> > To: dev 
> > >> > Subject: Re: Caching modes
> > >> >
> > >> > This cache mode parameter does not exist in "CreateDiskOfferingCmd"
> > >> > command. I also checked some commits from 2, 3, 4 and 5 years ago,
> and
> > >> > this parameter was never there. If you check the API in [1], you can
> > see
> > >> > that it is not an expected parameter. Moreover, I do not see any use
> > of
> > >> > "setCacheMode" in the code (in case it is updated by some other
> > method).
> > >> > Interestingly enough, the code uses "getCacheMode".
> > >> >
> > >> > In summary, it is not a feature, and it does not work. It looks like
> > >> some
> > >> > leftover from dark ages when people could commit anything and then
> > they
> > >> > would just 

Re: Caching modes

[Rafael Weingärtner]

Yes. Weird enough, the code is using the value in the database if it is
provided there, but there is no easy way for users to change that
configuration in the database. ¯\_(ツ)_/¯

On Tue, Feb 20, 2018 at 11:45 AM, Andrija Panic 
wrote:

> So it seems that just passing the cachemode value to API is not there, or
> somehow messedup, but deployVM process does read DB values from
> disk_offering table for sure, and applies it to XML file for KVM.
> This is above ACS 4.8.x.
>
>
> On 20 February 2018 at 15:44, Andrija Panic 
> wrote:
>
> > I have edited the disk_offering table, in the cache_mode just enter
> > "writeback". Stop and start VM, and it will pickup/inherit the cache_mode
> > from it's parrent offering
> > This also applies to Compute/Service offering, again inside disk_offering
> > table - just tested both
> >
> > i.e.
> >
> > UPDATE `cloud`.`disk_offering` SET `cache_mode`='writeback' WHERE
> > `id`=102; # Compute Offering (Service offering)
> > UPDATE `cloud`.`disk_offering` SET `cache_mode`='writeback' WHERE
> > `id`=114; #data disk offering
> >
> > Before SQL:
> >
> > root@ix1-c7-4:~# virsh dumpxml i-2-10-VM | grep cache -A2
> >   
> >   
> >   
> > --
> >   
> >   
> >   
> > --
> >
> > STOP and START VM = after SQL
> >
> > root@ix1-c7-4:~# virsh dumpxml i-2-10-VM | grep cache -A2
> >   
> >   
> >   
> > --
> >   
> >   
> >   
> > --
> >
> >
> >
> > On 20 February 2018 at 14:03, Rafael Weingärtner <
> > rafaelweingart...@gmail.com> wrote:
> >
> >> I have no idea how it can change the performance. If you look at the
> >> content of the commit you provided, it is only the commit that enabled
> the
> >> use of getCacheMode from disk offerings. However, it is not exposing any
> >> way to users to change that value/configuration in the database. I might
> >> have missed it; do you see any API methods that receive the parameter
> >> "cacheMode" and then pass this parameter to a "diskOffering" object, and
> >> then persist/update this object in the database?
> >>
> >> May I ask how are you guys changing the cacheMode configuration?
> >>
> >> On Tue, Feb 20, 2018 at 9:56 AM, Paul Angus 
> >> wrote:
> >>
> >> > I'm working with some guys who are experimenting with the setting as
> if
> >> > definitely seems to change the performance of data disks.  It also
> >> changes
> >> > the XML of the VM which is created.
> >> >
> >> > p.s.
> >> > I've found this commit;
> >> >
> >> > https://github.com/apache/cloudstack/commit/1edaa36cc68e845a
> >> 42339d5f267d49
> >> > c82343aefb
> >> >
> >> > so I've got something to investigate now, but API documentation must
> >> > definitely be askew.
> >> >
> >> >
> >> >
> >> > paul.an...@shapeblue.com
> >> > www.shapeblue.com
> >> > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> >> > @shapeblue
> >> >
> >> >
> >> >
> >> >
> >> > -Original Message-
> >> > From: Rafael Weingärtner [mailto:rafaelweingart...@gmail.com]
> >> > Sent: 20 February 2018 12:31
> >> > To: dev 
> >> > Subject: Re: Caching modes
> >> >
> >> > This cache mode parameter does not exist in "CreateDiskOfferingCmd"
> >> > command. I also checked some commits from 2, 3, 4 and 5 years ago, and
> >> > this parameter was never there. If you check the API in [1], you can
> see
> >> > that it is not an expected parameter. Moreover, I do not see any use
> of
> >> > "setCacheMode" in the code (in case it is updated by some other
> method).
> >> > Interestingly enough, the code uses "getCacheMode".
> >> >
> >> > In summary, it is not a feature, and it does not work. It looks like
> >> some
> >> > leftover from dark ages when people could commit anything and then
> they
> >> > would just leave a half implementation there in our code base.
> >> >
> >> > [1]
> >> > https://cloudstack.apache.org/api/apidocs-4.11/apis/
> >> > createDiskOffering.html
> >> >
> >> >
> >> > On Tue, Feb 20, 2018 at 9:19 AM, Andrija Panic <
> andrija.pa...@gmail.com
> >> >
> >> > wrote:
> >> >
> >> > > I can also assume that "cachemode" as API parameter is not
> supported,
> >> > > since when creating data disk offering via GUI also doesn't set it
> in
> >> > DB/table.
> >> > >
> >> > > CM:create diskoffering name=xxx displaytext=xxx
> storagetype=shared
> >> > > disksize=1024 cachemode=writeback
> >> > >
> >> > > this also does not set cachemode in table... my guess it's not
> >> > > implemented in API
> >> > >
> >> > > Let me know if I can help with any testing here.
> >> > >
> >> > > Cheers
> >> > >
> >> > > On 20 February 2018 at 13:09, Andrija Panic <
> andrija.pa...@gmail.com>
> >> > > wrote:
> >> > >
> >> > > > Hi Paul,
> >> > > >
> >> > > > not helping directly answering your question, but here are some
> >> > > > observations and "warning" if client's are using write-back cache
> on
> >> > > > KVM level
> >> > > >
> >> > > >
> >> > > > I have (long time ago) tested 

Re: Caching modes

[Andrija Panic]

So it seems that just passing the cachemode value to API is not there, or
somehow messedup, but deployVM process does read DB values from
disk_offering table for sure, and applies it to XML file for KVM.
This is above ACS 4.8.x.


On 20 February 2018 at 15:44, Andrija Panic  wrote:

> I have edited the disk_offering table, in the cache_mode just enter
> "writeback". Stop and start VM, and it will pickup/inherit the cache_mode
> from it's parrent offering
> This also applies to Compute/Service offering, again inside disk_offering
> table - just tested both
>
> i.e.
>
> UPDATE `cloud`.`disk_offering` SET `cache_mode`='writeback' WHERE
> `id`=102; # Compute Offering (Service offering)
> UPDATE `cloud`.`disk_offering` SET `cache_mode`='writeback' WHERE
> `id`=114; #data disk offering
>
> Before SQL:
>
> root@ix1-c7-4:~# virsh dumpxml i-2-10-VM | grep cache -A2
>   
>   
>   
> --
>   
>   
>   
> --
>
> STOP and START VM = after SQL
>
> root@ix1-c7-4:~# virsh dumpxml i-2-10-VM | grep cache -A2
>   
>   
>   
> --
>   
>   
>   
> --
>
>
>
> On 20 February 2018 at 14:03, Rafael Weingärtner <
> rafaelweingart...@gmail.com> wrote:
>
>> I have no idea how it can change the performance. If you look at the
>> content of the commit you provided, it is only the commit that enabled the
>> use of getCacheMode from disk offerings. However, it is not exposing any
>> way to users to change that value/configuration in the database. I might
>> have missed it; do you see any API methods that receive the parameter
>> "cacheMode" and then pass this parameter to a "diskOffering" object, and
>> then persist/update this object in the database?
>>
>> May I ask how are you guys changing the cacheMode configuration?
>>
>> On Tue, Feb 20, 2018 at 9:56 AM, Paul Angus 
>> wrote:
>>
>> > I'm working with some guys who are experimenting with the setting as if
>> > definitely seems to change the performance of data disks.  It also
>> changes
>> > the XML of the VM which is created.
>> >
>> > p.s.
>> > I've found this commit;
>> >
>> > https://github.com/apache/cloudstack/commit/1edaa36cc68e845a
>> 42339d5f267d49
>> > c82343aefb
>> >
>> > so I've got something to investigate now, but API documentation must
>> > definitely be askew.
>> >
>> >
>> >
>> > paul.an...@shapeblue.com
>> > www.shapeblue.com
>> > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>> > @shapeblue
>> >
>> >
>> >
>> >
>> > -Original Message-
>> > From: Rafael Weingärtner [mailto:rafaelweingart...@gmail.com]
>> > Sent: 20 February 2018 12:31
>> > To: dev 
>> > Subject: Re: Caching modes
>> >
>> > This cache mode parameter does not exist in "CreateDiskOfferingCmd"
>> > command. I also checked some commits from 2, 3, 4 and 5 years ago, and
>> > this parameter was never there. If you check the API in [1], you can see
>> > that it is not an expected parameter. Moreover, I do not see any use of
>> > "setCacheMode" in the code (in case it is updated by some other method).
>> > Interestingly enough, the code uses "getCacheMode".
>> >
>> > In summary, it is not a feature, and it does not work. It looks like
>> some
>> > leftover from dark ages when people could commit anything and then they
>> > would just leave a half implementation there in our code base.
>> >
>> > [1]
>> > https://cloudstack.apache.org/api/apidocs-4.11/apis/
>> > createDiskOffering.html
>> >
>> >
>> > On Tue, Feb 20, 2018 at 9:19 AM, Andrija Panic > >
>> > wrote:
>> >
>> > > I can also assume that "cachemode" as API parameter is not supported,
>> > > since when creating data disk offering via GUI also doesn't set it in
>> > DB/table.
>> > >
>> > > CM:create diskoffering name=xxx displaytext=xxx storagetype=shared
>> > > disksize=1024 cachemode=writeback
>> > >
>> > > this also does not set cachemode in table... my guess it's not
>> > > implemented in API
>> > >
>> > > Let me know if I can help with any testing here.
>> > >
>> > > Cheers
>> > >
>> > > On 20 February 2018 at 13:09, Andrija Panic 
>> > > wrote:
>> > >
>> > > > Hi Paul,
>> > > >
>> > > > not helping directly answering your question, but here are some
>> > > > observations and "warning" if client's are using write-back cache on
>> > > > KVM level
>> > > >
>> > > >
>> > > > I have (long time ago) tested performance in 3 combinations (this
>> > > > was not really thorough testing but a brief testing with FIO and
>> > > > random IO WRITE)
>> > > >
>> > > > - just CEPH rbd cache (on KVM side)
>> > > >i.e. [client]
>> > > >  rbd cache = true
>> > > >  rbd cache writethrough until flush = true
>> > > >  #(this is default 32MB per volume, afaik
>> > > >
>> > > > - just KMV write-back cache (had to manually edit disk_offering
>> > > > table to activate cache mode, since when creating new disk offering
>> > > 

Re: Caching modes

[Andrija Panic]

I have edited the disk_offering table, in the cache_mode just enter
"writeback". Stop and start VM, and it will pickup/inherit the cache_mode
from it's parrent offering
This also applies to Compute/Service offering, again inside disk_offering
table - just tested both

i.e.

UPDATE `cloud`.`disk_offering` SET `cache_mode`='writeback' WHERE
`id`=102; # Compute Offering (Service offering)
UPDATE `cloud`.`disk_offering` SET `cache_mode`='writeback' WHERE
`id`=114; #data disk offering

Before SQL:

root@ix1-c7-4:~# virsh dumpxml i-2-10-VM | grep cache -A2
  
  
  
--
  
  
  
--

STOP and START VM = after SQL

root@ix1-c7-4:~# virsh dumpxml i-2-10-VM | grep cache -A2
  
  
  
--
  
  
  
--



On 20 February 2018 at 14:03, Rafael Weingärtner <
rafaelweingart...@gmail.com> wrote:

> I have no idea how it can change the performance. If you look at the
> content of the commit you provided, it is only the commit that enabled the
> use of getCacheMode from disk offerings. However, it is not exposing any
> way to users to change that value/configuration in the database. I might
> have missed it; do you see any API methods that receive the parameter
> "cacheMode" and then pass this parameter to a "diskOffering" object, and
> then persist/update this object in the database?
>
> May I ask how are you guys changing the cacheMode configuration?
>
> On Tue, Feb 20, 2018 at 9:56 AM, Paul Angus 
> wrote:
>
> > I'm working with some guys who are experimenting with the setting as if
> > definitely seems to change the performance of data disks.  It also
> changes
> > the XML of the VM which is created.
> >
> > p.s.
> > I've found this commit;
> >
> > https://github.com/apache/cloudstack/commit/
> 1edaa36cc68e845a42339d5f267d49
> > c82343aefb
> >
> > so I've got something to investigate now, but API documentation must
> > definitely be askew.
> >
> >
> >
> > paul.an...@shapeblue.com
> > www.shapeblue.com
> > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > @shapeblue
> >
> >
> >
> >
> > -Original Message-
> > From: Rafael Weingärtner [mailto:rafaelweingart...@gmail.com]
> > Sent: 20 February 2018 12:31
> > To: dev 
> > Subject: Re: Caching modes
> >
> > This cache mode parameter does not exist in "CreateDiskOfferingCmd"
> > command. I also checked some commits from 2, 3, 4 and 5 years ago, and
> > this parameter was never there. If you check the API in [1], you can see
> > that it is not an expected parameter. Moreover, I do not see any use of
> > "setCacheMode" in the code (in case it is updated by some other method).
> > Interestingly enough, the code uses "getCacheMode".
> >
> > In summary, it is not a feature, and it does not work. It looks like some
> > leftover from dark ages when people could commit anything and then they
> > would just leave a half implementation there in our code base.
> >
> > [1]
> > https://cloudstack.apache.org/api/apidocs-4.11/apis/
> > createDiskOffering.html
> >
> >
> > On Tue, Feb 20, 2018 at 9:19 AM, Andrija Panic 
> > wrote:
> >
> > > I can also assume that "cachemode" as API parameter is not supported,
> > > since when creating data disk offering via GUI also doesn't set it in
> > DB/table.
> > >
> > > CM:create diskoffering name=xxx displaytext=xxx storagetype=shared
> > > disksize=1024 cachemode=writeback
> > >
> > > this also does not set cachemode in table... my guess it's not
> > > implemented in API
> > >
> > > Let me know if I can help with any testing here.
> > >
> > > Cheers
> > >
> > > On 20 February 2018 at 13:09, Andrija Panic 
> > > wrote:
> > >
> > > > Hi Paul,
> > > >
> > > > not helping directly answering your question, but here are some
> > > > observations and "warning" if client's are using write-back cache on
> > > > KVM level
> > > >
> > > >
> > > > I have (long time ago) tested performance in 3 combinations (this
> > > > was not really thorough testing but a brief testing with FIO and
> > > > random IO WRITE)
> > > >
> > > > - just CEPH rbd cache (on KVM side)
> > > >i.e. [client]
> > > >  rbd cache = true
> > > >  rbd cache writethrough until flush = true
> > > >  #(this is default 32MB per volume, afaik
> > > >
> > > > - just KMV write-back cache (had to manually edit disk_offering
> > > > table to activate cache mode, since when creating new disk offering
> > > > via GUI, the disk_offering tables was NOT populated with
> > > > "write-back" sertting/value
> > > ! )
> > > >
> > > > - both CEPH and KVM write-back cahce active
> > > >
> > > > My observations were like following, but would be good to actually
> > > confirm
> > > > by someone else:
> > > >
> > > > - same performance with only CEPH caching or with only KVM caching
> > > > - a bit worse performance with both CEPH and KVM caching active
> > > > (nonsense combination, I know...)
> > > 

Save the date: ApacheCon North America, September 24-27 in Montréal

[Rich Bowen]

Dear Apache Enthusiast,

(You’re receiving this message because you’re subscribed to a user@ or 
dev@ list of one or more Apache Software Foundation projects.)


We’re pleased to announce the upcoming ApacheCon [1] in Montréal, 
September 24-27. This event is all about you — the Apache project community.


We’ll have four tracks of technical content this time, as well as lots 
of opportunities to connect with your project community, hack on the 
code, and learn about other related (and unrelated!) projects across the 
foundation.


The Call For Papers (CFP) [2] and registration are now open. Register 
early to take advantage of the early bird prices and secure your place 
at the event hotel.


Important dates
March 30: CFP closes
April 20: CFP notifications sent
	August 24: Hotel room block closes (please do not wait until the last 
minute)


Follow @ApacheCon on Twitter to be the first to hear announcements about 
keynotes, the schedule, evening events, and everything you can expect to 
see at the event.


See you in Montréal!

Sincerely, Rich Bowen, V.P. Events,
on behalf of the entire ApacheCon team

[1] http://www.apachecon.com/acna18
[2] https://cfp.apachecon.com/conference.html?apachecon-north-america-2018

Re: Caching modes

[Rafael Weingärtner]

I have no idea how it can change the performance. If you look at the
content of the commit you provided, it is only the commit that enabled the
use of getCacheMode from disk offerings. However, it is not exposing any
way to users to change that value/configuration in the database. I might
have missed it; do you see any API methods that receive the parameter
"cacheMode" and then pass this parameter to a "diskOffering" object, and
then persist/update this object in the database?

May I ask how are you guys changing the cacheMode configuration?

On Tue, Feb 20, 2018 at 9:56 AM, Paul Angus 
wrote:

> I'm working with some guys who are experimenting with the setting as if
> definitely seems to change the performance of data disks.  It also changes
> the XML of the VM which is created.
>
> p.s.
> I've found this commit;
>
> https://github.com/apache/cloudstack/commit/1edaa36cc68e845a42339d5f267d49
> c82343aefb
>
> so I've got something to investigate now, but API documentation must
> definitely be askew.
>
>
>
> paul.an...@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
>
>
>
>
> -Original Message-
> From: Rafael Weingärtner [mailto:rafaelweingart...@gmail.com]
> Sent: 20 February 2018 12:31
> To: dev 
> Subject: Re: Caching modes
>
> This cache mode parameter does not exist in "CreateDiskOfferingCmd"
> command. I also checked some commits from 2, 3, 4 and 5 years ago, and
> this parameter was never there. If you check the API in [1], you can see
> that it is not an expected parameter. Moreover, I do not see any use of
> "setCacheMode" in the code (in case it is updated by some other method).
> Interestingly enough, the code uses "getCacheMode".
>
> In summary, it is not a feature, and it does not work. It looks like some
> leftover from dark ages when people could commit anything and then they
> would just leave a half implementation there in our code base.
>
> [1]
> https://cloudstack.apache.org/api/apidocs-4.11/apis/
> createDiskOffering.html
>
>
> On Tue, Feb 20, 2018 at 9:19 AM, Andrija Panic 
> wrote:
>
> > I can also assume that "cachemode" as API parameter is not supported,
> > since when creating data disk offering via GUI also doesn't set it in
> DB/table.
> >
> > CM:create diskoffering name=xxx displaytext=xxx storagetype=shared
> > disksize=1024 cachemode=writeback
> >
> > this also does not set cachemode in table... my guess it's not
> > implemented in API
> >
> > Let me know if I can help with any testing here.
> >
> > Cheers
> >
> > On 20 February 2018 at 13:09, Andrija Panic 
> > wrote:
> >
> > > Hi Paul,
> > >
> > > not helping directly answering your question, but here are some
> > > observations and "warning" if client's are using write-back cache on
> > > KVM level
> > >
> > >
> > > I have (long time ago) tested performance in 3 combinations (this
> > > was not really thorough testing but a brief testing with FIO and
> > > random IO WRITE)
> > >
> > > - just CEPH rbd cache (on KVM side)
> > >i.e. [client]
> > >  rbd cache = true
> > >  rbd cache writethrough until flush = true
> > >  #(this is default 32MB per volume, afaik
> > >
> > > - just KMV write-back cache (had to manually edit disk_offering
> > > table to activate cache mode, since when creating new disk offering
> > > via GUI, the disk_offering tables was NOT populated with
> > > "write-back" sertting/value
> > ! )
> > >
> > > - both CEPH and KVM write-back cahce active
> > >
> > > My observations were like following, but would be good to actually
> > confirm
> > > by someone else:
> > >
> > > - same performance with only CEPH caching or with only KVM caching
> > > - a bit worse performance with both CEPH and KVM caching active
> > > (nonsense combination, I know...)
> > >
> > >
> > > Please keep in mind, that some ACS functionality, KVM
> > > live-migrations on shared storage (NFS/CEPH) are NOT supported when
> > > you use KVM write-back cache, since this is considered "unsafe"
> migration, more info here:
> > > https://doc.opensuse.org/documentation/leap/virtualization/
> > html/book.virt/
> > > cha.cachemodes.html#sec.cache.mode.live.migration
> > >
> > > or in short:
> > > "
> > > The libvirt management layer includes checks for migration
> > > compatibility based on several factors. If the guest storage is
> > > hosted on a clustered file system, is read-only or is marked
> > > shareable, then the cache mode is ignored when determining if
> > > migration can be allowed. Otherwise libvirt will not allow migration
> > > unless the cache mode is set to none. However, this restriction can
> > > be overridden with the “unsafe” option to the migration APIs, which
> > > is also supported by virsh, as for example in
> > >
> > > virsh migrate --live --unsafe
> > > "
> > >
> > > Cheers
> > > Andrija
> > >
> > >
> 

RE: Caching modes

[Paul Angus]

I'm working with some guys who are experimenting with the setting as if 
definitely seems to change the performance of data disks.  It also changes the 
XML of the VM which is created.

p.s.
I've found this commit; 

https://github.com/apache/cloudstack/commit/1edaa36cc68e845a42339d5f267d49c82343aefb

so I've got something to investigate now, but API documentation must definitely 
be askew.



paul.an...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 


-Original Message-
From: Rafael Weingärtner [mailto:rafaelweingart...@gmail.com] 
Sent: 20 February 2018 12:31
To: dev 
Subject: Re: Caching modes

This cache mode parameter does not exist in "CreateDiskOfferingCmd"
command. I also checked some commits from 2, 3, 4 and 5 years ago, and this 
parameter was never there. If you check the API in [1], you can see that it is 
not an expected parameter. Moreover, I do not see any use of "setCacheMode" in 
the code (in case it is updated by some other method).
Interestingly enough, the code uses "getCacheMode".

In summary, it is not a feature, and it does not work. It looks like some 
leftover from dark ages when people could commit anything and then they would 
just leave a half implementation there in our code base.

[1]
https://cloudstack.apache.org/api/apidocs-4.11/apis/createDiskOffering.html


On Tue, Feb 20, 2018 at 9:19 AM, Andrija Panic 
wrote:

> I can also assume that "cachemode" as API parameter is not supported, 
> since when creating data disk offering via GUI also doesn't set it in 
> DB/table.
>
> CM:create diskoffering name=xxx displaytext=xxx storagetype=shared
> disksize=1024 cachemode=writeback
>
> this also does not set cachemode in table... my guess it's not 
> implemented in API
>
> Let me know if I can help with any testing here.
>
> Cheers
>
> On 20 February 2018 at 13:09, Andrija Panic 
> wrote:
>
> > Hi Paul,
> >
> > not helping directly answering your question, but here are some 
> > observations and "warning" if client's are using write-back cache on 
> > KVM level
> >
> >
> > I have (long time ago) tested performance in 3 combinations (this 
> > was not really thorough testing but a brief testing with FIO and 
> > random IO WRITE)
> >
> > - just CEPH rbd cache (on KVM side)
> >i.e. [client]
> >  rbd cache = true
> >  rbd cache writethrough until flush = true
> >  #(this is default 32MB per volume, afaik
> >
> > - just KMV write-back cache (had to manually edit disk_offering 
> > table to activate cache mode, since when creating new disk offering 
> > via GUI, the disk_offering tables was NOT populated with 
> > "write-back" sertting/value
> ! )
> >
> > - both CEPH and KVM write-back cahce active
> >
> > My observations were like following, but would be good to actually
> confirm
> > by someone else:
> >
> > - same performance with only CEPH caching or with only KVM caching
> > - a bit worse performance with both CEPH and KVM caching active 
> > (nonsense combination, I know...)
> >
> >
> > Please keep in mind, that some ACS functionality, KVM 
> > live-migrations on shared storage (NFS/CEPH) are NOT supported when 
> > you use KVM write-back cache, since this is considered "unsafe" migration, 
> > more info here:
> > https://doc.opensuse.org/documentation/leap/virtualization/
> html/book.virt/
> > cha.cachemodes.html#sec.cache.mode.live.migration
> >
> > or in short:
> > "
> > The libvirt management layer includes checks for migration 
> > compatibility based on several factors. If the guest storage is 
> > hosted on a clustered file system, is read-only or is marked 
> > shareable, then the cache mode is ignored when determining if 
> > migration can be allowed. Otherwise libvirt will not allow migration 
> > unless the cache mode is set to none. However, this restriction can 
> > be overridden with the “unsafe” option to the migration APIs, which 
> > is also supported by virsh, as for example in
> >
> > virsh migrate --live --unsafe
> > "
> >
> > Cheers
> > Andrija
> >
> >
> > On 20 February 2018 at 11:24, Paul Angus 
> wrote:
> >
> >> Hi Wido,
> >>
> >> This is for KVM (with Ceph backend as it happens), the API 
> >> documentation is out of sync with UI capabilities, so I'm trying to 
> >> figure out if we
> >> *should* be able to set cacheMode for root disks.  It seems to make
> quite a
> >> difference to performance.
> >>
> >>
> >>
> >> paul.an...@shapeblue.com
> >> www.shapeblue.com
> >> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue
> >>
> >>
> >>
> >>
> >> -Original Message-
> >> From: Wido den Hollander [mailto:w...@widodh.nl]
> >> Sent: 20 February 2018 09:03
> >> To: dev@cloudstack.apache.org
> >> Subject: Re: Caching modes
> >>
> >>
> >>
> >> On 02/20/2018 09:46 AM, Paul Angus wrote:
> >> > Hey guys,
> >> >
> >> > Can 

[4.11] Management to VR connection issues

[Rene Moser]

Hi

We upgraded from 4.9 to 4.11. VMware 6.5.0. (Testing environment).

VR upgrade went through. But we noticed that the communication between
the management server and the VR are not working properly.

We do not yet fully understand the issue, one thing we noted is that the
networks configs seems not be bound to the same interfaces after every
reboot. As a result, after a reboot you may can connect to the VR by
SSH, after another reboot you can't anymore.

The Network name eth0 switched from the NIC id 3 to 4 after reboot.

The VR is kept in "starting" state, of course as a consequence we get
many issues related to this, no VM deployments (kept in starting state),
VM expunging failure (cleanup fails), a.s.o.

Have anyone experienced similar issues?

Regards
René

Re: Caching modes

[Rafael Weingärtner]

This cache mode parameter does not exist in "CreateDiskOfferingCmd"
command. I also checked some commits from 2, 3, 4 and 5 years ago, and this
parameter was never there. If you check the API in [1], you can see that it
is not an expected parameter. Moreover, I do not see any use of
"setCacheMode" in the code (in case it is updated by some other method).
Interestingly enough, the code uses "getCacheMode".

In summary, it is not a feature, and it does not work. It looks like some
leftover from dark ages when people could commit anything and then they
would just leave a half implementation there in our code base.

[1]
https://cloudstack.apache.org/api/apidocs-4.11/apis/createDiskOffering.html


On Tue, Feb 20, 2018 at 9:19 AM, Andrija Panic 
wrote:

> I can also assume that "cachemode" as API parameter is not supported, since
> when creating data disk offering via GUI also doesn't set it in DB/table.
>
> CM:create diskoffering name=xxx displaytext=xxx storagetype=shared
> disksize=1024 cachemode=writeback
>
> this also does not set cachemode in table... my guess it's not implemented
> in API
>
> Let me know if I can help with any testing here.
>
> Cheers
>
> On 20 February 2018 at 13:09, Andrija Panic 
> wrote:
>
> > Hi Paul,
> >
> > not helping directly answering your question, but here are some
> > observations and "warning" if client's are using write-back cache on KVM
> > level
> >
> >
> > I have (long time ago) tested performance in 3 combinations (this was not
> > really thorough testing but a brief testing with FIO and random IO WRITE)
> >
> > - just CEPH rbd cache (on KVM side)
> >i.e. [client]
> >  rbd cache = true
> >  rbd cache writethrough until flush = true
> >  #(this is default 32MB per volume, afaik
> >
> > - just KMV write-back cache (had to manually edit disk_offering table to
> > activate cache mode, since when creating new disk offering via GUI, the
> > disk_offering tables was NOT populated with "write-back" sertting/value
> ! )
> >
> > - both CEPH and KVM write-back cahce active
> >
> > My observations were like following, but would be good to actually
> confirm
> > by someone else:
> >
> > - same performance with only CEPH caching or with only KVM caching
> > - a bit worse performance with both CEPH and KVM caching active (nonsense
> > combination, I know...)
> >
> >
> > Please keep in mind, that some ACS functionality, KVM live-migrations on
> > shared storage (NFS/CEPH) are NOT supported when you use KVM write-back
> > cache, since this is considered "unsafe" migration, more info here:
> > https://doc.opensuse.org/documentation/leap/virtualization/
> html/book.virt/
> > cha.cachemodes.html#sec.cache.mode.live.migration
> >
> > or in short:
> > "
> > The libvirt management layer includes checks for migration compatibility
> > based on several factors. If the guest storage is hosted on a clustered
> > file system, is read-only or is marked shareable, then the cache mode is
> > ignored when determining if migration can be allowed. Otherwise libvirt
> > will not allow migration unless the cache mode is set to none. However,
> > this restriction can be overridden with the “unsafe” option to the
> > migration APIs, which is also supported by virsh, as for example in
> >
> > virsh migrate --live --unsafe
> > "
> >
> > Cheers
> > Andrija
> >
> >
> > On 20 February 2018 at 11:24, Paul Angus 
> wrote:
> >
> >> Hi Wido,
> >>
> >> This is for KVM (with Ceph backend as it happens), the API documentation
> >> is out of sync with UI capabilities, so I'm trying to figure out if we
> >> *should* be able to set cacheMode for root disks.  It seems to make
> quite a
> >> difference to performance.
> >>
> >>
> >>
> >> paul.an...@shapeblue.com
> >> www.shapeblue.com
> >> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> >> @shapeblue
> >>
> >>
> >>
> >>
> >> -Original Message-
> >> From: Wido den Hollander [mailto:w...@widodh.nl]
> >> Sent: 20 February 2018 09:03
> >> To: dev@cloudstack.apache.org
> >> Subject: Re: Caching modes
> >>
> >>
> >>
> >> On 02/20/2018 09:46 AM, Paul Angus wrote:
> >> > Hey guys,
> >> >
> >> > Can anyone shed any light on write caching in CloudStack?   cacheMode
> >> is available through the UI for data disks (but not root disks), but not
> >> documented as an API option for data or root disks (although is
> documented
> >> as a response for data disks).
> >> >
> >>
> >> What hypervisor?
> >>
> >> In case of KVM it's passed down to XML which then passes it to Qemu/KVM
> >> which then handles the caching.
> >>
> >> The implementation varies per hypervisor, so that should be the
> question.
> >>
> >> Wido
> >>
> >>
> >> > #huh?
> >> >
> >> > thanks
> >> >
> >> >
> >> >
> >> > paul.an...@shapeblue.com
> >> > www.shapeblue.com
> >> > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue
> >> >
> >> >
> >> >
> >>
> >
> >
> >

Re: Caching modes

[Andrija Panic]

I can also assume that "cachemode" as API parameter is not supported, since
when creating data disk offering via GUI also doesn't set it in DB/table.

CM:create diskoffering name=xxx displaytext=xxx storagetype=shared
disksize=1024 cachemode=writeback

this also does not set cachemode in table... my guess it's not implemented
in API

Let me know if I can help with any testing here.

Cheers

On 20 February 2018 at 13:09, Andrija Panic  wrote:

> Hi Paul,
>
> not helping directly answering your question, but here are some
> observations and "warning" if client's are using write-back cache on KVM
> level
>
>
> I have (long time ago) tested performance in 3 combinations (this was not
> really thorough testing but a brief testing with FIO and random IO WRITE)
>
> - just CEPH rbd cache (on KVM side)
>i.e. [client]
>  rbd cache = true
>  rbd cache writethrough until flush = true
>  #(this is default 32MB per volume, afaik
>
> - just KMV write-back cache (had to manually edit disk_offering table to
> activate cache mode, since when creating new disk offering via GUI, the
> disk_offering tables was NOT populated with "write-back" sertting/value ! )
>
> - both CEPH and KVM write-back cahce active
>
> My observations were like following, but would be good to actually confirm
> by someone else:
>
> - same performance with only CEPH caching or with only KVM caching
> - a bit worse performance with both CEPH and KVM caching active (nonsense
> combination, I know...)
>
>
> Please keep in mind, that some ACS functionality, KVM live-migrations on
> shared storage (NFS/CEPH) are NOT supported when you use KVM write-back
> cache, since this is considered "unsafe" migration, more info here:
> https://doc.opensuse.org/documentation/leap/virtualization/html/book.virt/
> cha.cachemodes.html#sec.cache.mode.live.migration
>
> or in short:
> "
> The libvirt management layer includes checks for migration compatibility
> based on several factors. If the guest storage is hosted on a clustered
> file system, is read-only or is marked shareable, then the cache mode is
> ignored when determining if migration can be allowed. Otherwise libvirt
> will not allow migration unless the cache mode is set to none. However,
> this restriction can be overridden with the “unsafe” option to the
> migration APIs, which is also supported by virsh, as for example in
>
> virsh migrate --live --unsafe
> "
>
> Cheers
> Andrija
>
>
> On 20 February 2018 at 11:24, Paul Angus  wrote:
>
>> Hi Wido,
>>
>> This is for KVM (with Ceph backend as it happens), the API documentation
>> is out of sync with UI capabilities, so I'm trying to figure out if we
>> *should* be able to set cacheMode for root disks.  It seems to make quite a
>> difference to performance.
>>
>>
>>
>> paul.an...@shapeblue.com
>> www.shapeblue.com
>> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>> @shapeblue
>>
>>
>>
>>
>> -Original Message-
>> From: Wido den Hollander [mailto:w...@widodh.nl]
>> Sent: 20 February 2018 09:03
>> To: dev@cloudstack.apache.org
>> Subject: Re: Caching modes
>>
>>
>>
>> On 02/20/2018 09:46 AM, Paul Angus wrote:
>> > Hey guys,
>> >
>> > Can anyone shed any light on write caching in CloudStack?   cacheMode
>> is available through the UI for data disks (but not root disks), but not
>> documented as an API option for data or root disks (although is documented
>> as a response for data disks).
>> >
>>
>> What hypervisor?
>>
>> In case of KVM it's passed down to XML which then passes it to Qemu/KVM
>> which then handles the caching.
>>
>> The implementation varies per hypervisor, so that should be the question.
>>
>> Wido
>>
>>
>> > #huh?
>> >
>> > thanks
>> >
>> >
>> >
>> > paul.an...@shapeblue.com
>> > www.shapeblue.com
>> > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue
>> >
>> >
>> >
>>
>
>
>
> --
>
> Andrija Panić
>



-- 

Andrija Panić

Re: Caching modes

[Andrija Panic]

Hi Paul,

not helping directly answering your question, but here are some
observations and "warning" if client's are using write-back cache on KVM
level


I have (long time ago) tested performance in 3 combinations (this was not
really thorough testing but a brief testing with FIO and random IO WRITE)

- just CEPH rbd cache (on KVM side)
   i.e. [client]
 rbd cache = true
 rbd cache writethrough until flush = true
 #(this is default 32MB per volume, afaik

- just KMV write-back cache (had to manually edit disk_offering table to
activate cache mode, since when creating new disk offering via GUI, the
disk_offering tables was NOT populated with "write-back" sertting/value ! )

- both CEPH and KVM write-back cahce active

My observations were like following, but would be good to actually confirm
by someone else:

- same performance with only CEPH caching or with only KVM caching
- a bit worse performance with both CEPH and KVM caching active (nonsense
combination, I know...)


Please keep in mind, that some ACS functionality, KVM live-migrations on
shared storage (NFS/CEPH) are NOT supported when you use KVM write-back
cache, since this is considered "unsafe" migration, more info here:
https://doc.opensuse.org/documentation/leap/virtualization/html/book.virt/cha.cachemodes.html#sec.cache.mode.live.migration

or in short:
"
The libvirt management layer includes checks for migration compatibility
based on several factors. If the guest storage is hosted on a clustered
file system, is read-only or is marked shareable, then the cache mode is
ignored when determining if migration can be allowed. Otherwise libvirt
will not allow migration unless the cache mode is set to none. However,
this restriction can be overridden with the “unsafe” option to the
migration APIs, which is also supported by virsh, as for example in

virsh migrate --live --unsafe
"

Cheers
Andrija


On 20 February 2018 at 11:24, Paul Angus  wrote:

> Hi Wido,
>
> This is for KVM (with Ceph backend as it happens), the API documentation
> is out of sync with UI capabilities, so I'm trying to figure out if we
> *should* be able to set cacheMode for root disks.  It seems to make quite a
> difference to performance.
>
>
>
> paul.an...@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
>
>
>
>
> -Original Message-
> From: Wido den Hollander [mailto:w...@widodh.nl]
> Sent: 20 February 2018 09:03
> To: dev@cloudstack.apache.org
> Subject: Re: Caching modes
>
>
>
> On 02/20/2018 09:46 AM, Paul Angus wrote:
> > Hey guys,
> >
> > Can anyone shed any light on write caching in CloudStack?   cacheMode is
> available through the UI for data disks (but not root disks), but not
> documented as an API option for data or root disks (although is documented
> as a response for data disks).
> >
>
> What hypervisor?
>
> In case of KVM it's passed down to XML which then passes it to Qemu/KVM
> which then handles the caching.
>
> The implementation varies per hypervisor, so that should be the question.
>
> Wido
>
>
> > #huh?
> >
> > thanks
> >
> >
> >
> > paul.an...@shapeblue.com
> > www.shapeblue.com
> > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue
> >
> >
> >
>



-- 

Andrija Panić

Problem re-adding host to Cloudstack

[Marc Poll Garcia]

Hello guys,

we've just upgraded one of our hosts from Cloudstack Cluster, to ESXI
6.5.0, doing the following steps (Host 192.168.150.14):

- Change host state to "maintenance mode" on Cloudstack
- Remove the host as an infrastructure
- Put the host in maintenance mode on VCenter.
- Remove the host from the vmware cluster
- Attach the base line of vsphere 6.5
- Scan to verify that it is viable
- Remmediate
- Add the host to the cluster in VCenter
- Remove from maintenance (VCenter)
- Add the host on Cloudstack (Infraestructure --> Hosts --> Add host.

Unfortunatly we have problems to migrate any of our virtual machines, and
we have detected the following error on the logs:

2018-02-19 08:07:11,286 ERROR [c.c.h.v.r.VmwareResource]
(DirectAgent-356:ctx-57b3df7c 192.168.150.14, cmd: GetStorageStatsCommand)
(logid:046342d1) Unable to execute GetStorageStatsCommand(storageId :
de3b198a-9bc7-36ce-8597-43eed531ee61, localPath:
/Cloud-UPC/cloud-primary-pro-04, poolType: VMFS) due to Exception:
com.cloud.utils.exception.CloudRuntimeException
2018-02-19 08:07:11,365 ERROR [c.c.h.v.r.VmwareResource]
(DirectAgent-102:ctx-5b61f7f2 192.168.150.14, cmd: GetStorageStatsCommand)
(logid:df92288f) *Unable to connect to vSphere server: 192.168.150.20*

Could you help us please?

Thank you very much.
Kind regards.

-- 
Marc Poll Garcia
Technology Infrastructure . Àrea de Serveis TIC
Telèfon:  93.405.43.57

[image: UPCnet]

--
Aquest correu electrònic pot contenir informació confidencial o legalment
protegida i està exclusivament dirigit a la persona o entitat destinatària.
Si vostè no és el destinatari final o persona encarregada de recollir-lo,
no està autoritzat a llegir-lo, retenir-lo, modificar-lo, distribuir-lo,
copiar-lo ni a revelar el seu contingut. Si ha rebut aquest correu
electrònic per error, li preguem que informi al remitent i elimini del seu
sistema el missatge i el material annex que pugui contenir.
Gràcies per la seva col.laboració.
--

*** Si us plau, no m'imprimeixis. Vull seguir sent digital ***
*** Por favor, no me imprimas. Quiero seguir siendo digital ***
*** Please, don't print me. I want to remain digital ***
--

RE: Caching modes

[Paul Angus]

Hi Wido,

This is for KVM (with Ceph backend as it happens), the API documentation is out 
of sync with UI capabilities, so I'm trying to figure out if we *should* be 
able to set cacheMode for root disks.  It seems to make quite a difference to 
performance.



paul.an...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 


-Original Message-
From: Wido den Hollander [mailto:w...@widodh.nl] 
Sent: 20 February 2018 09:03
To: dev@cloudstack.apache.org
Subject: Re: Caching modes



On 02/20/2018 09:46 AM, Paul Angus wrote:
> Hey guys,
> 
> Can anyone shed any light on write caching in CloudStack?   cacheMode is 
> available through the UI for data disks (but not root disks), but not 
> documented as an API option for data or root disks (although is documented as 
> a response for data disks).
> 

What hypervisor?

In case of KVM it's passed down to XML which then passes it to Qemu/KVM which 
then handles the caching.

The implementation varies per hypervisor, so that should be the question.

Wido


> #huh?
> 
> thanks
> 
> 
> 
> paul.an...@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue
>
>   
> 

Re: Caching modes

[Wido den Hollander]

On 02/20/2018 09:46 AM, Paul Angus wrote:

Hey guys,

Can anyone shed any light on write caching in CloudStack?   cacheMode is 
available through the UI for data disks (but not root disks), but not 
documented as an API option for data or root disks (although is documented as a 
response for data disks).



What hypervisor?

In case of KVM it's passed down to XML which then passes it to Qemu/KVM 
which then handles the caching.


The implementation varies per hypervisor, so that should be the question.

Wido



#huh?

thanks



paul.an...@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
   
  

Caching modes

[Paul Angus]

Hey guys,

Can anyone shed any light on write caching in CloudStack?   cacheMode is 
available through the UI for data disks (but not root disks), but not 
documented as an API option for data or root disks (although is documented as a 
response for data disks).

#huh?

thanks



paul.an...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue