Re: How to re-create virtual router

[Andrija Panic]

Right. Another user did same thing here on ML (Jevgeni, from 19th March),
and had exact same issues (on basic zone) - please check it.

I did advised back then to try to "undelete' the router, by changing a
DB/table values in vm_instance table, to mark VR's "state" column back to
"Stopped" and set the NULL to the "removed" column for the VR.

After that I would try to restart Network (with cleanup) which should
trigger proper delete/recreated of the VR - this is how things SHOULD work,
but again I'm not using Basic Zone actively...

On Fri, 29 Mar 2019 at 12:09, Fariborz Navidan 
wrote:

> My zone is a basic one. Unfortunately.,It does not work for me  It think
> router exists for the network but it does not!
>
> On Fri, Mar 29, 2019 at 3:18 PM Andrija Panic 
> wrote:
>
> > what ACS version, what kind of network/zone ?
> >
> > In 4.11.2, Advanced Zone, Isolated network (KVM), I have deleted router,
> > then started a VM, which triggered a creation of new VM.
> >
> > Again, destroyed VR, restarted network withOUT the "clean up" option, and
> > again new VR was created.
> >
> > On Fri, 29 Mar 2019 at 11:39, Fariborz Navidan 
> > wrote:
> >
> > > It fails to restart networl. Log says it cannot find virtual router.
> > Bellow
> > > is the log:
> > >
> > > 2019-03-29 10:53:41,621 DEBUG [c.c.c.ClusterManagerImpl]
> > > (Cluster-Heartbeat-1:ctx-277a1d9d) (logid:9902f1d7) Detected management
> > > node left, id:8, nodeIP:178.33.230.41
> > > 2019-03-29 10:53:41,621 INFO  [c.c.c.ClusterManagerImpl]
> > > (Cluster-Heartbeat-1:ctx-277a1d9d) (logid:9902f1d7) Trying to connect
> to
> > > 178.33.230.41
> > > 2019-03-29 10:53:41,621 INFO  [c.c.c.ClusterManagerImpl]
> > > (Cluster-Heartbeat-1:ctx-277a1d9d) (logid:9902f1d7) Management node 8
> is
> > > detected inactive by timestamp but is pingable
> > > 2019-03-29 10:53:42,526 DEBUG [c.c.a.ApiServlet]
> > > (qtp788117692-21:ctx-219919fa) (logid:c3be28b0) ===START===
> 2.190.177.97
> > > -- GET
> > >
> > >
> >
> command=restartNetwork&response=json&id=6e644551-5aee-4c9b-a75f-134f544ee97c&cleanup=false&makeredundant=false&_=1553853224533
> > > 2019-03-29 10:53:42,528 DEBUG [c.c.a.ApiServer]
> > > (qtp788117692-21:ctx-219919fa ctx-3b77ed02) (logid:c3be28b0) CIDRs from
> > > which account 'Acct[27cd01ef-3907-11e9-87ab-a4bf012ed1a6-admin]' is
> > allowed
> > > to perform API calls: 0.0.0.0/0,::/0
> > > 2019-03-29 10:53:42,536 INFO  [o.a.c.f.j.i.AsyncJobMonitor]
> > > (API-Job-Executor-2:ctx-5a2a6bbe job-16321) (logid:b95430aa) Add
> > job-16321
> > > into job monitoring
> > > 2019-03-29 10:53:42,539 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > (qtp788117692-21:ctx-219919fa ctx-3b77ed02) (logid:c3be28b0) submit
> async
> > > job-16321, details: AsyncJobVO {id:16321, userId: 2, accountId: 2,
> > > instanceType: None, instanceId: null, cmd:
> > > org.apache.cloudstack.api.command.user.network.RestartNetworkCmd,
> > cmdInfo:
> > >
> > >
> >
> {"makeredundant":"false","cleanup":"false","response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"3200","id":"6e644551-5aee-4c9b-a75f-134f544ee97c","ctxDetails":"{\"interface
> > >
> > >
> >
> com.cloud.network.Network\":\"6e644551-5aee-4c9b-a75f-134f544ee97c\"}","ctxAccountId":"2","uuid":"6e644551-5aee-4c9b-a75f-134f544ee97c","cmdEventType":"NETWORK.RESTART","_":"1553853224533"},
> > > cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> > > result: null, initMsid: 279278805450993, completeMsid: null,
> lastUpdated:
> > > null, lastPolled: null, created: null}
> > > 2019-03-29 10:53:42,540 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > (API-Job-Executor-2:ctx-5a2a6bbe job-16321) (logid:155f349b) Executing
> > > AsyncJobVO {id:16321, userId: 2, accountId: 2, instanceType: None,
> > > instanceId: null, cmd:
> > > org.apache.cloudstack.api.command.user.network.RestartNetworkCmd,
> > cmdInfo:
> > >
> > >
> >
> {"makeredundant":"false","cleanup":"false","response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEvent

Re: How to re-create virtual router

[Andrija Panic]

what ACS version, what kind of network/zone ?

In 4.11.2, Advanced Zone, Isolated network (KVM), I have deleted router,
then started a VM, which triggered a creation of new VM.

Again, destroyed VR, restarted network withOUT the "clean up" option, and
again new VR was created.

On Fri, 29 Mar 2019 at 11:39, Fariborz Navidan 
wrote:

> It fails to restart networl. Log says it cannot find virtual router. Bellow
> is the log:
>
> 2019-03-29 10:53:41,621 DEBUG [c.c.c.ClusterManagerImpl]
> (Cluster-Heartbeat-1:ctx-277a1d9d) (logid:9902f1d7) Detected management
> node left, id:8, nodeIP:178.33.230.41
> 2019-03-29 10:53:41,621 INFO  [c.c.c.ClusterManagerImpl]
> (Cluster-Heartbeat-1:ctx-277a1d9d) (logid:9902f1d7) Trying to connect to
> 178.33.230.41
> 2019-03-29 10:53:41,621 INFO  [c.c.c.ClusterManagerImpl]
> (Cluster-Heartbeat-1:ctx-277a1d9d) (logid:9902f1d7) Management node 8 is
> detected inactive by timestamp but is pingable
> 2019-03-29 10:53:42,526 DEBUG [c.c.a.ApiServlet]
> (qtp788117692-21:ctx-219919fa) (logid:c3be28b0) ===START===  2.190.177.97
> -- GET
>
> command=restartNetwork&response=json&id=6e644551-5aee-4c9b-a75f-134f544ee97c&cleanup=false&makeredundant=false&_=1553853224533
> 2019-03-29 10:53:42,528 DEBUG [c.c.a.ApiServer]
> (qtp788117692-21:ctx-219919fa ctx-3b77ed02) (logid:c3be28b0) CIDRs from
> which account 'Acct[27cd01ef-3907-11e9-87ab-a4bf012ed1a6-admin]' is allowed
> to perform API calls: 0.0.0.0/0,::/0
> 2019-03-29 10:53:42,536 INFO  [o.a.c.f.j.i.AsyncJobMonitor]
> (API-Job-Executor-2:ctx-5a2a6bbe job-16321) (logid:b95430aa) Add job-16321
> into job monitoring
> 2019-03-29 10:53:42,539 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (qtp788117692-21:ctx-219919fa ctx-3b77ed02) (logid:c3be28b0) submit async
> job-16321, details: AsyncJobVO {id:16321, userId: 2, accountId: 2,
> instanceType: None, instanceId: null, cmd:
> org.apache.cloudstack.api.command.user.network.RestartNetworkCmd, cmdInfo:
>
> {"makeredundant":"false","cleanup":"false","response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"3200","id":"6e644551-5aee-4c9b-a75f-134f544ee97c","ctxDetails":"{\"interface
>
> com.cloud.network.Network\":\"6e644551-5aee-4c9b-a75f-134f544ee97c\"}","ctxAccountId":"2","uuid":"6e644551-5aee-4c9b-a75f-134f544ee97c","cmdEventType":"NETWORK.RESTART","_":"1553853224533"},
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> result: null, initMsid: 279278805450993, completeMsid: null, lastUpdated:
> null, lastPolled: null, created: null}
> 2019-03-29 10:53:42,540 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-2:ctx-5a2a6bbe job-16321) (logid:155f349b) Executing
> AsyncJobVO {id:16321, userId: 2, accountId: 2, instanceType: None,
> instanceId: null, cmd:
> org.apache.cloudstack.api.command.user.network.RestartNetworkCmd, cmdInfo:
>
> {"makeredundant":"false","cleanup":"false","response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"3200","id":"6e644551-5aee-4c9b-a75f-134f544ee97c","ctxDetails":"{\"interface
>
> com.cloud.network.Network\":\"6e644551-5aee-4c9b-a75f-134f544ee97c\"}","ctxAccountId":"2","uuid":"6e644551-5aee-4c9b-a75f-134f544ee97c","cmdEventType":"NETWORK.RESTART","_":"1553853224533"},
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> result: null, initMsid: 279278805450993, completeMsid: null, lastUpdated:
> null, lastPolled: null, created: null}
> 2019-03-29 10:53:42,541 DEBUG [c.c.a.ApiServlet]
> (qtp788117692-21:ctx-219919fa ctx-3b77ed02) (logid:c3be28b0) ===END===
> 2.190.177.97 -- GET
>
> command=restartNetwork&response=json&id=6e644551-5aee-4c9b-a75f-134f544ee97c&cleanup=false&makeredundant=false&_=1553853224533
> 2019-03-29 10:53:42,551 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> (API-Job-Executor-2:ctx-5a2a6bbe job-16321 ctx-a30b75bf) (logid:155f349b)
> Restarting network 204...
> 2019-03-29 10:53:42,552 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> (API-Job-Executor-2:ctx-5a2a6bbe job-16321 ctx-a30b75bf) (logid:155f349b)
> Implementing the network Ntwk[204|Guest|7] elements and resources as a part
> of network restart without cleanup
> 2019-03-29 10:53:42,554 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> (API-Job-Executor-2:ctx-5a2a6bbe job-16321 ctx-a30b75bf) (logid:155f349b)
> Asking VirtualRouter to implemenet Ntwk[204|Guest|7]
> 2019-03-29 10:53:42,558 ERROR [o.c.n.r.d.RouterDeploymentDefinition]
> (API-Job-Executor-2:ctx-5a2a6bbe job-16321 ctx-a30b75bf) (logid:155f349b)
> Didn't support redundant virtual router without public network!
> 2019-03-29 10:53:42,560 WARN  [o.a.c.e.o.NetworkOrchestrator]
> (API-Job-Executor-2:ctx-5a2a6bbe job-16321 ctx-a30b75bf) (logid:155f349b)
> Failed to implement network Ntwk[204|Guest|7] elements and resources as a
> part of network restart due to
> com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1]
> is unreachable: Can't find all necessary running routers!
> at
>
> com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:243)
> at

Re: Apply modified service offering

[Andrija Panic]

While on custom offering with 1 cpu, 1000 MHz/core, 512MB RAM:

http://10.2.x.y:8080/client/api?command=scaleVirtualMachine&response=json&id=19a67e20-c747-43bb-b149-c2b2294002f9&serviceofferingid=1b86713b-0131-4b25-8392-46518104bb90&details%5B0%5D.cpuSpeed=1300&details%5B0%5D.cpuNumber=2&details%5B0%5D.memory=768

You can craft a CloudMonkey command from above API (what I did here ^^^, I
used developer tools in Chrome, to catch the API call that changed offering
(from static) to custom one, and then I repeated the same with just
different cpu number, MHz, RAM...

It's always good to "learn" API syntax by catching GUI calls via Developer
tools or such.

Andrija

On Thu, 28 Mar 2019 at 22:39, Fariborz Navidan 
wrote:

> What if I want to change cpu and memory. Native UI does not have such
> capability even for customized service offering. How do I change them via
> cloudmonkey? I have already tried the "change
> serviceofferingforvirtualmachine" command but it always complain s  about
> invalid cpu cores.
>
> Thanks
>
> On Fri, Mar 29, 2019 at 1:55 AM Fariborz Navidan 
> wrote:
>
> >
> >
> > On Fri, Mar 29, 2019 at 1:24 AM Andrija Panic 
> > wrote:
> >
> >> Hi,
> >>
> >> you don't have to re-apply (nor there is such thing) same offering to a
> VM
> >> (you could change to other offering and back to original one - but that
> >> doesn't change a thing) - when VM is booted, some parameters are read
> from
> >> it's offering and applied acordingly (while some parameters might be
> >> cloned
> >> from the disk_offering / service_offering table to the vm_instance table
> >> during VM creation - but this doesn't apply for network rate)
> >>
> >> Depending on hypervisor, it should be enoughto just stop and start VM -
> >> that would trigger the update of network rate for that VM.
> >> I'm personally not sure if 0 means unlimited, worth trying with very
> high
> >> value (i.e.9) - anyway, try to confirm on hypervisor level if the
> new
> >> value is applied
> >>
> >> Andrija
> >>
> >> On Thu, 28 Mar 2019 at 21:33, Fariborz Navidan 
> >> wrote:
> >>
> >> > Hello,
> >> >
> >> > I have custom service offering with custom cpu and  memory. I have
> >> modified
> >> > the offering's network rate to be set to 0 for unlimited bandwidth.
> >> > CloudStack's native UI does not allow to change service offering of a
> VM
> >> > due to a bug. How can I re-apply the service offering to VMs deployed
> >> with
> >> > it via cloudmonkey?
> >> >
> >> > Thanks
> >> >
> >>
> >>
> >> --
> >>
> >> Andrija Panić
> >>
> >
>


-- 

Andrija Panić

Re: Apply modified service offering

[Andrija Panic]

Hi,

you don't have to re-apply (nor there is such thing) same offering to a VM
(you could change to other offering and back to original one - but that
doesn't change a thing) - when VM is booted, some parameters are read from
it's offering and applied acordingly (while some parameters might be cloned
from the disk_offering / service_offering table to the vm_instance table
during VM creation - but this doesn't apply for network rate)

Depending on hypervisor, it should be enoughto just stop and start VM -
that would trigger the update of network rate for that VM.
I'm personally not sure if 0 means unlimited, worth trying with very high
value (i.e.9) - anyway, try to confirm on hypervisor level if the new
value is applied

Andrija

On Thu, 28 Mar 2019 at 21:33, Fariborz Navidan 
wrote:

> Hello,
>
> I have custom service offering with custom cpu and  memory. I have modified
> the offering's network rate to be set to 0 for unlimited bandwidth.
> CloudStack's native UI does not allow to change service offering of a VM
> due to a bug. How can I re-apply the service offering to VMs deployed with
> it via cloudmonkey?
>
> Thanks
>


-- 

Andrija Panić

Re: CloudStack affinity group and vSphere DRS

[Andrija Panic]

Hi Yiping,

unless I'm mistaken, CloudStack should have nothing to do with DRS - it
only initially chooses the host to which to deploy a VM to, but later
VMware DRS can kick in and move it on it's own.

Cheers,
Andrija

On Thu, 28 Mar 2019 at 21:10, Yiping Zhang 
wrote:

> Hi, All:
>
> I started playing with vSphere 6.5 hypervisors under ACS 4.11.2.0 in our
> lab and have some questions on this set up.
>
> According to release notes, VMware DRS support in CloudStack was added in
> ACS 4.4.4 (here:
> http://docs.cloudstack.apache.org/projects/cloudstack-release-notes/en/4.4.4/about.html#vmware-support-for-drs).
> However,  in my CPU load tests,  VM instances are moved around by DRS in
> violation of CloudStack anti-host affinity group assignment for these VM
> instances.
>
> Did I configure it wrong or missed some step?  If anyone else are using
> these features, please share your experiences.
>
> Thanks
>
> Yiping
>


-- 

Andrija Panić

Re: cannot start system VMs: disaster after maintenance followup

[Andrija Panic]

Stick to 4.11.2 - 4.12 should be released withing few days officially.

As for qemu-kvm-ev - yes, it's supposed to work - make sure to test new
versions obviously.

Did you got your new installation running fine ?

On Thu, 21 Mar 2019 at 19:26, Jevgeni Zolotarjov 
wrote:

> Andrija,
>
> I asked here in the group if its safe to try new version of KVM and got
> reply, that it works. It was back in September. So we installed it with
> yum install centos-release-qemu-ev
> yum install qemu-kvm-ev
>
> It worked fine ever since.
> But with new maintenance (yum update) apparently some breaking changes were
> introduced.
> So, take care.
>
> Anyway, thanks. for help.
>
> As for your suggestion to use CS4.12. I haven't managed to find systemvm
> images for 4.12. Should I continue to use 4.11.12 systemvm?
>
>
>
>
>
>
> On Thu, Mar 21, 2019 at 7:19 PM Andrija Panic 
> wrote:
>
> > Jevgeni, qemu-kvm 1.5.3 is the lastest official one for CentoS 7.6.XXX
> > (latest) which I'm running atm in my lab (just checked for update) - how
> > did you manage to go to 2.0 (custom repo ?)
> >
> > On Thu, 21 Mar 2019 at 18:13, Ivan Kudryavtsev  >
> > wrote:
> >
> > > Jevgeniy, simplest and the most obvious way is to flatten their images
> > with
> > > "qemu-img convert", next import them as templates and recreate VMs from
> > > those templates.
> > >
> > > чт, 21 мар. 2019 г. в 13:05, Jevgeni Zolotarjov <
> j.zolotar...@gmail.com
> > >:
> > >
> > > > What happened in the end was: qemu-kvm got updated to version 2.0
> > during
> > > > the maintenance.  We could not manage to make this KVM to work with
> > > > Cloudstack.
> > > > So we rolled back to version 1.5.3.
> > > >
> > > > And now we have clean cloudstack fully operational. We can create new
> > VMs
> > > > and it works. I am almost happy.
> > > >
> > > > Now question - how do I get my old VMs to work, considering I have
> only
> > > > their volumes?
> > > >
> > > > On Thu, Mar 21, 2019 at 6:24 PM Andrija Panic <
> andrija.pa...@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Just replace the URL for systemVM template from 4.11.1 with 4.11.2
> > > (there
> > > > > is a PR for this now).
> > > > >
> > > > > On Thu, 21 Mar 2019 at 16:53, Andrija Panic <
> andrija.pa...@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > Please use the one, updated specifically for CentOS 7 -
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/cloudstack-documentation/blob/master/source/quickinstallationguide/qig.rst
> > > > > >
> > > > > > And please avoid collocating KVM and MGMT on same server
> > (especially
> > > in
> > > > > > any production-like system)
> > > > > >
> > > > > > Please let me know if the guide above gives you problem - we had
> > > > multiple
> > > > > > users explicitly following it - and successfully installed (with
> > some
> > > > > minor
> > > > > > modification, which we committed back to that guide).
> > > > > >
> > > > > > Thanks
> > > > > > Andrija
> > > > > >
> > > > > > On Thu, 21 Mar 2019 at 16:34, Jevgeni Zolotarjov <
> > > > j.zolotar...@gmail.com
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > >> OS management - centos 7 (1810)
> > > > > >> OS hypervisor - centos 7 (1810)
> > > > > >>
> > > > > >> Basic zone - yes
> > > > > >> I am following this quide
> > > > > >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> http://docs.cloudstack.apache.org/en/4.11.2.0/quickinstallationguide/qig.html
> > > > > >>
> > > > > >> Right now from scratch - management ans hypervisor on the same
> > > machine
> > > > > >> qemu - version 1.5.3
> > > > > >> libvirt - libvirt version: 4.5.0, package: 10.el7_6.6
> > > > > >>
> > > > > >> Basically - everything out of the box of clean centos install
> > 

Re: cannot start system VMs: disaster after maintenance followup

[Andrija Panic]

Jevgeni, qemu-kvm 1.5.3 is the lastest official one for CentoS 7.6.XXX
(latest) which I'm running atm in my lab (just checked for update) - how
did you manage to go to 2.0 (custom repo ?)

On Thu, 21 Mar 2019 at 18:13, Ivan Kudryavtsev 
wrote:

> Jevgeniy, simplest and the most obvious way is to flatten their images with
> "qemu-img convert", next import them as templates and recreate VMs from
> those templates.
>
> чт, 21 мар. 2019 г. в 13:05, Jevgeni Zolotarjov :
>
> > What happened in the end was: qemu-kvm got updated to version 2.0 during
> > the maintenance.  We could not manage to make this KVM to work with
> > Cloudstack.
> > So we rolled back to version 1.5.3.
> >
> > And now we have clean cloudstack fully operational. We can create new VMs
> > and it works. I am almost happy.
> >
> > Now question - how do I get my old VMs to work, considering I have only
> > their volumes?
> >
> > On Thu, Mar 21, 2019 at 6:24 PM Andrija Panic 
> > wrote:
> >
> > > Just replace the URL for systemVM template from 4.11.1 with 4.11.2
> (there
> > > is a PR for this now).
> > >
> > > On Thu, 21 Mar 2019 at 16:53, Andrija Panic 
> > > wrote:
> > >
> > > > Please use the one, updated specifically for CentOS 7 -
> > > >
> > >
> >
> https://github.com/apache/cloudstack-documentation/blob/master/source/quickinstallationguide/qig.rst
> > > >
> > > > And please avoid collocating KVM and MGMT on same server (especially
> in
> > > > any production-like system)
> > > >
> > > > Please let me know if the guide above gives you problem - we had
> > multiple
> > > > users explicitly following it - and successfully installed (with some
> > > minor
> > > > modification, which we committed back to that guide).
> > > >
> > > > Thanks
> > > > Andrija
> > > >
> > > > On Thu, 21 Mar 2019 at 16:34, Jevgeni Zolotarjov <
> > j.zolotar...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > >> OS management - centos 7 (1810)
> > > >> OS hypervisor - centos 7 (1810)
> > > >>
> > > >> Basic zone - yes
> > > >> I am following this quide
> > > >>
> > > >>
> > >
> >
> http://docs.cloudstack.apache.org/en/4.11.2.0/quickinstallationguide/qig.html
> > > >>
> > > >> Right now from scratch - management ans hypervisor on the same
> machine
> > > >> qemu - version 1.5.3
> > > >> libvirt - libvirt version: 4.5.0, package: 10.el7_6.6
> > > >>
> > > >> Basically - everything out of the box of clean centos install
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> On Thu, Mar 21, 2019 at 5:08 PM Andrija Panic <
> > andrija.pa...@gmail.com>
> > > >> wrote:
> > > >>
> > > >> > Hey Jevgeni,
> > > >> >
> > > >> > what OS mgmt, what OS hypervisor, what qemu/libvirt versions -
> still
> > > in
> > > >> > Basic Zone, SG ?
> > > >> >
> > > >> > Andrija
> > > >> >
> > > >> > On Thu, 21 Mar 2019 at 13:06, Jevgeni Zolotarjov <
> > > >> j.zolotar...@gmail.com>
> > > >> > wrote:
> > > >> >
> > > >> > > I reinstalled cloudstack from scratch - everything
> > > >> > >
> > > >> > > But looks like I hit the same wall now
> > > >> > >
> > > >> > > In the last step of installation it cannot create system VMs.
> > > >> > >
> > > >> > > service libvirtd status -l
> > > >> > > gives me
> > > >> > > 
> > > >> > > ● libvirtd.service - Virtualization daemon
> > > >> > >Loaded: loaded (/usr/lib/systemd/system/libvirtd.service;
> > > enabled;
> > > >> > > vendor preset: enabled)
> > > >> > >Active: active (running) since Thu 2019-03-21 11:45:00 GMT;
> > 18min
> > > >> ago
> > > >> > >  Docs: man:libvirtd(8)
> > > >> > >https://libvirt.org
> > > >> > >  Main PID: 537 (libvirtd)
> > > >> > >  

Re: cannot start system VMs: disaster after maintenance followup

[Andrija Panic]

Just replace the URL for systemVM template from 4.11.1 with 4.11.2 (there
is a PR for this now).

On Thu, 21 Mar 2019 at 16:53, Andrija Panic  wrote:

> Please use the one, updated specifically for CentOS 7 -
> https://github.com/apache/cloudstack-documentation/blob/master/source/quickinstallationguide/qig.rst
>
> And please avoid collocating KVM and MGMT on same server (especially in
> any production-like system)
>
> Please let me know if the guide above gives you problem - we had multiple
> users explicitly following it - and successfully installed (with some minor
> modification, which we committed back to that guide).
>
> Thanks
> Andrija
>
> On Thu, 21 Mar 2019 at 16:34, Jevgeni Zolotarjov 
> wrote:
>
>> OS management - centos 7 (1810)
>> OS hypervisor - centos 7 (1810)
>>
>> Basic zone - yes
>> I am following this quide
>>
>> http://docs.cloudstack.apache.org/en/4.11.2.0/quickinstallationguide/qig.html
>>
>> Right now from scratch - management ans hypervisor on the same machine
>> qemu - version 1.5.3
>> libvirt - libvirt version: 4.5.0, package: 10.el7_6.6
>>
>> Basically - everything out of the box of clean centos install
>>
>>
>>
>>
>> On Thu, Mar 21, 2019 at 5:08 PM Andrija Panic 
>> wrote:
>>
>> > Hey Jevgeni,
>> >
>> > what OS mgmt, what OS hypervisor, what qemu/libvirt versions - still in
>> > Basic Zone, SG ?
>> >
>> > Andrija
>> >
>> > On Thu, 21 Mar 2019 at 13:06, Jevgeni Zolotarjov <
>> j.zolotar...@gmail.com>
>> > wrote:
>> >
>> > > I reinstalled cloudstack from scratch - everything
>> > >
>> > > But looks like I hit the same wall now
>> > >
>> > > In the last step of installation it cannot create system VMs.
>> > >
>> > > service libvirtd status -l
>> > > gives me
>> > > 
>> > > ● libvirtd.service - Virtualization daemon
>> > >Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled;
>> > > vendor preset: enabled)
>> > >Active: active (running) since Thu 2019-03-21 11:45:00 GMT; 18min
>> ago
>> > >  Docs: man:libvirtd(8)
>> > >https://libvirt.org
>> > >  Main PID: 537 (libvirtd)
>> > > Tasks: 20 (limit: 32768)
>> > >CGroup: /system.slice/libvirtd.service
>> > >├─  537 /usr/sbin/libvirtd -l
>> > >├─12206 /usr/sbin/dnsmasq
>> > > --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro
>> > > --dhcp-script=/usr/libexec/libvirt_leaseshelper
>> > >└─12207 /usr/sbin/dnsmasq
>> > > --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro
>> > > --dhcp-script=/usr/libexec/libvirt_leaseshelper
>> > >
>> > > Mar 21 11:45:01 mtl1-apphst03.mt.pbt.com.mt libvirtd[537]: 2019-03-21
>> > > 11:45:01.168+: 566: info : libvirt version: 4.5.0, package:
>> > 10.el7_6.6
>> > > (CentOS BuildSystem <http://bugs.centos.org>, 2019-03-14-10:21:47,
>> > > x86-01.bsys.centos.org)
>> > > Mar 21 11:45:01 mtl1-apphst03.mt.pbt.com.mt libvirtd[537]: 2019-03-21
>> > > 11:45:01.168+: 566: info : hostname: mtl1-apphst03.mt.pbt.com.mt
>> > > Mar 21 11:45:01 mtl1-apphst03.mt.pbt.com.mt libvirtd[537]: 2019-03-21
>> > > 11:45:01.168+: 566: error : virFirewallApplyRuleDirect:709 :
>> internal
>> > > error: Failed to apply firewall rules /usr/sbin/iptables -w --table
>> nat
>> > > --insert POSTROUTING --source 192.168.122.0/24 '!' --destination
>> > > 192.168.122.0/24 --jump MASQUERADE: iptables v1.4.21: can't
>> initialize
>> > > iptables table `nat': Table does not exist (do you need to insmod?)
>> > > Mar 21 11:45:01 mtl1-apphst03.mt.pbt.com.mt libvirtd[537]: Perhaps
>> > > iptables
>> > > or your kernel needs to be upgraded.
>> > > Mar 21 11:45:01 mtl1-apphst03.mt.pbt.com.mt dnsmasq[12206]: read
>> > > /etc/hosts
>> > > - 4 addresses
>> > > Mar 21 11:45:01 mtl1-apphst03.mt.pbt.com.mt dnsmasq[12206]: read
>> > > /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses
>> > > Mar 21 11:45:01 mtl1-apphst03.mt.pbt.com.mt dnsmasq-dhcp[12206]: read
>> > > /var/lib/libvirt/dnsmasq/default.hostsfile
>> > > Mar 21 11:45:01 mtl1-apphst03.mt.pbt.com.mt libvirtd[537]: 2019-03-21
>> > > 11:45:01.354+: 566: warning : virSecurityManagerNew:189 :
>> Configured
>> > > security driver "none" disables default policy to create confined
>> guests
>> > > Mar 21 11:49:57 mtl1-apphst03.mt.pbt.com.mt libvirtd[537]: 2019-03-21
>> > > 11:49:57.354+: 542: warning : qemuDomainObjTaint:7521 : Domain
>> id=2
>> > > name='s-1-VM' uuid=1a06d3a7-4e3f-4cba-912f-74ae24569bac is tainted:
>> > > high-privileges
>> > > Mar 21 11:49:59 mtl1-apphst03.mt.pbt.com.mt libvirtd[537]: 2019-03-21
>> > > 11:49:59.402+: 540: warning : qemuDomainObjTaint:7521 : Domain
>> id=3
>> > > name='v-2-VM' uuid=af2a8342-cd9b-4b55-ba12-480634a31d65 is tainted:
>> > > high-privileges
>> > >
>> > >
>> > > What can be done about that ?
>> > >
>> >
>> >
>> > --
>> >
>> > Andrija Panić
>> >
>>
>
>
> --
>
> Andrija Panić
>


-- 

Andrija Panić

Re: cannot start system VMs: disaster after maintenance followup

[Andrija Panic]

Please use the one, updated specifically for CentOS 7 -
https://github.com/apache/cloudstack-documentation/blob/master/source/quickinstallationguide/qig.rst

And please avoid collocating KVM and MGMT on same server (especially in any
production-like system)

Please let me know if the guide above gives you problem - we had multiple
users explicitly following it - and successfully installed (with some minor
modification, which we committed back to that guide).

Thanks
Andrija

On Thu, 21 Mar 2019 at 16:34, Jevgeni Zolotarjov 
wrote:

> OS management - centos 7 (1810)
> OS hypervisor - centos 7 (1810)
>
> Basic zone - yes
> I am following this quide
>
> http://docs.cloudstack.apache.org/en/4.11.2.0/quickinstallationguide/qig.html
>
> Right now from scratch - management ans hypervisor on the same machine
> qemu - version 1.5.3
> libvirt - libvirt version: 4.5.0, package: 10.el7_6.6
>
> Basically - everything out of the box of clean centos install
>
>
>
>
> On Thu, Mar 21, 2019 at 5:08 PM Andrija Panic 
> wrote:
>
> > Hey Jevgeni,
> >
> > what OS mgmt, what OS hypervisor, what qemu/libvirt versions - still in
> > Basic Zone, SG ?
> >
> > Andrija
> >
> > On Thu, 21 Mar 2019 at 13:06, Jevgeni Zolotarjov  >
> > wrote:
> >
> > > I reinstalled cloudstack from scratch - everything
> > >
> > > But looks like I hit the same wall now
> > >
> > > In the last step of installation it cannot create system VMs.
> > >
> > > service libvirtd status -l
> > > gives me
> > > 
> > > ● libvirtd.service - Virtualization daemon
> > >Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled;
> > > vendor preset: enabled)
> > >Active: active (running) since Thu 2019-03-21 11:45:00 GMT; 18min
> ago
> > >  Docs: man:libvirtd(8)
> > >https://libvirt.org
> > >  Main PID: 537 (libvirtd)
> > > Tasks: 20 (limit: 32768)
> > >CGroup: /system.slice/libvirtd.service
> > >├─  537 /usr/sbin/libvirtd -l
> > >├─12206 /usr/sbin/dnsmasq
> > > --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro
> > > --dhcp-script=/usr/libexec/libvirt_leaseshelper
> > >└─12207 /usr/sbin/dnsmasq
> > > --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro
> > > --dhcp-script=/usr/libexec/libvirt_leaseshelper
> > >
> > > Mar 21 11:45:01 mtl1-apphst03.mt.pbt.com.mt libvirtd[537]: 2019-03-21
> > > 11:45:01.168+: 566: info : libvirt version: 4.5.0, package:
> > 10.el7_6.6
> > > (CentOS BuildSystem <http://bugs.centos.org>, 2019-03-14-10:21:47,
> > > x86-01.bsys.centos.org)
> > > Mar 21 11:45:01 mtl1-apphst03.mt.pbt.com.mt libvirtd[537]: 2019-03-21
> > > 11:45:01.168+: 566: info : hostname: mtl1-apphst03.mt.pbt.com.mt
> > > Mar 21 11:45:01 mtl1-apphst03.mt.pbt.com.mt libvirtd[537]: 2019-03-21
> > > 11:45:01.168+: 566: error : virFirewallApplyRuleDirect:709 :
> internal
> > > error: Failed to apply firewall rules /usr/sbin/iptables -w --table nat
> > > --insert POSTROUTING --source 192.168.122.0/24 '!' --destination
> > > 192.168.122.0/24 --jump MASQUERADE: iptables v1.4.21: can't initialize
> > > iptables table `nat': Table does not exist (do you need to insmod?)
> > > Mar 21 11:45:01 mtl1-apphst03.mt.pbt.com.mt libvirtd[537]: Perhaps
> > > iptables
> > > or your kernel needs to be upgraded.
> > > Mar 21 11:45:01 mtl1-apphst03.mt.pbt.com.mt dnsmasq[12206]: read
> > > /etc/hosts
> > > - 4 addresses
> > > Mar 21 11:45:01 mtl1-apphst03.mt.pbt.com.mt dnsmasq[12206]: read
> > > /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses
> > > Mar 21 11:45:01 mtl1-apphst03.mt.pbt.com.mt dnsmasq-dhcp[12206]: read
> > > /var/lib/libvirt/dnsmasq/default.hostsfile
> > > Mar 21 11:45:01 mtl1-apphst03.mt.pbt.com.mt libvirtd[537]: 2019-03-21
> > > 11:45:01.354+: 566: warning : virSecurityManagerNew:189 :
> Configured
> > > security driver "none" disables default policy to create confined
> guests
> > > Mar 21 11:49:57 mtl1-apphst03.mt.pbt.com.mt libvirtd[537]: 2019-03-21
> > > 11:49:57.354+: 542: warning : qemuDomainObjTaint:7521 : Domain id=2
> > > name='s-1-VM' uuid=1a06d3a7-4e3f-4cba-912f-74ae24569bac is tainted:
> > > high-privileges
> > > Mar 21 11:49:59 mtl1-apphst03.mt.pbt.com.mt libvirtd[537]: 2019-03-21
> > > 11:49:59.402+: 540: warning : qemuDomainObjTaint:7521 : Domain id=3
> > > name='v-2-VM' uuid=af2a8342-cd9b-4b55-ba12-480634a31d65 is tainted:
> > > high-privileges
> > >
> > >
> > > What can be done about that ?
> > >
> >
> >
> > --
> >
> > Andrija Panić
> >
>


-- 

Andrija Panić

Re: cannot start system VMs: disaster after maintenance followup

[Andrija Panic]

Hey Jevgeni,

what OS mgmt, what OS hypervisor, what qemu/libvirt versions - still in
Basic Zone, SG ?

Andrija

On Thu, 21 Mar 2019 at 13:06, Jevgeni Zolotarjov 
wrote:

> I reinstalled cloudstack from scratch - everything
>
> But looks like I hit the same wall now
>
> In the last step of installation it cannot create system VMs.
>
> service libvirtd status -l
> gives me
> 
> ● libvirtd.service - Virtualization daemon
>Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled;
> vendor preset: enabled)
>Active: active (running) since Thu 2019-03-21 11:45:00 GMT; 18min ago
>  Docs: man:libvirtd(8)
>https://libvirt.org
>  Main PID: 537 (libvirtd)
> Tasks: 20 (limit: 32768)
>CGroup: /system.slice/libvirtd.service
>├─  537 /usr/sbin/libvirtd -l
>├─12206 /usr/sbin/dnsmasq
> --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro
> --dhcp-script=/usr/libexec/libvirt_leaseshelper
>└─12207 /usr/sbin/dnsmasq
> --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro
> --dhcp-script=/usr/libexec/libvirt_leaseshelper
>
> Mar 21 11:45:01 mtl1-apphst03.mt.pbt.com.mt libvirtd[537]: 2019-03-21
> 11:45:01.168+: 566: info : libvirt version: 4.5.0, package: 10.el7_6.6
> (CentOS BuildSystem , 2019-03-14-10:21:47,
> x86-01.bsys.centos.org)
> Mar 21 11:45:01 mtl1-apphst03.mt.pbt.com.mt libvirtd[537]: 2019-03-21
> 11:45:01.168+: 566: info : hostname: mtl1-apphst03.mt.pbt.com.mt
> Mar 21 11:45:01 mtl1-apphst03.mt.pbt.com.mt libvirtd[537]: 2019-03-21
> 11:45:01.168+: 566: error : virFirewallApplyRuleDirect:709 : internal
> error: Failed to apply firewall rules /usr/sbin/iptables -w --table nat
> --insert POSTROUTING --source 192.168.122.0/24 '!' --destination
> 192.168.122.0/24 --jump MASQUERADE: iptables v1.4.21: can't initialize
> iptables table `nat': Table does not exist (do you need to insmod?)
> Mar 21 11:45:01 mtl1-apphst03.mt.pbt.com.mt libvirtd[537]: Perhaps
> iptables
> or your kernel needs to be upgraded.
> Mar 21 11:45:01 mtl1-apphst03.mt.pbt.com.mt dnsmasq[12206]: read
> /etc/hosts
> - 4 addresses
> Mar 21 11:45:01 mtl1-apphst03.mt.pbt.com.mt dnsmasq[12206]: read
> /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses
> Mar 21 11:45:01 mtl1-apphst03.mt.pbt.com.mt dnsmasq-dhcp[12206]: read
> /var/lib/libvirt/dnsmasq/default.hostsfile
> Mar 21 11:45:01 mtl1-apphst03.mt.pbt.com.mt libvirtd[537]: 2019-03-21
> 11:45:01.354+: 566: warning : virSecurityManagerNew:189 : Configured
> security driver "none" disables default policy to create confined guests
> Mar 21 11:49:57 mtl1-apphst03.mt.pbt.com.mt libvirtd[537]: 2019-03-21
> 11:49:57.354+: 542: warning : qemuDomainObjTaint:7521 : Domain id=2
> name='s-1-VM' uuid=1a06d3a7-4e3f-4cba-912f-74ae24569bac is tainted:
> high-privileges
> Mar 21 11:49:59 mtl1-apphst03.mt.pbt.com.mt libvirtd[537]: 2019-03-21
> 11:49:59.402+: 540: warning : qemuDomainObjTaint:7521 : Domain id=3
> name='v-2-VM' uuid=af2a8342-cd9b-4b55-ba12-480634a31d65 is tainted:
> high-privileges
>
>
> What can be done about that ?
>


-- 

Andrija Panić

Re: Disaster after maintenance

[Andrija Panic]

Hi Jevgeni,

I would perhaps consider you continue with plan B from your separate email
thread (root volumes --> create snapshots, convert snaps to template,
download template somewhere safe - for DATA volumes, also create snapshots,
then convert to volume and download it (or simply directly download
existing DATA volume if VM is stopped).
Once you are safe, and all templates, and VM volumes are safe, you are good
to reinstall.
Seriously, I'm not sure how to proceed via ML - if this was my own setup,
probably would be able to fix it...

In next installment, start with clean 4.11.2 (4.10 was never released as an
official release and was SERIOUSLY broken), or even 4.12 which has just
been released (will be in 1-2 days).
In this new installment, please dedicate a VM (or physical server) that
will host mgmt+DB+NFS (or even better separate NFS on different server
etc.) - but certainly do NOT collocate management components with KVM role.
When you build zone successfully, you can then import all templates and
upload all volumes (which you saved previously to some external place, web
server)
This will allow you to restore your VMs - possibly with just different IPs
versus original ones.

Considering failed DB upgrades and issues you see now, I assume your env,
might be severely broken at this point, and warrants starting from
scratch...

Hope that makes sense - so again, download all root and data volumes to
safe place (consider some petrol + matches fun) and then reinstall with a
fresh and shiny infra.

Alternatively, I would try to wipe all new zones (this takes some time and
certain steps) and then continue troubleshooting with failed-to-start VRs.

Cheers

On Wed, 20 Mar 2019 at 17:59, Jevgeni Zolotarjov 
wrote:

> It started with 4.10 and then gradually upgraded with all stops, when new
> releases were available.
>
>
> >>> Why do you have 3 zones in this installation - what is the setup ?
> >>> SSVM and CPVM (for whatever zone) are failing to be created...
> Its a result of attempts to create new zone and somehow move VMs to this
> new zone. These all are unsuccessful attempts.
> Before problem started there was 1 zone and There should be just 1 zone in
> reality.
>
>
> >>> yes, the VR can't be started, it get's timeout - in AGENT logs, I see
> that
> >>> it attemps to create a volume on primary storage...
> I guess this is the root cause. I checked, and primary storage is
> accessible via NFS share on both hosts. How to troubleshoot it?
>
>
> On Wed, Mar 20, 2019 at 6:29 PM Andrija Panic 
> wrote:
>
> > Hi,
> >
> > 2019-03-20 06:41:50,446 INFO  [c.c.u.DatabaseUpgradeChecker] (main:null)
> > (logid:) DB version = 4.10.0.0 Code Version = 4.11.2.0
> > 2019-03-20 06:41:50,447 DEBUG [c.c.u.DatabaseUpgradeChecker] (main:null)
> > (logid:) Running upgrade Upgrade41000to41100 to upgrade from
> > 4.10.0.0-4.11.0.0 to 4.11.0.0
> > fails due to
> > java.sql.SQLException: Error on rename of './cloud/ldap_trust_map' to
> > './cloud/#sql2-2f01-13d' (errno: 152)
> >
> > Then later...
> >
> > com.cloud.exception.InsufficientServerCapacityException: Unable to
> create a
> > deployment for VM[SecondaryStorageVm|s-734-VM]Scope=interface
> > com.cloud.dc.DataCenter; id=3
> > com.cloud.exception.InsufficientServerCapacityException: Unable to
> create a
> > deployment for VM[ConsoleProxy|v-733-VM]Scope=interface
> > com.cloud.dc.DataCenter; id=3
> >
> > 2019-03-20 15:02:39,113 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
> > (secstorage-1:ctx-059f87f3) (logid:cf6cf89a) Zone 1 is ready to launch
> > secondary storage VM
> > 2019-03-20 15:02:39,117 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
> > (secstorage-1:ctx-059f87f3) (logid:cf6cf89a) Zone 2 is not ready to
> launch
> > secondary storage VM yet
> > 2019-03-20 15:02:39,122 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
> > (secstorage-1:ctx-059f87f3) (logid:cf6cf89a) Zone 3 is ready to launch
> > secondary storage VM
> >
> > so did you start with clean 4.11.2 install, or was it upgraded one - I
> see
> > in logs an upgrade from DB version 4.10 to 4.11 was tried and failed...
> > Why do you have 3 zones in this installation - what is the setup ?
> > SSVM and CPVM (for whatever zone) are failing to be created...
> >
> > yes, the VR can't be started, it get's timeout - in AGENT logs, I see
> that
> > it attemps to create a volume on primary storage...
> >
> >
> > Also, for SSVM I got this one...
> > 2019-03-20 14:38:09,227 DEBUG [c.c.d.FirstFitPlanner]
> > (Work-Job-Executor-96:ctx-04c5c9f2 job-5120/job-6960 ctx-fde3d4d7)
> > (logid:49483c7a) No clusters found having

Re: Disaster after maintenance

[Andrija Panic]

Hi,

2019-03-20 06:41:50,446 INFO  [c.c.u.DatabaseUpgradeChecker] (main:null)
(logid:) DB version = 4.10.0.0 Code Version = 4.11.2.0
2019-03-20 06:41:50,447 DEBUG [c.c.u.DatabaseUpgradeChecker] (main:null)
(logid:) Running upgrade Upgrade41000to41100 to upgrade from
4.10.0.0-4.11.0.0 to 4.11.0.0
fails due to
java.sql.SQLException: Error on rename of './cloud/ldap_trust_map' to
'./cloud/#sql2-2f01-13d' (errno: 152)

Then later...

com.cloud.exception.InsufficientServerCapacityException: Unable to create a
deployment for VM[SecondaryStorageVm|s-734-VM]Scope=interface
com.cloud.dc.DataCenter; id=3
com.cloud.exception.InsufficientServerCapacityException: Unable to create a
deployment for VM[ConsoleProxy|v-733-VM]Scope=interface
com.cloud.dc.DataCenter; id=3

2019-03-20 15:02:39,113 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
(secstorage-1:ctx-059f87f3) (logid:cf6cf89a) Zone 1 is ready to launch
secondary storage VM
2019-03-20 15:02:39,117 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
(secstorage-1:ctx-059f87f3) (logid:cf6cf89a) Zone 2 is not ready to launch
secondary storage VM yet
2019-03-20 15:02:39,122 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
(secstorage-1:ctx-059f87f3) (logid:cf6cf89a) Zone 3 is ready to launch
secondary storage VM

so did you start with clean 4.11.2 install, or was it upgraded one - I see
in logs an upgrade from DB version 4.10 to 4.11 was tried and failed...
Why do you have 3 zones in this installation - what is the setup ?
SSVM and CPVM (for whatever zone) are failing to be created...

yes, the VR can't be started, it get's timeout - in AGENT logs, I see that
it attemps to create a volume on primary storage...


Also, for SSVM I got this one...
2019-03-20 14:38:09,227 DEBUG [c.c.d.FirstFitPlanner]
(Work-Job-Executor-96:ctx-04c5c9f2 job-5120/job-6960 ctx-fde3d4d7)
(logid:49483c7a) No clusters found having a host with enough capacity,
returning.

Andrija

On Wed, 20 Mar 2019 at 16:39, Jevgeni Zolotarjov 
wrote:

> Basic Zone - Yes
>
> router has been actually started/created on KVM side - not created, not
> started. Thats the main problem, I guess
>
> agent.log
> https://drive.google.com/open?id=1rATxHKqgNKo2kD23BtlrZy_9gFXC-Bq-
>
> management log
> https://drive.google.com/open?id=1H2jI0roeiWxtzReB8qV6QxDkNpaki99A
>
> >> Can you confirm your zone/pod/cluster/hosts are all in Enabled state,
> i.e.
> YES, all green
>
> >> Can you connect your both KVM hosts can access/mount both Primary and
> Secondary Storage
> YES. Double checked
>
> >>>Can you also explain your infrastructure - you said you have two hosts
> only, where does CloudStack management run?
> 2 hosts:
> host1: 192.168.1.14
> host2: 192.168.1.5
>
> Servers are standing next to each other - connected to the same switch
> Management server runs on the same physical server with host1
>
> I noticed, that Virtual router gets created after I try to start any of the
> existing guest VM
> Here are logs
> management:
> https://drive.google.com/open?id=1H2jI0roeiWxtzReB8qV6QxDkNpaki99A
>
> agent on host1:
> https://drive.google.com/open?id=1u8YHYIuyU2MA2UKY7G5z7q8p5XxU1zsy
>
> agent on host2:
> https://drive.google.com/open?id=1YzkCL-FmTgPva-QHHp5vTM5Nb3qAXxz4
>
> But this virtual router stays in Starting state forever and hence VMs do
> not start either.
>
> On Wed, Mar 20, 2019 at 2:49 PM Andrija Panic 
> wrote:
>
> > Just to confirm, you are using Basic Zone in CloudStack, right ?
> >
> > Can you confirm that router has been actually started/created on KVM
> side,
> > again, as requested please post logs (mgmt and agent - and note the time
> > around which you tried to start VR last time it partially succeeded) - we
> > can't guess what went wrong without logs.
> >
> > I would push more effort solving this one, instead of reinstalling - you
> > might hit the issue again and then it's no good.
> >
> > Can you confirm your zone/pod/cluster/hosts are all in Enabled state,
> i.e.
> > not disabled and hosts connected AND both SSVM and CPVM are
> > connectedUP/green
> > Is your dashboard in GUI all green - no issues there ?
> > Can you connect your both KVM hosts can access/mount both Primary and
> > Secondary Storage
> >
> > On Wed, 20 Mar 2019 at 13:15, Jevgeni Zolotarjov  >
> > wrote:
> >
> > > After dozen of attempts, the Virtual Router could finally be recreated.
> > But
> > > its in eternal Starting status, and console prompts it required upgrade
> > and
> > > Version is UNKNOWN
> > >
> > > It does not resolve the problem, I cannot move further form this point.
> > > Any hints?
> > >
> > > Or I am condemned to do reinstall clo

Re: Disaster after maintenance

[Andrija Panic]

Just to confirm, you are using Basic Zone in CloudStack, right ?

Can you confirm that router has been actually started/created on KVM side,
again, as requested please post logs (mgmt and agent - and note the time
around which you tried to start VR last time it partially succeeded) - we
can't guess what went wrong without logs.

I would push more effort solving this one, instead of reinstalling - you
might hit the issue again and then it's no good.

Can you confirm your zone/pod/cluster/hosts are all in Enabled state, i.e.
not disabled and hosts connected AND both SSVM and CPVM are
connectedUP/green
Is your dashboard in GUI all green - no issues there ?
Can you connect your both KVM hosts can access/mount both Primary and
Secondary Storage

On Wed, 20 Mar 2019 at 13:15, Jevgeni Zolotarjov 
wrote:

> After dozen of attempts, the Virtual Router could finally be recreated. But
> its in eternal Starting status, and console prompts it required upgrade and
> Version is UNKNOWN
>
> It does not resolve the problem, I cannot move further form this point.
> Any hints?
>
> Or I am condemned to do reinstall cloudstack from scratch?
>
> On Wed, Mar 20, 2019 at 11:08 AM Jevgeni Zolotarjov <
> j.zolotar...@gmail.com>
> wrote:
>
> > Under this defaultGuestNetwork, I go to Virtual Appliances. There is no
> > VMS - "no data to show"
> >
> > I dont have any network, other than this single default one.
> >
> > I've tried adding new network - Add guest network. But I am not able to
> do
> > so, cause in the wizard popup, it offers empty dropdown with Zones
> > selection. And this wizard doesnt not allow to go further without
> selecting
> > Zone
> >
> > On Wed, Mar 20, 2019 at 10:28 AM Andrija Panic 
> > wrote:
> >
> >> you need to delete/remove all VMs inside this network (tick the
> "Expunge"
> >> button during VM deletion - if you want to really delete the VMs) in
> order
> >> to be able to delete the network - OR simply attach this VM to another
> >> network, make this new network a DEFAULT one (NIC that is...), and then
> >> detach from old network - and then effectively your VM was "removed"
> from
> >> old network - after this you should be able to delete the old network. I
> >> assume some DB incosistencies perhaps, being the reason you can not
> >> restart
> >> the network.
> >>
> >> Did you try restarting some other Network - or deploying a new network,
> >> spin a VM in it, then again try to restart this new network - does it
> >> work ?
> >>
> >> Andrija
> >>
> >> On Wed, 20 Mar 2019 at 08:58, Jevgeni Zolotarjov <
> j.zolotar...@gmail.com>
> >> wrote:
> >>
> >> > >>>Stop mgmt,
> >> > >>>Stop all agents
> >> > >>>Restart libvirtd (and check libvirt logs afterwards)
> >> > >>>Start agents
> >> > >>>Start mgmt.
> >> >
> >> > I did that numerous time. Nothing really suspicious
> >> > I can see that systems VMs are running - both in cloudstack console
> and
> >> > with virsh list -all
> >> >
> >> > It is apparently problem with network.
> >> > Is there a way to force recreation of defaultGuestNetwork? or force
> >> > recreation of Virtual Router.
> >> > I am unable to delete network, which is supposed to rebuild network
> with
> >> > its router. Thats the issue
> >> >
> >> > The issue with libvirtd was, that eventually at some point it was
> >> updated
> >> > during 4 months of running, and not rebooted. It still worked. We had
> to
> >> > add listen_tcp = 1 for libvirtd to start working again.
> >> >
> >> > On Wed, Mar 20, 2019 at 9:49 AM Andrija Panic <
> andrija.pa...@gmail.com>
> >> > wrote:
> >> >
> >> > > As Sergey suggested... but i would also verify no libvirt issues or
> >> > storage
> >> > > pool issues - so perhaps:
> >> > >
> >> > > Stop mgmt,
> >> > > Stop all agents
> >> > > Restart libvirtd (and check libvirt logs afterwards)
> >> > > Start agents
> >> > > Start mgmt.
> >> > >
> >> > > What was originally issue with libvirtd ?
> >> > > That sounds fishy to me...
> >> > >
> >> > > Andrija
> >> > >
> >> > > On Wed, Mar 20, 2019, 02:15 Sergey Levitskiy 
> >> > wr

Re: Disaster after maintenance

[Andrija Panic]

you need to delete/remove all VMs inside this network (tick the "Expunge"
button during VM deletion - if you want to really delete the VMs) in order
to be able to delete the network - OR simply attach this VM to another
network, make this new network a DEFAULT one (NIC that is...), and then
detach from old network - and then effectively your VM was "removed" from
old network - after this you should be able to delete the old network. I
assume some DB incosistencies perhaps, being the reason you can not restart
the network.

Did you try restarting some other Network - or deploying a new network,
spin a VM in it, then again try to restart this new network - does it work ?

Andrija

On Wed, 20 Mar 2019 at 08:58, Jevgeni Zolotarjov 
wrote:

> >>>Stop mgmt,
> >>>Stop all agents
> >>>Restart libvirtd (and check libvirt logs afterwards)
> >>>Start agents
> >>>Start mgmt.
>
> I did that numerous time. Nothing really suspicious
> I can see that systems VMs are running - both in cloudstack console and
> with virsh list -all
>
> It is apparently problem with network.
> Is there a way to force recreation of defaultGuestNetwork? or force
> recreation of Virtual Router.
> I am unable to delete network, which is supposed to rebuild network with
> its router. Thats the issue
>
> The issue with libvirtd was, that eventually at some point it was updated
> during 4 months of running, and not rebooted. It still worked. We had to
> add listen_tcp = 1 for libvirtd to start working again.
>
> On Wed, Mar 20, 2019 at 9:49 AM Andrija Panic 
> wrote:
>
> > As Sergey suggested... but i would also verify no libvirt issues or
> storage
> > pool issues - so perhaps:
> >
> > Stop mgmt,
> > Stop all agents
> > Restart libvirtd (and check libvirt logs afterwards)
> > Start agents
> > Start mgmt.
> >
> > What was originally issue with libvirtd ?
> > That sounds fishy to me...
> >
> > Andrija
> >
> > On Wed, Mar 20, 2019, 02:15 Sergey Levitskiy 
> wrote:
> >
> > > select * from networks where removed is null;
> > > select * from vm_instance where id=87;
> > > select id,name from vm_instance where name like 'r%' and removed is
> null;
> > >
> > > Basically since the network offering is not redundant this error is
> only
> > > thrown when there is no router associated with your network. Usually
> > > management server restart tries to implement network again. Please
> > restart
> > > management server, save and share management server log.
> > >
> > >
> > >
> > >
> > > On 3/19/19, 3:31 PM, "Jevgeni Zolotarjov" 
> > wrote:
> > >
> > > >>>>Check network_offering table for  value in column
> > > redundant_router_service  for the network offering you use.
> > > in table network_offering_table all records have
> > > redundant_router_service =
> > > 0
> > >
> > > Can you also run the following:
> > > >>>select name, state, removed  from host where name like 'r%'
> > > returns zero rows - nothing
> > >
> > > >>>select * from domain_router;
> > > # id, element_id, public_mac_address, public_ip_address,
> > > public_netmask,
> > > guest_netmask, guest_ip_address, is_redundant_router, priority,
> > > redundant_state, stop_pending, role, template_version,
> > scripts_version,
> > > vpc_id, update_state
> > > '4', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0',
> > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 15:37:28
> UTC
> > > 2018',
> > > '57db7bd8118977a5f2cd3ef1c7503633\n', NULL, NULL
> > > '49', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0',
> > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 15:37:28
> UTC
> > > 2018',
> > > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL
> > > '73', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0',
> > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 15:37:28
> UTC
> > > 2018',
> > > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL
> > > '74', '1', NULL,

Re: Disaster after maintenance

[Andrija Panic]

 '2', '49', '204', 'Shared'
> '3', '73', '204', 'Shared'
> '4', '75', '204', 'Shared'
> '5', '76', '204', 'Shared'
> '6', '77', '204', 'Shared'
> '7', '80', '204', 'Shared'
> '8', '85', '204', 'Shared'
> '9', '86', '204', 'Shared'
> '10', '87', '204', 'Shared'
>
>
> On Wed, Mar 20, 2019 at 12:18 AM Sergey Levitskiy  >
> wrote:
>
> > Check network_offering table for  value in column
> > redundant_router_service  for the network offering you use.
> > Can you also run the following:
> > select name, state, removed  from host where name like 'r%'
> > select * from domain_router;
> > select * from router_network_ref;
> >
> > Cloudstack is supposed to recreate you VR. If it is not happening
> there is
> > something fundamentally wrong. I would advise to destroy your VR
> again.
> > Stop you management server. Rotate management server log and start it
> > again. If your VR doesn't start in few min, post your complete
> management
> > server log  and agent log again.
> >
> >
> >
> >
> > On 3/19/19, 2:56 PM, "Jevgeni Zolotarjov" 
> wrote:
> >
> > >>>Network offering needs to be change to non-redundant
> > How do I do that?
> >
> > On Tue, Mar 19, 2019 at 11:47 PM Sergey Levitskiy <
> serg...@hotmail.com
> > >
> > wrote:
> >
> > > Network offering needs to be change to non-redundant. Most
> likely
> > old bug
> > > is resurfaced.
> > > https://issues.apache.org/jira/browse/CLOUDSTACK-9024
> > >
> > >
> > >
> > > On 3/19/19, 2:19 PM, "Jevgeni Zolotarjov" <
> j.zolotar...@gmail.com>
> > wrote:
> > >
> > > Hello
> > >
> > > I did exactly like you suggested.
> > >
> > > After UPDATE on db, I can see the router in cloudstack
> console.
> > > But attempt to restart network fails.
> > > I get error:
> > > Resource [DataCenter:1] is unreachable: Can't find all
> necessary
> > > running
> > > routers!
> > >
> > > I rechecked agents on both servers. They look running OK.
> > >
> > > On Tue, Mar 19, 2019 at 10:00 PM Andrija Panic <
> > > andrija.pa...@gmail.com>
> > > wrote:
> > >
> > > > Ok, so:
> > > >
> > > > 1.BACKUP YOUR DB - in case there is issues, you will
> want to
> > restore
> > > > 2. I tried to reproduce your case on 4.11.2 (though
> didn't do
> > any
> > > > maintenance etc.) - and could not - I have artificially
> marked
> > > existing VR
> > > > as removed in DB - and tried to restart network and it
> worked
> > just
> > > fine.
> > > >
> > > > Let's try to "undelete" the VR for that network - though
> I
> > can't be
> > > sure if
> > > > this will work or not.
> > > >
> > > > Find VR based on the network UUID:
> > > >   SELECT * FROM vm_instance WHERE id IN (SELECT
> > instance_id from
> > > nics
> > > > where network_id IN (SELECT id FROM networks WHERE
> > > > UUID="65ca9a05-ff96-4563-ab9c-ffb610dc8b73"));
> > > > Obviously, replace this UUID with your network UUID.
> > > > SQL above will return all VMs (user VMs, VRs, etc,
> delete or
> > not...)
> > > that
> > > > have NIC inside the network with that UUID - look for the
> > r-XX-VM
> > > being a
> > > > router.
> > > &

Re: [RESULT][VOTE] Apache CloudStack 4.12.0.0

[Andrija Panic]

Good work Gabriel !

On Tue, Mar 19, 2019, 22:37 Gabriel Beims Bräscher 
wrote:

> Hi all,
>
> After 3 business days, the vote for CloudStack 4.12.0.0 *passes* with 4 PMC
> + 2 non-PMC votes.
>
> +1 (PMC / binding)
> * Wido den Hollander
> * Simon Weller
> * Rafael Weingärtner
> * Rohit Yadav
>
> +1 (nonbinding)
> * Gabriel Bräscher
> * Nicolas Vazquez
>
> 0
> none
>
> -1
> none
>
> Thanks to everyone participating.
>
> I will now prepare the release announcement to go out after 24 hours to
> give the mirrors time to catch up.
>
> Best regards,
> Gabriel
>

Re: cloudshell for individual users

[Andrija Panic]

Hi Richard,

No built-in way of doing so, but you could spin a new VM template and be
creative, possibly with user-data etc. It boils down, as you know, on
having CloudMonkey (we have just released a v6.0 writen in Go !) Available
to user...

Andrija

On Wed, Mar 20, 2019, 03:23 Richard Persaud 
wrote:

> Hello,
>
> How can I enable a "cloudshell" instance for each account/user so they are
> able to manage their VPCs via the CLI? It would be great to offer a
> cloudMonkey cloudshell instance - similar to how Azure and GCP offer CLI
> options.
>
> Regards,
>
> Richard Persaud
>
>

Re: Disaster after maintenance

[Andrija Panic]

Ok, so:

1.BACKUP YOUR DB - in case there is issues, you will want to restore
2. I tried to reproduce your case on 4.11.2 (though didn't do any
maintenance etc.) - and could not - I have artificially marked existing VR
as removed in DB - and tried to restart network and it worked just fine.

Let's try to "undelete" the VR for that network - though I can't be sure if
this will work or not.

Find VR based on the network UUID:
  SELECT * FROM vm_instance WHERE id IN (SELECT instance_id from nics
where network_id IN (SELECT id FROM networks WHERE
UUID="65ca9a05-ff96-4563-ab9c-ffb610dc8b73"));
Obviously, replace this UUID with your network UUID.
SQL above will return all VMs (user VMs, VRs, etc, delete or not...) that
have NIC inside the network with that UUID - look for the r-XX-VM being a
router.

Now that you have the name of the VR (i.e. r-4-VM) - you want to set 2
fields to some other value than what it might be already, as below:

Need to set field "state" to Stopped:
  UPDATE cloud.vm_instance SET state='Stopped' WHERE name="r-4-VM";
 (what state is set currently ???)

Need to set "removed" to NULL value:
  UPDATE cloud.vm_instance SET removed = NULL WHERE NAME="r-4-VM"; (is
there a removed date already set ???)

Obviously make sure that this VR is not running on any hypervisor and if
running, virsh destroy it...

AFTER above has been done (VR is considered to exist but stopped, from
CloudStack point of view) - try to RESTART the network (don't bother
deleteing it, since there are VMs in that network)

I'm not 100% positive this will fix your issue, but doesn't hurt to try

If above doesn't work - I would still take a look into agents and if they
are still connected - optionally, restart agents on both hosts once more
and confirm they are connected and up.

Let us know how it goes (and please backup DB once more before any actions
!)

Andrija


On Tue, 19 Mar 2019 at 20:41, Jevgeni Zolotarjov 
wrote:

> Yes.  Just a single network.
>
> On Tue, 19 Mar 2019, 21:39 Andrija Panic,  wrote:
>
> > Just one more clarification - this is Isolate single network (not Shared
> > Network, not VPC) ?
> >
> >
> >
> > On Tue, Mar 19, 2019, 19:36 Jevgeni Zolotarjov 
> > wrote:
> >
> > > Name defaultGuestNetwork
> > > ID 4ba834ed-48f3-468f-b667-9bb2d2c258f1
> > > Zone PBT zone 1
> > > Description defaultGuestNetwork
> > > Type Shared
> > > State Setup
> > > VPC ID N/A
> > > Persistent No
> > >
> > > On Tue, Mar 19, 2019 at 8:29 PM Andrija Panic  >
> > > wrote:
> > >
> > > > Share the network id and name as seen from GUI...
> > > >
> > > > On Tue, Mar 19, 2019, 19:27 Jevgeni Zolotarjov <
> j.zolotar...@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > >>> 1. Confirm please rolling restart is set to false please -
> double
> > > > check
> > > > > Double checked - It is set to false
> > > > >
> > > > > >>>>2. If so - do you know the name of VR which you deleted ? Is it
> > > last
> > > > > one
> > > > > >>> ever created - if so we can find it easily...
> > > > > I dont know the name
> > > > > Is there a way to fetch it from DB?
> > > > >
> > > > > On Tue, Mar 19, 2019 at 7:58 PM Andrija Panic <
> > andrija.pa...@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Right...still complaining on missing running routers.
> > > > > >
> > > > > > 1. Confirm please rolling restart is set to false please - double
> > > check
> > > > > > 2. If so - do you know the name of VR which you deleted ? Is it
> > last
> > > > one
> > > > > > ever created - if so we can find it easily...
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, Mar 19, 2019, 18:40 Jevgeni Zolotarjov <
> > > j.zolotar...@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Here is management server log
> > > > > > >
> > https://drive.google.com/open?id=1H2jI0roeiWxtzReB8qV6QxDkNpaki99A
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Mar 19, 2019 at 7:29 PM Andrija Panic <
> > > > andrija.pa...@gmail.com
> > > > > >
> > > > > > > wr

Re: Disaster after maintenance

[Andrija Panic]

Just one more clarification - this is Isolate single network (not Shared
Network, not VPC) ?



On Tue, Mar 19, 2019, 19:36 Jevgeni Zolotarjov 
wrote:

> Name defaultGuestNetwork
> ID 4ba834ed-48f3-468f-b667-9bb2d2c258f1
> Zone PBT zone 1
> Description defaultGuestNetwork
> Type Shared
> State Setup
> VPC ID N/A
> Persistent No
>
> On Tue, Mar 19, 2019 at 8:29 PM Andrija Panic 
> wrote:
>
> > Share the network id and name as seen from GUI...
> >
> > On Tue, Mar 19, 2019, 19:27 Jevgeni Zolotarjov 
> > wrote:
> >
> > > >>> 1. Confirm please rolling restart is set to false please - double
> > check
> > > Double checked - It is set to false
> > >
> > > >>>>2. If so - do you know the name of VR which you deleted ? Is it
> last
> > > one
> > > >>> ever created - if so we can find it easily...
> > > I dont know the name
> > > Is there a way to fetch it from DB?
> > >
> > > On Tue, Mar 19, 2019 at 7:58 PM Andrija Panic  >
> > > wrote:
> > >
> > > > Right...still complaining on missing running routers.
> > > >
> > > > 1. Confirm please rolling restart is set to false please - double
> check
> > > > 2. If so - do you know the name of VR which you deleted ? Is it last
> > one
> > > > ever created - if so we can find it easily...
> > > >
> > > >
> > > >
> > > > On Tue, Mar 19, 2019, 18:40 Jevgeni Zolotarjov <
> j.zolotar...@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Here is management server log
> > > > > https://drive.google.com/open?id=1H2jI0roeiWxtzReB8qV6QxDkNpaki99A
> > > > >
> > > > >
> > > > > On Tue, Mar 19, 2019 at 7:29 PM Andrija Panic <
> > andrija.pa...@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > I would also try to "undelete" VR in DB, but let's keep this as
> > last
> > > > > step.
> > > > > >
> > > > > > On Tue, Mar 19, 2019, 18:24 Andrija Panic <
> andrija.pa...@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Disclaimer: from mobile device.
> > > > > > >
> > > > > > > I don't see any special failure in agent log, except some long
> > > > running
> > > > > > > migration, timeout for graceful VM shutdown etc (and agent
> > restart)
> > > > > > >
> > > > > > > You did not send mgmt log after last network restart failure ?
> > > > > > >
> > > > > > > On Tue, Mar 19, 2019, 17:29 Jevgeni Zolotarjov <
> > > > j.zolotar...@gmail.com
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Under
> > > > > > >> Infrastructure / Hosts:
> > > > > > >> both hosts are enabled and green (Unsecure though)
> > > > > > >>
> > > > > > >> agent.log -
> > > > > > >>
> > > https://drive.google.com/open?id=1rATxHKqgNKo2kD23BtlrZy_9gFXC-Bq-
> > > > > > >>
> > > > > > >> I set network.rolling.restart to false now. Restarted
> management
> > > > > server
> > > > > > -
> > > > > > >> same problem - cannot restart not delete network
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> On Tue, Mar 19, 2019 at 5:48 PM Boris Stoyanov <
> > > > > > >> boris.stoya...@shapeblue.com>
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >> > Hi, you shouldn’t be worried about data as long as you use a
> > > > > separate
> > > > > > >> > shared storage.
> > > > > > >> >
> > > > > > >> > What does the cloudstack console say about your host? Is it
> > up?
> > > If
> > > > > > it’s
> > > > > > >> up
> > > > > > >> > then you should be able to deploy a VM on it.
> > > > > > >> >
> > > > > > >> > If not, you’ll need to investigate the 

Re: Disaster after maintenance

[Andrija Panic]

Share the network id and name as seen from GUI...

On Tue, Mar 19, 2019, 19:27 Jevgeni Zolotarjov 
wrote:

> >>> 1. Confirm please rolling restart is set to false please - double check
> Double checked - It is set to false
>
> >>>>2. If so - do you know the name of VR which you deleted ? Is it last
> one
> >>> ever created - if so we can find it easily...
> I dont know the name
> Is there a way to fetch it from DB?
>
> On Tue, Mar 19, 2019 at 7:58 PM Andrija Panic 
> wrote:
>
> > Right...still complaining on missing running routers.
> >
> > 1. Confirm please rolling restart is set to false please - double check
> > 2. If so - do you know the name of VR which you deleted ? Is it last one
> > ever created - if so we can find it easily...
> >
> >
> >
> > On Tue, Mar 19, 2019, 18:40 Jevgeni Zolotarjov 
> > wrote:
> >
> > > Here is management server log
> > > https://drive.google.com/open?id=1H2jI0roeiWxtzReB8qV6QxDkNpaki99A
> > >
> > >
> > > On Tue, Mar 19, 2019 at 7:29 PM Andrija Panic  >
> > > wrote:
> > >
> > > > I would also try to "undelete" VR in DB, but let's keep this as last
> > > step.
> > > >
> > > > On Tue, Mar 19, 2019, 18:24 Andrija Panic 
> > > wrote:
> > > >
> > > > > Disclaimer: from mobile device.
> > > > >
> > > > > I don't see any special failure in agent log, except some long
> > running
> > > > > migration, timeout for graceful VM shutdown etc (and agent restart)
> > > > >
> > > > > You did not send mgmt log after last network restart failure ?
> > > > >
> > > > > On Tue, Mar 19, 2019, 17:29 Jevgeni Zolotarjov <
> > j.zolotar...@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > >> Under
> > > > >> Infrastructure / Hosts:
> > > > >> both hosts are enabled and green (Unsecure though)
> > > > >>
> > > > >> agent.log -
> > > > >>
> https://drive.google.com/open?id=1rATxHKqgNKo2kD23BtlrZy_9gFXC-Bq-
> > > > >>
> > > > >> I set network.rolling.restart to false now. Restarted management
> > > server
> > > > -
> > > > >> same problem - cannot restart not delete network
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Tue, Mar 19, 2019 at 5:48 PM Boris Stoyanov <
> > > > >> boris.stoya...@shapeblue.com>
> > > > >> wrote:
> > > > >>
> > > > >> > Hi, you shouldn’t be worried about data as long as you use a
> > > separate
> > > > >> > shared storage.
> > > > >> >
> > > > >> > What does the cloudstack console say about your host? Is it up?
> If
> > > > it’s
> > > > >> up
> > > > >> > then you should be able to deploy a VM on it.
> > > > >> >
> > > > >> > If not, you’ll need to investigate the reason cloudstack-agent
> is
> > > not
> > > > >> > connected, is the agent running? If not just start it and see if
> > it
> > > > >> > connects.
> > > > >> >
> > > > >> > If it’s running but still not connected please share the
> > agent.log.
> > > > >> >
> > > > >> > If nothing helps at all you can remove the host and add it
> again,
> > > but
> > > > >> > let’s try to troubleshoot first.
> > > > >> >
> > > > >> > Bobby.
> > > > >> >
> > > > >> >
> > > > >> > boris.stoya...@shapeblue.com
> > > > >> > www.shapeblue.com
> > > > >> > Amadeus House, Floral Street, London  WC2E 9DPUK
> > > > >> > @shapeblue
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > > On 19 Mar 2019, at 17:29, Jevgeni Zolotarjov <
> > > > j.zolotar...@gmail.com>
> > > > >> > wrote:
> > > > >> > >
> > > > >> > > Guys, please help with it.
> > > > >> > > What can be done here?
> > > > >> &

Re: Disaster after maintenance

[Andrija Panic]

Right...still complaining on missing running routers.

1. Confirm please rolling restart is set to false please - double check
2. If so - do you know the name of VR which you deleted ? Is it last one
ever created - if so we can find it easily...



On Tue, Mar 19, 2019, 18:40 Jevgeni Zolotarjov 
wrote:

> Here is management server log
> https://drive.google.com/open?id=1H2jI0roeiWxtzReB8qV6QxDkNpaki99A
>
>
> On Tue, Mar 19, 2019 at 7:29 PM Andrija Panic 
> wrote:
>
> > I would also try to "undelete" VR in DB, but let's keep this as last
> step.
> >
> > On Tue, Mar 19, 2019, 18:24 Andrija Panic 
> wrote:
> >
> > > Disclaimer: from mobile device.
> > >
> > > I don't see any special failure in agent log, except some long running
> > > migration, timeout for graceful VM shutdown etc (and agent restart)
> > >
> > > You did not send mgmt log after last network restart failure ?
> > >
> > > On Tue, Mar 19, 2019, 17:29 Jevgeni Zolotarjov  >
> > > wrote:
> > >
> > >> Under
> > >> Infrastructure / Hosts:
> > >> both hosts are enabled and green (Unsecure though)
> > >>
> > >> agent.log -
> > >> https://drive.google.com/open?id=1rATxHKqgNKo2kD23BtlrZy_9gFXC-Bq-
> > >>
> > >> I set network.rolling.restart to false now. Restarted management
> server
> > -
> > >> same problem - cannot restart not delete network
> > >>
> > >>
> > >>
> > >> On Tue, Mar 19, 2019 at 5:48 PM Boris Stoyanov <
> > >> boris.stoya...@shapeblue.com>
> > >> wrote:
> > >>
> > >> > Hi, you shouldn’t be worried about data as long as you use a
> separate
> > >> > shared storage.
> > >> >
> > >> > What does the cloudstack console say about your host? Is it up? If
> > it’s
> > >> up
> > >> > then you should be able to deploy a VM on it.
> > >> >
> > >> > If not, you’ll need to investigate the reason cloudstack-agent is
> not
> > >> > connected, is the agent running? If not just start it and see if it
> > >> > connects.
> > >> >
> > >> > If it’s running but still not connected please share the agent.log.
> > >> >
> > >> > If nothing helps at all you can remove the host and add it again,
> but
> > >> > let’s try to troubleshoot first.
> > >> >
> > >> > Bobby.
> > >> >
> > >> >
> > >> > boris.stoya...@shapeblue.com
> > >> > www.shapeblue.com
> > >> > Amadeus House, Floral Street, London  WC2E 9DPUK
> > >> > @shapeblue
> > >> >
> > >> >
> > >> >
> > >> > > On 19 Mar 2019, at 17:29, Jevgeni Zolotarjov <
> > j.zolotar...@gmail.com>
> > >> > wrote:
> > >> > >
> > >> > > Guys, please help with it.
> > >> > > What can be done here?
> > >> > > There is too much valuable data.
> > >> > >
> > >> > > On Tue, Mar 19, 2019 at 4:21 PM Jevgeni Zolotarjov <
> > >> > j.zolotar...@gmail.com>
> > >> > > wrote:
> > >> > >
> > >> > >> Tried that just now and got error:
> > >> > >> Resource [DataCenter:1] is unreachable: Can't find all necessary
> > >> running
> > >> > >> routers!
> > >> > >>
> > >> > >> In the log I see:
> > >> > >> =
> > >> > >>
> > >> > >> 2019-03-19 14:20:39,644 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> > >> > >> (API-Job-Executor-4:ctx-7b6b69eb job-5093 ctx-9be30648)
> > >> (logid:265a6099)
> > >> > >> Restarting network 204...
> > >> > >> 2019-03-19 14:20:39,645 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> > >> > >> (API-Job-Executor-4:ctx-7b6b69eb job-5093 ctx-9be30648)
> > >> (logid:265a6099)
> > >> > >> Performing rolling restart of routers of network
> Ntwk[204|Guest|6]
> > >> > >> 2019-03-19 14:20:39,649 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> > >> > >> (API-Job-Executor-4:ctx-7b6b69eb job-5093 ctx-9be30648)
> > >> (logid:265a6099)
> > >> > >> Asking VirtualRouter 

Re: Disaster after maintenance

[Andrija Panic]

I would also try to "undelete" VR in DB, but let's keep this as last step.

On Tue, Mar 19, 2019, 18:24 Andrija Panic  wrote:

> Disclaimer: from mobile device.
>
> I don't see any special failure in agent log, except some long running
> migration, timeout for graceful VM shutdown etc (and agent restart)
>
> You did not send mgmt log after last network restart failure ?
>
> On Tue, Mar 19, 2019, 17:29 Jevgeni Zolotarjov 
> wrote:
>
>> Under
>> Infrastructure / Hosts:
>> both hosts are enabled and green (Unsecure though)
>>
>> agent.log -
>> https://drive.google.com/open?id=1rATxHKqgNKo2kD23BtlrZy_9gFXC-Bq-
>>
>> I set network.rolling.restart to false now. Restarted management server -
>> same problem - cannot restart not delete network
>>
>>
>>
>> On Tue, Mar 19, 2019 at 5:48 PM Boris Stoyanov <
>> boris.stoya...@shapeblue.com>
>> wrote:
>>
>> > Hi, you shouldn’t be worried about data as long as you use a separate
>> > shared storage.
>> >
>> > What does the cloudstack console say about your host? Is it up? If it’s
>> up
>> > then you should be able to deploy a VM on it.
>> >
>> > If not, you’ll need to investigate the reason cloudstack-agent is not
>> > connected, is the agent running? If not just start it and see if it
>> > connects.
>> >
>> > If it’s running but still not connected please share the agent.log.
>> >
>> > If nothing helps at all you can remove the host and add it again, but
>> > let’s try to troubleshoot first.
>> >
>> > Bobby.
>> >
>> >
>> > boris.stoya...@shapeblue.com
>> > www.shapeblue.com
>> > Amadeus House, Floral Street, London  WC2E 9DPUK
>> > @shapeblue
>> >
>> >
>> >
>> > > On 19 Mar 2019, at 17:29, Jevgeni Zolotarjov 
>> > wrote:
>> > >
>> > > Guys, please help with it.
>> > > What can be done here?
>> > > There is too much valuable data.
>> > >
>> > > On Tue, Mar 19, 2019 at 4:21 PM Jevgeni Zolotarjov <
>> > j.zolotar...@gmail.com>
>> > > wrote:
>> > >
>> > >> Tried that just now and got error:
>> > >> Resource [DataCenter:1] is unreachable: Can't find all necessary
>> running
>> > >> routers!
>> > >>
>> > >> In the log I see:
>> > >> =
>> > >>
>> > >> 2019-03-19 14:20:39,644 DEBUG [o.a.c.e.o.NetworkOrchestrator]
>> > >> (API-Job-Executor-4:ctx-7b6b69eb job-5093 ctx-9be30648)
>> (logid:265a6099)
>> > >> Restarting network 204...
>> > >> 2019-03-19 14:20:39,645 DEBUG [o.a.c.e.o.NetworkOrchestrator]
>> > >> (API-Job-Executor-4:ctx-7b6b69eb job-5093 ctx-9be30648)
>> (logid:265a6099)
>> > >> Performing rolling restart of routers of network Ntwk[204|Guest|6]
>> > >> 2019-03-19 14:20:39,649 DEBUG [o.a.c.e.o.NetworkOrchestrator]
>> > >> (API-Job-Executor-4:ctx-7b6b69eb job-5093 ctx-9be30648)
>> (logid:265a6099)
>> > >> Asking VirtualRouter to implemenet Ntwk[204|Guest|6]
>> > >> 2019-03-19 14:20:39,658 ERROR [c.c.a.ApiAsyncJobDispatcher]
>> > >> (API-Job-Executor-4:ctx-7b6b69eb job-5093) (logid:265a6099)
>> Unexpected
>> > >> exception while executing
>> > >> org.apache.cloudstack.api.command.user.network.RestartNetworkCmd
>> > >> com.cloud.exception.ResourceUnavailableException: Resource
>> > [DataCenter:1]
>> > >> is unreachable: Can't find all necessary running routers!
>> > >>at
>> > >>
>> >
>> com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:243)
>> > >>at
>> > >>
>> >
>> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElements(NetworkOrchestrator.java:1203)
>> > >>at
>> > >>
>> >
>> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.rollingRestartRouters(NetworkOrchestrator.java:2948)
>> > >>at
>> > >>
>> >
>> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:2853)
>> > >>at
>> > >>
>> >
>> com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java

Re: Disaster after maintenance

[Andrija Panic]

ework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
> > >>at
> > >>
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> > >>at
> > >>
> >
> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
> > >>at
> > >>
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
> > >>at
> > >>
> >
> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
> > >>at
> > >>
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
> > >>at
> > >>
> >
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
> > >>at
> > >>
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
> > >>at
> > >>
> >
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
> > >>at com.sun.proxy.$Proxy229.restartNetwork(Unknown Source)
> > >>at
> > >>
> >
> org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:99)
> > >>at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
> > >>at
> > >>
> >
> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
> > >>at
> > >>
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
> > >>at
> > >>
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> > >>at
> > >>
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> > >>    at
> > >>
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> > >>at
> > >>
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> > >>at
> > >>
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> > >>at
> > >>
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
> > >>at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
> > >> Source)
> > >>at java.util.concurrent.FutureTask.run(Unknown Source)
> > >>at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
> > >> Source)
> > >>at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> > >> Source)
> > >>at java.lang.Thread.run(Unknown Source)
> > >> 2019-03-19 14:20:39,659 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > >> (API-Job-Executor-4:ctx-7b6b69eb job-5093) (logid:265a6099) Complete
> > async
> > >> job-5093, jobStatus: FAILED, resultCode: 530, result:
> > >>
> >
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Resource
> > >> [DataCenter:1] is un
> > >>
> > >> On Tue, Mar 19, 2019 at 4:19 PM Andrija Panic <
> > andrija.pa...@shapeblue.com>
> > >> wrote:
> > >>
> > >>> ​​
> > >>> Your network can't be deleted due to "Can't delete the network, not
> all
> > >>> user vms are expunged. Vm
> > >>> VM[User|i-2-11-VM] is in Stopped state" - which is fine.
> > >>>
> > >>> You should be able to just start the user VM - but if you have
> actually
> > >>> delete the VR itself, then just do Network restart with "cleanup" and
> > it
> > >>> will recreate a new VR, after which you should be able to start the
> VM.
> > >>>
> > >>> Andrija
> > >>>
> > >>> andrija.pa...@shapeblue.com
> > >>> www.shapeblue.com
> > >&

Re: Disaster after maintenance

[Andrija Panic]

tiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
> >>at
> >>
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
> >>at com.sun.proxy.$Proxy229.restartNetwork(Unknown Source)
> >>at
> >>
> org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:99)
> >>at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
> >>at
> >>
> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
> >>at
> >>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
> >>at
> >>
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> >>at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> >>at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> >>at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> >>at
> >>
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> >>at
> >>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
> >>at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
> >> Source)
> >>at java.util.concurrent.FutureTask.run(Unknown Source)
> >>at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
> >> Source)
> >>at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> >> Source)
> >>at java.lang.Thread.run(Unknown Source)
> >> 2019-03-19 14:20:39,659 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >> (API-Job-Executor-4:ctx-7b6b69eb job-5093) (logid:265a6099) Complete
> async
> >> job-5093, jobStatus: FAILED, resultCode: 530, result:
> >>
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Resource
> >> [DataCenter:1] is un
> >>
> >> On Tue, Mar 19, 2019 at 4:19 PM Andrija Panic <
> andrija.pa...@shapeblue.com>
> >> wrote:
> >>
> >>> ​​
> >>> Your network can't be deleted due to "Can't delete the network, not all
> >>> user vms are expunged. Vm
> >>> VM[User|i-2-11-VM] is in Stopped state" - which is fine.
> >>>
> >>> You should be able to just start the user VM - but if you have actually
> >>> delete the VR itself, then just do Network restart with "cleanup" and
> it
> >>> will recreate a new VR, after which you should be able to start the VM.
> >>>
> >>> Andrija
> >>>
> >>> andrija.pa...@shapeblue.com
> >>> www.shapeblue.com
> >>> Amadeus House, Floral Street, London  WC2E 9DPUK
> >>> @shapeblue
> >>>
> >>>
> >>>
> >>>
> >>> -Original Message-
> >>> From: Jevgeni Zolotarjov 
> >>> Sent: 19 March 2019 15:10
> >>> To: users@cloudstack.apache.org
> >>> Subject: Re: Disaster after maintenance
> >>>
> >>> I mean I cannot delete network: In the management server log I see
> >>>
> >>> ==
> >>> 019-03-19 14:06:36,316 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>> (API-Job-Executor-1:ctx-1c0fd4dc job-5090) (logid:c734edfc) Executing
> >>> AsyncJobVO {id:5090, userId: 2, accountId: 2, instanceType: Network,
> >>> instanceId: 204, cmd:
> >>> org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd,
> cmdInfo:
> >>>
> >>>
> {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"2641","id":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","ctxDetails":"{\"interface
> >>>
> >>>
> com.cloud.network.Network\":\"4ba834ed-48f3-468f-b667-9bb2d2c258f1\"}","ctxAccountId":"2","uuid":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","cmdEventType":"NETWORK.DELETE","_":"1553004396247"},
> >>>

RE: Disaster after maintenance

[Andrija Panic]

​​
Your network can't be deleted due to "Can't delete the network, not all user 
vms are expunged. Vm
VM[User|i-2-11-VM] is in Stopped state" - which is fine.

You should be able to just start the user VM - but if you have actually delete 
the VR itself, then just do Network restart with "cleanup" and it will recreate 
a new VR, after which you should be able to start the VM.

Andrija

andrija.pa...@shapeblue.com 
www.shapeblue.com
Amadeus House, Floral Street, London  WC2E 9DPUK
@shapeblue
  
 


-Original Message-
From: Jevgeni Zolotarjov  
Sent: 19 March 2019 15:10
To: users@cloudstack.apache.org
Subject: Re: Disaster after maintenance

I mean I cannot delete network: In the management server log I see

==
019-03-19 14:06:36,316 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] 
(API-Job-Executor-1:ctx-1c0fd4dc job-5090) (logid:c734edfc) Executing 
AsyncJobVO {id:5090, userId: 2, accountId: 2, instanceType: Network,
instanceId: 204, cmd:
org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd, cmdInfo:
{"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"2641","id":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","ctxDetails":"{\"interface
com.cloud.network.Network\":\"4ba834ed-48f3-468f-b667-9bb2d2c258f1\"}","ctxAccountId":"2","uuid":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","cmdEventType":"NETWORK.DELETE","_":"1553004396247"},
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
result: null, initMsid: 264216221068220, completeMsid: null, lastUpdated:
null, lastPolled: null, created: null}
2019-03-19 14:06:36,351 WARN  [o.a.c.e.o.NetworkOrchestrator] 
(API-Job-Executor-1:ctx-1c0fd4dc job-5090 ctx-134954fa) (logid:c734edfc) Can't 
delete the network, not all user vms are expunged. Vm VM[User|i-2-11-VM] is in 
Stopped state
2019-03-19 14:06:36,356 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] 
(API-Job-Executor-1:ctx-1c0fd4dc job-5090) (logid:c734edfc) Complete async 
job-5090, jobStatus: FAILED, resultCode: 530, result:
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
to delete network"}
==


I deleted a router, expecting it to be recreated on deleting network. But I am 
unable to delete network because of above error

On Tue, Mar 19, 2019 at 3:58 PM Jevgeni Zolotarjov 
wrote:

> I've managed to make libvirtd running
> Now cloudstack console shows both hosts - running
>
> But now as I have removed network, VMs are unable to start.
>
> How can I recreate the network now?
>
> On Tue, Mar 19, 2019 at 3:14 PM Ivan Kudryavtsev 
> 
> wrote:
>
>> Jevgeniy, it may be a documentation bug. Take s look:
>> https://github.com/apache/cloudstack-documentation/pull/27/files
>>
>> вт, 19 мар. 2019 г., 9:09 Jevgeni Zolotarjov :
>>
>> > That's it - libvirtd failed to start on second host.
>> > Tried restarting, but it does not start.
>> >
>> >
>> > >> Do you have some NUMA constraints or anything which requires
>> particular
>> > RAM configuration?
>> > No
>> >
>> >  libvirtd.service - Virtualization daemon
>> >Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; 
>> > enabled; vendor preset: enabled)
>> >Active: failed (Result: start-limit) since Tue 2019-03-19 
>> > 13:03:07
>> GMT;
>> > 12s ago
>> >  Docs: man:libvirtd(8)
>> >https://libvirt.org
>> >   Process: 892 ExecStart=/usr/sbin/libvirtd $LIBVIRTD_ARGS 
>> > (code=exited,
>> > status=1/FAILURE)
>> >  Main PID: 892 (code=exited, status=1/FAILURE)
>> > Tasks: 19 (limit: 32768)
>> >CGroup: /system.slice/libvirtd.service
>> >├─11338 /usr/sbin/libvirtd -d -l
>> >├─11909 /usr/sbin/dnsmasq 
>> > --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro 
>> > --dhcp-script=/usr/libexec/libvirt_leaseshelper
>> >└─11910 /usr/sbin/dnsmasq 
>> > --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro 
>> > --dhcp-script=/usr/libexec/libvirt_leaseshelper
>> >
>> > Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: Failed to 
>> > start Virtualization daemon.
>> > Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: Unit 
>> > libvirtd.service entered failed state.
>> > Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]:
>> libvirtd.service
>> > failed.
>> > Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]:
>> libvirtd.service
>> > holdoff time over, scheduling restart.
>> > Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: Stopped 
>> > Virtualization daemon.
>> > Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: start 
>> > request repeated too quickly for libvirtd.service Mar 19 13:03:07 
>> > mtl1-apphst04.mt.pbt.com.mt systemd[1]: Failed to start 
>> > Virtualization daemon.
>> > Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: Unit 
>> > libvirtd.service entered failed state.
>> > Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]:
>> libvirtd.servic

RE: CloudStack on VirtualBox - NFS failed

[Andrija Panic]

Hi Larry,

As you might already know, this has nothing to do with CloudStack itself, and I 
would advice actually reading the system logs to understand why the basic 
service failed to start.

Best,
Anrija

andrija.pa...@shapeblue.com 
www.shapeblue.com
Amadeus House, Floral Street, London  WC2E 9DPUK
@shapeblue
  
 


-Original Message-
From: Larry Sevilla  
Sent: 15 March 2019 02:08
To: users@cloudstack.apache.org
Subject: CloudStack on VirtualBox - NFS failed

Attn: ShapeBlue

I'm trying to install CloudStack using VirtualBox using:
https://www.shapeblue.com/virtualbox-test-env/
(not sure if this is still applicable, more than 4 years past)

I used the latest CentOS 6.10.

Starting NFS failed.

# service nfs start
Starting NFS services:   [OK]
Starting mountd:
[FAILED]
Starting daemon:
svc: failed to register nfsdv2 RPC service (errno 111).
svc: failed to register nfsaclv2 RPC service (errno 111).
nfsd: last server has exited, flushing export cache
rpc.nfsd: writing fd to kernel failed: errno 111 (Connection refused)
svc: failed to register nfsdv2 RPC service (errno 97).
svc: failed to register nfsaclv2 RPC service (errno 97).
nfsd: last server has exited, flushing export cache
rpc.nfsd: unable to set any sockets for nfsd

[FAILED]

Re: Limited network upload rate

[Andrija Panic]

Hi,

please test with smaller values, to confirm that the outside server can
handle it - 500 and 1000 Mbps might be just too muc (test with 10-50 range,
to make sense)

For KVM, if nothing has changed recently, the way it worked was following:

VM will inherit network rate from the compute offering - afaik all NICs (if
VM is attached to more than 1 network) - but if you DON'T set network rate
on compute offering, then limits are inherited (on VM level) from global
settings (can't be sure which one of the two existing)

VR (all it's NICs) will inherit network rate in similar fashion, if you set
it on Service Offering level, otherwise again they are inherited from
global settings.

For KVM specifically, you should see same download/upload rate in general.

Andrija


On Mon, 18 Mar 2019 at 16:37, Fariborz Navidan 
wrote:

> I am using KVM
>
> On Mon, Mar 18, 2019 at 3:11 PM Melanie Desaive <
> m.desa...@heinlein-support.de> wrote:
>
> > Hi Fariborz,
> >
> > what Hypervisor are you using? I tested the network limits for
> > XenServer some months ago and think to remember, that I learned there
> > are differences how upload and download rates are realized.
> >
> > If you are interested, I could look into it once again and share my
> > findings with you.
> >
> > Greetings,
> >
> > Melanie
> >
> > Am Freitag, den 15.03.2019, 21:41 +0330 schrieb Fariborz Navidan:
> > > Hello,
> > >
> > > My server has 1Gbps network connectivity with no limits and I have
> > > set
> > > 500Mbps network rate for the service offering. VMs deployed with this
> > > service offering seems to have 500Mbps download speed capacity,
> > > however
> > > looks like their upload speed rate is limited to 100Mbps by the
> > > virtual
> > > router. Also I have network throttling rate to 1 Mbps in the
> > > zone's
> > > settings.
> > >
> > > Any idea what's limiting the upload speeds?
> > >
> > > Best Regards
> > --
> > --
> > Heinlein Support GmbH
> > Schwedter Str. 8/9b, 10119 Berlin
> >
> > https://www.heinlein-support.de
> >
> > Tel: 030 / 40 50 51 - 62
> > Fax: 030 / 40 50 51 - 19
> >
> > Amtsgericht Berlin-Charlottenburg - HRB 93818 B
> > Geschäftsführer: Peer Heinlein - Sitz: Berlin
> >
> >
>


-- 

Andrija Panić

Re: Effect of disk caching on performance

[Andrija Panic]

https://doc.opensuse.org/documentation/leap/virtualization/html/book.virt/cha.cachemodes.html

On Wed, 13 Mar 2019 at 15:48, Fariborz Navidan 
wrote:

> What is difference between NONE and WRITE THROUGH?
>
> On Wed, Mar 13, 2019 at 7:10 PM Andrija Panic 
> wrote:
>
> > Not sure how this directly has anything to do with CloudStack - it's
> > standalone Hypervisor thing :)
> >
>


-- 

Andrija Panić

Re: Effect of disk caching on performance

[Andrija Panic]

Not sure how this directly has anything to do with CloudStack - it's
standalone Hypervisor thing :)

RE: New VP of CloudStack: Paul Angus

[Andrija Panic]

Congrats Paul!

Mike thanks for your effort over the last year !

andrija.pa...@shapeblue.com 
www.shapeblue.com
Amadeus House, Floral Street, London  WC2E 9DPUK
@shapeblue
  
 


-Original Message-
From: David Merrill  
Sent: 11 March 2019 16:33
To: users@cloudstack.apache.org
Subject: Re: New VP of CloudStack: Paul Angus

Congratulations Paul!

David Merrill
Senior Systems Engineer,
Managed and Private/Hybrid Cloud Services OTELCO
92 Oak Street, Portland ME 04101
office 207.772.5678  www.otelco.com 
/business/managed-services



Confidentiality Message
The information contained in this e-mail transmission may be confidential and 
legally privileged. If you are not the intended recipient, you are notified 
that any dissemination, distribution, copying or other use of this information, 
including attachments, is prohibited. If you received this message in error, 
please call me at 207.772.5678  so this error can be 
corrected.
 

On 3/11/19, 11:16 AM, "Tutkowski, Mike"  wrote:

Hi everyone,

As you may know, the role of VP of CloudStack (Chair of the CloudStack PMC) 
has a one-year term. My term has now come and gone.

I’m happy to announce that the CloudStack PMC has elected Paul Angus as our 
new VP of CloudStack.

As many already know, Paul has been an active member of the CloudStack 
Community for over six years now. I’ve worked with Paul on and off throughout 
much of that time and I believe he’ll be a great fit for this role.

Please join me in welcoming Paul as the new VP of Apache CloudStack!

Thanks,
Mike

Re: Cloudstack is unable to create new VM

[Andrija Panic]

It was enough to just stop VMs and start again (assuming you change
cpu.overprovisioning on the cluster level - i.e. settings tab of the actual
cluster). If global settings changed, then yes, restart of mgmt server and
again stop/start existing VMs, but definitely NOT removing and re-adding
the host.

Glad you have it solved.

On Sun, Mar 10, 2019, 17:29 Fariborz Navidan  wrote:

> Hello All,
>
> I solved the issue. When you have running VMs, after changing
> cpu.overprovising.factor and restart management server, you need to stop
> all VMs, put the host in Maintenance mode and remove it. The host will be
> automatically re-added to the cluster and you will get additional CPU
> capacity as I wanted.
>
> Thank you for all your ideas.
>
> Best Regards
>
> On Sun, Mar 10, 2019 at 7:18 PM Andrija Panic 
> wrote:
>
> > Please send compete log to pastebin.com or such...and link here.
> >
> > On Sun, Mar 10, 2019, 14:36 Rafael Weingärtner <
> > rafaelweingart...@gmail.com>
> > wrote:
> >
> > > Ah, this happens for a simple reason.
> > >
> > > When you allocate a VM with over-provision of 1, you are charging by
> > that,
> > > and later if you change the over-provisioning to two; what would you
> > expect
> > > to happen with that VM that was allocated when over-provisioning was
> set
> > to
> > > 1? ACS is doing that do maintain the resources originally allocated by
> > the
> > > VM. If you want to change this, you need to restart the VMs.
> > >
> > > On Sun, Mar 10, 2019 at 10:19 AM Fariborz Navidan <
> mdvlinqu...@gmail.com
> > >
> > > wrote:
> > >
> > > > I guess the issue is that allocated cpu is also grown by
> > overprovisioning
> > > > factor such thatusage percentage remains the same. So, by increasing
> > cpu
> > > > overprovisiong factor you don't gain access to un-utilized cpu
> > capacity.
> > > My
> > > > real cpu utilization on the host is 31% but 84.82% is allocated and
> by
> > > > increasing the factor, the usage is also increased!
> > > >
> > > > On Sun, Mar 10, 2019 at 4:40 PM Fariborz Navidan <
> > mdvlinqu...@gmail.com>
> > > > wrote:
> > > >
> > > > > Even with cpu overprovisioning  factor of 20 set on both cluster
> and
> > > > > global settings, still I cannot deploy new  VMs. It show 672GHz in
> > > > > dashboardinstead of 33.6GHz (actual capacity) but still I get in
> the
> > > log
> > > > > "The host is in avoid list".
> > > > >
> > > > > Please advise.
> > > > > Thanks
> > > > >
> > > > > On Sun, Mar 10, 2019 at 4:04 PM Andrija Panic <
> > andrija.pa...@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > >> Cpu.overprovisioning.ratio or such - set it to 2.0 - means
> > CloudStack
> > > > will
> > > > >> see your cluster having 200% cpu of that it really has, 3.5= 350%
> > etc.
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Sun, Mar 10, 2019, 10:24 Fariborz Navidan <
> mdvlinqu...@gmail.com
> > >
> > > > >> wrote:
> > > > >>
> > > > >> > I've already set
> cluster.cpu.allocated.capacity.disablethreshold
> > to
> > > > >> > maximum value in both cluster and global settings. Is it
> possible
> > to
> > > > to
> > > > >> > makeACS ignore cpu capacity check?
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > On Sun, Mar 10, 2019 at 12:43 PM Andrija Panic <
> > > > andrija.pa...@gmail.com
> > > > >> >
> > > > >> > wrote:
> > > > >> >
> > > > >> > > Try to o provision CPU on a cluster level, thought I can't say
> > if
> > > > >> > shortage
> > > > >> > > of CPU or RAM, on my mobile atm...
> > > > >> > >
> > > > >> > > On Sun, Mar 10, 2019, 10:03 Andrija Panic <
> > > andrija.pa...@gmail.com>
> > > > >> > wrote:
> > > > >> > >
> > > > >> > > > Cannot allocate cluster list [1] for vm creation since their
> > > > >> allocated
> > > 

Re: Cloudstack is unable to create new VM

[Andrija Panic]

Please send compete log to pastebin.com or such...and link here.

On Sun, Mar 10, 2019, 14:36 Rafael Weingärtner 
wrote:

> Ah, this happens for a simple reason.
>
> When you allocate a VM with over-provision of 1, you are charging by that,
> and later if you change the over-provisioning to two; what would you expect
> to happen with that VM that was allocated when over-provisioning was set to
> 1? ACS is doing that do maintain the resources originally allocated by the
> VM. If you want to change this, you need to restart the VMs.
>
> On Sun, Mar 10, 2019 at 10:19 AM Fariborz Navidan 
> wrote:
>
> > I guess the issue is that allocated cpu is also grown by overprovisioning
> > factor such thatusage percentage remains the same. So, by increasing cpu
> > overprovisiong factor you don't gain access to un-utilized cpu capacity.
> My
> > real cpu utilization on the host is 31% but 84.82% is allocated and by
> > increasing the factor, the usage is also increased!
> >
> > On Sun, Mar 10, 2019 at 4:40 PM Fariborz Navidan 
> > wrote:
> >
> > > Even with cpu overprovisioning  factor of 20 set on both cluster and
> > > global settings, still I cannot deploy new  VMs. It show 672GHz in
> > > dashboardinstead of 33.6GHz (actual capacity) but still I get in the
> log
> > > "The host is in avoid list".
> > >
> > > Please advise.
> > > Thanks
> > >
> > > On Sun, Mar 10, 2019 at 4:04 PM Andrija Panic  >
> > > wrote:
> > >
> > >> Cpu.overprovisioning.ratio or such - set it to 2.0 - means CloudStack
> > will
> > >> see your cluster having 200% cpu of that it really has, 3.5= 350% etc.
> > >>
> > >>
> > >>
> > >> On Sun, Mar 10, 2019, 10:24 Fariborz Navidan 
> > >> wrote:
> > >>
> > >> > I've already set  cluster.cpu.allocated.capacity.disablethreshold to
> > >> > maximum value in both cluster and global settings. Is it possible to
> > to
> > >> > makeACS ignore cpu capacity check?
> > >> >
> > >> >
> > >> >
> > >> > On Sun, Mar 10, 2019 at 12:43 PM Andrija Panic <
> > andrija.pa...@gmail.com
> > >> >
> > >> > wrote:
> > >> >
> > >> > > Try to o provision CPU on a cluster level, thought I can't say if
> > >> > shortage
> > >> > > of CPU or RAM, on my mobile atm...
> > >> > >
> > >> > > On Sun, Mar 10, 2019, 10:03 Andrija Panic <
> andrija.pa...@gmail.com>
> > >> > wrote:
> > >> > >
> > >> > > > Cannot allocate cluster list [1] for vm creation since their
> > >> allocated
> > >> > > > percentage crosses the disable capacity threshold defined at
> each
> > >> > > cluster/
> > >> > > > at global value for capacity Type : 1, skipping these clusters
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > On Sat, Mar 9, 2019, 20:54 Fariborz Navidan <
> > mdvlinqu...@gmail.com>
> > >> > > wrote:
> > >> > > >
> > >> > > >> Any idea?
> > >> > > >>
> > >> > > >> On Sat, Mar 9, 2019 at 10:05 PM Fariborz Navidan <
> > >> > mdvlinqu...@gmail.com
> > >> > > >
> > >> > > >> wrote:
> > >> > > >>
> > >> > > >> > Hi,
> > >> > > >> >
> > >> > > >> > Another noticeable issue about system is that cpu cores
> > >> allocation
> > >> > is
> > >> > > >> 19/8
> > >> > > >> > (237%) which may be thereason for error. However the RAM and
> > CPU
> > >> > > >> > utilization is fairly low and I should be able to create more
> > >> VMs.
> > >> > How
> > >> > > >> can
> > >> > > >> > I change settings for cpu cores overprovisioning factor?
> > >> > > >> >
> > >> > > >> > Thanks
> > >> > > >> >
> > >> > > >> > On Sat, Mar 9, 2019 at 9:53 PM Fariborz Navidan <
> > >> > > mdvlinqu...@gmail.com>
> > >> > > >> > wrote:
> > >> > > &g

Re: Cloudstack is unable to create new VM

[Andrija Panic]

Cpu.overprovisioning.ratio or such - set it to 2.0 - means CloudStack will
see your cluster having 200% cpu of that it really has, 3.5= 350% etc.



On Sun, Mar 10, 2019, 10:24 Fariborz Navidan  wrote:

> I've already set  cluster.cpu.allocated.capacity.disablethreshold to
> maximum value in both cluster and global settings. Is it possible to to
> makeACS ignore cpu capacity check?
>
>
>
> On Sun, Mar 10, 2019 at 12:43 PM Andrija Panic 
> wrote:
>
> > Try to o provision CPU on a cluster level, thought I can't say if
> shortage
> > of CPU or RAM, on my mobile atm...
> >
> > On Sun, Mar 10, 2019, 10:03 Andrija Panic 
> wrote:
> >
> > > Cannot allocate cluster list [1] for vm creation since their allocated
> > > percentage crosses the disable capacity threshold defined at each
> > cluster/
> > > at global value for capacity Type : 1, skipping these clusters
> > >
> > >
> > >
> > > On Sat, Mar 9, 2019, 20:54 Fariborz Navidan 
> > wrote:
> > >
> > >> Any idea?
> > >>
> > >> On Sat, Mar 9, 2019 at 10:05 PM Fariborz Navidan <
> mdvlinqu...@gmail.com
> > >
> > >> wrote:
> > >>
> > >> > Hi,
> > >> >
> > >> > Another noticeable issue about system is that cpu cores allocation
> is
> > >> 19/8
> > >> > (237%) which may be thereason for error. However the RAM and CPU
> > >> > utilization is fairly low and I should be able to create more VMs.
> How
> > >> can
> > >> > I change settings for cpu cores overprovisioning factor?
> > >> >
> > >> > Thanks
> > >> >
> > >> > On Sat, Mar 9, 2019 at 9:53 PM Fariborz Navidan <
> > mdvlinqu...@gmail.com>
> > >> > wrote:
> > >> >
> > >> >> Any idea?
> > >> >>
> > >> >> On Sat, Mar 9, 2019 at 8:38 PM Fariborz Navidan <
> > mdvlinqu...@gmail.com
> > >> >
> > >> >> wrote:
> > >> >>
> > >> >>> Memory Total:  62.69 GB
> > >> >>> Memory Allocated: *14.25 GB*
> > >> >>> Memory Used:  19.03 GB
> > >> >>>
> > >> >>> There are also lots of storage available. As the log states, it is
> > >> >>> complaining about resource type 1 (cpu).
> > >> >>>
> > >> >>> Please advise.
> > >> >>>
> > >> >>> Thanks
> > >> >>>
> > >> >>> On Sat, Mar 9, 2019 at 8:16 PM Ivan Kudryavtsev <
> > >> >>> kudryavtsev...@bw-sw.com> wrote:
> > >> >>>
> > >> >>>> What about RAM and storage?
> > >> >>>>
> > >> >>>> сб, 9 мар. 2019 г., 9:22 Fariborz Navidan  >:
> > >> >>>>
> > >> >>>> > I have already set
> > cluster.cpu.allocated.capacity.disablethreshold
> > >> to
> > >> >>>> 1
> > >> >>>> > and cpu.overprovisioning.factor to 10 both in global settings
> and
> > >> >>>> cluster
> > >> >>>> > level.
> > >> >>>> >
> > >> >>>> > Best Regards
> > >> >>>> >
> > >> >>>> > On Sat, Mar 9, 2019 at 6:47 PM Fariborz Navidan <
> > >> >>>> mdvlinqu...@gmail.com>
> > >> >>>> > wrote:
> > >> >>>> >
> > >> >>>> > > Hello
> > >> >>>> > >
> > >> >>>> > > This is host's cpu resource statistics:
> > >> >>>> > >
> > >> >>>> > > CPU Utilized: 30.2%
> > >> >>>> > > CPU Allocated for VMs: 84.82%
> > >> >>>> > >
> > >> >>>> > > On Sat, Mar 9, 2019 at 4:20 PM Ivan Kudryavtsev <
> > >> >>>> > kudryavtsev...@bw-sw.com>
> > >> >>>> > > wrote:
> > >> >>>> > >
> > >> >>>> > >> Looks like your cluster is pretty full. Increase thresholds
> in
> > >> >>>> cluster
> > >> >>>> > >> vars
> > >> >>>> > >> or add resour

Re: Cloudstack is unable to create new VM

[Andrija Panic]

Cannot allocate cluster list [1] for vm creation since their allocated
percentage crosses the disable capacity threshold defined at each cluster/
at global value for capacity Type : 1, skipping these clusters



On Sat, Mar 9, 2019, 20:54 Fariborz Navidan  wrote:

> Any idea?
>
> On Sat, Mar 9, 2019 at 10:05 PM Fariborz Navidan 
> wrote:
>
> > Hi,
> >
> > Another noticeable issue about system is that cpu cores allocation is
> 19/8
> > (237%) which may be thereason for error. However the RAM and CPU
> > utilization is fairly low and I should be able to create more VMs. How
> can
> > I change settings for cpu cores overprovisioning factor?
> >
> > Thanks
> >
> > On Sat, Mar 9, 2019 at 9:53 PM Fariborz Navidan 
> > wrote:
> >
> >> Any idea?
> >>
> >> On Sat, Mar 9, 2019 at 8:38 PM Fariborz Navidan 
> >> wrote:
> >>
> >>> Memory Total:  62.69 GB
> >>> Memory Allocated: *14.25 GB*
> >>> Memory Used:  19.03 GB
> >>>
> >>> There are also lots of storage available. As the log states, it is
> >>> complaining about resource type 1 (cpu).
> >>>
> >>> Please advise.
> >>>
> >>> Thanks
> >>>
> >>> On Sat, Mar 9, 2019 at 8:16 PM Ivan Kudryavtsev <
> >>> kudryavtsev...@bw-sw.com> wrote:
> >>>
>  What about RAM and storage?
> 
>  сб, 9 мар. 2019 г., 9:22 Fariborz Navidan :
> 
>  > I have already set cluster.cpu.allocated.capacity.disablethreshold
> to
>  1
>  > and cpu.overprovisioning.factor to 10 both in global settings and
>  cluster
>  > level.
>  >
>  > Best Regards
>  >
>  > On Sat, Mar 9, 2019 at 6:47 PM Fariborz Navidan <
>  mdvlinqu...@gmail.com>
>  > wrote:
>  >
>  > > Hello
>  > >
>  > > This is host's cpu resource statistics:
>  > >
>  > > CPU Utilized: 30.2%
>  > > CPU Allocated for VMs: 84.82%
>  > >
>  > > On Sat, Mar 9, 2019 at 4:20 PM Ivan Kudryavtsev <
>  > kudryavtsev...@bw-sw.com>
>  > > wrote:
>  > >
>  > >> Looks like your cluster is pretty full. Increase thresholds in
>  cluster
>  > >> vars
>  > >> or add resources.
>  > >>
>  > >> сб, 9 мар. 2019 г., 6:55 Rafael Weingärtner <
>  > rafaelweingart...@gmail.com
>  > >> >:
>  > >>
>  > >> > In the log there is this message:
>  > >> > > Cannot allocate cluster list [1] for vm creation since their
>  > allocated
>  > >> > percentage crosses the disable capacity threshold defined at
> each
>  > >> > cluster/at global value for capacity Type : 1, skipping these
>  clusters
>  > >> >
>  > >> > What is the status of your cluster's host?
>  > >> >
>  > >> > On Sat, Mar 9, 2019 at 7:14 AM Fariborz Navidan <
>  > mdvlinqu...@gmail.com>
>  > >> > wrote:
>  > >> >
>  > >> > > Hello,
>  > >> > >
>  > >> > > Here is the log for VM deployment:
>  > >> > >
>  > >> > > 2019-03-09 11:01:34,984 DEBUG [c.c.a.ApiServlet]
>  > >> > > (qtp788117692-18:ctx-f51fd578) (logid:113f467d) ===START===
>  > >> 137.74.35.65
>  > >> > > -- GET
>  > >> > >
>  > >> > >
>  > >> >
>  > >>
>  >
> 
> command=deployVirtualMachine&response=json&zoneid=bc4565d8-4029-4dbd-93eb-47137ff45188&templateid=4c10414a-51f2-4859-b673-e92c3f32cbfd&hypervisor=KVM&rootdisksize=10&serviceofferingid=cd1235b8-acdf-4ce3-a3ac-3f7ddcea8870&displayname=centos&name=centos&_=1552125695843
>  > >> > > 2019-03-09 11:01:34,989 DEBUG [c.c.a.ApiServer]
>  > >> > > (qtp788117692-18:ctx-f51fd578 ctx-a82cdf12) (logid:113f467d)
>  CIDRs
>  > >> from
>  > >> > > which account
>  'Acct[27cd01ef-3907-11e9-87ab-a4bf012ed1a6-admin]' is
>  > >> > allowed
>  > >> > > to perform API calls: 0.0.0.0/0,::/0
>  > >> > > 2019-03-09 11:01:35,040 DEBUG [c.c.u.AccountManagerImpl]
>  > >> > > (qtp788117692-18:ctx-f51fd578 ctx-a82cdf12) (logid:113f467d)
>  Access
>  > >> > granted
>  > >> > > to Acct[27cd01ef-3907-11e9-87ab-a4bf012ed1a6-admin] to
>  > >> > >
>  > >> > >
>  > >> >
>  > >>
>  >
> 
> org.apache.cloudstack.quota.vo.ServiceOfferingVO$$EnhancerByCGLIB$$c370fca@68de8a96
>  > >> > > by AffinityGroupAccessChecker
>  > >> > > 2019-03-09 11:01:35,042 DEBUG [c.c.u.AccountManagerImpl]
>  > >> > > (qtp788117692-18:ctx-f51fd578 ctx-a82cdf12) (logid:113f467d)
>  Access
>  > >> > granted
>  > >> > > to Acct[27cd01ef-3907-11e9-87ab-a4bf012ed1a6-admin] to null
> by
>  > >> > > AffinityGroupAccessChecker
>  > >> > > 2019-03-09 11:01:35,046 DEBUG [c.c.n.NetworkModelImpl]
>  > >> > > (qtp788117692-18:ctx-f51fd578 ctx-a82cdf12) (logid:113f467d)
>  Service
>  > >> > > SecurityGroup is not supported in the network id=204
>  > >> > > 2019-03-09 11:01:35,058 DEBUG [c.c.n.NetworkModelImpl]
>  > >> > > (qtp788117692-18:ctx-f51fd578 ctx-a82cdf12) (logid:113f467d)
>  Service
>  > >> > > SecurityGroup is not supported in the network id=204
>  > >> > > 2019-03-09 11:01:35,

Re: Cloudstack is unable to create new VM

[Andrija Panic]

Try to o provision CPU on a cluster level, thought I can't say if shortage
of CPU or RAM, on my mobile atm...

On Sun, Mar 10, 2019, 10:03 Andrija Panic  wrote:

> Cannot allocate cluster list [1] for vm creation since their allocated
> percentage crosses the disable capacity threshold defined at each cluster/
> at global value for capacity Type : 1, skipping these clusters
>
>
>
> On Sat, Mar 9, 2019, 20:54 Fariborz Navidan  wrote:
>
>> Any idea?
>>
>> On Sat, Mar 9, 2019 at 10:05 PM Fariborz Navidan 
>> wrote:
>>
>> > Hi,
>> >
>> > Another noticeable issue about system is that cpu cores allocation is
>> 19/8
>> > (237%) which may be thereason for error. However the RAM and CPU
>> > utilization is fairly low and I should be able to create more VMs. How
>> can
>> > I change settings for cpu cores overprovisioning factor?
>> >
>> > Thanks
>> >
>> > On Sat, Mar 9, 2019 at 9:53 PM Fariborz Navidan 
>> > wrote:
>> >
>> >> Any idea?
>> >>
>> >> On Sat, Mar 9, 2019 at 8:38 PM Fariborz Navidan > >
>> >> wrote:
>> >>
>> >>> Memory Total:  62.69 GB
>> >>> Memory Allocated: *14.25 GB*
>> >>> Memory Used:  19.03 GB
>> >>>
>> >>> There are also lots of storage available. As the log states, it is
>> >>> complaining about resource type 1 (cpu).
>> >>>
>> >>> Please advise.
>> >>>
>> >>> Thanks
>> >>>
>> >>> On Sat, Mar 9, 2019 at 8:16 PM Ivan Kudryavtsev <
>> >>> kudryavtsev...@bw-sw.com> wrote:
>> >>>
>> >>>> What about RAM and storage?
>> >>>>
>> >>>> сб, 9 мар. 2019 г., 9:22 Fariborz Navidan :
>> >>>>
>> >>>> > I have already set cluster.cpu.allocated.capacity.disablethreshold
>> to
>> >>>> 1
>> >>>> > and cpu.overprovisioning.factor to 10 both in global settings and
>> >>>> cluster
>> >>>> > level.
>> >>>> >
>> >>>> > Best Regards
>> >>>> >
>> >>>> > On Sat, Mar 9, 2019 at 6:47 PM Fariborz Navidan <
>> >>>> mdvlinqu...@gmail.com>
>> >>>> > wrote:
>> >>>> >
>> >>>> > > Hello
>> >>>> > >
>> >>>> > > This is host's cpu resource statistics:
>> >>>> > >
>> >>>> > > CPU Utilized: 30.2%
>> >>>> > > CPU Allocated for VMs: 84.82%
>> >>>> > >
>> >>>> > > On Sat, Mar 9, 2019 at 4:20 PM Ivan Kudryavtsev <
>> >>>> > kudryavtsev...@bw-sw.com>
>> >>>> > > wrote:
>> >>>> > >
>> >>>> > >> Looks like your cluster is pretty full. Increase thresholds in
>> >>>> cluster
>> >>>> > >> vars
>> >>>> > >> or add resources.
>> >>>> > >>
>> >>>> > >> сб, 9 мар. 2019 г., 6:55 Rafael Weingärtner <
>> >>>> > rafaelweingart...@gmail.com
>> >>>> > >> >:
>> >>>> > >>
>> >>>> > >> > In the log there is this message:
>> >>>> > >> > > Cannot allocate cluster list [1] for vm creation since their
>> >>>> > allocated
>> >>>> > >> > percentage crosses the disable capacity threshold defined at
>> each
>> >>>> > >> > cluster/at global value for capacity Type : 1, skipping these
>> >>>> clusters
>> >>>> > >> >
>> >>>> > >> > What is the status of your cluster's host?
>> >>>> > >> >
>> >>>> > >> > On Sat, Mar 9, 2019 at 7:14 AM Fariborz Navidan <
>> >>>> > mdvlinqu...@gmail.com>
>> >>>> > >> > wrote:
>> >>>> > >> >
>> >>>> > >> > > Hello,
>> >>>> > >> > >
>> >>>> > >> > > Here is the log for VM deployment:
>> >>>> > >> > >
>> >>>> > >> &

Re: Quick User's Guide?

[Andrija Panic]

https://www.youtube.com/playlist?list=PLEr0fbgkyLKxeyTghZ34jUOcQXw--YgiK

A vry old, and very amateur videos of mine, VPC, network, VMs, etc -
might get you going.

Kind regards,
Andrija

On Fri, 8 Mar 2019 at 15:33, Larry Sevilla 
wrote:

> Hi Konstantin,
>
> Already done the Quick Installation Guide.
> Successfully installed on CentOS7.
>
> Install Guide / Configuration is for Advanced items.
>
> My concern is "Quick User's Guide" specially on the GUI Management web
> console.
> The basic / fundamental way of creating a VM, install an OS from an ISO
> installer, something like that.
>
> On Fri, Mar 8, 2019 at 5:27 PM Konstantin 
> wrote:
>
> > You can try this
> >
> >
> http://docs.cloudstack.apache.org/en/4.11.2.0/quickinstallationguide/qig.html
> > but I was successful after 4-5 attempts. Would recommend to snapshot
> > cloudstack VM for fast rollbacks in case of mistakes
> > Also please ignore quick setup wizard and add your first zone cluster and
> > storage's yourself.
> > here is valuable link
> >
> >
> http://docs.cloudstack.apache.org/en/4.11.2.0/installguide/configuration.html
> >
> >
> > On Fri, Mar 8, 2019 at 11:49 AM Larry Sevilla <
> sevilla.larry@gmail.com
> > >
> > wrote:
> >
> > > Thx for the site...
> > >
> > > On Fri, Mar 8, 2019 at 4:00 PM Andrija Panic <
> > andrija.pa...@shapeblue.com>
> > > wrote:
> > >
> > > > Hi Larry,
> > > >
> > > > CloudStack can be comprehensive for novice users, but it the
> > > > default/official documentation to follow and is probably best suited
> > for
> > > > learning. You can also take a look at
> > > >
> > >
> >
> https://www.slideshare.net/search/slideshow?searchfrom=header&q=cloudstack
> > > > - there are many nice slides from past presentations, where you can
> > also
> > > > learn in more visual way.
> > > >
> > > > Kind regards,
> > > > Andrija
> > > >
> > > > andrija.pa...@shapeblue.com
> > > > www.shapeblue.com
> > > > Amadeus House, Floral Street, London  WC2E 9DPUK
> > > > @shapeblue
> > > >
> > > >
> > > >
> > > >
> > > > -Original Message-
> > > > From: Larry Sevilla 
> > > > Sent: 08 March 2019 07:52
> > > > To: users@cloudstack.apache.org
> > > > Subject: Quick User's Guide?
> > > >
> > > > Hi,
> > > >
> > > > I'm trying to Learn How to Use CloudStack.
> > > > http://docs.cloudstack.apache.org/en/4.11.2.0/adminguide/index.html
> > > >
> > > > But it seems comprehensive.
> > > >
> > > > Is there something like a Quick User's Guide?
> > > >
> > >
> >
>


-- 

Andrija Panić

RE: [VOTE] Apache CloudStack 4.12.0.0 [RC4]

[Andrija Panic]

Hi,

You are going to hate me, but I need to vote again as:

-1

Usage server seems broken or severely misbehaving - please see: 
https://github.com/apache/cloudstack/issues/3203
Reproduced the issue myself.

Kind regards,
Andrija

andrija.pa...@shapeblue.com 
www.shapeblue.com
Amadeus House, Floral Street, London  WC2E 9DPUK
@shapeblue
  
 


-Original Message-
From: Gabriel Beims Bräscher  
Sent: 07 March 2019 14:27
To: dev 
Cc: users 
Subject: Re: [VOTE] Apache CloudStack 4.12.0.0 [RC4]

Rohit, the testing packages are located at:

Ubuntu:
http://cloudstack.apt-get.eu/ubuntu/dists/trusty/4.12/
http://cloudstack.apt-get.eu/ubuntu/dists/xenial/4.12/
http://cloudstack.apt-get.eu/ubuntu/dists/bionic/4.12/

Centos:
http://cloudstack.apt-get.eu/centos/6/4.12/
http://cloudstack.apt-get.eu/centos/7/4.12/

Em qui, 7 de mar de 2019 às 04:57, Rohit Yadav 
escreveu:

> Hi Gabriel,
>
>
> Can you advise the test repo url? I don't see RC4 here:
> http://cloudstack.apt-get.eu/testing/
>
>
>
> Regards,
>
> Rohit Yadav
>
> Software Architect, ShapeBlue
>
> https://www.shapeblue.com
>
> 
> From: Gabriel Beims Br?scher 
> Sent: Thursday, March 7, 2019 1:23:27 PM
> To: dev; users
> Subject: [VOTE] Apache CloudStack 4.12.0.0 [RC4]
>
> Hi All,
>
> I've created a 4.12.0.0 release (RC4), with the following artifacts up 
> for a vote:
> The changes since RC3 are listed at the end of this email.
>
> Git Branch and Commit SH:
> https://github.com/apache/cloudstack/tree/4.12.0.0-RC20190306T1109
>
> https://gitbox.apache.org/repos/asf?p=cloudstack.git;a=shortlog;h=refs
> /heads/4.12.0.0-RC20190306T1109
>
>
> https://github.com/apache/cloudstack/commit/9c0cbf91caefad3aa9340b88dd
> 3f45a5a93323f4
> Commit: 9c0cbf91caefad3aa9340b88dd3f45a5a93323f4
>
> Source release for 4.12.0.0-RC20190306T1109:
> https://dist.apache.org/repos/dist/dev/cloudstack/4.12.0.0/
>
> PGP release keys (signed using 25908455):
> https://dist.apache.org/repos/dist/release/cloudstack/KEYS
>
> The vote will be open for 3 business days (until 11th March).
>
> For sanity in tallying the vote, can PMC members please be sure to 
> indicate "(binding)" with their vote?
>
> [ ] +1  approve
> [ ] +0  no opinion
> [ ] -1  disapprove (and reason why)
>
> Additional information:
>
> All Travis, Jenkins, and CI (Blue Orangutan) checks have passed [1].
>
> For users' convenience, packages are available in 
> http://cloudstack.apt-get.eu/
> 4.12.0.0 RC3 is available for the following distros:
> - Ubuntu 14.04, 16.04, and 18.04;
> - CentOS6 and CentOS7.
>
> Please, use the template 4.11.2 (located in [2]) when testing the RC4.
> The release notes [3] still need to be updated.
>
> Changes Since RC3:
> Merged #3197 Fix XenServer Security Groups 'vmops' script [1]
>
> [1] https://github.com/apache/cloudstack/pull/3197
> [2] http://download.cloudstack.org/systemvm/4.11/
> [3]
>
> http://docs.cloudstack.apache.org/projects/cloudstack-release-notes/en
> /latest/index.html
>
> rohit.ya...@shapeblue.com
> www.shapeblue.com
> Amadeus House, Floral Street, London  WC2E 9DPUK @shapeblue
>
>
>
>

RE: cloudstack usage server issue

[Andrija Panic]

Hi Konstantin,

Running hourly versus daily will case you DB to have 24x more records and be 24 
times bigger (trust me on this one, it can be a HUGE problem once your 
infrastructure grows with exponentially slower usage job runs, which will make 
you billing issues).

From usage records point of view, if you VM was running i.e. 25.5h, hourly 
aggregation period will make 26 records in total - 25 records saying "your VM 
was running for 1h" and another one record saying "your VM was running for .05h,
while daily aggregation period (1440) will have 2 records saying "your VM was 
running 24h" and another one "your VM was running for 1.5h"

Avoid hourly by all means and modify your billing software as needed.

Kind regards,
Andrija


andrija.pa...@shapeblue.com 
www.shapeblue.com
Amadeus House, Floral Street, London  WC2E 9DPUK
@shapeblue
  
 


-Original Message-
From: Konstantin  
Sent: 08 March 2019 12:34
To: users@cloudstack.apache.org
Subject: Re: cloudstack usage server issue

Dear Andrija,

May you give more info about aggregation period of 60 minutes?
My billing software will calc cloud usage by hours, should I keep aggregation 
period for 60 minutes or update it to default 1440?



On Thu, Mar 7, 2019 at 10:58 PM Andrija Panic 
wrote:

> Hi Konstantin,
>
> Issues/bugs are raised here 
> https://github.com/apache/cloudstack/issues - so please feel free to 
> raise it ASAP - since we have a RC4 voting process in place ! (it's 
> very simple) - also feel free to jump on the voting email thread 
> (before giving  -1, let's first try to reproduce the issue once more? 
> )
>
> I will try to look into this problem, but can't promise anything (into
> 4.12 issue).
>
> As for the very first link you shared - the jump in time visible in 
> logs - is this because of restart, right ?
>
> Also as for aggregation period, 1440 is default (and in production, 
> believe me, should be kept this way if possible).
> Hourly jobs are ok for testing (value of 60 minutes), not sure setting 
> 5
> (minutes) makes sense (perhaps for testing, but I would better stick 
> to 60 min period, and then change execution time to 5min from now() to 
> be able to test (and restart usage service) - please give it some time 
> to actually start processing data...
>
> As for the issues changing db.properties file, you should have it like
> following:
>
> ls -la /etc/cloudstack/usage/
> total 4
> lrwxrwxrwx. 1 root root   40 Feb 13 20:27 db.properties ->
> /etc/cloudstack/management/db.properties
> lrwxrwxrwx. 1 root root   30 Feb 13 20:27 key ->
> /etc/cloudstack/management/key
> -rw-r--r--. 1 root root 2980 Mar  7 17:22 log4j-cloud.xml First two  
> are links as you see - assuming you tried to change correct values in 
> db.properties (to make usage fail to connect to DB), that should work, 
> otherwise, it's a bug...
>
> Kind regards,
> Andrija
>
> andrija.pa...@shapeblue.com
> www.shapeblue.com
> Amadeus House, Floral Street, London  WC2E 9DPUK @shapeblue
>
>
>
>
> -Original Message-
> From: Konstantin 
> Sent: 07 March 2019 16:59
> To: users@cloudstack.apache.org
> Subject: Re: cloudstack usage server issue
>
> I reinstalled 4.11.2 and logs are changed. New error appear
>
> https://i.pozitis.ru/77a661462b182d54a6405484074dcee2.txt
>
> 2019-03-07 15:50:00,001 INFO  [cloud.usage.UsageManagerImpl]
> (Usage-Job-1:null) (logid:) starting usage job...
> 2019-03-07 15:50:00,014 DEBUG [cloud.usage.UsageManagerImpl]
> (Usage-Job-1:null) (logid:) Not owner of usage job, skipping...
> 2019-03-07 15:50:00,014 INFO  [cloud.usage.UsageManagerImpl]
> (Usage-Job-1:null) (logid:) usage job complete
>
> I cleaned the usage_job table from the records and FINALLY its start
> working!
>
>
> How to submit a bug report for the version  4.12.0.0 ?
>
> Could you please do it on my behalf?
>
>
>
>
> On Thu, Mar 7, 2019 at 6:23 PM Konstantin 
> wrote:
>
> > Dear Andrija,
> >
> > Thanks a lot for the attention to my issue.
> >
> > There is logs from usage server:
> >
> > https://i.pozitis.ru/80d76e3cc5e605be037a088fd014a986.txt
> >
> > There is config
> > https://i.pozitis.ru/8419e465b3ffa63b4d677ae5be85688a.txt
> >
> > its looks like usage server just ignore cloud_usage DB settings, I
> > tried to put wrong credentials to db.usage.name=cloud_usage params, No
> > matter, Its load the usage details from db.cloud.name=cloud, I think
> >
> > Here is my global settings
> >
> > https://i.pozitis.ru/9d3d53fd86997dd649b0e8171ca14c73.txt
> >
> > I did checked management server log, parsed by "usage", here is
> > result, I se

RE: downloaded template vs disk service offering

[Andrija Panic]

Hi Piotr,

https://doc.opensuse.org/documentation/leap/virtualization/html/book.virt/cha.cachemodes.html#sec.cache.mode.live.migration
 

So yes, CEPH being considered "clustered storage" - live migration works - but 
in case of QCOW2 (NFS) it doesn't actually work.

BTW, as for CEPH, you would probably want to also check RBD client side 
write-back cache... (versus/instead qemu cache=writeback) (i.e. 32MB writeback 
cache in librbd per each volume, etc.).
I believe I did test one versus another caching (was operating CEPH backed 
CloudStack installation myself a while ago) - afaik, there were no visible 
performance/latency differences in RBD write-back caching versus qemu writeback 
caching (both active = issues with performance)

Kind regards,
Andrija

andrija.pa...@shapeblue.com 
www.shapeblue.com
Amadeus House, Floral Street, London  WC2E 9DPUK
@shapeblue
  
 


-Original Message-
From: Piotr Pisz  
Sent: 08 March 2019 12:44
To: users@cloudstack.apache.org
Subject: RE: downloaded template vs disk service offering

Hey Andrija,

Thank you for the explanation, now I finally understand how it works :-) As for 
live migration, the migration of such machines (with cache = writeback) in ceph 
rbd (centos 7 kvm) works without any problem.

Regards,
Piotr


-Original Message-
From: Andrija Panic 
Sent: Friday, March 8, 2019 9:22 AM
To: users@cloudstack.apache.org
Subject: RE: downloaded template vs disk service offering

Hi Piotr,

It's true that when setting cache mode for Disk offering via GUI doesn't get it 
implemented in DB (does API works fine, did you test it ? if so please raise 
the GitHub issue with description).

In general, you can initially set cache mode for disk only on Disk Offering 
(possibly also on Compute Offering for the Root disk).
When you make new template from existing disk, this new template will have 
source_template_id field in vm_templates table (on it's row) set to your 
original template from which you created the volum (template --> disk --> new 
template)

Also worth noting - all volumes are inheriting "on he fly" (when you start VM) 
this cache mode setting from it's template (all volumes have "template_id" 
field in "volumes" table)

So if you set cache_mode (via DB) for specific template, it will affect ALL VMs 
created from that template...(once you stop and start those VMs, obviously) - 
i.e. when you deploy new VM, some column values are copied over to the actual 
volume row/table, but some are just read on the fly, as this cache_mode.

Nevertheless, I would strongly discourage using write-back cache for disks, 
since:

- it can be severely risky, in case of power loss, kernel panic, etc - you will 
end up with corrupted volumes.
- VMs can NOT be live migrated (at least with KVM), with cache set to anything 
else than none (google it yourself) - happy to learn if this limitation is 
present for other Hypervisors as well

Fine to play with, but I would skip it in production.

Kind regards,
Andrija

andrija.pa...@shapeblue.com
www.shapeblue.com
Amadeus House, Floral Street, London  WC2E 9DPUK @shapeblue
  
 


-Original Message-
From: Piotr Pisz 
Sent: 08 March 2019 08:32
To: users@cloudstack.apache.org
Subject: downloaded template vs disk service offering

Hi Users :-)

I have a question.
If from the disk for which the cache = writeback paramter was set, I make a 
template, all new machines have cache = writeback. And that's ok. 
If I load a template from outside, volume has cache = none. I have not found a 
place in DB where I could improve this parameter.
Do you know where we can set the template cache?

PS. Disk offering made with GUI does not set the cache parameter in DB...

Regards,
Piotr

RE: Network Bridge - error

[Andrija Panic]

Hi Larry,

If you have account on GitHub, just click the pencil (EDIT file in your 
fork...) icon/button (from 
https://github.com/apache/cloudstack-documentation/blob/master/source/quickinstallationguide/qig.rst),
 it will automatically fork the upstream project into your account and open 
EDIT page for that specific page - you make changes - and simply commit with 
message and description- it will then ask you if you want to make a Pull 
Request, confirm yes, and you are good to go !

Very simple really.

Kind regards,
Andrija

andrija.pa...@shapeblue.com 
www.shapeblue.com
Amadeus House, Floral Street, London  WC2E 9DPUK
@shapeblue
  
 


-Original Message-
From: Larry Sevilla  
Sent: 08 March 2019 09:53
To: users@cloudstack.apache.org
Subject: Re: Network Bridge - error

Hi Andrija,

Thx for your trust, but I'm not familiar with GitHub & it's editing.  I might 
messed something up.


On Fri, Mar 8, 2019 at 3:54 PM Andrija Panic 
wrote:

> Hi Larry,
>
> As a valuable community member - please feel free to edit the 
> documentation for CentOS7 (you can do it all via GitHub, from the link 
> for
> CentOS7 guide), and include somewhere the fact that NetworkManager 
> needs to be disabled (copy/paste the commands needed) - this will help 
> future novice users get it up and running from the first time.
>
> Kind regards,
> Andrija
>
> andrija.pa...@shapeblue.com
> www.shapeblue.com
> Amadeus House, Floral Street, London  WC2E 9DPUK @shapeblue
>
>
>
>
> -Original Message-
> From: Larry Sevilla 
> Sent: 08 March 2019 05:20
> To: users@cloudstack.apache.org
> Subject: Re: Network Bridge - error
>
> Hi,
>
> Successfully configured using:
>
> CentOS 6
>
> http://docs.cloudstack.apache.org/en/4.11.2.0/quickinstallationguide/qig.html
> but needs #yum install epel-release
> before #yum install mysql-connector-python
>
> And
>
> CentOS 7
>
> https://github.com/apache/cloudstack-documentation/blob/master/source/quickinstallationguide/qig.rst
> but needs to disable NetworkManager
>
>
> Thx...
>
> Currently, learning how to use.
>
>
> On Fri, Mar 8, 2019 at 9:14 AM Larry Sevilla 
> wrote:
>
> > Hi Dag,
> >
> > Disabling NetworkManager works.
> >
> > # systemctl stop NetworkManager
> > # systemctl disable NetworkManager
> >
> >
> > thx...
> >
> >
> > On Thu, Mar 7, 2019 at 5:36 PM Dag Sonstebo
> > 
> > wrote:
> >
> >> Hi Larry,
> >>
> >> Looks to me like your network configuration isn't consistent - your
> >> enp2s0 interface is still NetworkManager controlled, whilst the
> >> bridge
> >> isn't:
> >>
> >> Mar 07 10:05:07 localhost.localdomain NetworkManager[4025]: 
> >> [1551924307.4967] ifcfg-rh: Ignoring connection
> >> /etc/sysconfig/network-scripts/ifcfg-cloudbr0
> >> (d4b789ba-7321-548d-dabd-5c4150da0266,"System cloudbr0") due to
> >> NM_CONTROLLED=no.
> >> . . .
> >> Mar 07 10:05:07 localhost.localdomain NetworkManager[4025]: 
> >> [1551924307.6674] audit: op="connection-activate"
> >> uuid="740fc662-1d75-42ae-98ed-0ce92a0ca3dc" name="enp2s0" pid=14783
> >> uid=0 result="fail" reason="Can not find a master for enp2s0: Master
> >> connection not found or invalid"
> >> Mar 07 10:05:07 localhost.localdomain network[14650]: Bringing up
> >> interface enp2s0:  Error: Connection activation failed: Can not find
> >> a master for enp2s0: Master connection not found or invalid
> >>
> >>
> >> I recommend you disable NM altogether - it causes nothing but problems.
> >> If you want a bit more detailed network configuration overview check
> >> https://www.shapeblue.com/networking-kvm-for-cloudstack-2018-revisit-
> >> for-centos7-and-ubuntu-18-04/
> >>
> >> Regards,
> >> Dag Sonstebo
> >> Cloud Architect
> >> ShapeBlue
> >>
> >>
> >> On 07/03/2019, 03:05, "Larry Sevilla" 
> >> wrote:
> >>
> >> Hi Andrija,
> >>
> >> Got the same error message, but different details.
> >>
> >> systemctl status network.service
> >> https://pastebin.com/NjtXUwci
> >>
> >> journalctl -xe
> >> https://pastebin.com/Kn0Dggmm
> >>
> >> My network settings:
> >> https://pastebin.com/JDGJJYVc
> >>
> >>
> >> On Wed, Mar 6, 2019 at 4:57 PM Andrija Panic <
> >> andrija.pa...@shapeblue.com>

RE: downloaded template vs disk service offering

[Andrija Panic]

Hi Piotr,

It's true that when setting cache mode for Disk offering via GUI doesn't get it 
implemented in DB (does API works fine, did you test it ? if so please raise 
the GitHub issue with description).

In general, you can initially set cache mode for disk only on Disk Offering 
(possibly also on Compute Offering for the Root disk).
When you make new template from existing disk, this new template will have 
source_template_id field in vm_templates table (on it's row) set to your 
original template from which you created the volum (template --> disk --> new 
template)

Also worth noting - all volumes are inheriting "on he fly" (when you start VM) 
this cache mode setting from it's template (all volumes have "template_id" 
field in "volumes" table)

So if you set cache_mode (via DB) for specific template, it will affect ALL VMs 
created from that template...(once you stop and start those VMs, obviously) - 
i.e. when you deploy new VM, some column values are copied over to the actual 
volume row/table, but some are just read on the fly, as this cache_mode.

Nevertheless, I would strongly discourage using write-back cache for disks, 
since:

- it can be severely risky, in case of power loss, kernel panic, etc - you will 
end up with corrupted volumes.
- VMs can NOT be live migrated (at least with KVM), with cache set to anything 
else than none (google it yourself) - happy to learn if this limitation is 
present for other Hypervisors as well

Fine to play with, but I would skip it in production.

Kind regards,
Andrija

andrija.pa...@shapeblue.com 
www.shapeblue.com
Amadeus House, Floral Street, London  WC2E 9DPUK
@shapeblue
  
 


-Original Message-
From: Piotr Pisz  
Sent: 08 March 2019 08:32
To: users@cloudstack.apache.org
Subject: downloaded template vs disk service offering

Hi Users :-)

I have a question.
If from the disk for which the cache = writeback paramter was set, I make a 
template, all new machines have cache = writeback. And that's ok. 
If I load a template from outside, volume has cache = none. I have not found a 
place in DB where I could improve this parameter.
Do you know where we can set the template cache?

PS. Disk offering made with GUI does not set the cache parameter in DB...

Regards,
Piotr

RE: Quick User's Guide?

[Andrija Panic]

Hi Larry,

CloudStack can be comprehensive for novice users, but it the default/official 
documentation to follow and is probably best suited for learning. You can also 
take a look at 
https://www.slideshare.net/search/slideshow?searchfrom=header&q=cloudstack - 
there are many nice slides from past presentations, where you can also learn in 
more visual way.

Kind regards,
Andrija

andrija.pa...@shapeblue.com 
www.shapeblue.com
Amadeus House, Floral Street, London  WC2E 9DPUK
@shapeblue
  
 


-Original Message-
From: Larry Sevilla  
Sent: 08 March 2019 07:52
To: users@cloudstack.apache.org
Subject: Quick User's Guide?

Hi,

I'm trying to Learn How to Use CloudStack.
http://docs.cloudstack.apache.org/en/4.11.2.0/adminguide/index.html

But it seems comprehensive.

Is there something like a Quick User's Guide?

RE: Network Bridge - error

[Andrija Panic]

Hi Larry,

As a valuable community member - please feel free to edit the documentation for 
CentOS7 (you can do it all via GitHub, from the link for CentOS7 guide), and 
include somewhere the fact that NetworkManager needs to be disabled (copy/paste 
the commands needed) - this will help future novice users get it up and running 
from the first time.

Kind regards,
Andrija

andrija.pa...@shapeblue.com 
www.shapeblue.com
Amadeus House, Floral Street, London  WC2E 9DPUK
@shapeblue
  
 


-Original Message-
From: Larry Sevilla  
Sent: 08 March 2019 05:20
To: users@cloudstack.apache.org
Subject: Re: Network Bridge - error

Hi,

Successfully configured using:

CentOS 6
http://docs.cloudstack.apache.org/en/4.11.2.0/quickinstallationguide/qig.html
but needs #yum install epel-release
before #yum install mysql-connector-python

And

CentOS 7
https://github.com/apache/cloudstack-documentation/blob/master/source/quickinstallationguide/qig.rst
but needs to disable NetworkManager


Thx...

Currently, learning how to use.


On Fri, Mar 8, 2019 at 9:14 AM Larry Sevilla 
wrote:

> Hi Dag,
>
> Disabling NetworkManager works.
>
> # systemctl stop NetworkManager
> # systemctl disable NetworkManager
>
>
> thx...
>
>
> On Thu, Mar 7, 2019 at 5:36 PM Dag Sonstebo 
> 
> wrote:
>
>> Hi Larry,
>>
>> Looks to me like your network configuration isn't consistent - your
>> enp2s0 interface is still NetworkManager controlled, whilst the 
>> bridge
>> isn't:
>>
>> Mar 07 10:05:07 localhost.localdomain NetworkManager[4025]:  
>> [1551924307.4967] ifcfg-rh: Ignoring connection
>> /etc/sysconfig/network-scripts/ifcfg-cloudbr0
>> (d4b789ba-7321-548d-dabd-5c4150da0266,"System cloudbr0") due to 
>> NM_CONTROLLED=no.
>> . . .
>> Mar 07 10:05:07 localhost.localdomain NetworkManager[4025]:  
>> [1551924307.6674] audit: op="connection-activate"
>> uuid="740fc662-1d75-42ae-98ed-0ce92a0ca3dc" name="enp2s0" pid=14783 
>> uid=0 result="fail" reason="Can not find a master for enp2s0: Master 
>> connection not found or invalid"
>> Mar 07 10:05:07 localhost.localdomain network[14650]: Bringing up 
>> interface enp2s0:  Error: Connection activation failed: Can not find 
>> a master for enp2s0: Master connection not found or invalid
>>
>>
>> I recommend you disable NM altogether - it causes nothing but problems.
>> If you want a bit more detailed network configuration overview check 
>> https://www.shapeblue.com/networking-kvm-for-cloudstack-2018-revisit-
>> for-centos7-and-ubuntu-18-04/
>>
>> Regards,
>> Dag Sonstebo
>> Cloud Architect
>> ShapeBlue
>>
>>
>> On 07/03/2019, 03:05, "Larry Sevilla" 
>> wrote:
>>
>> Hi Andrija,
>>
>> Got the same error message, but different details.
>>
>> systemctl status network.service
>> https://pastebin.com/NjtXUwci
>>
>> journalctl -xe
>> https://pastebin.com/Kn0Dggmm
>>
>> My network settings:
>> https://pastebin.com/JDGJJYVc
>>
>>
>> On Wed, Mar 6, 2019 at 4:57 PM Andrija Panic < 
>> andrija.pa...@shapeblue.com>
>> wrote:
>>
>> > Hi Larry,
>> >
>> > Log says: Bridge support not available: brctl not found
>> >
>> > Fix with: yum install bridge-utils
>> >
>> > Let me know once you are finished with guide - you are perfect 
>> for testing
>> > this guide (since new to CentOS completely)
>> >
>> > I will update the guide once you are done with it - let me know 
>> please!
>> >
>> > Kind regards,
>> > Andrija
>> >
>> > andrija.pa...@shapeblue.com
>> > www.shapeblue.com
>> > Amadeus House, Floral Street, London  WC2E 9DPUK
>> > @shapeblue
>> >
>> >
>> >
>> >
>> > -Original Message-
>> > From: Larry Sevilla 
>> > Sent: 06 March 2019 09:45
>> > To: users@cloudstack.apache.org; cen...@centos.org
>> > Subject: Network Bridge - error
>> >
>> > Hi,
>> >
>> > I'm new to CentOS. (I have tried Debian, Ubuntu, Devuan & Gentoo).
>> > And also new to CloudStack.
>> >
>> > I'm trying the Quick Installation Guide of CloudStack as pointed by
>> > Andrija Panic of ShapeBlue
>> >
>> https://github.com/apache/cloudstack-documentation/blob/master/source/quickinstallationguide/qig.rst
>> >
>> > At Configuring the Network topic, I got an error:
>> >
>> > [root@localhost ~]# systemctl restart network Job for 
>> network.service
>> > failed because the control process exited with error code.  See 
>> "systemctl
>> > status network.service" and "journalctl -xe"
>> > for details.
>> >
>> > systemctl status network.service
>> > https://pastebin.com/KzBqJN65
>> >
>> > journalctl -xe
>> > https://pastebin.com/PLQGuLeR
>> >
>>
>>
>>
>> dag.sonst...@shapeblue.com
>> www.shapeblue.com
>> Amadeus House, Floral Street, London  WC2E 9DPUK @shapeblue
>>
>>
>>
>>

RE: cloudstack usage server issue

[Andrija Panic]

Hi Konstantin,

Thanks for raising the issue - seems it's really broken in 4.12 (usage server)

[root@ref-trl-1013-k-M7-apanic-mgmt1 usage]# grep -i usage 
/etc/cloudstack/usage/db.properties | grep invalid
db.usage.name=invalid_database_name
db.usage.port=invalid_port
db.usage.username=invalid_username
db.usage.password=invalid_password!
[root@ref-trl-1013-k-M7-apanic-mgmt1 usage]# tail -n 5 
/var/log/cloudstack/usage/usage.log
2019-03-07 22:40:36,738 INFO  [cloud.usage.UsageManagerImpl] (main:null) 
(logid:) Implementation Version is 4.12.0.0
2019-03-07 22:40:37,569 DEBUG [cloud.usage.UsageManagerImpl] (main:null) 
(logid:) Usage stats aggregation time zone: GMT
2019-03-07 22:40:37,570 DEBUG [cloud.usage.UsageManagerImpl] (main:null) 
(logid:) Execution Time: Thu Mar 07 22:28:00 UTC 2019
2019-03-07 22:40:37,570 DEBUG [cloud.usage.UsageManagerImpl] (main:null) 
(logid:) Current Time: Thu Mar 07 22:40:37 UTC 2019
2019-03-07 22:40:37,575 INFO  [cloud.usage.UsageServer] (main:null) (logid:) 
UsageServer ready..

So even with all usage server settings in db.properties, it's still running 
happily 

And yes, the usage job doesn't kick in at all - but let me let it wait until 
morning and if no job run (I configured hourly run), I will raise the blocker 
issue - or eve  better - you can do it for that matter - as a valuable 
community member 😊 )

Feel free to create an GitHub issue (on link I shared before) and I will 
comment and please feel free to vote -1 on the mailing list for 4.12 RC4 - and 
please reference the issue you raised previously on GitHub.

Kind regards,
Andrija

andrija.pa...@shapeblue.com 
www.shapeblue.com
Amadeus House, Floral Street, London  WC2E 9DPUK
@shapeblue
  
 


-Original Message-
From: Andrija Panic  
Sent: 07 March 2019 20:53
To: users@cloudstack.apache.org
Subject: RE: cloudstack usage server issue

Hi Konstantin,

Issues/bugs are raised here https://github.com/apache/cloudstack/issues - so 
please feel free to raise it ASAP - since we have a RC4 voting process in place 
! (it's very simple) - also feel free to jump on the voting email thread 
(before giving  -1, let's first try to reproduce the issue once more? )

I will try to look into this problem, but can't promise anything (into 4.12 
issue).

As for the very first link you shared - the jump in time visible in logs - is 
this because of restart, right ?

Also as for aggregation period, 1440 is default (and in production, believe me, 
should be kept this way if possible).
Hourly jobs are ok for testing (value of 60 minutes), not sure setting 5 
(minutes) makes sense (perhaps for testing, but I would better stick to 60 min 
period, and then change execution time to 5min from now() to be able to test 
(and restart usage service) - please give it some time to actually start 
processing data...

As for the issues changing db.properties file, you should have it like 
following:

ls -la /etc/cloudstack/usage/
total 4
lrwxrwxrwx. 1 root root   40 Feb 13 20:27 db.properties -> 
/etc/cloudstack/management/db.properties
lrwxrwxrwx. 1 root root   30 Feb 13 20:27 key -> /etc/cloudstack/management/key
-rw-r--r--. 1 root root 2980 Mar  7 17:22 log4j-cloud.xml First two  are links 
as you see - assuming you tried to change correct values in db.properties (to 
make usage fail to connect to DB), that should work, otherwise, it's a bug...

Kind regards,
Andrija

andrija.pa...@shapeblue.com
www.shapeblue.com
Amadeus House, Floral Street, London  WC2E 9DPUK @shapeblue
  
 


-Original Message-
From: Konstantin 
Sent: 07 March 2019 16:59
To: users@cloudstack.apache.org
Subject: Re: cloudstack usage server issue

I reinstalled 4.11.2 and logs are changed. New error appear

https://i.pozitis.ru/77a661462b182d54a6405484074dcee2.txt

2019-03-07 15:50:00,001 INFO  [cloud.usage.UsageManagerImpl]
(Usage-Job-1:null) (logid:) starting usage job...
2019-03-07 15:50:00,014 DEBUG [cloud.usage.UsageManagerImpl]
(Usage-Job-1:null) (logid:) Not owner of usage job, skipping...
2019-03-07 15:50:00,014 INFO  [cloud.usage.UsageManagerImpl]
(Usage-Job-1:null) (logid:) usage job complete

I cleaned the usage_job table from the records and FINALLY its start working!


How to submit a bug report for the version  4.12.0.0 ?

Could you please do it on my behalf?




On Thu, Mar 7, 2019 at 6:23 PM Konstantin 
wrote:

> Dear Andrija,
>
> Thanks a lot for the attention to my issue.
>
> There is logs from usage server:
>
> https://i.pozitis.ru/80d76e3cc5e605be037a088fd014a986.txt
>
> There is config
> https://i.pozitis.ru/8419e465b3ffa63b4d677ae5be85688a.txt
>
> its looks like usage server just ignore cloud_usage DB settings, I 
> tried to put wrong credentials to db.usage.name=cloud_usage params, No 
> matter, Its load the usage details from db.cloud.name=cloud, I think
>
> Here is my global settings
>
> https://i.pozitis.ru/9d3d53fd86997dd649b0e8171ca14c73

RE: cloudstack usage server issue

[Andrija Panic]

Hi Konstantin,

Issues/bugs are raised here https://github.com/apache/cloudstack/issues - so 
please feel free to raise it ASAP - since we have a RC4 voting process in place 
! (it's very simple) - also feel free to jump on the voting email thread 
(before giving  -1, let's first try to reproduce the issue once more? )

I will try to look into this problem, but can't promise anything (into 4.12 
issue).

As for the very first link you shared - the jump in time visible in logs - is 
this because of restart, right ?

Also as for aggregation period, 1440 is default (and in production, believe me, 
should be kept this way if possible).
Hourly jobs are ok for testing (value of 60 minutes), not sure setting 5 
(minutes) makes sense (perhaps for testing, but I would better stick to 60 min 
period, and then change execution time to 5min from now() to be able to test 
(and restart usage service) - please give it some time to actually start 
processing data...

As for the issues changing db.properties file, you should have it like 
following:

ls -la /etc/cloudstack/usage/
total 4
lrwxrwxrwx. 1 root root   40 Feb 13 20:27 db.properties -> 
/etc/cloudstack/management/db.properties
lrwxrwxrwx. 1 root root   30 Feb 13 20:27 key -> /etc/cloudstack/management/key
-rw-r--r--. 1 root root 2980 Mar  7 17:22 log4j-cloud.xml
First two  are links as you see - assuming you tried to change correct values 
in db.properties (to make usage fail to connect to DB), that should work, 
otherwise, it's a bug...

Kind regards,
Andrija

andrija.pa...@shapeblue.com 
www.shapeblue.com
Amadeus House, Floral Street, London  WC2E 9DPUK
@shapeblue
  
 


-Original Message-
From: Konstantin  
Sent: 07 March 2019 16:59
To: users@cloudstack.apache.org
Subject: Re: cloudstack usage server issue

I reinstalled 4.11.2 and logs are changed. New error appear

https://i.pozitis.ru/77a661462b182d54a6405484074dcee2.txt

2019-03-07 15:50:00,001 INFO  [cloud.usage.UsageManagerImpl]
(Usage-Job-1:null) (logid:) starting usage job...
2019-03-07 15:50:00,014 DEBUG [cloud.usage.UsageManagerImpl]
(Usage-Job-1:null) (logid:) Not owner of usage job, skipping...
2019-03-07 15:50:00,014 INFO  [cloud.usage.UsageManagerImpl]
(Usage-Job-1:null) (logid:) usage job complete

I cleaned the usage_job table from the records and FINALLY its start working!


How to submit a bug report for the version  4.12.0.0 ?

Could you please do it on my behalf?




On Thu, Mar 7, 2019 at 6:23 PM Konstantin 
wrote:

> Dear Andrija,
>
> Thanks a lot for the attention to my issue.
>
> There is logs from usage server:
>
> https://i.pozitis.ru/80d76e3cc5e605be037a088fd014a986.txt
>
> There is config
> https://i.pozitis.ru/8419e465b3ffa63b4d677ae5be85688a.txt
>
> its looks like usage server just ignore cloud_usage DB settings, I 
> tried to put wrong credentials to db.usage.name=cloud_usage params, No 
> matter, Its load the usage details from db.cloud.name=cloud, I think
>
> Here is my global settings
>
> https://i.pozitis.ru/9d3d53fd86997dd649b0e8171ca14c73.txt
>
> I did checked management server log, parsed by "usage", here is 
> result, I see error checking health of usage server, usage server 
> running? false,
> heartbeat: Wed Mar 06 20:00:06 UTC 2019
>
> https://i.pozitis.ru/fd56fdc8a3f7d80fabdec8d4cf563325.txt
>
> I restarted both services, here is debug results
>
> https://i.pozitis.ru/d8a8c06ea31a6ec89e0ca3e5aa89b178.txt
>
> here is usage server logs after restart 
> https://i.pozitis.ru/3bf38d723d2ccfc0e3bb6be785225c48.txt
>
> Would you recommend something to check more?
>
>
> Regads,
> Konstantin
>
>
> On Thu, Mar 7, 2019 at 2:23 PM Andrija Panic 
> 
> wrote:
>
>> Hi Konstantin,
>>
>> Can you please usage upload the logs to somewhere (pastebin is OK) so 
>> we can check ?
>>
>> Did you/can you try with 4.11.2 ?
>>
>> Kind regards,
>> Andrija
>>
>> andrija.pa...@shapeblue.com
>> www.shapeblue.com
>> Amadeus House, Floral Street, London  WC2E 9DPUK @shapeblue
>>
>>
>>
>>
>> -Original Message-
>> From: Konstantin 
>> Sent: 06 March 2019 23:20
>> To: users@cloudstack.apache.org
>> Subject: cloudstack usage server issue
>>
>>  I did installed the management server, connect it to ESX HV, the control
>> part working fine.
>>
>> I deployed the usage server with all guidelines awareness but have no
>> luck with processing a usage data from "cloud" DB to "cloud_usage" DB by
>> usage server.
>>
>> I tried to configure, reconfigure and even reinstall everything, but the
>> last usage server log record is always same:
>>
>> 2019-03-06 18:21:28,903 DEBUG [cloud.usage.UsageManagerImpl] (main:null

RE: cloudstack usage server issue

[Andrija Panic]

Sorry, what OS are you using for management/usage server (they are collocated 
on same server, right ?)
Is this RC4 release or some previous RC release, that you are testing ?

andrija.pa...@shapeblue.com 
www.shapeblue.com
Amadeus House, Floral Street, London  WC2E 9DPUK
@shapeblue
  
 


-Original Message-
From: Andrija Panic 
Sent: 07 March 2019 20:53
To: users@cloudstack.apache.org
Subject: RE: cloudstack usage server issue

Hi Konstantin,

Issues/bugs are raised here https://github.com/apache/cloudstack/issues - so 
please feel free to raise it ASAP - since we have a RC4 voting process in place 
! (it's very simple) - also feel free to jump on the voting email thread 
(before giving  -1, let's first try to reproduce the issue once more? )

I will try to look into this problem, but can't promise anything (into 4.12 
issue).

As for the very first link you shared - the jump in time visible in logs - is 
this because of restart, right ?

Also as for aggregation period, 1440 is default (and in production, believe me, 
should be kept this way if possible).
Hourly jobs are ok for testing (value of 60 minutes), not sure setting 5 
(minutes) makes sense (perhaps for testing, but I would better stick to 60 min 
period, and then change execution time to 5min from now() to be able to test 
(and restart usage service) - please give it some time to actually start 
processing data...

As for the issues changing db.properties file, you should have it like 
following:

ls -la /etc/cloudstack/usage/
total 4
lrwxrwxrwx. 1 root root   40 Feb 13 20:27 db.properties -> 
/etc/cloudstack/management/db.properties
lrwxrwxrwx. 1 root root   30 Feb 13 20:27 key -> /etc/cloudstack/management/key
-rw-r--r--. 1 root root 2980 Mar  7 17:22 log4j-cloud.xml First two  are links 
as you see - assuming you tried to change correct values in db.properties (to 
make usage fail to connect to DB), that should work, otherwise, it's a bug...

Kind regards,
Andrija

-Original Message-
From: Konstantin 
Sent: 07 March 2019 16:59
To: users@cloudstack.apache.org
Subject: Re: cloudstack usage server issue

I reinstalled 4.11.2 and logs are changed. New error appear

https://i.pozitis.ru/77a661462b182d54a6405484074dcee2.txt

2019-03-07 15:50:00,001 INFO  [cloud.usage.UsageManagerImpl]
(Usage-Job-1:null) (logid:) starting usage job...
2019-03-07 15:50:00,014 DEBUG [cloud.usage.UsageManagerImpl]
(Usage-Job-1:null) (logid:) Not owner of usage job, skipping...
2019-03-07 15:50:00,014 INFO  [cloud.usage.UsageManagerImpl]
(Usage-Job-1:null) (logid:) usage job complete

I cleaned the usage_job table from the records and FINALLY its start working!


How to submit a bug report for the version  4.12.0.0 ?

Could you please do it on my behalf?




On Thu, Mar 7, 2019 at 6:23 PM Konstantin 
wrote:

> Dear Andrija,
>
> Thanks a lot for the attention to my issue.
>
> There is logs from usage server:
>
> https://i.pozitis.ru/80d76e3cc5e605be037a088fd014a986.txt
>
> There is config
> https://i.pozitis.ru/8419e465b3ffa63b4d677ae5be85688a.txt
>
> its looks like usage server just ignore cloud_usage DB settings, I 
> tried to put wrong credentials to db.usage.name=cloud_usage params, No 
> matter, Its load the usage details from db.cloud.name=cloud, I think
>
> Here is my global settings
>
> https://i.pozitis.ru/9d3d53fd86997dd649b0e8171ca14c73.txt
>
> I did checked management server log, parsed by "usage", here is 
> result, I see error checking health of usage server, usage server 
> running? false,
> heartbeat: Wed Mar 06 20:00:06 UTC 2019
>
> https://i.pozitis.ru/fd56fdc8a3f7d80fabdec8d4cf563325.txt
>
> I restarted both services, here is debug results
>
> https://i.pozitis.ru/d8a8c06ea31a6ec89e0ca3e5aa89b178.txt
>
> here is usage server logs after restart 
> https://i.pozitis.ru/3bf38d723d2ccfc0e3bb6be785225c48.txt
>
> Would you recommend something to check more?
>
>
> Regads,
> Konstantin
>
>
> On Thu, Mar 7, 2019 at 2:23 PM Andrija Panic 
> 
> wrote:
>
>> Hi Konstantin,
>>
>> Can you please usage upload the logs to somewhere (pastebin is OK) so 
>> we can check ?
>>
>> Did you/can you try with 4.11.2 ?
>>
>> Kind regards,
>> Andrija
>>
>> andrija.pa...@shapeblue.com
>> www.shapeblue.com
>> Amadeus House, Floral Street, London  WC2E 9DPUK @shapeblue
>>
>>
>>
>>
>> -Original Message-
>> From: Konstantin 
>> Sent: 06 March 2019 23:20
>> To: users@cloudstack.apache.org
>> Subject: cloudstack usage server issue
>>
>>  I did installed the management server, connect it to ESX HV, the 
>> control part working fine.
>>
>> I deployed the usage server with all guidelines awareness but have no 
>> luck with processin

RE: cloudstack usage server issue

[Andrija Panic]

Hi Konstantin,

Can you please usage upload the logs to somewhere (pastebin is OK) so we can 
check ?

Did you/can you try with 4.11.2 ?
 
Kind regards,
Andrija

andrija.pa...@shapeblue.com 
www.shapeblue.com
Amadeus House, Floral Street, London  WC2E 9DPUK
@shapeblue
  
 


-Original Message-
From: Konstantin  
Sent: 06 March 2019 23:20
To: users@cloudstack.apache.org
Subject: cloudstack usage server issue

 I did installed the management server, connect it to ESX HV, the control part 
working fine.

I deployed the usage server with all guidelines awareness but have no luck with 
processing a usage data from "cloud" DB to "cloud_usage" DB by usage server.

I tried to configure, reconfigure and even reinstall everything, but the last 
usage server log record is always same:

2019-03-06 18:21:28,903 DEBUG [cloud.usage.UsageManagerImpl] (main:null)
(logid:) Checking to see if usage.vmops.pid exists.
2019-03-06 18:21:28,903 INFO  [cloud.usage.UsageManagerImpl] (main:null)
(logid:) Implementation Version is 4.12.0.0
2019-03-06 18:21:30,865 DEBUG [cloud.usage.UsageManagerImpl] (main:null)
(logid:) Usage stats aggregation time zone: UTC
2019-03-06 18:21:30,866 DEBUG [cloud.usage.UsageManagerImpl] (main:null)
(logid:) Execution Time: Wed Mar 06 18:22:00 UTC 2019
2019-03-06 18:21:30,866 DEBUG [cloud.usage.UsageManagerImpl] (main:null)
(logid:) Current Time: Wed Mar 06 18:21:30 UTC 2019
2019-03-06 18:21:30,874 INFO  [cloud.usage.UsageServer] (main:null)
(logid:) UsageServer ready...

As you may see, I played with starting time to force usage server to start the 
job, but no luck...
Its never gave me any logs written and any action taken after " UsageServer 
ready... "
I tried to call

[root@cloudstack usage]# cloudmonkey generateUsageRecords
startdate=2018-09-01 enddate=2019-09-30
success = True

but the "cloud_usage" DB is always empty, the all tables are empty, no single 
line in it.
In the same time, the "cloud" DB and usage_event table is full of data with 
prossessed column "0" for any record in the table.

Do you know how to force usage server to start the job?

here is my usage params info:

enable.usage.servertrue
publish.usage.eventstrue
quota.usage.smtp.connection.timeout60
quota.usage.smtp.host
quota.usage.smtp.password
quota.usage.smtp.port
quota.usage.smtp.sender
quota.usage.smtp.useAuth
quota.usage.smtp.user
usage.aggregation.timezoneUTC
usage.execution.timezoneUTC
usage.sanity.check.interval1
usage.snapshot.virtualsize.selectfalse
usage.stats.job.aggregation.range5 //I changed it from 5 to 60 and 1440
with same negative result
usage.stats.job.exec.time18:22 /changed many times to fire job, no luck


Regards,
Konstantin

RE: TCP port of libvird, possible typo in Quick Installation Guide - CentOS 6

[Andrija Panic]

Hi Larry, 

strictly technically speaking that is acceptable port - but you are right, 
should be a default one from libvirtd.conf - updated in docs MASTER branch. 
(centos6 guide will still keep old one)

Kind regards
Andrija

andrija.pa...@shapeblue.com 
www.shapeblue.com
Amadeus House, Floral Street, London  WC2E 9DPUK
@shapeblue
  
 


-Original Message-
From: Larry Sevilla  
Sent: 07 March 2019 09:08
To: users@cloudstack.apache.org
Subject: TCP port of libvird, possible typo in Quick Installation Guide - 
CentOS 6

Hi,

attn: Andrija

I have followed the Quick Installation Guide using CentOS 6 at:
http://docs.cloudstack.apache.org/en/4.11.2.0/quickinstallationguide/qig.html

In the Libvirt Configuration and file /etc/libvirt/libvirtd.conf, the commented 
line is:
#tcp_port = "16509"

In the QIG, its "16059".
There's a possible typo the QIG.

anyway, I have successfully configured my installation.

Currently, learning how to use.

RE: Network Bridge - error

[Andrija Panic]

Hi Larry,

Log says: Bridge support not available: brctl not found

Fix with: yum install bridge-utils 

Let me know once you are finished with guide - you are perfect for testing this 
guide (since new to CentOS completely)

I will update the guide once you are done with it - let me know please!

Kind regards,
Andrija

andrija.pa...@shapeblue.com 
www.shapeblue.com
Amadeus House, Floral Street, London  WC2E 9DPUK
@shapeblue
  
 


-Original Message-
From: Larry Sevilla  
Sent: 06 March 2019 09:45
To: users@cloudstack.apache.org; cen...@centos.org
Subject: Network Bridge - error

Hi,

I'm new to CentOS. (I have tried Debian, Ubuntu, Devuan & Gentoo).
And also new to CloudStack.

I'm trying the Quick Installation Guide of CloudStack as pointed by Andrija 
Panic of ShapeBlue 
https://github.com/apache/cloudstack-documentation/blob/master/source/quickinstallationguide/qig.rst

At Configuring the Network topic, I got an error:

[root@localhost ~]# systemctl restart network Job for network.service failed 
because the control process exited with error code.  See "systemctl status 
network.service" and "journalctl -xe"
for details.

systemctl status network.service
https://pastebin.com/KzBqJN65

journalctl -xe
https://pastebin.com/PLQGuLeR

RE: Quick Installation Guide - Network Bridge - DELAY - two entries

[Andrija Panic]

Hi Larry,

That is redundant - please use delay=5 - I will update that guide now...

thx

andrija.pa...@shapeblue.com 
www.shapeblue.com
Amadeus House, Floral Street, London  WC2E 9DPUK
@shapeblue
  
 


-Original Message-
From: Larry Sevilla  
Sent: 06 March 2019 07:10
To: users@cloudstack.apache.org
Subject: Quick Installation Guide - Network Bridge - DELAY - two entries

Hi,

I followed the QIG as pointed by Andrija Panic (shapeblue) 
https://github.com/apache/cloudstack-documentation/blob/master/source/quickinstallationguide/qig.rst

But on the Network Bridge configuration
/etc/sysconfig/network-scripts/ifcfg-cloudbr0, there are two (2) settings of 
DELAY.  One is =5, then below =0.

Is this correct?

RE: No package mysql-connector-python available

[Andrija Panic]

Hi Larry,

Please use updated version of Quick install guide: 
https://github.com/apache/cloudstack-documentation/blob/master/source/quickinstallationguide/qig.rst

It's with CentOS 7.

Kind regards,
Andrija

andrija.pa...@shapeblue.com 
www.shapeblue.com
Amadeus House, Floral Street, London  WC2E 9DPUK
@shapeblue
  
 


-Original Message-
From: Dag Sonstebo  
Sent: 05 March 2019 10:02
To: users@cloudstack.apache.org
Subject: Re: No package mysql-connector-python available

It's in the EPEL repo:

mysql-connector-python.noarch1.1.6-1.el7   @epel

yum install epel-release
yum install mysql-connector-python

Regards,
Dag Sonstebo
Cloud Architect
ShapeBlue
 

On 05/03/2019, 08:18, "Larry Sevilla"  wrote:

thx for the reply.

But the page cannot be opened (on my side).  Loading (forever)...
the "pbone.net" site cannot be accessed.

What does it basically say?


dag.sonst...@shapeblue.com
www.shapeblue.com
Amadeus House, Floral Street, London  WC2E 9DPUK @shapeblue
  
 

On Tue, Mar 5, 2019 at 3:52 PM Thomas Joseph  wrote:

> Did you search for it on Google?
>
>
> 
http://rpm.pbone.net/index.php3/stat/4/idpl/37815819/dir/redhat_el_6/com/mysql-connector-python-2.1.7-1.el6.x86_64.rpm.html
>
> On Tue, 5 Mar 2019, 7:47 am Larry Sevilla, 
> wrote:
>
> > Hi,
> >
> > I'm new to CloudStack.
> >
> > I'm trying to install ClouldStack by following the quick installation
> > guide:
> >
> >
> 
http://docs.cloudstack.apache.org/en/4.11.2.0/quickinstallationguide/qig.html
> >
> > I have installed CentOS 6.10 x86_64 minimal install CD.
> > (latest of ver 6)
> >
> > when I went to:
> >
> > # yum install mysql-connector-python
> >
> > it gives a message:
> > "No package mysql-connector-python available"
> >
>

Re: KVM Host HA and power lost to host.

[Andrija Panic]

Jon,

not an expert on particular implementation, but obviously your host needs
power, so its IPMI/BMC/iLo/iDRAC/etc. controller can be contacted and host
fenced. Redundant PSU with different power sources is expected (defacto
standard in production).

Kind regards,
Andrija

On Mon, 4 Mar 2019 at 12:19, Jon Marshall  wrote:

>
> I have KVM Host HA enabled and power is lost to one of the compute nodes.
>  The host has it's state marked as alert and the HA states go through
> degraded to suspect to Fencing.
>
> The problem is that the host is never fenced because there is no power to
> it so none of the OOBM commands work which means the VMs are never migrated.
>
>  From the management server logs -
>
> 2019-03-04 11:02:48,288 WARN  [o.a.c.h.t.BaseHATask]
> (pool-6-thread-9:null) (logid:d0a19f20) Exception occurred while running
> FenceTask on a resource:
> org.apache.cloudstack.ha.provider.HAFenceException: OOBM service is not
> configured or enabled for this host dcp-cscn2.local
> org.apache.cloudstack.ha.provider.HAFenceException: OOBM service is not
> configured or enabled for this host dcp-cscn2.local
> at
> org.apache.cloudstack.kvm.ha.KVMHAProvider.fence(KVMHAProvider.java:99)
> at
> org.apache.cloudstack.kvm.ha.KVMHAProvider.fence(KVMHAProvider.java:42)
> at
> org.apache.cloudstack.ha.task.FenceTask.performAction(FenceTask.java:42)
> at
> org.apache.cloudstack.ha.task.BaseHATask$1.call(BaseHATask.java:86)
> at
> org.apache.cloudstack.ha.task.BaseHATask$1.call(BaseHATask.java:83)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: com.cloud.utils.exception.CloudRuntimeException: Out-of-band
> Management action (OFF) on host (b53122bc-1446-4ffd-a179-e363ad0d541f)
> failed with error: Get Auth Capabilities error
> Error issuing Get Channel Authentication Capabilities request
> Error: Unable to establish IPMI v2 / RMCP+ session
>
> at
> org.apache.cloudstack.outofbandmanagement.OutOfBandManagementServiceImpl.executePowerOperation(OutOfBandManagementServiceImpl.java:423)
> at sun.reflect.GeneratedMethodAccessor225.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ... 21 more
>
>
> which begs the question how is this meant to work for a host whose power
> has failed.
>
>
> If I turn off KVM Host HA and change the ping interval to 30 and ping
> timeout to 2 then the VMs failover to another host within 5 mins.
>
> I understand what Host HA is meant for but it seems for a failed host in
> terms of power it doesn't work.
>
> Jon
>


-- 

Andrija Panić

Re: Max template size to download

[Andrija Panic]

max.template.iso.size The maximum size for a downloaded template or ISO (in
GB).
global setting.

defaults to 50, afaik.

On Sat, 2 Mar 2019 at 13:08, Fariborz Navidan  wrote:

> Hello,
>
> I have converted a 65GB vmdk disk to QCOW2 and want to upload it as a
> template so I can easily deploy a VM from it in order to migrate the VM to
> KVM. However, when registering the template from URL, I get an error that
> states the download size is too large. I've looked into global settings but
> couldn't find any setting for it.
>
> Any idea how to increase max download size for templates?
>
> Best Regards
>


-- 

Andrija Panić

Re: Not able to access the vm from outside network

[Andrija Panic]

Hi,

ACS version, hypervisor (version) ?

On Thu, 28 Feb 2019 at 14:58, soundar rajan  wrote:

> Hi,
>
> VM outbound is working fine. Inbound is not  not able to access from
> outside network
>
> Error Log
> 2019-02-28 18:12:25,112 - Failed to network rule !
> Traceback (most recent call last):
>   File "/usr/share/cloudstack-common/scripts/vm/network/security_group.py",
> line 995, in add_network_rules
> default_network_rules(vmName, vm_id, vm_ip, vm_ip6, vmMac, vif, brname,
> sec_ips)
>   File "/usr/share/cloudstack-common/scripts/vm/network/security_group.py",
> line 490, in default_network_rules
> if ips[0] == "0":
> IndexError: list index out of range
> 2019-02-28 18:13:16,635 - Executing command: cleanup_rules
> 2019-02-28 18:13:16,645 -  Vms on the host : ['i-2-40-VM', 'i-2-90-VM',
> 'i-2-112-VM']
> 2019-02-28 18:13:16,645 - iptables-save | grep -P '^:(?!.*-(def|eg))' | awk
> '{sub(/^:/, "", $1) ; print $1}' | sort | uniq
> 2019-02-28 18:13:16,671 -  iptables chains in the host :['BF-cloudbr0',
> 'BF-cloudbr0-IN', 'BF-cloudbr0-OUT', 'FORWARD', 'i-2-112-VM', 'i-2-40-VM',
> 'i-2-90-VM', 'INPUT', 'OUTPUT', 'POSTROUTING', 'PREROUTING', '']
> 2019-02-28 18:13:16,672 - grep -E '^ebtable_' /proc/modules | cut -f1 -d' '
> | sed s/ebtable_//
> 2019-02-28 18:13:16,693 - ebtables -t nat -L | awk '/chain:/ {
> gsub(/(^.*chain: |-(in|out|ips).*)/, ""); print $1}' | sort | uniq
> 2019-02-28 18:13:16,716 - ebtables -t filter -L | awk '/chain:/ {
> gsub(/(^.*chain: |-(in|out|ips).*)/, ""); print $1}' | sort | uniq
> 2019-02-28 18:13:16,738 -  ebtables chains in the host: ['FORWARD,',
> 'INPUT,', 'OUTPUT,', '']
> 2019-02-28 18:13:16,739 - Cleaned up rules for 0 chains
> 2019-02-28 18:13:23,959 - Executing command: get_rule_logs_for_vms
>
> It happens to particular vm
>
> Please help..
>


-- 

Andrija Panić

Re: Re:Re: VMs Connection break under two isolate network

[Andrija Panic]

Hi,

in general yes it may make sense, a few years ago I was hitting same issue
with 10GB Intel NICs, but with KVM on Ubuntu 14 due to either issues with
kernel or the Intel driver itself - TSO and LRO should be disabled with
routing and bridging, since it's incompatible in those scenarios. ( i.e.
https://downloadmirror.intel.com/14687/eng/readme.txt and search for
"incompatible" , or ,
http://ehaselwanter.com/en/blog/2014/11/02/mtu-issue--nope-it-is-lro-with-bridge-and-bond/
).
i.e. LRO/TSO should be automatically turned off when you add NIC to a
bridge (i.e. CentOS6 was fine, but Ubuntu 14 had issues)...

Cheers
Andrija

On Sun, 24 Feb 2019 at 08:41, Haijiao <18602198...@163.com> wrote:

> Hi, Dag and All
>
>
> Yes, we are using active-active(mode 7) for bond.
>
> VM A1 ---> VR A(Isloated Network A) > VR B(Isolated Network B)  >
> VM B1
>
>
>
> After rounds of isoloation,  based on packet analysis,  it seems to us
> - the traffic between  VM A1 and VR A is normal
> - however, between  VR A and VM B1,   VR A receives packets
> aknowledges from VM B1 which VR A thinks they has not sent thru it yet.
> - Then,  VR A reset the session, causing the traffic dropped.
>
>
> For  testing purpose, we turned off the TSO (tcp-segmentation-offload ）on
> XenServer network adpaters by command 'ethtool -k eth0 tso off',   the
> issue is just gone, we can run iperf for testing without any drop for a
> couple of hours.
>
>
> Does it make sense ?  Any improvement can be implemented from ACS side ?
>
>
> Thanks !
>
>
>
>
> 在2019年02月22 23时20分, "Haijiao"<18602198...@163.com>写道:
>
>
> Thanks Dag,  you are always helpful !
>
>
> We will look into your sharing and come back.
>
>
>
>
>
>
>
> 在2019年02月22 17时26分, "Dag Sonstebo"写道:
>
> Hi Haijiao,
>
> We've come across similar things in the past. In short - what is your
> XenServer bond mode? Is it active-active (mode 7) or LACP (mode 4)? (see
> https://support.citrix.com/article/CTX137599)
>
> In short if your switches don't keep up with MAC address changes on the XS
> hosts then you will get traffic flapping with intermittent loss of
> connectivity (root cause is a MAC address moves to another uplink, but the
> switch only checks for changes every X seconds so it takes a while for it
> to catch up). LACP mode 4 has a much more robust mechanism for this but
> obviously needs configured both XS and switch end. Normal active-active
> (mode 7) seems to always cause problems.
>
> My general advise would be to simplify and just go active-passive (mode 1)
> - unless you really need the bandwidth this gives you a much more stable
> network backend.
>
> Regards,
> Dag Sonstebo
> Cloud Architect
> ShapeBlue
>
>
> On 22/02/2019, 07:14, "Haijiao" <18602198...@163.com> wrote:
>
>Hi, Devs and Community Users
>
>
>To be more specific,  our environment is built with
>* 2 Dell R740XD Servers + Dell Compellent Storage w/ iSCSI
>* Each server equiped with two Mellanox Connect-4 LX 25GbE network
> adapters, and configured with bond mode(active+active) in XenServer
>* CloudStack 4.11.2 LTS + XenServer 7.1CU2(LTS) Enterprise
>
>
>Everything goes fine with shared network, but the weird thing is if we
> setup 2 isolated networks,  try to use 'iperf',  'wget' or 'SCP' to test
> the network performance betwen two VMs located in these 2 isolated
> networks,  the traffic will drop to zero in about 200-300 seconds,  even
> though we were still able to ping or SSH VM B1 from A1 or verse.
>
>
>VM A1 ---> VR A(Isloated Network A) > VR B(Isolated Network B)
> > VM B1
>
>  
> 
>We have checked the configuration on switches, upgraded Mellanox driver
> for XenServer,  but no luck.
>Meanwhile, we can not re-produce this issue in another environment
> (XenServer 7.1CU2+ACS 4.11.2+ Intel Gb network).
>
>
> It seems it might be related to Mellanox adapter, but we have no idea
> what part we could possibly miss in this case.
>
>
>Any advice would be highly appreciated !   Thank you !
>
>
>在2019年02月22 13时09分, "gu haven"写道:
>
>
>hi ，all
>  I try iperf wget scp connection will break after 200 seconds ，Do
> need any optimization in vr ?
>
>environment infomation below:
>
>cloudstack 4.11.2
>
>xenserver 7.1 CU2 Enterprise
>
>NIC :MLNX 25GbE 2P ConnectX4LX
>
>bond mode in xenserver : acitce-active
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> dag.sonst...@shapeblue.com
> www.shapeblue.com
> Amadeus House, Floral Street, London  WC2E 9DPUK
> @shapeblue
>
>
>
>
>
>
>
>
>



-- 

Andrija Panić

RE: [VOTE] Apache CloudStack 4.12.0.0 [RC3]

[Andrija Panic]

-1

Blocker

Security Group rules not applied at all for XenServer 6.5 / Advanced Zone

https://github.com/apache/cloudstack/issues/3192


andrija.pa...@shapeblue.com 
www.shapeblue.com
Amadeus House, Floral Street, London  WC2E 9DPUK
@shapeblue
  
 


-Original Message-
From: Gabriel Beims Bräscher  
Sent: 22 February 2019 02:56
To: dev ; users 
Subject: [VOTE] Apache CloudStack 4.12.0.0 [RC3]

Hi All,

I've created a 4.12.0.0 release (RC3), with the following artifacts up for a 
vote:

Git Branch and Commit SH:
https://github.com/apache/cloudstack/tree/4.12.0.0-RC20190212T2301
https://github.com/apache/cloudstack/commit/e5b3aa4b5a5d1a25c79313cecd3ae1c9f074baca
Commit: e5b3aa4b5a5d1a25c79313cecd3ae1c9f074baca

Source release for 4.12.0.0-RC20190212T2301:
https://dist.apache.org/repos/dist/dev/cloudstack/4.12.0.0/

PGP release keys (signed using 25908455):
https://dist.apache.org/repos/dist/release/cloudstack/KEYS

The vote will be open for 3 business days (until 26th February).

For sanity in tallying the vote, can PMC members please be sure to indicate 
"(binding)" with their vote?

[ ] +1  approve
[ ] +0  no opinion
[ ] -1  disapprove (and reason why)

Additional information:

*Note:* the RC2 VOTE ("[VOTE] Apache CloudStack 4.12.0.0 [RC2]") had no blocker 
bug to be fixed; however, the VOTE process led to the need of an RC3. For more 
details, please follow the discussions at the respective email thread.

All Travis and Jenkins checks have passed [1] for the branch 
4.12.0.0-RC20190212T2301.

For users' convenience, packages are available in http://cloudstack.apt-get.eu/
4.12.0.0 RC3 is available for the following distros:
- Ubuntu 14.04, 16.04, and 18.04;
- CentOS6 and CentOS7.

Please, use the template 4.11.2 (located in [2]) when testing the RC3.
The release notes [3] still need to be updated.

[1] https://github.com/apache/cloudstack/pull/3189
[2] http://download.cloudstack.org/systemvm/4.11/
[3]
http://docs.cloudstack.apache.org/projects/cloudstack-release-notes/en/latest/index.html

Re: Templates for Apache Cloudstack

[Andrija Panic]

Good stuff Lucian, thanks.

On Thu, 21 Feb 2019 at 17:30, Nux!  wrote:

> Hi,
>
> I have discovered where the problem lies. Cloud-init has finally caught up
> with Cloudstack, we can call the set-password module at every boot now.
> So to get everything working we need to:
> 1. get rid of custom scripts for password
> 2. replace "set-passwords" with "[set-passwords, always]" in
> /etc/cloud/cloud.cfg
>
> I have started new build jobs for Ubuntu 18.04 Xen and KVM with the fix
> and will correct the other templates in the next few days.
>
> --
> Sent from the Delta quadrant using Borg technology!
>
> Nux!
> www.nux.ro
>
> - Original Message -
> > From: "Nux!" 
> > To: "asender" 
> > Cc: "users" 
> > Sent: Wednesday, 20 February, 2019 09:31:41
> > Subject: Re: Templates for Apache Cloudstack
>
> > Thanks for testing.
> > Machinina uses just a simple script to retrieve and change the password,
> all the
> > other templates use cloud-init which sometimes gets in the way ...
> >
> > I'll try to find where the problem is, when I do I'll get back to the
> list.
> >
> > --
> > Sent from the Delta quadrant using Borg technology!
> >
> > Nux!
> > www.nux.ro
> >
> > - Original Message -
> >> From: "asender" 
> >> To: "users" 
> >> Cc: "Nux!" 
> >> Sent: Tuesday, 19 February, 2019 21:41:37
> >> Subject: Re: Templates for Apache Cloudstack
> >
> >> Hi Nux,
> >>
> >> That spin appears to be fine (macchinina-xen.vhd) - I can reset password
> >> multiple times and it changes as expected. I believe this is just a
> >> template issue.
> >>
> >> All other templates appear to exhibit the issue though.
> >>
> >> -AS.
> >>
> >> On 2019-02-19 23:53, Nux! wrote:
> >>> Asender,
> >>>
> >>> Access to the lab sounds good, I'm in an unfortunate situation where I
> >>> actually don't have access to ACS anymore to test ...
> >>>
> >>> Actually, before that, can you give this a try?
> >>> http://dl.openvm.eu/cloudstack/macchinina/x86_64/
> >>>
> >>> This template has not changed in a long time and it always sort of
> >>> worked. If on KVM you can add it as virtio (not scsi) and password
> >>> enabled.
> >>>
> >>> Regards
> >>>
> >>> --
> >>> Sent from the Delta quadrant using Borg technology!
> >>>
> >>> Nux!
> >>> www.nux.ro
> >>>
> >>> - Original Message -
> >>>> From: "asender" 
> >>>> To: "users" 
> >>>> Cc: "Andrija Panic" 
> >>>> Sent: Tuesday, 19 February, 2019 02:59:55
> >>>> Subject: Re: Templates for Apache Cloudstack
> >>>
> >>>> Hi Andrija,
> >>>>
> >>>> Thanks for your response! I am on version 4.11.2. Virtual router is
> >>>> behaving...
> >>>>
> >>>> This seems to be a problem with the specific templates, I just wanted
> >>>> someone to double check the templates to see if they have the same
> >>>> issue.
> >>>>
> >>>> I think Nux will need to check all of them. :( If you want access to
> >>>> my
> >>>> lab Nux send me a separate email.
> >>>>
> >>>> -Adrian Sender
> >>>>
> >>>> On 2019-02-19 09:52, Andrija Panic wrote:
> >>>>> Hi,
> >>>>>
> >>>>> since you have logged in first time with password that CloudStack
> >>>>> gave
> >>>>> you,
> >>>>> that should mean that templates are OK - i.e. VM did fetch for that
> >>>>> first
> >>>>> time the pass from VR and set it locally for ROOT user.
> >>>>>
> >>>>> What version of ACS are you using ?
> >>>>>
> >>>>> Try to troubleshoot/check inside VR file similar to
> >>>>> /var/cache/cloud/password-192.168.1.1 (this 192.168.1.1 being the VR
> >>>>> interface / gateway IP of the network where VMs exist). If VM has
> >>>>> successfully fetched the pass from VR (from this file), than it
> >>>>> should
> >>>>> write a line like "192.168.1.10

Re: Change an instances NIC

[Andrija Panic]

Adam,

Makes sure to test it thoroughly.

i.e. you can also change Settings tab of the Stopped VM (add details like
rootDiskController = pvscsi or nicAdapter=Vmxnet3 etc) - in case of VMware
no changes are actually in place after VM is booted (i.e it doesn't work
yet), in contrast to KVM, where new setting ARE being used/applied when you
boot a VM...
This is a way ti override whatever is implicit setting of the OS Type you
are using (what Ivan suggested).

Cheers
Andrija

On Wed, 20 Feb 2019 at 15:59, Adam Witwicki  wrote:

> Cheers Ivan, that does the job
>
> Adam
>
> -Original Message-
> From: Ivan Kudryavtsev 
> Sent: 20 February 2019 14:27
> To: users 
> Subject: Re: Change an instances NIC
>
> ** This mail originated from OUTSIDE the Oakford corporate network. Treat
> hyperlinks and attachments in this email with caution. **
>
> Adam,
>
> just change OS family to the Linux for templates, ISOs which use your VMs
> and restart VMs via stop/start.
>
> ср, 20 февр. 2019 г., 8:39 Adam Witwicki awitwi...@oakfordis.com:
>
> > Hello,
> >
> > I would like to change an instances nic from e1000 to vmxnet3 or even
> > virtio, How can I do this?
> >
> > Is there something I can add to the user_vm_details table?
> >
> >
> > Thanks
> >
> > Adam
> >
> >
> > Disclaimer Notice:
> > This email has been sent by Oakford Technology Limited, while we have
> > checked this e-mail and any attachments for viruses, we can not
> > guarantee that they are virus-free. You must therefore take full
> > responsibility for virus checking.
> > This message and any attachments are confidential and should only be
> > read by those to whom they are addressed. If you are not the intended
> > recipient, please contact us, delete the message from your computer
> > and destroy any copies. Any distribution or copying without our prior
> > permission is prohibited.
> > Internet communications are not always secure and therefore Oakford
> > Technology Limited does not accept legal responsibility for this message.
> > The recipient is responsible for verifying its authenticity before
> > acting on the contents. Any views or opinions presented are solely
> > those of the author and do not necessarily represent those of Oakford
> Technology Limited.
> > Registered address: Oakford Technology Limited, The Manor House,
> > Potterne, Wiltshire. SN10 5PN.
> > Registered in England and Wales No. 5971519
> >
> >
> Disclaimer Notice:
> This email has been sent by Oakford Technology Limited, while we have
> checked this e-mail and any attachments for viruses, we can not guarantee
> that they are virus-free. You must therefore take full responsibility for
> virus checking.
> This message and any attachments are confidential and should only be read
> by those to whom they are addressed. If you are not the intended recipient,
> please contact us, delete the message from your computer and destroy any
> copies. Any distribution or copying without our prior permission is
> prohibited.
> Internet communications are not always secure and therefore Oakford
> Technology Limited does not accept legal responsibility for this message.
> The recipient is responsible for verifying its authenticity before acting
> on the contents. Any views or opinions presented are solely those of the
> author and do not necessarily represent those of Oakford Technology Limited.
> Registered address: Oakford Technology Limited, The Manor House, Potterne,
> Wiltshire. SN10 5PN.
> Registered in England and Wales No. 5971519
>
>

-- 

Andrija Panić

Re: CPU live migration question

[Andrija Panic]

Protip: don't restart via ACS Restart VM button (since identical XML will
be used as before) - do full stop /start cycle.

On Tue, Feb 19, 2019, 15:24 Ivan Kudryavtsev  The only problem is with live migration and when cpu frequency is set to
> the highest rate possible, while new cpu has lower rate.
>
> Other problem is when passthrough is set and cpus belong to various
> families like x5600 and E5-2600 V4. Then you should avoid passthrough and
> set in agent config lowest possible cpu among all cluster hosts. Next,
> restart agents and restart every VM to get new cpu model.
>
> вт, 19 февр. 2019 г., 8:40 Ugo Vasi ugo.v...@procne.it.invalid:
>
>> Hi Andrija,
>> is it possible to change agent.properties "on fly"?
>>
>> Il 19/02/19 12:59, Andrija Panic ha scritto:
>> > Ugo,
>> >
>> > as long as the CPU flags are same or newer than the existing hosts in
>> > cluster, it's 100% safe operation in regards to live migrations.
>> >
>> > But as Piotr said, if you are using some level of emulation of CPU model
>> > (versus hos-passthrough), then you might be good even with slighly older
>> > CPU on this new hosts that you want to add to a cluster.
>> >
>> > Do some reading around libvirt CPU model and simply just add one host
>> and
>> > test live migration - as simple as that.
>> >
>> > Kind regards,
>> >
>> > Andrija
>> >
>> > On Tue, 19 Feb 2019 at 12:39, Piotr Pisz  wrote:
>> >
>> >> Hi Ugo,
>> >>
>> >> [root@cstack-1 ~]# cat /etc/cloudstack/agent/agent.properties  |grep
>> passt
>> >> guest.cpu.mode=host-passthrough
>> >>
>> >> The CPU model determines everything, if you have a passthrough it will
>> be
>> >> difficult or not possible.
>> >> But if you have emulated it is no problem (eg you set up SandyBridge
>> >> everywhere).
>> >>
>> >> Regards,
>> >> Piotr
>> >>
>> >>
>> >> -Original Message-
>> >> From: Ugo Vasi 
>> >> Sent: Tuesday, February 19, 2019 12:27 PM
>> >> To: users@cloudstack.apache.org
>> >> Subject: CPU live migration question
>> >>
>> >> Hi all,
>> >> I would like to know if it is possible to add to an existing cluster a
>> host
>> >> that has the cpu different from that of the pre-existing hosts or which
>> >> features must be the same.
>> >>
>> >> I have read around that there may be problems at the libvirt level but
>> it
>> >> is
>> >> not clear to me which parameters must match.
>> >>
>> >> Thanks
>> >> --
>> >>
>> >> *Ugo Vasi* / System Administrator
>> >> ugo.v...@procne.it <mailto:ugo.v...@procne.it>
>> >>
>> >>
>> >>
>> >>
>> >> *Procne S.r.l.*
>> >> +39 0432 486 523
>> >> via Cotonificio, 45
>> >> 33010 Tavagnacco (UD)
>> >> www.procne.it <http://www.procne.it/>
>> >>
>> >>
>> >> Le informazioni contenute nella presente comunicazione ed i relativi
>> >> allegati possono essere riservate e sono, comunque, destinate
>> >> esclusivamente
>> >> alle persone od alla Società sopraindicati. La diffusione,
>> distribuzione
>> >> e/o
>> >> copiatura del documento trasmesso da parte di qualsiasi soggetto
>> diverso
>> >> dal
>> >> destinatario è proibita sia ai sensi dell'art. 616 c.p., che ai sensi
>> del
>> >> Decreto Legislativo n. 196/2003 "Codice in materia di protezione dei
>> dati
>> >> personali". Se avete ricevuto questo messaggio per errore, vi
>> preghiamo di
>> >> distruggerlo e di informare immediatamente Procne S.r.l. scrivendo all'
>> >> indirizzo e-mail i...@procne.it <mailto:i...@procne.it>.
>> >>
>> >>
>> >>
>>
>>
>> --
>>
>> *Ugo Vasi* / System Administrator
>> ugo.v...@procne.it <mailto:ugo.v...@procne.it>
>>
>>
>>
>>
>> *Procne S.r.l.*
>> +39 0432 486 523
>> via Cotonificio, 45
>> 33010 Tavagnacco (UD)
>> www.procne.it <http://www.procne.it/>
>>
>>
>> Le informazioni contenute nella presente comunicazione ed i relativi
>> allegati possono essere riservate e sono, comunque, destinate
>> esclusivamente alle persone od alla Società sopraindicati. La
>> diffusione, distribuzione e/o copiatura del documento trasmesso da parte
>> di qualsiasi soggetto diverso dal destinatario è proibita sia ai sensi
>> dell'art. 616 c.p., che ai sensi del Decreto Legislativo n. 196/2003
>> "Codice in materia di protezione dei dati personali". Se avete ricevuto
>> questo messaggio per errore, vi preghiamo di distruggerlo e di informare
>> immediatamente Procne S.r.l. scrivendo all' indirizzo e-mail
>> i...@procne.it <mailto:i...@procne.it>.
>>
>>

Re: CPU live migration question

[Andrija Panic]

Not on the fly - but yes, change model of cpu, and restart agent. Affects
only VMs started after that change.

On Tue, Feb 19, 2019, 14:40 Ugo Vasi  Hi Andrija,
> is it possible to change agent.properties "on fly"?
>
> Il 19/02/19 12:59, Andrija Panic ha scritto:
> > Ugo,
> >
> > as long as the CPU flags are same or newer than the existing hosts in
> > cluster, it's 100% safe operation in regards to live migrations.
> >
> > But as Piotr said, if you are using some level of emulation of CPU model
> > (versus hos-passthrough), then you might be good even with slighly older
> > CPU on this new hosts that you want to add to a cluster.
> >
> > Do some reading around libvirt CPU model and simply just add one host and
> > test live migration - as simple as that.
> >
> > Kind regards,
> >
> > Andrija
> >
> > On Tue, 19 Feb 2019 at 12:39, Piotr Pisz  wrote:
> >
> >> Hi Ugo,
> >>
> >> [root@cstack-1 ~]# cat /etc/cloudstack/agent/agent.properties  |grep
> passt
> >> guest.cpu.mode=host-passthrough
> >>
> >> The CPU model determines everything, if you have a passthrough it will
> be
> >> difficult or not possible.
> >> But if you have emulated it is no problem (eg you set up SandyBridge
> >> everywhere).
> >>
> >> Regards,
> >> Piotr
> >>
> >>
> >> -Original Message-
> >> From: Ugo Vasi 
> >> Sent: Tuesday, February 19, 2019 12:27 PM
> >> To: users@cloudstack.apache.org
> >> Subject: CPU live migration question
> >>
> >> Hi all,
> >> I would like to know if it is possible to add to an existing cluster a
> host
> >> that has the cpu different from that of the pre-existing hosts or which
> >> features must be the same.
> >>
> >> I have read around that there may be problems at the libvirt level but
> it
> >> is
> >> not clear to me which parameters must match.
> >>
> >> Thanks
> >> --
> >>
> >> *Ugo Vasi* / System Administrator
> >> ugo.v...@procne.it <mailto:ugo.v...@procne.it>
> >>
> >>
> >>
> >>
> >> *Procne S.r.l.*
> >> +39 0432 486 523
> >> via Cotonificio, 45
> >> 33010 Tavagnacco (UD)
> >> www.procne.it <http://www.procne.it/>
> >>
> >>
> >> Le informazioni contenute nella presente comunicazione ed i relativi
> >> allegati possono essere riservate e sono, comunque, destinate
> >> esclusivamente
> >> alle persone od alla Società sopraindicati. La diffusione, distribuzione
> >> e/o
> >> copiatura del documento trasmesso da parte di qualsiasi soggetto diverso
> >> dal
> >> destinatario è proibita sia ai sensi dell'art. 616 c.p., che ai sensi
> del
> >> Decreto Legislativo n. 196/2003 "Codice in materia di protezione dei
> dati
> >> personali". Se avete ricevuto questo messaggio per errore, vi preghiamo
> di
> >> distruggerlo e di informare immediatamente Procne S.r.l. scrivendo all'
> >> indirizzo e-mail i...@procne.it <mailto:i...@procne.it>.
> >>
> >>
> >>
>
>
> --
>
> *Ugo Vasi* / System Administrator
> ugo.v...@procne.it <mailto:ugo.v...@procne.it>
>
>
>
>
> *Procne S.r.l.*
> +39 0432 486 523
> via Cotonificio, 45
> 33010 Tavagnacco (UD)
> www.procne.it <http://www.procne.it/>
>
>
> Le informazioni contenute nella presente comunicazione ed i relativi
> allegati possono essere riservate e sono, comunque, destinate
> esclusivamente alle persone od alla Società sopraindicati. La
> diffusione, distribuzione e/o copiatura del documento trasmesso da parte
> di qualsiasi soggetto diverso dal destinatario è proibita sia ai sensi
> dell'art. 616 c.p., che ai sensi del Decreto Legislativo n. 196/2003
> "Codice in materia di protezione dei dati personali". Se avete ricevuto
> questo messaggio per errore, vi preghiamo di distruggerlo e di informare
> immediatamente Procne S.r.l. scrivendo all' indirizzo e-mail
> i...@procne.it <mailto:i...@procne.it>.
>
>

Re: CPU live migration question

[Andrija Panic]

Ugo,

as long as the CPU flags are same or newer than the existing hosts in
cluster, it's 100% safe operation in regards to live migrations.

But as Piotr said, if you are using some level of emulation of CPU model
(versus hos-passthrough), then you might be good even with slighly older
CPU on this new hosts that you want to add to a cluster.

Do some reading around libvirt CPU model and simply just add one host and
test live migration - as simple as that.

Kind regards,

Andrija

On Tue, 19 Feb 2019 at 12:39, Piotr Pisz  wrote:

> Hi Ugo,
>
> [root@cstack-1 ~]# cat /etc/cloudstack/agent/agent.properties  |grep passt
> guest.cpu.mode=host-passthrough
>
> The CPU model determines everything, if you have a passthrough it will be
> difficult or not possible.
> But if you have emulated it is no problem (eg you set up SandyBridge
> everywhere).
>
> Regards,
> Piotr
>
>
> -Original Message-
> From: Ugo Vasi 
> Sent: Tuesday, February 19, 2019 12:27 PM
> To: users@cloudstack.apache.org
> Subject: CPU live migration question
>
> Hi all,
> I would like to know if it is possible to add to an existing cluster a host
> that has the cpu different from that of the pre-existing hosts or which
> features must be the same.
>
> I have read around that there may be problems at the libvirt level but it
> is
> not clear to me which parameters must match.
>
> Thanks
> --
>
> *Ugo Vasi* / System Administrator
> ugo.v...@procne.it 
>
>
>
>
> *Procne S.r.l.*
> +39 0432 486 523
> via Cotonificio, 45
> 33010 Tavagnacco (UD)
> www.procne.it 
>
>
> Le informazioni contenute nella presente comunicazione ed i relativi
> allegati possono essere riservate e sono, comunque, destinate
> esclusivamente
> alle persone od alla Società sopraindicati. La diffusione, distribuzione
> e/o
> copiatura del documento trasmesso da parte di qualsiasi soggetto diverso
> dal
> destinatario è proibita sia ai sensi dell'art. 616 c.p., che ai sensi del
> Decreto Legislativo n. 196/2003 "Codice in materia di protezione dei dati
> personali". Se avete ricevuto questo messaggio per errore, vi preghiamo di
> distruggerlo e di informare immediatamente Procne S.r.l. scrivendo all'
> indirizzo e-mail i...@procne.it .
>
>
>

-- 

Andrija Panić

Re: Templates for Apache Cloudstack

[Andrija Panic]

Hi,

since you have logged in first time with password that CloudStack gave you,
that should mean that templates are OK - i.e. VM did fetch for that first
time the pass from VR and set it locally for ROOT user.

What version of ACS are you using ?

Try to troubleshoot/check inside VR file similar to
/var/cache/cloud/password-192.168.1.1 (this 192.168.1.1 being the VR
interface / gateway IP of the network where VMs exist). If VM has
successfully fetched the pass from VR (from this file), than it should
write a line like "192.168.1.10=saved_password" inside that file - if not,
if password is still present inside, that means VM did not fetch the pass -
and then you can troubleshoot inside the VM itself (not sure if Nux's
templates use password reset script or cloud-init)

In some of previous CloudStack versions, you would need to do another round
of VM reboots to actually fetch the password (i.e. don't repeat the reset
procedure, just simply restart the VM/OS).

Best
Andrija

On Mon, 18 Feb 2019 at 23:00,  wrote:

> Hi Nux!
>
> I have noticed the same problem with all the templates.
>
> -AS.
>
> On 2019-02-18 19:36, Nux! wrote:
> > Hi,
> >
> > Have you noticed the same problem with the Ubuntu ones?
> >
> > --
> > Sent from the Delta quadrant using Borg technology!
> >
> > Nux!
> > www.nux.ro
> >
> > - Original Message -
> >> From: asen...@testlabs.com.au
> >> To: "users" 
> >> Cc: "Nux!" 
> >> Sent: Monday, 18 February, 2019 02:54:41
> >> Subject: Re: Templates for Apache Cloudstack
> >
> >> Hi Guys,
> >>
> >> Could someone double check Nux's templates for me, note the username
> >> for
> >> centos is "centos"
> >>
> >> 1. Download centos xenserver vhd template to cloudstack
> >> http://dl.openvm.eu/cloudstack/centos/x86_64/centos-7-xen.vhd.bz2
> >> 2. Deploy the template with password option set.
> >> 3. Login with password provided by cloudstack (first login works)
> >> 4. Turn off VM and reset password.
> >> 5. Try to login again with the new password. (password still remains
> >> same as old password, cannot login with new password)
> >>
> >> -Adrian Sender
> >>
> >>
> >>
> >>
> >> On 2019-02-12 19:52, Nux! wrote:
> >>> Hi,
> >>>
> >>> Can you give this a try? Let me know if it works as expected or not,
> >>> thanks.
> >>>
> >>> http://dl.openvm.eu/cloudstack/ubuntu/x86_64/ubuntu-18.04-xen.vhd.bz2
> >>>
> >>> --
> >>> Sent from the Delta quadrant using Borg technology!
> >>>
> >>> Nux!
> >>> www.nux.ro
> >>>
> >>> - Original Message -
>  From: "Nux!" 
>  To: "users" 
>  Sent: Thursday, 7 February, 2019 09:15:51
>  Subject: Re: Templates for Apache Cloudstack
> >>>
>  Hello Adrian,
> 
>  I can have a go at it when I get some time these days.
> 
>  --
>  Sent from the Delta quadrant using Borg technology!
> 
>  Nux!
>  www.nux.ro
> 
>  - Original Message -
> > From: asen...@testlabs.com.au
> > To: "users" 
> > Sent: Thursday, 7 February, 2019 00:04:40
> > Subject: Templates for Apache Cloudstack
> 
> > Hi All.
> >
> > Would it be possible to get the templates updated from
> > http://dl.openvm.eu/cloudstack/
> >
> > I would like to see a ubuntu18 vhd for xenserver if possible :)
> >
> > Does anyone else have vhd public templates for xenserver ?
> >
> > Thank You.
> >
> >> >> > Adrian Sender
>


-- 

Andrija Panić

RE: connection refused errors on 4.11.2

[Andrija Panic]

Hi Ivan,

What hypervisor, what kind of zone (basic or advanced), do you have explicitly 
defined Storage traffic in the zone or does it goes over management network ?

secstorage.allowed.internal.sites should be a CIDR, comma separated list 
(i.e.192.168.16.0/24), that systemVM should be able to access normally - i.e. 
login to inside systemVM and try to wget that image/template (check DB records 
if you don't know what is the URL of the template - vm_templates table).
Connection refused, is clear I guess - simply SSVM could not connect to web 
server.
/mnt/SecStorage/ Is the mount point inside SSVM.

If your SSVM is up and running and all in nice green color, then still I would 
do the following simple script, which should verify the health of your SSVM - 
pay attention to output - should NOT return any errors and should say something 
like "secondary storage mounted and can write to it" etc.
https://cwiki.apache.org/confluence/display/CLOUDSTACK/SSVM%2C+templates%2C+Secondary+storage+troubleshooting
(ssvm-check.sh ^^^ either connect via ssh or via console proxy)

Let us know how is it going with above.

Kind regards
Andrija

andrija.pa...@shapeblue.com 
www.shapeblue.com
Amadeus House, Floral Street, London  WC2E 9DPUK
@shapeblue
  
 


-Original Message-
From: Richard Persaud  
Sent: 14 February 2019 22:11
To: users@cloudstack.apache.org
Subject: RE: connection refused errors on 4.11.2

HI Ivan,

This is specifically with the SystemVM template .. all the other templates are 
fine.

It will show downloaded in the GUI, and after some time, I will start to get 
these connection refused errors and the SystemVM template will show as “timeout 
waiting for host”. Around the same time,  the console proxy becomes 
unresponsive.

Currently, both console and storage proxies are functioning, and SystemVM 
template shows as Timeout waiting for response from storage host.

It would be helpful if I could figure out which storage host(s) are causing the 
message.


Regards,

Richard Persaud

From: Ivan Kudryavtsev 
Sent: Thursday, February 14, 2019 3:54 PM
To: users 
Subject: Re: connection refused errors on 4.11.2

⚠ EXT MSG:

Richard, probably the error is with bridge mess. Check you have you bridges 
configured right and they matches to cloudstack networking labels for 
configured network topology...

May be I misunderstood you and it worked before, but stopped occasionally?

чт, 14 февр. 2019 г., 15:46 Richard Persaud 
richard.pers...@macys.com:

> Hello,
>
> (4.11.2 on Ubuntu 16.04)
>
> I am at my wits end trying to figure what these connection refused 
> errors are
>
>
>   *   The secstorage.allowed.internal.sites global option is set to blank
> (had it set to a /8, /16, /24 and /32s to no avail.)
>   *   All the storage/compute devices are using NFS and they can all
> communicate with each other and the management server.
>   *   There are no firewalls in between the devices and IPTABLES and UFW
> are all turned off.
>   *   I have properly followed the documentation and installed the system
> template.
>   *   *It is interesting to note that the "downloadPath" of
> /mnt/SecStorage does not exist.
>  *   I have created that path in case it may help, but has not stopped
> the error messages.
>
> Any advice you can give is greatly appreciated!
>
> 2019-02-14 13:35:11,462 DEBUG [c.c.a.t.Request]
> (AgentManager-Handler-5:null) (logid:) Seq 20-8098879504895705293:
> Processing:  { Ans: , MgmtId: 2483719209125, via: 20, Ver: v1, Flags: 
> 10, 
> [{"com.cloud.agent.api.storage.DownloadAnswer":{"jobId":"e0393396-44bc
> -44cd-ab93-f7b4b197ac39","downloadPct":0,"errorString":"Connection
> refused (Connection
> refused)","downloadStatus":"DOWNLOAD_ERROR","downloadPath":"/mnt/SecSt
> orage/e6ed90a3-a377-3c18-8edf-54299f3a0fc5/template/tmpl/2/218/dnld912
> 4079927637432602tmp_","installPath":"template/tmpl/2/218","templateSiz
> e":0,"templatePhySicalSize":0,"checkSum":"bd43d41e01c2a46b3cb23eb9139d
> ce4b","result":true,"details":"Connection
> refused (Connection refused)","wait":0}}] }
> 2019-02-14 13:35:11,464 DEBUG [c.c.a.t.Request]
> (AgentManager-Handler-11:null) (logid:) Seq 20-8098879504895705294:
> Processing:  { Ans: , MgmtId: 2483719209125, via: 20, Ver: v1, Flags: 
> 10, 
> [{"com.cloud.agent.api.storage.DownloadAnswer":{"jobId":"ea138e2a-1397
> -43f2-b37e-c9f684f0b3f8","downloadPct":0,"errorString":"Connection
> refused (Connection
> refused)","downloadStatus":"DOWNLOAD_ERROR","downloadPath":"/mnt/SecSt
> orage/e6ed90a3-a377-3c18-8edf-54299f3a0fc5/template/tmpl/2/220/dnld112
> 3458749647429711tmp_","installPath":"template/tmpl/2/220","templateSiz
> e":0,"templatePhySicalSize":0,"checkSum":"e8adeebcd8076702593469e33cc2
> d092","result":true,"details":"Connection
> refused (Connection refused)","wait":0}}] }
> 2019-02-14 13:35:11,464 DEBUG [c.c.a.t.Request]
> (AgentManager-Handler-3:null) (logid:) Seq 20-8098879504895705295:
> Processing:  { Ans: , MgmtId: 248371920912

RE: Mysql got timeout and problems with UI

[Andrija Panic]

Ugo,

Please do proper log analysis, otherwise we can only guess.

If you restart network with clean up - that means a BRAND NEW VR was created 
for your networks..
That sounds like something was wrong with old one.
And that needs investigation of log (mgmt and agent) and/or inside VR, before 
you destroy it.
I don't get the DNS part of your reply...

Kind regards,
Andrija


andrija.pa...@shapeblue.com 
www.shapeblue.com
Amadeus House, Floral Street, London  WC2E 9DPUK
@shapeblue
  
 


-Original Message-
From: Ugo Vasi  
Sent: 07 February 2019 11:47
To: users@cloudstack.apache.org; Andrija Panic 
Subject: Re: Mysql got timeout and problems with UI

Hi Adrija,
I restarted the network with the appropriate button (enabling the clean
option) and then the firewall responds correctly, this makes me think that it 
is not a problem of communication between browsers and mgmt.

It would seem to me a DNS problem that delays the answers, how can I check the 
calls that the software agent makes inside the router?

Is there a query to count the number of rules of that firewall or for ACL? I've 
never used more than 15 rules for single IP.




Il 07/02/19 09:19, Andrija Panic ha scritto:
> Hi Ugo,
>
> I have never observed such behavior in any release so far (general 
> sluggishness of GUI) - I hope you don't mind me asking you  to make sure that 
> your network connectivity is good enough and so on, between browser/API 
> client and the mgmt server itself (and of course between mgmt and mysql 
> server) - reason being that in some of my ex companies I got one (high level) 
> guy complaining on lost packet when doing ping to the DC, but it was his WiFi 
> which has problems (once moved to wire, everything magically worked).
>
> To be fair, I did see dead gui when a customer has been abusing the ACL 
> rules, i.e. had like 200 rules on single ACL, and browser could not handle 
> showing it and it was completely unmanageable.
>
> As for other issue, this seems to be something you had before (if I remember 
> ML correctly) - please check the logs inside VR and any lines in mgmt. log 
> ABOVE that final error string (usually it will tell you that i.e. ssh 
> connection to VR could not be done).
>
> Also worth checking Agent logs on specific host where VR is running (make 
> sure to increase log level to debug first)
> For MySQL issues, again check network, and make sure no LB with short TCP 
> timeout is set in between.
>
> Kind regards,
> Andrija
>
> andrija.pa...@shapeblue.com
> www.shapeblue.com
> Amadeus House, Floral Street, London  WC2E 9DPUK
> @shapeblue
>
>   
>
>
> -Original Message-
> From: Ugo Vasi 
> Sent: 06 February 2019 14:53
> To: users@cloudstack.apache.org
> Subject: Re: Mysql got timeout and problems with UI
>
> Hi all,
> adding a new firewall rule I receive this error (note, after 1189 seconds):
>
> 2019-02-06 14:42:08,441 WARN  [o.a.c.f.j.i.AsyncJobMonitor]
> (Timer-0:ctx-c62615c7) (logid:37033038) Task (job-886) has been pending for 
> 1189 seconds
> 2019-02-06 14:42:20,358 DEBUG [c.c.a.t.Request]
> (API-Job-Executor-5:ctx-6106b5b5 job-886 ctx-ac682bcd) (logid:b4ed6266) Seq 
> 7-419397715298887: Received:  { Ans: , MgmtId: 220777304233416,
> via: 7(cshp143), Ver: v1, Flags: 0, { GroupAnswer } }
> 2019-02-06 14:42:20,358 WARN  [c.c.n.f.FirewallManagerImpl]
> (API-Job-Executor-5:ctx-6106b5b5 job-886 ctx-ac682bcd) (logid:b4ed6266) 
> Failed to apply firewall rules due to : Resource [DataCenter:1] is
> unreachable: Unable to apply firewall rules on router
> 2019-02-06 14:42:20,366 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-5:ctx-6106b5b5 job-886) (logid:b4ed6266) Complete async 
> job-886, jobStatus: FAILED, resultCode: 530, result:
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
> to create firewall rule"}
>
>
> The rule now is in "Deleting" state.
>
>
> Il 06/02/19 10:59, Ugo Vasi ha scritto:
>> Hi all,
>> in a recente ACS 4.11.2.0 installation (Ubuntu 16.04 + KVM + Advanced
>> network) I have some problems in cloudstack usage.
>>
>> During the usual operations, I find a slowdown in the responses of the
>> UI and occasionally errors.
>>
>> This morning I created a new VM and I assigned an IP (xx.xx.xx.xx) of
>> a specific network (net-name).
>>
>> The  UI has stopped in the assignment phase even if in the logs (which
>> I report at the bottom of the email) I see that the job is executed in
>> a few seconds and updating the list of associated IPs in the network
>> list I see that the association is correct.
>>
>> I noticed that a series of these mess

RE: Failed to create the 33rd static nat

[Andrija Panic]

Hi Ivan,

Consider pasting a log on pastebin and sending the link here.

If you are sure this is bug in 4.11, please raise an issue in Github: 
https://github.com/apache/cloudstack/issues

(worth checking global limits etc)

Kind regards
Andrija

andrija.pa...@shapeblue.com 
www.shapeblue.com
Amadeus House, Floral Street, London  WC2E 9DPUK
@shapeblue
  
 


-Original Message-
From: Ivan X Yue  
Sent: 07 February 2019 07:12
To: users@cloudstack.apache.org
Subject: Failed to create the 33rd static nat

I have a weird problem in CloudStack 4.11.2.  And want to see if there is any 
suggestion on how to debug / fix this issue.

I created a VPC network, with one guest network.   I have added 32 VMs, 
requested 32 public IPs, and setup static NAT to associate the IP to each of 
the VM.  These works without any issue.  However, then I added one more VM, and 
request 1 public IP.  When I try to associate the public IP with the VM, I get 
the error that it failed to enable the static NAT:

2019-02-07 00:52:41,474 WARN  [c.c.n.r.RulesManagerImpl]
(qtp788117692-1109:ctx-5b6f9e33 ctx-351aca5a) (logid:56d8d140) Failed to create 
static nat rule due to
com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] is 
unreachable: Unable to apply static nat rules on router
at
org.apache.cloudstack.network.topology.BasicNetworkTopology.applyRules(BasicNetworkTopology.java:403)
at
org.apache.cloudstack.network.topology.BasicNetworkTopology.applyStaticNats(BasicNetworkTopology.java:245)
at
com.cloud.network.element.VirtualRouterElement.applyStaticNats(VirtualRouterElement.java:599)
at
com.cloud.network.IpAddressManagerImpl.applyStaticNats(IpAddressManagerImpl.java:1899)
at
com.cloud.network.rules.RulesManagerImpl.applyStaticNatForIp(RulesManagerImpl.java:1338)
at
com.cloud.network.rules.RulesManagerImpl.enableStaticNat(RulesManagerImpl.java:598)


It seems that the virtual route has problem associating the 33rd static NAT.  
We have tried CloudStack 4.9.3 and does not have this problem. 

Is there any suggestion on what I should do? 

Thanks
Ivan

RE: Contrail (tungsten fabric) integration

[Andrija Panic]

Considered VXLAN ?

andrija.pa...@shapeblue.com 
www.shapeblue.com
Amadeus House, Floral Street, London  WC2E 9DPUK
@shapeblue
  
 


-Original Message-
From: Sergei Hanus  
Sent: 05 February 2019 09:22
To: users@cloudstack.apache.org; swel...@ena.com
Subject: Re: Contrail (tungsten fabric) integration

  Simon, thank you for the reply.

Is there any opinion for now - what will be the way to go with SDN/NFV? 
What options are being considered? We are not vendor-biased, and are trying to 
find a complete cloud solution. Just Tungsten looks right in networking portion 
from our point of view.

Sergei.

RE: Templates for Apache Cloudstack

[Andrija Panic]

There is only one @nux 😊

andrija.pa...@shapeblue.com 
www.shapeblue.com
Amadeus House, Floral Street, London  WC2E 9DPUK
@shapeblue
  
 


-Original Message-
From: asen...@testlabs.com.au  
Sent: 07 February 2019 01:05
To: users@cloudstack.apache.org
Subject: Templates for Apache Cloudstack

Hi All.

Would it be possible to get the templates updated from 
http://dl.openvm.eu/cloudstack/

I would like to see a ubuntu18 vhd for xenserver if possible :)

Does anyone else have vhd public templates for xenserver ?

Thank You.

Adrian Sender

RE: Mysql got timeout and problems with UI

[Andrija Panic]

Hi Ugo,

I have never observed such behavior in any release so far (general sluggishness 
of GUI) - I hope you don't mind me asking you  to make sure that your network 
connectivity is good enough and so on, between browser/API client and the mgmt 
server itself (and of course between mgmt and mysql server) - reason being that 
in some of my ex companies I got one (high level) guy complaining on lost 
packet when doing ping to the DC, but it was his WiFi which has problems (once 
moved to wire, everything magically worked).

To be fair, I did see dead gui when a customer has been abusing the ACL rules, 
i.e. had like 200 rules on single ACL, and browser could not handle showing it 
and it was completely unmanageable.

As for other issue, this seems to be something you had before (if I remember ML 
correctly) - please check the logs inside VR and any lines in mgmt. log ABOVE 
that final error string (usually it will tell you that i.e. ssh connection to 
VR could not be done). 

Also worth checking Agent logs on specific host where VR is running (make sure 
to increase log level to debug first)
For MySQL issues, again check network, and make sure no LB with short TCP 
timeout is set in between.

Kind regards,
Andrija

andrija.pa...@shapeblue.com 
www.shapeblue.com
Amadeus House, Floral Street, London  WC2E 9DPUK
@shapeblue
  
 


-Original Message-
From: Ugo Vasi  
Sent: 06 February 2019 14:53
To: users@cloudstack.apache.org
Subject: Re: Mysql got timeout and problems with UI

Hi all,
adding a new firewall rule I receive this error (note, after 1189 seconds):

2019-02-06 14:42:08,441 WARN  [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-c62615c7) (logid:37033038) Task (job-886) has been pending for 
1189 seconds
2019-02-06 14:42:20,358 DEBUG [c.c.a.t.Request]
(API-Job-Executor-5:ctx-6106b5b5 job-886 ctx-ac682bcd) (logid:b4ed6266) Seq 
7-419397715298887: Received:  { Ans: , MgmtId: 220777304233416,
via: 7(cshp143), Ver: v1, Flags: 0, { GroupAnswer } }
2019-02-06 14:42:20,358 WARN  [c.c.n.f.FirewallManagerImpl]
(API-Job-Executor-5:ctx-6106b5b5 job-886 ctx-ac682bcd) (logid:b4ed6266) Failed 
to apply firewall rules due to : Resource [DataCenter:1] is
unreachable: Unable to apply firewall rules on router
2019-02-06 14:42:20,366 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-5:ctx-6106b5b5 job-886) (logid:b4ed6266) Complete async 
job-886, jobStatus: FAILED, resultCode: 530, result: 
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
to create firewall rule"}


The rule now is in "Deleting" state.


Il 06/02/19 10:59, Ugo Vasi ha scritto:
> Hi all,
> in a recente ACS 4.11.2.0 installation (Ubuntu 16.04 + KVM + Advanced
> network) I have some problems in cloudstack usage.
>
> During the usual operations, I find a slowdown in the responses of the 
> UI and occasionally errors.
>
> This morning I created a new VM and I assigned an IP (xx.xx.xx.xx) of 
> a specific network (net-name).
>
> The  UI has stopped in the assignment phase even if in the logs (which 
> I report at the bottom of the email) I see that the job is executed in 
> a few seconds and updating the list of associated IPs in the network 
> list I see that the association is correct.
>
> I noticed that a series of these messages appear in the mysql log:
>
> 2019-02-06T00:50:58.758699Z 6298 [Note] Aborted connection 6298 to db: 
> 'dbname' user: 'username' host: 'localhost' (Got timeout reading 
> communication packets) 2019-02-06T00:51:59.352203Z 6269 [Note] Aborted 
> connection 6269 to db:
> 'dbname' user: 'username' host: 'localhost' (Got timeout reading 
> communication packets) 2019-02-06T00:51:59.421092Z 6289 [Note] Aborted 
> connection 6289 to db:
> 'dbname' user: 'username' host: 'localhost' (Got timeout reading 
> communication packets) 2019-02-06T08:00:35.717673Z 6101 [Note] Aborted 
> connection 6101 to db:
> 'dbname_usage' user: 'username' host: 'localhost' (Got timeout reading 
> communication packets
>
>
> Note that the mysql server is used only by cloustack and is configured 
> according to the instructions in the cloudstack manual.
>
> I've already had a number of similar problems before (the version of 
> ACS was 4.11.1) and the situation has been getting worse and worse 
> until the interface and the whole cloudstack system has stopped 
> working (see "URGENT: Unable to schedule async job for command 
> com.cloud.vm.VmWorkStart, unexpected exception" and "urgent: Unable to 
> apply firewall rules on router").
>
>
> Has anyone had similar experiences or can help me understand where the 
> problem is?
>
>
>
> Job log (extracted from management-server.log):
>
> 2019-02-06 10:21:07,282 INFO  [o.a.c.f.j.i.AsyncJobMonitor] 
> (API-Job-Executor-6:ctx-5d20ed8a job-876) (logid:130cd0a6) Add job-876 
> into job monitoring 2019-02-06 10:21:07,294 DEBUG 
> [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-6:ctx-5d20ed8a
> job-876) (logid:ce8dc8fb) Executing AsyncJobVO {i

RE: Converting a VPC

[Andrija Panic]

Hi Adam,

You probably want to do a "migration" to another network - i.e. go to the 
specific VM's NIC tab:
-  Add a new Network to the VM (add an existing L2 network - TBH, I never tried 
this with L2 network, but I can confirm it works for any VPC, Shared or 
Isolated network) - effectively you are adding additional NIC to your VM
-  Make this new network Default, so you can...
-  ...remove original Network from the VM

(When you say L2, I assume the new model in 4.11 that has no IP settings no VR, 
nothing...)

This way you have performed "migration" of VM from on network to another.

Kind regards,
Andrija

andrija.pa...@shapeblue.com 
www.shapeblue.com
Amadeus House, Floral Street, London  WC2E 9DPUK
@shapeblue
  
 


-Original Message-
From: Adam Witwicki  
Sent: 06 February 2019 17:29
To: users@cloudstack.apache.org
Subject: Converting a VPC

Hi Everybody.

Is there a way to convert a VPC to a Layer2 network to avoid me recreating all 
the instances?

Thanks

Adam



Disclaimer Notice:
This email has been sent by Oakford Technology Limited, while we have checked 
this e-mail and any attachments for viruses, we can not guarantee that they are 
virus-free. You must therefore take full responsibility for virus checking.
This message and any attachments are confidential and should only be read by 
those to whom they are addressed. If you are not the intended recipient, please 
contact us, delete the message from your computer and destroy any copies. Any 
distribution or copying without our prior permission is prohibited.
Internet communications are not always secure and therefore Oakford Technology 
Limited does not accept legal responsibility for this message. The recipient is 
responsible for verifying its authenticity before acting on the contents. Any 
views or opinions presented are solely those of the author and do not 
necessarily represent those of Oakford Technology Limited.
Registered address: Oakford Technology Limited, The Manor House, Potterne, 
Wiltshire. SN10 5PN.
Registered in England and Wales No. 5971519

RE: how to run rhel 6.x VM as PV VM on xenserver 7.1CU1?

[Andrija Panic]

Hi Yiping,

Please check how this was done in previous releases (i.e. added support for 
XenServer 7.1.0 and some additional missing guest os mapping etc:

https://github.com/apache/cloudstack/blob/master/engine/schema/src/main/resources/META-INF/db/schema-41000to41100.sql

Just make sure that your SQL queries don't depend on ID, i.e. will work across 
different people's installation...
... and then being a good community member 😊 please try to commit this back to 
community - someone will review for sure and give you feedback
i.e. here 
https://github.com/apache/cloudstack/blob/master/engine/schema/src/main/resources/META-INF/db/schema-41120to41200.sql
 though today/tomorrow guys will freeze the 4.12/master for a new release 
candidate etc. - so not sure if you will have to send your pull request against 
some other branch

Hope that helps

Kind regards,
Andrija

andrija.pa...@shapeblue.com 
www.shapeblue.com
Amadeus House, Floral Street, London  WC2E 9DPUK
@shapeblue
  
 


-Original Message-
From: Yiping Zhang  
Sent: 31 January 2019 22:41
To: users@cloudstack.apache.org
Subject: Re: how to run rhel 6.x VM as PV VM on xenserver 7.1CU1?

Hi, Andrija:

I am willing to try this approach given that we are working in a lab 
environment. Otherwise we would have to downgrade to use XenServer 7.1 + 
installing security patches afterwards

Since we also need to add one new entry in hypervisor_capabilities table for 
XenServer 7.1.1 and there is no API to *add* new entry into this table, we 
decided to use SQL directly.  After reading source code in 
cloudstack/engine/schema/src/main/resources/META-INF/db directory on Github, we 
come up with following SQL statements to seed DB tables for XenServer 7.1CU1 
support:

INSERT IGNORE INTO hypervisor_capabilities (uuid, hypervisor_type, 
hypervisor_version, max_guests_limit, max_data_volumes_limit, 
storage_motion_supported) values (UUID(), "XenServer", "7.1.1", 500, 13, 1);

INSERT IGNORE INTO guest_os_hypervisor (uuid,hypervisor_type, 
hypervisor_version, guest_os_name, guest_os_id, created, is_user_defined)
  SELECT UUID(),"Xenserver", "7.1.1", guest_os_name, guest_os_id, 
utc_timestamp(), 0
  FROM guest_os_hypervisor
  WHERE hypervisor_type="Xenserver"
  AND hypervisor_version="7.1.0";

After executing these two SQL statements,  and restarting management service, 
all my RHEL 6.x VM instances can be started successfully as PV instances. Now 
we just have to do a lot more validation checks to make sure all is well, 
especially with our own particular setups and usages.  I'd appreciate very much 
if anyone else could send their feedbacks and gotchas if they have done 
anything similar and any area I may need to pay more attention with.

Thanks

Yiping



On 1/30/19, 2:34 AM, "Andrija Panic"  wrote:

Hi Yiping,

As far as I can expect, patch level should not break any functionality with 
ACS (except, obviously, guest OS mappings...) so I assume it should work same 
was as unpatched/vanila version.


https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcloudstack.apache.org%2Fapi%2Fapidocs-4.11%2Fapis%2FaddGuestOsMapping.html&data=02%7C01%7Cyipzhang%40adobe.com%7C7be8d8c0b54748293cc308d6869e9335%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C1%7C636844412977844955&sdata=x6cRVNkrIw2bN87nibCtwlgwfZ3GfWm%2Buxsg4R7oyVg%3D&reserved=0

Please use above API call to make needed guest OS mapping - i.e. observe 
needed (or all ???) OS types (ID value from guest_os table for each guest OS 
type in ACS) and use it to generate appropriate API calls which will create 
missing mapping records inside guest_os_hypervisor table).

Alternatively,  just copy all 192 rows that you have for 7.1.0 - duplicate 
all these rows with changing hypervisor_version to 7.1.1 - I assume mgmt. 
restart might be needed, but since test env, doesn't hurt.

Let me know how this worked for you,

Best
Andrija

andrija.pa...@shapeblue.com 

https://na01.safelinks.protection.outlook.com/?url=www.shapeblue.com&data=02%7C01%7Cyipzhang%40adobe.com%7C7be8d8c0b54748293cc308d6869e9335%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C1%7C636844412977844955&sdata=XeE6PL2DHQZhBlkXLA6o3bjIqfRZFTklrFBp85naDeU%3D&reserved=0
Amadeus House, Floral Street, London  WC2E 9DPUK
@shapeblue
  
 


-Original Message-
From: Yiping Zhang  
Sent: 29 January 2019 21:54
To: users@cloudstack.apache.org
Subject: Re: how to run rhel 6.x VM as PV VM on xenserver 7.1CU1?

Hi, Andrija:

I think you are

 on to something here:

Here are my query results for these sql statements:

mysql> select id,name,hypervisor_type,hypervisor_version from host where 
type="Routing" and removed is NULL;
  

RE: how to run rhel 6.x VM as PV VM on xenserver 7.1CU1?

[Andrija Panic]

Hi Yiping,

As far as I can expect, patch level should not break any functionality with ACS 
(except, obviously, guest OS mappings...) so I assume it should work same was 
as unpatched/vanila version.

https://cloudstack.apache.org/api/apidocs-4.11/apis/addGuestOsMapping.html

Please use above API call to make needed guest OS mapping - i.e. observe needed 
(or all ???) OS types (ID value from guest_os table for each guest OS type in 
ACS) and use it to generate appropriate API calls which will create missing 
mapping records inside guest_os_hypervisor table).

Alternatively,  just copy all 192 rows that you have for 7.1.0 - duplicate all 
these rows with changing hypervisor_version to 7.1.1 - I assume mgmt. restart 
might be needed, but since test env, doesn't hurt.

Let me know how this worked for you,

Best
Andrija

andrija.pa...@shapeblue.com 
www.shapeblue.com
Amadeus House, Floral Street, London  WC2E 9DPUK
@shapeblue
  
 


-Original Message-
From: Yiping Zhang  
Sent: 29 January 2019 21:54
To: users@cloudstack.apache.org
Subject: Re: how to run rhel 6.x VM as PV VM on xenserver 7.1CU1?

Hi, Andrija:

I think you are

 on to something here:

Here are my query results for these sql statements:

mysql> select id,name,hypervisor_type,hypervisor_version from host where 
type="Routing" and removed is NULL;
++---+-++
| id | name  | hypervisor_type | hypervisor_version |
++---+-++
| 56 | lab-hv03 | XenServer   | 7.1.1  |
| 57 | lab-hv02 | XenServer   | 7.1.1  |
| 58 | lab-hv04 | XenServer   | 7.1.1  |
++---+-++
3 rows in set (0.00 sec)

mysql> SELECT count(*) FROM guest_os_hypervisor WHERE 
hypervisor_type="Xenserver" AND hypervisor_version = "7.1.0";
+--+
| count(*) |
+--+
|  192 |
+--+
1 row in set (0.00 sec)

mysql> SELECT count(*) FROM guest_os_hypervisor WHERE 
hypervisor_type="Xenserver" AND hypervisor_version = "7.1.1";
+--+
| count(*) |
+--+
|0 |
+--+
1 row in set (0.01 sec)

mysql>


As can be seen,  there are 192 entries for hypervisor_version "7.1.0", but zero 
entries for "7.1.1" which is what my hypervisors are.  I went back to read 
4.11.2.0 release notes again on supported hypervisor lists and sure enough,  
there is no mention of XenServer 7.1CU1 being supported.

What changes are required on CloudStack side to add support for XenServer 
7.1CU1 (aka 7.1.1)?  Would it be sufficient by adding missing mappings in 
guest_os_hypervsior table and adding new entry in hypervisor_capabilities 
table?  Is there any code change required?

Thanks

Yiping

On 1/29/19, 12:34 AM, "Andrija Panic"  wrote:

Yiping,

Here is one more thing to check - please check your hypervisor version, as 
reported inside cloud.host table:

i.e.

select id,name,hypervisor_type,hypervisor_version from host where 
type="Routing" and removed is NULL;

would in my lab return something like this:

+++-++
| id | name   | hypervisor_type | hypervisor_version |
+++-++
|  1 | 10.2.2.202 | VMware  | 6.5|
+++-++

Check the version reported for your XenServer, I'm assuming it might report 
true version after upgrade i.e. "7.1.2", for which there are NO mappings inside 
guest_os_hypervisor table - if so, you would need to duplicate all the mapping 
records for XenServer 7.1.0 (or 7.0.0).

Different way to say it :
SELECT * FROM guest_os_hypervisor WHERE hypervisor_type="Xenserver" AND 
hypervisor_version = "7.1.2";
...gives no result (there is no mapping from ACS guest OS to hypervisor 
guest OS)

while

SELECT * FROM guest_os_hypervisor WHERE hypervisor_type="Xenserver" AND 
hypervisor_version = "7.1.0";
...would return many rows with proper mappings

Check this, and we can work from there, once you confirm this might be the 
issue (there are API calls to do this, or we can do it with SQL etc - add 
missing mappins).

Kind regards,
Andrija

andrija.pa...@shapeblue.com 

https://na01.safelinks.protection.outlook.com/?url=www.shapeblue.com&data=02%7C01%7Cyipzhang%40adobe.com%7Caf6dbfb87b194e4324f508d685c48f5d%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C1%7C636843476632252646&sdata=BPs3N2CbfNepxGyN4Mkt9ct863GiJ6kWgPLWRlFVEA4%3D&reserved=0
Amadeus House, Floral Street, London  WC2E 9DPUK
@shapeblue
  
 


-Original Message-
From: Andrija Pan

RE: how to run rhel 6.x VM as PV VM on xenserver 7.1CU1?

[Andrija Panic]

Yiping,

Here is one more thing to check - please check your hypervisor version, as 
reported inside cloud.host table:

i.e.

select id,name,hypervisor_type,hypervisor_version from host where 
type="Routing" and removed is NULL;

would in my lab return something like this:

+++-++
| id | name   | hypervisor_type | hypervisor_version |
+++-++
|  1 | 10.2.2.202 | VMware  | 6.5|
+++-++

Check the version reported for your XenServer, I'm assuming it might report 
true version after upgrade i.e. "7.1.2", for which there are NO mappings inside 
guest_os_hypervisor table - if so, you would need to duplicate all the mapping 
records for XenServer 7.1.0 (or 7.0.0).

Different way to say it :
SELECT * FROM guest_os_hypervisor WHERE hypervisor_type="Xenserver" AND 
hypervisor_version = "7.1.2";
...gives no result (there is no mapping from ACS guest OS to hypervisor guest 
OS)

while

SELECT * FROM guest_os_hypervisor WHERE hypervisor_type="Xenserver" AND 
hypervisor_version = "7.1.0";
...would return many rows with proper mappings

Check this, and we can work from there, once you confirm this might be the 
issue (there are API calls to do this, or we can do it with SQL etc - add 
missing mappins).

Kind regards,
Andrija

andrija.pa...@shapeblue.com 
www.shapeblue.com
Amadeus House, Floral Street, London  WC2E 9DPUK
@shapeblue
  
 


-Original Message-
From: Andrija Panic  
Sent: 29 January 2019 00:54
To: users@cloudstack.apache.org
Subject: RE: how to run rhel 6.x VM as PV VM on xenserver 7.1CU1?

Additionally, I expect that following table is correct in sense of paper vs 
reality, but always good to check: 
https://docs.citrix.com/en-us/xenserver/7-1/system-requirements/guest-os-support.html


andrija.pa...@shapeblue.com
www.shapeblue.com
Amadeus House, Floral Street, London  WC2E 9DPUK @shapeblue
  
 


-Original Message-
From: Andrija Panic 
Sent: 29 January 2019 00:52
To: users@cloudstack.apache.org
Subject: RE: how to run rhel 6.x VM as PV VM on xenserver 7.1CU1?

Hi Yiping,

If you do the following SQL:SELECT * FROM guest_os_hypervisor WHERE 
hypervisor_type="XenServer" and hypervisor_version="7.1.0" AND guest_os_id IN 
(SELECT id FROM guest_os WHERE display_name="CentOS 6.4 (64-bit)");
+---+
| guest_os_name |
+---+
| CentOS 6 (64-bit) |
+---+

It basically shows you, that for ACS OS type called "CentOS 6.4 (64-bit)", this 
is translated/matches/mapped to the "CentOS 6 (64-bit)" as seen from the 
XenServer 7.1.x hypervisor. 
Now, there IS a possibility that some of these mappings is incorrect...
I would just go to XenServer and try to deploy manually (via XenCenter etc) a 
VM with selecting the OS type from above SQL results set (CentOS 6 (64-bit)) 
and observe if it gets provisioned as HVM or PV - if XenServer brings it as 
HVM, then you can see it's XenServer making it HVM. But if XenServer starts 
that manually deployed VM as PV - then we can assume some wrong mapping from 
ACS side.

If you have time to test this, I'm also interested in root cause - since I saw 
with 7.1.x XenServer that you can't even restore a VM from snapshot, if you 
change OS type on existing VM in ACS from, i.e. centos 6.4 to 6.5 etc.

Kind regards,
Andrija

andrija.pa...@shapeblue.com
www.shapeblue.com
Amadeus House, Floral Street, London  WC2E 9DPUK @shapeblue
  
 


-Original Message-
From: Yiping Zhang 
Sent: 28 January 2019 23:16
To: users@cloudstack.apache.org
Subject: how to run rhel 6.x VM as PV VM on xenserver 7.1CU1?

Hi, All:

I have a large number of RHEL 6.x VM instances running in our ACS environment. 
Last time when I upgraded our XenServer from 6.5SP1 to 7.0,   I have to change 
my templates to assign OS TYPE as “RHEL 6.4 (64bit)” so that my VM instances 
can be started as PV instances. Anything above “RHEL 6.5 (64bits)” would be 
started as HVM instances and they would get stuck during boot.

Last week, after I upgraded my lab hypervisors to use XenServer 7.1CU1,  all my 
(lab) rhel 6.x VM instances would get started as HVM instances, thus stuck 
during boot.  I even tried to change template’s OS TYPE to other types such as 
“rhel 5.10/5.0/6.0/, Other PV(64bit)” etc without any luck.

What did I miss?  My lab is running ACS 4.11.2.0 packages from Shapeblue.  
According to Citrix document, 
https://docs.citrix.com/en-us/xenserver/7-1/downloads/vm-users-guide.pdf,  RHEL 
6.x should always be started as PV instance. So why I only get HVM instances 
instead?

Is it CloudStack or XenServer which decides an instance to be started as PV or 
HVM instance?

Thanks, all helps are appreciated.

Yiping

RE: how to run rhel 6.x VM as PV VM on xenserver 7.1CU1?

[Andrija Panic]

Additionally, I expect that following table is correct in sense of paper vs 
reality, but always good to check: 
https://docs.citrix.com/en-us/xenserver/7-1/system-requirements/guest-os-support.html


andrija.pa...@shapeblue.com 
www.shapeblue.com
Amadeus House, Floral Street, London  WC2E 9DPUK
@shapeblue
  
 


-Original Message-
From: Andrija Panic  
Sent: 29 January 2019 00:52
To: users@cloudstack.apache.org
Subject: RE: how to run rhel 6.x VM as PV VM on xenserver 7.1CU1?

Hi Yiping,

If you do the following SQL:SELECT * FROM guest_os_hypervisor WHERE 
hypervisor_type="XenServer" and hypervisor_version="7.1.0" AND guest_os_id IN 
(SELECT id FROM guest_os WHERE display_name="CentOS 6.4 (64-bit)");
+---+
| guest_os_name |
+---+
| CentOS 6 (64-bit) |
+---+

It basically shows you, that for ACS OS type called "CentOS 6.4 (64-bit)", this 
is translated/matches/mapped to the "CentOS 6 (64-bit)" as seen from the 
XenServer 7.1.x hypervisor. 
Now, there IS a possibility that some of these mappings is incorrect...
I would just go to XenServer and try to deploy manually (via XenCenter etc) a 
VM with selecting the OS type from above SQL results set (CentOS 6 (64-bit)) 
and observe if it gets provisioned as HVM or PV - if XenServer brings it as 
HVM, then you can see it's XenServer making it HVM. But if XenServer starts 
that manually deployed VM as PV - then we can assume some wrong mapping from 
ACS side.

If you have time to test this, I'm also interested in root cause - since I saw 
with 7.1.x XenServer that you can't even restore a VM from snapshot, if you 
change OS type on existing VM in ACS from, i.e. centos 6.4 to 6.5 etc.

Kind regards,
Andrija

andrija.pa...@shapeblue.com
www.shapeblue.com
Amadeus House, Floral Street, London  WC2E 9DPUK @shapeblue
  
 


-Original Message-
From: Yiping Zhang 
Sent: 28 January 2019 23:16
To: users@cloudstack.apache.org
Subject: how to run rhel 6.x VM as PV VM on xenserver 7.1CU1?

Hi, All:

I have a large number of RHEL 6.x VM instances running in our ACS environment. 
Last time when I upgraded our XenServer from 6.5SP1 to 7.0,   I have to change 
my templates to assign OS TYPE as “RHEL 6.4 (64bit)” so that my VM instances 
can be started as PV instances. Anything above “RHEL 6.5 (64bits)” would be 
started as HVM instances and they would get stuck during boot.

Last week, after I upgraded my lab hypervisors to use XenServer 7.1CU1,  all my 
(lab) rhel 6.x VM instances would get started as HVM instances, thus stuck 
during boot.  I even tried to change template’s OS TYPE to other types such as 
“rhel 5.10/5.0/6.0/, Other PV(64bit)” etc without any luck.

What did I miss?  My lab is running ACS 4.11.2.0 packages from Shapeblue.  
According to Citrix document, 
https://docs.citrix.com/en-us/xenserver/7-1/downloads/vm-users-guide.pdf,  RHEL 
6.x should always be started as PV instance. So why I only get HVM instances 
instead?

Is it CloudStack or XenServer which decides an instance to be started as PV or 
HVM instance?

Thanks, all helps are appreciated.

Yiping

RE: how to run rhel 6.x VM as PV VM on xenserver 7.1CU1?

[Andrija Panic]

Hi Yiping,

If you do the following SQL:SELECT * FROM guest_os_hypervisor WHERE 
hypervisor_type="XenServer" and hypervisor_version="7.1.0" AND guest_os_id IN 
(SELECT id FROM guest_os WHERE display_name="CentOS 6.4 (64-bit)");
+---+
| guest_os_name |
+---+
| CentOS 6 (64-bit) |
+---+

It basically shows you, that for ACS OS type called "CentOS 6.4 (64-bit)", this 
is translated/matches/mapped to the "CentOS 6 (64-bit)" as seen from the 
XenServer 7.1.x hypervisor. 
Now, there IS a possibility that some of these mappings is incorrect...
I would just go to XenServer and try to deploy manually (via XenCenter etc) a 
VM with selecting the OS type from above SQL results set (CentOS 6 (64-bit)) 
and observe if it gets provisioned as HVM or PV - if XenServer brings it as 
HVM, then you can see it's XenServer making it HVM. But if XenServer starts 
that manually deployed VM as PV - then we can assume some wrong mapping from 
ACS side.

If you have time to test this, I'm also interested in root cause - since I saw 
with 7.1.x XenServer that you can't even restore a VM from snapshot, if you 
change OS type on existing VM in ACS from, i.e. centos 6.4 to 6.5 etc.

Kind regards,
Andrija

andrija.pa...@shapeblue.com 
www.shapeblue.com
Amadeus House, Floral Street, London  WC2E 9DPUK
@shapeblue
  
 


-Original Message-
From: Yiping Zhang  
Sent: 28 January 2019 23:16
To: users@cloudstack.apache.org
Subject: how to run rhel 6.x VM as PV VM on xenserver 7.1CU1?

Hi, All:

I have a large number of RHEL 6.x VM instances running in our ACS environment. 
Last time when I upgraded our XenServer from 6.5SP1 to 7.0,   I have to change 
my templates to assign OS TYPE as “RHEL 6.4 (64bit)” so that my VM instances 
can be started as PV instances. Anything above “RHEL 6.5 (64bits)” would be 
started as HVM instances and they would get stuck during boot.

Last week, after I upgraded my lab hypervisors to use XenServer 7.1CU1,  all my 
(lab) rhel 6.x VM instances would get started as HVM instances, thus stuck 
during boot.  I even tried to change template’s OS TYPE to other types such as 
“rhel 5.10/5.0/6.0/, Other PV(64bit)” etc without any luck.

What did I miss?  My lab is running ACS 4.11.2.0 packages from Shapeblue.  
According to Citrix document, 
https://docs.citrix.com/en-us/xenserver/7-1/downloads/vm-users-guide.pdf,  RHEL 
6.x should always be started as PV instance. So why I only get HVM instances 
instead?

Is it CloudStack or XenServer which decides an instance to be started as PV or 
HVM instance?

Thanks, all helps are appreciated.

Yiping

RE: Reassigning same public IP

[Andrija Panic]

​Hi,

you can either try adding multiple Public IPs (not the most elegant solution, 
but sometimes work), until you eventually add the old IP you want, then remove 
all other public IPs.

Or you can just edit the table "user_ip_address" - here, I suggest to add one 
additional IP address, and check what different fields were updated in the 
table for that specific row/IP address - then replicate same  field changes for 
the old public IP address you want (later release the unneeded public IP).

Best,
Andrija​

andrija.pa...@shapeblue.com 
www.shapeblue.com
Amadeus House, Floral Street, London  WC2E 9DPUK
@shapeblue
  
 


-Original Message-
From: Makrand  
Sent: 28 January 2019 08:52
To: users@cloudstack.apache.org
Subject: Reassigning same public IP

Hello there,

Is there any way I can reassign accidentally released public IP to VM (Static 
NAT)

If I have to edit DB - what tables - rows should I be updating?


--
Makrand

Re: 答复: CloudStack 4.11.2 Snapshot Revert fail

[Andrija Panic]

Hi

I have just reproduced your issue (7.1 though, no updates, and not relevant
I guess...)
I changed VM (edit VM, while stopped) after it was previously snapshoot-ed
successfully

OS types vs restore results below:
Centos 6.6 x64 fails to restore,
CentOS 6.7 x64 fails to restore
CentOS 6.5 x64 CAN restore just fine !
CentOS 7.x x64 fails to restore

I also could NOT make a VM snapshot while VM was set (OS Type) as CentOS
7.0 (6.6x64 also, etc) with "Unexpected exception" message in mgmt logs.

Please file a bug here https://github.com/apache/cloudstack/issues and
hopefully someone will  pick it up in some of next releases.
Please also do a little bit of more testing, to see what works, what not,
so you can "properly" file a bug.

Thx

On Wed, 23 Jan 2019 at 10:44, li jerry  wrote:

>
> I modified and recompiled cloudstack4.11.2, and then I can complete the
> xenserver vm snapshot recovery.
>
>
> plugins/hypervisors/xenserver/src/com/cloud/hypervisor/xenserver/resource/CitrixResourceBase.java
>
>
> public VM createWorkingVM(final Connection conn, final String vmName,
> final String guestOSType, final String platformEmulator, final
> List listVolumeTo)
>  throws BadServerResponse, Types.VmBadPowerState, Types.SrFull,
> Types.OperationNotAllowed, XenAPIException, XmlRpcException {
>  // below is redundant but keeping for consistency and code readabilty
>  //final String guestOsTypeName = platformEmulator;
>  String guestOsTypeName = platformEmulator;
> if (guestOsTypeName == null) {
>  final String msg = " Hypervisor " + this.getClass().getName() + " doesn't
> support guest OS type " + guestOSType + ". you can choose 'Other install
> media' to run it as HVM";
>  s_logger.warn(msg);
>  //throw new CloudRuntimeException(msg);
>   guestOsTypeName = "Other install media";
> }
>
>
> This rude modification, I don’t know if it will create other new problems….
>
>
>
>
>
> 发件人: li jerry<mailto:div...@hotmail.com>
> 发送时间: 2019年1月23日 17:16
> 收件人: users@cloudstack.apache.org<mailto:users@cloudstack.apache.org>;
> dev<mailto:d...@cloudstack.apache.org>
> 抄送: users@cloudstack.apache.org<mailto:users@cloudstack.apache.org>
> 主题: 答复: 答复: CloudStack 4.11.2 Snapshot Revert fail
>
> I tried, modified the VM to CENTOS6.5 (64-bit) and reported the same error.
>
>
>
> 发件人: Andrija Panic<mailto:andrija.pa...@gmail.com>
> 发送时间: 2019年1月23日 16:38
> 收件人: dev<mailto:d...@cloudstack.apache.org>
> 抄送: users@cloudstack.apache.org<mailto:users@cloudstack.apache.org>
> 主题: Re: 答复: CloudStack 4.11.2 Snapshot Revert fail
>
> Hi,
>
> Did you try with other OS types - message above suggests that your Xen is
> seen as 6.5 version and doesn't support CentOS 7 - can you try to edit the
> VM to CentOS 6 or CentOS 7.1/7.2 etc ? Does the revert works then ?
>
> On Wed, 23 Jan 2019 at 02:58, Haijiao <18602198...@163.com> wrote:
>
> > This is a fresh installation (XenSever 7.1.2+ACS 4.11.2).
> >
> >
> >
> > We simply can not revert the VM snapshot though there's no changes at all
> > to VM or template.
> >
> >
> > We can consistently reproduce this issue in our environtment with the
> > error log attached. And we believe the issue doesn't exisit in another
> > environment ( XenServer 6.5+ACS 4.7.1).
> >
> >
> >
> >
>
> --
>
> Andrija Panić
>
>

-- 

Andrija Panić

Re: 答复: CloudStack 4.11.2 Snapshot Revert fail

[Andrija Panic]

Hi,

Did you try with other OS types - message above suggests that your Xen is
seen as 6.5 version and doesn't support CentOS 7 - can you try to edit the
VM to CentOS 6 or CentOS 7.1/7.2 etc ? Does the revert works then ?

On Wed, 23 Jan 2019 at 02:58, Haijiao <18602198...@163.com> wrote:

> This is a fresh installation (XenSever 7.1.2+ACS 4.11.2).
>
>
>
> We simply can not revert the VM snapshot though there's no changes at all
> to VM or template.
>
>
> We can consistently reproduce this issue in our environtment with the
> error log attached. And we believe the issue doesn't exisit in another
> environment ( XenServer 6.5+ACS 4.7.1).
>
>
>
>

-- 

Andrija Panić

Re: CloudStack 4.11.2 Snapshot Revert fail

[Andrija Panic]

Hi there,

after VM was deployed and snapshots created - was there any changes to VM
or template from which VM was created - did ACS version get upgraded ?

Best

On Tue, 22 Jan 2019 at 17:52, li jerry  wrote:

> HI ALL
>
> I use CloudStack 4.11.2 to manage Xenserver 7.1.2 (XenServer CU2).
>
> VM snapshot for revert failure (snapshot does not contain memorysnapshot).
>
> 2019-01-23 00:06:54,210 DEBUG [c.c.a.m.ClusteredAgentAttache]
> (Work-Job-Executor-156:ctx-28f7465a job-2867/job-2869 ctx-a04e0ed9)
> (logid:a9ef7fe7) Seq 5-6201456686889173919: Forwarding Seq
> 5-6201456686889173919:  { Cmd , MgmtId: 240661250348494, via: 5(wxac6001),
> Ver: v1, Flags: 100011,
> [{"com.cloud.agent.api.RevertToVMSnapshotCommand":{"reloadVm":false,"vmUuid":"b2a78e9c-06ab-4200-ad6d-fe095f622502","volumeTOs":[{"uuid":"7a58ffdc-b02c-41bf-963c-be56c2da0e9b","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"WXACP01CL01_LUN10","id":19,"poolType":"PreSetup","host":"localhost","path":"/WXACP01CL01_LUN10","port":0,"url":"PreSetup://localhost/WXACP01CL01_LUN10/?ROLE=Primary&STOREUUID=WXACP01CL01_LUN10","isManaged":false}},"name":"ROOT-33","size":21474836480,"path":"dd1cf43d-d5a4-4633-9c3e-8f73d1ccc484","volumeId":93,"vmName":"i-2-33-VM","accountId":2,"format":"VHD","provisioningType":"THIN","id":93,"deviceId":0,"hypervisorType":"XenServer"},{"uuid":"74268aa2-b4e5-4574-a981-027e55b5383f","volumeType":"DATADISK","dataStore":{"org.apache.
> cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"WXACP01CL01_LUN01","id":1,"poolType":"PreSetup","host":"localhost","path":"/WXACP01CL01_LUN01","port":0,"url":"PreSetup://localhost/WXACP01CL01_LUN01/?ROLE=Primary&STOREUUID=WXACP01CL01_LUN01","isManaged":false}},"name":"DATA-33","size":1099511627776,"path":"e2ead686-d0bb-49f2-b656-77c2bf497990","volumeId":95,"vmName":"i-2-33-VM","accountId":2,"format":"VHD","provisioningType":"THIN","id":95,"deviceId":1,"hypervisorType":"XenServer"}],"target":{"id":27,"snapshotName":"i-2-33-VM_VS_20190122155503","type":"Disk","createTime":1548172503000,"current":true,"description":"asdfasdf","quiescevm":true},"vmName":"i-2-33-VM","guestOSType":"CentOS
> 7","wait":0}}] } to 55935224135780
>
> 2019-01-23 00:06:54,222 DEBUG [c.c.a.t.Request]
> (AgentManager-Handler-14:null) (logid:) Seq 5-6201456686889173919:
> Processing:  { Ans: , MgmtId: 240661250348494, via: 5, Ver: v1, Flags: 10,
> [{"com.cloud.agent.api.RevertToVMSnapshotAnswer":{"result":false,"details":"
> Hypervisor com.cloud.hypervisor.xenserver.resource.XenServer650Resource
> doesn't support guest OS type CentOS 7. you can choose 'Other install
> media' to run it as HVM","wait":0}}] }
> 2019-01-23 00:06:54,223 DEBUG [c.c.a.t.Request]
> (Work-Job-Executor-156:ctx-28f7465a job-2867/job-2869 ctx-a04e0ed9)
> (logid:a9ef7fe7) Seq 5-6201456686889173919: Received:  { Ans: , MgmtId:
> 240661250348494, via: 5(wxac6001), Ver: v1, Flags: 10, {
> RevertToVMSnapshotAnswer } }
> 2019-01-23 00:06:54,223 ERROR [o.a.c.s.v.DefaultVMSnapshotStrategy]
> (Work-Job-Executor-156:ctx-28f7465a job-2867/job-2869 ctx-a04e0ed9)
> (logid:a9ef7fe7) Revert VM: i-2-33-VM to snapshot:
> i-2-33-VM_VS_20190122155503 failed due to  Hypervisor
> com.cloud.hypervisor.xenserver.resource.XenServer650Resource doesn't
> support guest OS type CentOS 7. you can choose 'Other install media' to run
> it as HVM
> 2019-01-23 00:06:54,226 DEBUG [c.c.v.s.VMSnapshotManagerImpl]
> (Work-Job-Executor-156:ctx-28f7465a job-2867/job-2869 ctx-a04e0ed9)
> (logid:a9ef7fe7) Failed to revert vmsnapshot: 27
> com.cloud.utils.exception.CloudRuntimeException: Revert VM: i-2-33-VM to
> snapshot: i-2-33-VM_VS_20190122155503 failed due to  Hypervisor
> com.cloud.hypervisor.xenserver.resource.XenServer650Resource doesn't
> support guest OS type CentOS 7. you can choose 'Other install media' to run
> it as HVM
>   at
> org.apache.cloudstack.storage.vmsnapshot.DefaultVMSnapshotStrategy.revertVMSnapshot(DefaultVMSnapshotStrategy.java:393)
>   at
> com.cloud.vm.snapshot.VMSnapshotManagerImpl.orchestrateRevertToVMSnapshot(VMSnapshotManagerImpl.java:846)
>   at
> com.cloud.vm.snapshot.VMSnapshotManagerImpl.orchestrateRevertToVMSnapshot(VMSnapshotManagerImpl.java:1211)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>   at
> com.cloud.vm.snapshot.VMSnapshotManagerImpl.handleVmWorkJob(VMSnapshotManagerImpl.java:1224)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor

Re: Problem with NAT

[Andrija Panic]

Hi Piotr,

not really helping...but 4.10 (afaik) was broken enough to not be
considered an official release - can you please upgrade to 4.11 ?

Cheers
Andrija

On Fri, 11 Jan 2019 at 07:55, Piotr Pisz  wrote:

> Hello community :-)
>
> I have two CS installations, one in version 4.10 and one 4.11. In both
> cases, the infrastructure is configured in the same way, we have an
> advanced network. In 4.11 everything works fine, while in 4.10, static NAT
> and port forwarding from public addresses do not work properly in the VPC
> network. The only difference I found was that in 4.11 in Virtual Router,
> the network looks like this:
> eth0 - link local
> eth1 - vr public / nat and port fw ip's (logically this is ok)
> eth2 - vpc
> In 4.10 instead:
> eth0 - link local
> eth1 - vr public ip
> eth2 - vpc / public ip with nat or port fw enabled (logically this is not
> ok)
> The question is, is VR in 4.10 configured correctly?
> How this diagnose? The VPC network has a default_allow policy, in both
> cases we can get outside.
> I can not find any error in the configuration, in the previously existing
> configuration 4.10 everything worked correctly (we test different
> possibilities).
>
> Regards,
> Piotr
>
>
>

-- 

Andrija Panić

Re: Unable to schedule async job

[Andrija Panic]

/etc/cloudstack/management/log4j-cloud.xml

more info:
https://image.slidesharecdn.com/cloudstacktroubleshooting-lily-150506063617-conversion-gate02/95/cloud-stack-troubleshooting-12-638.jpg?cb=1430894327

cheers

On Thu, 10 Jan 2019 at 19:17, Ivan X Yue  wrote:

> Dag,
>
> That's good idea.  How can I update log to debug / trace mode?
>
> Thanks
> Ivan
>
>
>
> From:   Dag Sonstebo 
> To: "users@cloudstack.apache.org" 
> Date:   2019/01/10 11:40 AM
> Subject:Re: Unable to schedule async job
>
>
>
> Hi Ivan,
>
> Probably a good idea to bump your logging up to debug or trace – these
> show the SQL queries being prepared in the logs.
>
> Regards,
> Dag Sonstebo
> Cloud Architect
> ShapeBlue
>
>
> From: Ivan X Yue 
> Reply-To: "users@cloudstack.apache.org" 
> Date: Thursday, 10 January 2019 at 16:18
> To: "users@cloudstack.apache.org" 
> Subject: Re: Unable to schedule async job
>
> Hi, Dag,
>
> Thank for the reply.
>
> Below is my async_job table.  Today, I try to delete some VM.
> Interestingly, I can delete some VM and some are failed withthis error:
>
> Caused by:
> com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
>
> Duplicate entry '387' for key 'PRIMARY'
>
> But in the async_job table, I don't see id=387 there.
>
> I have attached part of the Management Server log wrt the delete VM
> action.
>
> One minor correction.  I am actually using cloudstack 4.11.1 instead of
> 4.9.2.
>
> MariaDB [cloud]> select id, job_cmd from async_job order by id;
> +-+-+
> | id  | job_cmd |
> +-+-+
> | 275 | org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd |
> | 276 | NULL|
> | 281 | NULL|
> | 287 | NULL|
> | 288 | NULL|
> | 292 | NULL|
> | 305 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd |
> | 307 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd |
> | 309 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin  |
> | 313 | org.apache.cloudstack.api.command.admin.router.StartRouterCmd   |
> | 314 | NULL|
> | 315 | org.apache.cloudstack.api.command.admin.router.StartRouterCmd   |
> | 316 | com.cloud.vm.VmWorkStart|
> | 317 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd |
> | 319 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin  |
> | 321 | NULL|
> | 322 | com.cloud.vm.VmWorkStop |
> | 323 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd |
> | 327 | com.cloud.vm.VmWorkStart|
> | 328 | com.cloud.vm.VmWorkStart|
> | 329 | NULL|
> | 330 | com.cloud.vm.VmWorkStop |
> | 332 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd |
> | 334 | com.cloud.vm.VmWorkStop |
> | 335 | com.cloud.vm.VmWorkStart|
> | 336 | com.cloud.vm.VmWorkStart|
> | 338 | org.apache.cloudstack.api.command.admin.vpc.CreateVPCCmdByAdmin |
> | 340 | com.cloud.vm.VmWorkStop |
> | 341 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd |
> | 342 | com.cloud.vm.VmWorkStop |
> | 343 | org.apache.cloudstack.api.command.user.vpc.RestartVPCCmd|
> | 344 | com.cloud.vm.VmWorkStart|
> | 345 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd |
> | 346 | com.cloud.vm.VmWorkStop |
> | 347 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd |
> | 348 | com.cloud.vm.VmWorkStop |
> | 349 | org.apache.cloudstack.api.command.user.vpc.RestartVPCCmd|
> | 350 | com.cloud.vm.VmWorkStart|
> | 351 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin  |
> | 352 | com.cloud.vm.VmWorkStop |
> | 354 | org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd |
> | 355 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin  |
> | 356 | com.cloud.vm.VmWorkStop  

Re: Importing converted VMs to CloudStack

[Andrija Panic]

Hi there,

For KVM you should have QCOW2 files, or optionally RAW format (not
recommend due to obvious reasons) for both first (root disk) and any data
disk (second disk, third and so on...).

For first disk of VM (ROOT disk) you will want to Add/upload Template from
remote URL (web server, only plain HTTP supported - i.e. no HTTPS), or via
your browser (from your laptop) in the ACS GUI - and deploy VM from
template - it takes time to copy over qcow2 image from Secondary to Primary
Storage - so be patient.

For DATA disks (second volume of VM, third, and so on...) you will want to
upload Volume, again from remote url or via browser. You will attach volume
to VM and when VM boots, CloudStack will move the volume from Secondary
Storage NFS to Primary Storage - so it will take some time - this first
time, so be patient.

Make sure to experiment with different "OS Type" in CloudStack in order to
see when hardware will be emulated ( IDE drives, Intel e1000 nic) and when
it will use VirtIO (scsi hdd controller and VirtIO nic - a.k.a.
paravirtualizer drivers, for better performance) - this is especially
important for Windows VMs since they lack VirtIO out of the box.


Cheers
Andrija

On Sun, Dec 30, 2018, 20:58 Fariborz Navidan  Hello folks,
>
> Happy New Year!
>
> I have converted some VMs from VMware to KVM using virt-convert. How can I
> import them into CloudStack so can be managed in cloud environment?
>
> Kind Regards
>

Re: Storage VM not getting start new setup

[Andrija Panic]

Hi Vivek,

those screenshots clarify some things - so you have both SSVM and CPVM
running, but agent is NOT connected, which means there is some issue of
Agents (inside SSVM and CPVM) connecting to Management Server and receiving
additional inputs for configuration etc.

I would suggest you SSH into the SSVM or CPVM from KVM host (to
169.254.x..y address) - you should definitively be able to SSH to VM on
this control IP (169.254...) - as per the instructions I shared with you
previously (article).

Once inside VM, I would try to telnet to Mangement server IP to port  8250
- check if agents have proper network connectivity to it.

As for the VR - forget about this until you have SSVM  and CPVM working
properly. Also CentOS 5.5 default template, will be eventually downloaded
by SSVM from the internet (once SSVM is fully functional).

I would examine logs on both agent and management server
(/var/log/cloudstack/management OR /agent, then management.log OR
agent.log) - there is pretty nice human-readable log, very verbose by
default, where you should look for any kind of "exception" message etc - in
case things are not working.

Best
Andrija

On Sat, 29 Dec 2018 at 15:23, Vivek Singh  wrote:

> Hi Andrija,
>
> Guest range -- 192.168.100.211-192.168.100.220
> MGMt--192.168.100.201-192.168.100.210
>
> You are right "KVM traffic label" was not defined correctly. I reconfigure
> the VM and set label as cloudbr0 and destroy the SVM twice to double check
> but same error message .
> I attached all the required screenshots all the zone addresses and system
> VM download status as well.
>
> I noticed no VR  is running .Is it required for the SVM to work?
>
>
> Please let me know if further information required.
>
> Regards
> Vivek
>
> -Original Message-
> From: Andrija Panic [mailto:andrija.pa...@gmail.com]
> Sent: 28 December 2018 15:35
> To: Vivek Singh 
> Cc: users 
> Subject: Re: Storage VM not getting start new setup
>
> Hi Vivek,
>
> a few questions/notes:
>
> - ACS mgmt server being on ESXi is not of a concern nor important
> information in my opinion - but thx for sharing info.
> - it would be good to have your IP scheme for everything (IP of mgmt
> server, ip of KVM host, IP of NFS primary/secondary, and IPs reserved for
> POD), but I assume both MGMT server, NFS storage and Guest network range
> are on 192.168.100.x network (per your centos7 output) ?
> - did you preseed systemVM template fine - if both system VMs have been
> started on KVM host, then I assume yes - in GUI, when checking systemVM
> template for KVM, clicking on your Zone, should say Ready and Downloaded.
> - are you using same IP range for Guest network/VMs as is used for
> Management Network ?
>
>
> - I was referring to New Zone wizard with different coloed buttons - i.e.
> like this:
>
> http://docs.cloudstack.apache.org/projects/archived-cloudstack-getting-started/en/latest/_images/nuage_vsp_isolation_method_setting.png
>
> Cheers
>
>
>
>
> On Fri, 28 Dec 2018 at 08:42, Vivek Singh  wrote:
>
> > Hello Andrija,
> >
> > Thanks for the quick response.
> > Here is detail:
> > 1. I tried with latest version give in the link you shared
> >
> https://github.com/apache/cloudstack-documentation/blob/master/source/quickinstallationguide/qig.rst
> .
> > But same result All my installation endup with same result. I mean with
> > error   "There is no secondary storage VM for secondary storage host
> nfs".
> > Even With disabled firewall as well.
> >
> > 2.  when you says "KVM traffic label"  I think you are telling to use
> > bridge cloudbr0 .I am using the same as you can see in the attached
> > outputs. I didn't see any green/red/blue/yellow nice buttons in GUI
> > interface.
> >   Can you please guide me where I can find those?
> > 3. You said don't reboot SVM  manually. I agree but I can't find any
> > other way to find what is going on. It is not accessible through
> > console view in gui. Neither using this command "ssh -i
> > /var/cloudstack/management/.ssh/id_rsa  -p 3922 root@169.254.3.222".
> > However I can see the ip address assigned on gui  of cloudstackt manager.
> >
> > Please find attached all the detail you are looking for.  Let me know
> > if needed further information.
> >
> > Centos6 is with old setup and Centos7 with the link you shared .
> > Remember only one server was up a time.
> >
> > Regards
> > Vivek
> >
> >
> >
> >
> >
> > -Original Message-
> > From: Andrija Panic [mailto:andrija.pa...@gmail.com]
> > Sent: 27 December 2018 15:11
> > To: users 

Re: SystemVM template - timeout waiting for response from storage host

[Andrija Panic]

Hi,

did you preseed the systemVM template as in the guide ?
i.e.
 /usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt
-m /mnt/secondary -u systemvm-kvm-4.11.2 -h kvm -s
 -F

After starting the Management server (after you have preseeded the systemVM
template previously...), CloudStack should recognize systemVM template as
fully downloaded on Secondary Storage NFS and mark it as Ready and 100%
downloaded.

Can your KVM host access Secondary Storage NFS (mount it) ?
Can you Management server access Secondary Storage NFS ?

Cheers

On Fri, 28 Dec 2018 at 21:09, Richard Persaud 
wrote:

> Hello,
>
> I am running 4.11.2.0
>
> I am constantly getting an error on my SystemVM Template (KVM) that it is
> not in a ready state due to "Timeout waiting for response from storage
> host".
> If I reboot all the host machines, the SystemVM template is OK for a
> while. Then after some time it will enter the error state.
>
> The infrastructure looks fine, everything is up and seemingly working. I
> can connect to all of the NFS storage hosts without issue.
>
> Any ideas?
>
> Regards,
>


-- 

Andrija Panić

Re: Storage VM not getting start new setup

[Andrija Panic]

Hi Vivek,

a few questions/notes:

- ACS mgmt server being on ESXi is not of a concern nor important
information in my opinion - but thx for sharing info.
- it would be good to have your IP scheme for everything (IP of mgmt
server, ip of KVM host, IP of NFS primary/secondary, and IPs reserved for
POD), but I assume both MGMT server, NFS storage and Guest network range
are on 192.168.100.x network (per your centos7 output) ?
- did you preseed systemVM template fine - if both system VMs have been
started on KVM host, then I assume yes - in GUI, when checking systemVM
template for KVM, clicking on your Zone, should say Ready and Downloaded.
- are you using same IP range for Guest network/VMs as is used for
Management Network ?


- I was referring to New Zone wizard with different coloed buttons - i.e.
like this:
http://docs.cloudstack.apache.org/projects/archived-cloudstack-getting-started/en/latest/_images/nuage_vsp_isolation_method_setting.png

Cheers




On Fri, 28 Dec 2018 at 08:42, Vivek Singh  wrote:

> Hello Andrija,
>
> Thanks for the quick response.
> Here is detail:
> 1. I tried with latest version give in the link you shared
> https://github.com/apache/cloudstack-documentation/blob/master/source/quickinstallationguide/qig.rst.
> But same result All my installation endup with same result. I mean with
> error   "There is no secondary storage VM for secondary storage host nfs".
> Even With disabled firewall as well.
>
> 2.  when you says "KVM traffic label"  I think you are telling to use
> bridge cloudbr0 .I am using the same as you can see in the attached
> outputs. I didn't see any green/red/blue/yellow nice buttons in GUI
> interface.
>   Can you please guide me where I can find those?
> 3. You said don't reboot SVM  manually. I agree but I can't find any other
> way to find what is going on. It is not accessible through console view in
> gui. Neither using this command "ssh -i
> /var/cloudstack/management/.ssh/id_rsa  -p 3922 root@169.254.3.222".
> However I can see the ip address assigned on gui  of cloudstackt manager.
>
> Please find attached all the detail you are looking for.  Let me know if
> needed further information.
>
> Centos6 is with old setup and Centos7 with the link you shared . Remember
> only one server was up a time.
>
> Regards
> Vivek
>
>
>
>
>
> -Original Message-
> From: Andrija Panic [mailto:andrija.pa...@gmail.com]
> Sent: 27 December 2018 15:11
> To: users 
> Subject: Re: Storage VM not getting start new setup
>
> Hi Vivek,
>
> FYI: Please note that CentOS 6 is depreciated in latest CloudStack version
> - perhaps better to try with latest versions of Centos and CloudStack as
> noted here:
>
> https://github.com/apache/cloudstack-documentation/blob/master/source/quickinstallationguide/qig.rst
> - this a fresh guide, tested properly on Centos 7.5 and CLoudStack 4.11.2
>
> But down to your problem...
>
> Based on your input there are 2 VMs started - but not fully "patched" to
> become CPVM and SSVM.
>
> Don't ever manually reboot VMs since they are not meant to be rebooted -
> it will boot broken with i.e. no network config inside etc.
> Just use the "Destroy" button on both CPVM and SSVM  and they will be
> re-generated again brand new. After VMs are initially created, there will
> be special ISO attached to them to "patch" them properly and these vanila
> VMs will be configured as needed (with custom scripts and info from DB),
> including IP settings etc...
> It takes some time to bring up the VMs, so please be patient.
>
> I would also stronglu suggest, during troubleshooting, to disable firewall
> completely on KVM host - later you can play with it - and it can sometimes
> be a problem - so safer to disable during installation.
>
> Can you share your network setup on KVM host , and zone netwrok setup,
> hypervisor used, etc ? If you are following quick guide, I assume KVM on
> CentOS 6..
> Name of your bridge should be set as the "KVM traffic label" for each
> traffic type (Management, Guest,Storage etc) - when you click on these
> green/red/blue/yellow nice buttons, you can set "KVM traffic label" - this
> should be the name of your bridge that you previously created on KVM host.
>
> Another things to mention - after CPVM and SSVM are started fine - you can
> use Console access to SSVM and login with root/password, and then test SSVM
> operations by running script:  /usr/local/cloud/systemvm/ssvm-check.sh
> This script will check if SSVM can access internet, if DNS resolution
> works, if it can mount Secondary Storage to itself, and if it can connect
> it's Agent to the Management Server.
>
> Cheers
&

Re: Storage VM not getting start new setup

[Andrija Panic]

Hi Vivek,

FYI: Please note that CentOS 6 is depreciated in latest CloudStack version
- perhaps better to try with latest versions of Centos and CloudStack as
noted here:
https://github.com/apache/cloudstack-documentation/blob/master/source/quickinstallationguide/qig.rst
- this a fresh guide, tested properly on Centos 7.5 and CLoudStack 4.11.2

But down to your problem...

Based on your input there are 2 VMs started - but not fully "patched" to
become CPVM and SSVM.

Don't ever manually reboot VMs since they are not meant to be rebooted - it
will boot broken with i.e. no network config inside etc.
Just use the "Destroy" button on both CPVM and SSVM  and they will be
re-generated again brand new. After VMs are initially created, there will
be special ISO attached to them to "patch" them properly and these vanila
VMs will be configured as needed (with custom scripts and info from DB),
including IP settings etc...
It takes some time to bring up the VMs, so please be patient.

I would also stronglu suggest, during troubleshooting, to disable firewall
completely on KVM host - later you can play with it - and it can sometimes
be a problem - so safer to disable during installation.

Can you share your network setup on KVM host , and zone netwrok setup,
hypervisor used, etc ? If you are following quick guide, I assume KVM on
CentOS 6..
Name of your bridge should be set as the "KVM traffic label" for each
traffic type (Management, Guest,Storage etc) - when you click on these
green/red/blue/yellow nice buttons, you can set "KVM traffic label" - this
should be the name of your bridge that you previously created on KVM host.

Another things to mention - after CPVM and SSVM are started fine - you can
use Console access to SSVM and login with root/password, and then test SSVM
operations by running script:  /usr/local/cloud/systemvm/ssvm-check.sh
This script will check if SSVM can access internet, if DNS resolution
works, if it can mount Secondary Storage to itself, and if it can connect
it's Agent to the Management Server.

Cheers
Andrija




On Thu, 27 Dec 2018 at 12:22, Vivek Singh  wrote:

> Hello Friends,
>
>
>
> I am new to cloudstack and from last 2 weeks I am struggling to install
>  cloudstack using  quick installation procedure (
> http://docs.cloudstack.apache.org/projects/cloudstack-installation/en/4.9/qig.html
> ).
>
> I tried all the method that I can google and apply but no luck.
>
>
>
>
>
> I am using  VM on ESXi  as a management server. I enable promiscuous mode
> on ESXi switch. I am using Centos6.10
>
>
>
> My ssvm and cosole-proxy VM are not coming up. I'm login through
> virt-manager  into both system vm and found their network cards are not up.
>
> I assign static ip and rebooted but all the ip deleted from interfaces
> after reboot.
>
>
>
> I am attaching log when I started the installation utill I getting the
> error message “There is no secondary storage VM for secondary storage host”.
>
>
>
>
>
> I really appreciate any help
>
>
>
>
>
>
>
> *Vivek Singh *
>
>
>


-- 

Andrija Panić

Re: CloudStack Ceph Storage experiments / help requested

[Andrija Panic]

Hi Ivan,

I'm using zone-wide ceph in 4.5, upgrade to 4.8 version - so it should be
supported just fine. I didn't respond to your original email since I had no
idea why it would not work.

I'm even using it as we speak in DEV 4.11.2 zone - so that is not the
problem.

Cheers

On Wed, 26 Dec 2018 at 21:55, Ivan Kudryavtsev 
wrote:

> I removed zone-wide Ceph storage and created Cluster Wide. Now it works. I
> traced the problem to
>
>
> https://github.com/apache/cloudstack/blob/68b4b8410138a1a16337ff15ac6260e0ecae9bc0/engine/storage/src/org/apache/cloudstack/storage/allocator/AbstractStoragePoolAllocator.java#L118
>
> Looks like zone-wide Ceph is not supported.
>
> ср, 26 дек. 2018 г. в 15:07, Ivan Kudryavtsev :
>
> > Hello, colleagues. Happy Merry Christmas to you. Try to play with CS Ceph
> > Block Storage and stumbled upon the problem with the deployment of VM to
> > Ceph RBD.
> >
> > ACS 4.11.3
> >
> > 1. Created ZoneWide RBD Storage which is UP in CS with 'rbd' tag.
> > 2. Created SO with 'rbd' storage tag.
> >
> > Upon deployment I see next logs:
> >
> > 2018-12-27 02:58:55,691 DEBUG [o.a.c.s.a.ZoneWideStoragePoolAllocator]
> > (API-Job-Executor-102:ctx-fe2aab98 job-459769 ctx-bddac228)
> > (logid:dd88c73e) ZoneWideStoragePoolAllocator to find storage pool
> > 2018-12-27 02:58:55,696 DEBUG [o.a.c.s.a.AbstractStoragePoolAllocator]
> > (API-Job-Executor-102:ctx-fe2aab98 job-459769 ctx-bddac228)
> > (logid:dd88c73e) Checking if storage pool is suitable, name: null
> ,poolId:
> > 37
> > 2018-12-27 02:58:55,699 DEBUG [c.c.s.StorageManagerImpl]
> > (API-Job-Executor-102:ctx-fe2aab98 job-459769 ctx-bddac228)
> > (logid:dd88c73e) Destination pool id: 37
> > 2018-12-27 02:58:55,711 DEBUG [c.c.s.StorageManagerImpl]
> > (API-Job-Executor-102:ctx-fe2aab98 job-459769 ctx-bddac228)
> > (logid:dd88c73e) Pool ID for the volume with ID 21022 is null
> > 2018-12-27 02:58:55,716 DEBUG [c.c.s.StorageManagerImpl]
> > (API-Job-Executor-102:ctx-fe2aab98 job-459769 ctx-bddac228)
> > (logid:dd88c73e) Found storage pool HA-CEPH-SSD-R5 of type RBD with
> > overprovisioning factor 1
> > 2018-12-27 02:58:55,716 DEBUG [c.c.s.StorageManagerImpl]
> > (API-Job-Executor-102:ctx-fe2aab98 job-459769 ctx-bddac228)
> > (logid:dd88c73e) Total over provisioned capacity calculated is 1 *
> > 1099511627776
> > 2018-12-27 02:58:55,716 DEBUG [c.c.s.StorageManagerImpl]
> > (API-Job-Executor-102:ctx-fe2aab98 job-459769 ctx-bddac228)
> > (logid:dd88c73e) Total capacity of the pool HA-CEPH-SSD-R5 with ID 37 is
> > 1099511627776
> > 2018-12-27 02:58:55,717 DEBUG [c.c.s.StorageManagerImpl]
> > (API-Job-Executor-102:ctx-fe2aab98 job-459769 ctx-bddac228)
> > (logid:dd88c73e) Checking pool: 37 for storage allocation , maxSize :
> > 1099511627776, totalAllocatedSize : 10737418240, askingSize :
> 64424509440,
> > allocated disable threshold: 0.85
> > 2018-12-27 02:58:55,718 DEBUG [o.a.c.s.a.AbstractStoragePoolAllocator]
> > (API-Job-Executor-102:ctx-fe2aab98 job-459769 ctx-bddac228)
> > (logid:dd88c73e) List of pools in descending order of free capacity: []
> > 2018-12-27 02:58:55,718 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > (API-Job-Executor-102:ctx-fe2aab98 job-459769 ctx-bddac228)
> > (logid:dd88c73e) No suitable pools found for volume:
> > Vol[21022|vm=5641|ROOT] under cluster: 1
> >
> > So, It looks like that the storage is found (HA-CEPH-SSD-R5). Its size
> and
> > utilization is determined OK, but next an empty set as a result of the
> > calculation.
> >
> > Any help is appreciated.
> >
> > --
> > With best regards, Ivan Kudryavtsev
> > Bitworks LLC
> > Cell RU: +7-923-414-1515
> > Cell USA: +1-201-257-1512
> > WWW: http://bitworks.software/ 
> >
> >
>
> --
> With best regards, Ivan Kudryavtsev
> Bitworks LLC
> Cell RU: +7-923-414-1515
> Cell USA: +1-201-257-1512
> WWW: http://bitworks.software/ 
>


-- 

Andrija Panić

Re: How to Reorganise Storage Tags

[Andrija Panic]

Hi Melani,

for root volumes - I'm pretty sure it's like following - sorry long
copy/paste - here we examine sample VM (which we really did migrate from
NFS/CEPH to SolidFire)
(comments after code below...)

mysql> select name,service_offering_id from vm_instance where
uuid="5525edf1-94d7-4678-b484-a3292749c08f";
+--+-+
| name | service_offering_id |
+--+-+
| VM-X | 837 |
+--+-+
1 row in set (0.01 sec)

mysql> select name,disk_offering_id from volumes where instance_id in
(select id from vm_instance where
uuid="5525edf1-94d7-4678-b484-a3292749c08f") and name like "ROOT%";
+--+--+
| name | disk_offering_id |
+--+--+
| ROOT-226 |  837 |
+--+--+
1 row in set (0.00 sec)


mysql> select name,type,tags from disk_offering where id=837;
+--+-++
| name | type| tags   |
+--+-++
| 4vCPU-8GB-SSD-STD-SF | Service | SolidFire1 |
+--+-++
1 row in set (0.00 sec)

mysql> select cpu,speed,ram_size from service_offering where id=837;
+--+---+--+
| cpu  | speed | ram_size |
+--+---+--+
|4 |  2000 | 8192 |
+--+---+--+

So here you see both service offering (compute offering for user VMs, or
really service offering for system VMs) AND the disk offering (ROOT) have
same id of 837 (single offering that we examine here).

So it's should be enough to just
- change the disk_offering_id in the volumes table (for specific, migrated
root - to point to some offering that targets new storage (by storage tags)
- and later again change properly via UI//API to correct one
- change service_offering_id in vm_instance table (for specific VM whose
ROOT volume was migrated)
-these 2 above needs to match obviously..

Later when you change via UI/API the to correct/exact Compute Offering the
way you like it - make sure that both tables are updated accordingly with
new ID - in ACS 4.8 only service_offering_id (vm_instances table) was
updated to point to new service offering, while disk_offering_id in
"volume" table was NOT updated - we had an in-house patch for this to
update this one as well...

Above I always say "manually change to some new one - and later via UI"  -
in order to generate proper usage records for final offering chosen  -
otherwise you can target final offering directly with DB edit...)

Hope I did not confuse you even more :)

Cheers
andrija



On Thu, 20 Dec 2018 at 14:29, Melanie Desaive 
wrote:

> Hi Andrija,
>
> I tested your suggestion and they worked perfectly for data-volumes.
>
> Now I am trying to figure out how to change storage tags for
> root-volumes of VMs and SystemVMs.
>
> For the root-volumes of user VMs the storage tag seems to come from the
> service offering, I did not find any relationships to the table
> disk_offering up to now. Still I am not 100% shure through which fields
> the relationship is defined, and continue researching.
>
> For the root-volumes of the system-VMs the easiest way to change storage
> tags seems to define new offering and then destroy/redeploy...
>
> If you like I will summarize my knowledge about this issue when I am
> through with the task..
>
> Greetings,
>
> Melanie
>
>
>
> Am 13.12.18 um 19:58 schrieb Andrija Panic:
> > Hi Melanie,
> >
> > I did change it, yes (tags on existing offerings) and no need to restart
> > mgmt, just I once had to wait for a minute or two, but Im sure it was me
> > messed up something at that specific moment.
> >
> > Tags are evaluated during creation of the volume only (and obviously when
> > changing offering as you can see) and not relevant later for the volume -
> > vs. i.e. cache mode (writeback etc.) which is read during starting VM
> > (attaching volume to VM in boot process).
> >
> > Let me know if I can help more.
> >
> > Cheers
> >
> > On Thu, Dec 13, 2018, 18:41 Melanie Desaive <
> m.desa...@heinlein-support.de
> > wrote:
> >
> >> Hi andrija,
> >>
> >> thanks a lot for your answer.
> >>
> >> Indeed is absolutely sufficient for me to know that I may change
> >> disk_offering_id for a volume. I would assume it is not necessary to
> shut
> >> down/restart the VM or restart management service, but will try
> tomorrow.
> >>
> >> I will figure out a suitable path to migrate the volumes to their
> >> destination pools and also change the offering to those with the desired
> &

Re: Import VMs into CloudStack

[Andrija Panic]

Hi Mark,

unfortunately that is not possible (not in an official way at least) -
since CloudStack is keeping tons of meta-data (info about VM hardware and
such) in DB.

For simpler VMs, you could *experiment* with deploying brand new VM with
(empty) appropriate disk sizes, shutdown VM, then go and replace QCOW2
files on your NFS storage with original volumes, but again this is all
hacking and is not really guarantied to work - MAC and IP address would
certainly change etc, perhaps underlying hardware for VM and you would need
to play with drivers potentially (for sure if Windows VMs) - not the best
approach - but is possible to work.

Normal way would be to export original ROOT (system/OS, say, /dev/vda)
volume of your VM as qcow2 files to some web server (optionally RAW format,
but that is both inefficient and might have its issues) and then register
it as template inside CloudStack and finally deploy new VM in regular way -
still some play with drivers etc might be needed, possible mount points (if
not used UUID of volume in /etc/fstab), change MAC address of VM (i.e.
delete udev rules for network, reboot VM), etc.
For data volumes (i.e. vdb and onwards) you would just upload qcow2 file as
Volume to CloudStack - and attach it to your VM (this "attach" is "lazy"
attach - meaning when you power on VM very first time (or you hot-plugged
the volume during VM running...) CloudStack will move the qcow2 data file
that you uploaded, from Secondary NFS to Primary Storage - and this move
(qemu-img convert) takes time obviously...so you are not confused why it's
so long for the first time.

Just in case you have some Windows VMs - you would probably need to
initialize HAL again (google this or ping me if needed) - especially if
changing from emulated/IDE stuff to VirtIO (otherwise you will get nice
BSOD as usual with driver issues)

CloudStack is an awesome stuff, just give it a try.

Hope that helps
Andrija

On Thu, 20 Dec 2018 at 01:45, Werner, Mark  wrote:

> Hi,
>
>
>
> I am curious to know if I am doing migrations of workloads into a virtual
> infrastructure, in this case KVM, that is managed by CloudStack, and I
> perform the migrations directly to the hypervisor(‘s), is there a way then
> to register or import those migrated VM instances into the CloudStack
> environment / inventory so that CloudStack is aware of them and can manage
> them?
>
>
>
> Thanks,
>
>
>
> *Mark Werner* | Senior Systems Engineer | Cloud & Infrastructure Services
>
> Unisys | Mobile Phone 586.214.9017 | mark.wer...@unisys.com
>
> 11720 Plaza America Drive, Reston, VA 20190
>
> [image: unisys_logo] 
>
>
>
> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
> MATERIAL and is for use only by the intended recipient. If you received
> this in error, please contact the sender and delete the e-mail and its
> attachments from all devices.
>
> [image: Grey_LI]   [image:
> Grey_TW]  [image: Grey_GP]
> [image: Grey_YT]
> [image: Grey_FB]
> [image: Grey_Vimeo]
> [image: Grey_UB] 
>
>
>


-- 

Andrija Panić

Re: Resource Allocation Question

[Andrija Panic]

Hi Asai,

you have 3 servers of 8 cores @ 2.4GHz - but your turbo boost frequency is
3.2 GHZ (
https://ark.intel.com/products/83352/Intel-Xeon-Processor-E5-2620-v3-15M-Cache-2-40-GHz-)
and for unknown reason, this Turbo Boost frequency is what ACS sees - but
that only means, say 30% effective over-provisioning of CPU - not issues in
other things at all !

Your total CPU/MHz math looks like this: 3 servers x 8 cores (HT) x 3.2GHZ
= 24 x 3.2GHz = 76.8 GHz - so the math is clear :)

You should be able to use up to  "
cluster.cpu.allocated.capacity.disablethreshold " as percentage of your
total CPU/GHz - which by default (unless I'm wrong) 0.8, meaning 80% of
your total CPU/MHz = 0.8 x 76.8GHz = 61.44 GHz... Whatever is the color on
Dashboard - you should be able to use up to total of 61.44 GHz really -
after that you will receive error about "Insufficient Capacity..."

There is lower threshold used just to notify you
"cluster.cpu.allocated.capacity.notificationthreshold" when you are close
to limit... (you get email only once !)

These 2 thresholds can be set as part of Global Settings (requires MGMT
server restart afterwards) - or you can set it on the Cluster level (no
need for MGMT restart)

You can also set CPU over-provisioning as neeeded, for testing purposes,
say to 3.0 so you ACS will see 300% of your real GHz = 230.4 GHz

Hope that makes sense.

NOTE: one thing that I see perhaps confuses you - in Compute Offerings, you
define number of CPU cores (say, 4 cores), but also a speed of each CPU
core (say, 2.0 GHz), instead of real core frequency = this means your VM
will effectively be allocated 4 cores x 2.0GHz = 8GHz. So this means that
your whole cluster (but also a host which will be chosen eventually) needs
to have at least 8GHz free/available in order to successfully deploy VM.
So forget about real number of cores vs. virtual cores (as part of Compute
Offering) - better observe only GHz (Dashboard does display only GHz not
number of Cores, afaik?)

Cheers
Andrija

On Tue, 18 Dec 2018 at 18:56, Asai  wrote:

> Thank you for your response.
>
> So, is there any way we can take advantage of the processor speed that
> doesn’t seem to be allocated?  We’re not clear on how this works.  It seems
> like there’s a tremendous amount of underutilization here?  Although we
> have allocated 20 out of 24 cores, it seems like we’re not able to access
> the resources the processors like they should be able to be.
>
> What are we missing here?
>
> > On Dec 13, 2018, at 2:27 PM, Andrija Panic 
> wrote:
> >
> > In my case, hyperthreading number of cores (whatever OS sees as number of
> > cores) times the TurboBoost GHZ is what I get, so for 2x8 cored 2.6Ghz
> (32
> > cores with hyperthreading) I get 32 x 3.4 Ghz, which is not what you can
> > achieve at any time (that frequency can be obtained on only very few
> cores
> > at any moment, not nearly all cores).
> >
> > Go to Host in GUI and check its statistics, you will see - take into
> > account any overprovisioning applied
> >
> > Cheers
> >
> > On Thu, Dec 13, 2018, 20:19 Rafael Weingärtner <
> rafaelweingart...@gmail.com
> > wrote:
> >
> >> You have a four core CPU, with 8 threads each? Then, each thread is 2.4
> >> GHz. So, 2.4 * 4 * 8.
> >>
> >> On Thu, Dec 13, 2018 at 6:14 PM Asai 
> wrote:
> >>
> >>> Greetings,
> >>>
> >>> I have a simple question regarding Cloudstack resource allocation.
> >>>
> >>> In the dashboard, I’m seeing that our Memory is 25 out of 30 GB and the
> >>> circle is colored red.  # of CPUs is 20 out of 24 and is colored red.
> >> But
> >>> CPU is 33 GHz out of 76 GHz.  We have a single Xeon E5-2620 v3 @
> 2.40GHz.
> >>> This is a 4 core / 8 thread CPU.  How is that we have 76 GHz available?
> >> It
> >>> seems like we’d have around 57 or so GHz available.
> >>>
> >>> Thanks for your assistance.
> >>>
> >>> Asai
> >>>
> >>>
> >>>
> >>
> >> --
> >> Rafael Weingärtner
> >>
>
>

-- 

Andrija Panić

Re: Upgrading OS on hypervisors

[Andrija Panic]

No magic as Ivan said,

you may want to check dynamic auto-convergence in case of busy VMs (high
RAM change ram i.e. busy MSSQL servers and such) - qemu 2.5 needed for
this, can be consumed from 4.11 and onwards - but might not be relevant in
your case.

Some very short interruption IS done - i.e. source VM on source host is
paused - and last few bits/megabits of RAM are synced to new VM on new host
- then source destination VM is killed/destroyed and destination VM (which
was paused all time...) will be un-paused/resumed - this should have be
less than a second or so...

No magic to make it happen without a brief interruption to networking - the
faster management network the better (better throughput, so less pause
period for VM...)


Cheers

On Fri, 14 Dec 2018 at 07:45, Rakesh v  wrote:

> Hello Ivan
>
> We upgrade only when critical security patch is released like meltdown and
> L1TF else we don't do upgrades for every single kernel release
>
> Sent from my iPhone
>
> > On 13-Dec-2018, at 11:05 PM, Ivan Kudryavtsev 
> wrote:
> >
> > Rakesh,
> > No magic exists. Migrate forth, reboot, and migrate back. What I'm really
> > wondering about is why you upgrade every time, Ubuntu releases a new
> > kernel... There are a pretty small amount of fixes related to KVM and
> hosts
> > are located in the private networks, they are single-tenant. Maybe you
> just
> > have to change to policy for that?
> >
> > чт, 13 дек. 2018 г. в 16:54, Rakesh v :
> >
> >> Hello Folks
> >>
> >> I have a question regarding upgrading OS on hypervisor. We have few
> >> platforms with around 200 hypervisors in each platforms. Every time a
> new
> >> Ubuntu kernel comes in with new security patch like L1TF fix or other ,
> we
> >> need to upgrade kernel on all hypervisors. Before rebooting hypervisor
> we
> >> enable maintenance on it so that all vm's are migrated away. During this
> >> maintenance we always see some Network interruptions because of live
> >> migrations.
> >>
> >> How are you guys handling such situations and how are doing the OS
> >> upgrades? If you have some suggestions or using some automation tools
> then
> >> please let know
> >>
> >> I want to reduce time taken to upgrade all hypervisors without any
> impact
> >> to Networks or VM's
> >>
> >> Sent from my iPhone
> >
> >
> >
> > --
> > With best regards, Ivan Kudryavtsev
> > Bitworks LLC
> > Cell RU: +7-923-414-1515
> > Cell USA: +1-201-257-1512
> > WWW: http://bitworks.software/ 
>


-- 

Andrija Panić

Re: URGENT: Unable to schedule async job for command com.cloud.vm.VmWorkStart, unexpected exception

[Andrija Panic]

That's funny - does restarting mgmg helps ? I assume you restored full
cloud DB "property"...

On Fri, Dec 14, 2018, 09:42 Ugo Vasi  Hi Andrija,
> I restored database but now I can't see any instance, host or system vm.
> On dashboard I see a real situation statistics but I can't see any data,
> neither via webUI nor via cloudmonkey... I verify existence of the
> instances/hosts data on cloud's database tables.
>
>
>
> Il 13/12/18 17:22, Andrija Panic ha scritto:
> > Done it once (8h old DB)...make sure to kill any new VMs that were
> created
> > in that period of time (and volumes/templates on the end storage).
> >
> > You might run into situation that some VMs are i.e. powered on during
> that
> > last 8h, so DB state for that VM will not be in sync with reality, but
> that
> > should be synced shorty after.
> >
> > I.e. hope that a very few changes/creations of any resources have
> happened
> > during that period - and expect some stuff will need to be manually
> > synced/fixed/repeated - i.e. user created PF rule on VR, but now you have
> > restored DB so - that might be a challenge to fix without pissing off
> users
> > :)
> >
> > Fingers crossed!
> >
> > On Thu, Dec 13, 2018, 14:33 Ugo Vasi  >
> >> Hi all,
> >> is there anyone who tried to restore the database a day ago and
> >> restarted cloudstack manager?
> >>
> >> The two situations differ for the status of the VM / VR and I do not
> >> know what to expect and bearing in mind that the problem could recur.
> >>
> >> The statistics don't interest me.
> >>
> >>
> >> Il 13/12/18 10:39, Ugo Vasi ha scritto:
> >>> We have verified that the problem is not tied to a vm but it also
> >>> concerns the other VMs that are stopped. Trying to restart them gives
> >>> the same error message.
> >>>
> >>> Restoring the previous day's database which problems might imply?
> >>>
> >>>
> >>> Il 13/12/18 09:01, Ugo Vasi ha scritto:
> >>>> Hi all,
> >>>> I'm trying to reboot a vm after the host it ran on crashed and
> >>>> restarted from the HA system. All the VMs running on the rebooted
> >>>> host were restarted on other hosts except one.
> >>>> In the web interface and using cloudmonkey I get this message:
> >>>>"Unable to schedule async job for command com.cloud.vm.VmWorkStart,
> >>>> unexpected exception."
> >>>>
> >>>> In the management-server.log file there would seem to be a problem
> >>>> when creating an element that is duplicated (Duplicate entry ''
> >>>> for key 'PRIMARY'):
> >>>>
> >>>> 2018-12-13 08:44:06,659 DEBUG [c.c.a.ApiServlet]
> >>>> (qtp1096283470-445:ctx-6c065e06) (logid:87edf8d7) ===START===
> >>>> 10.80.0.6 -- GET
> >>>>
> >>
> command=startVirtualMachine&response=json&id=dde566b2-ef2c-4f86-a82b-c8286f0c24f7&_=1544687047015
> >>>> 2018-12-13 08:44:06,665 DEBUG [c.c.a.ApiServer]
> >>>> (qtp1096283470-445:ctx-6c065e06 ctx-c8b27e4a) (logid:87edf8d7) CIDRs
> >>>> from which account 'Acct[26e597f2-b5ca-11e8-a619-c8cbb8cb15cd-admin]'
> >>>> is allowed to perform API calls: 0.0.0.0/0,::/0
> >>>> 2018-12-13 08:44:06,693 INFO  [o.a.c.f.j.i.AsyncJobMonitor]
> >>>> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:7e0c4dc9) Add
> >>>> job-1343 into job monitoring
> >>>> 2018-12-13 08:44:06,698 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>> (qtp1096283470-445:ctx-6c065e06 ctx-c8b27e4a) (logid:87edf8d7) submit
> >>>> async job-1343, details: AsyncJobVO {id:1343, userId: 2, accountId:
> >>>> 2, instanceType: VirtualMachine, instanceId: 8, cmd:
> >>>> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin,
> >>>> cmdInfo:
> >>>>
> >>
> {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"6170","id":"dde566b2-ef2c-4f86-a82b-c8286f0c24f7","ctxDetails":"{\"interface
> >>
> >>
> com.cloud.vm.VirtualMachine\":\"dde566b2-ef2c-4f86-a82b-c8286f0c24f7\"}","ctxAccountId":"2","uuid":"dde566b2-ef2c-4f86-a82b-c8286f0c24f7","cmdEventType":&quo

Re: Resource Allocation Question

[Andrija Panic]

In my case, hyperthreading number of cores (whatever OS sees as number of
cores) times the TurboBoost GHZ is what I get, so for 2x8 cored 2.6Ghz (32
cores with hyperthreading) I get 32 x 3.4 Ghz, which is not what you can
achieve at any time (that frequency can be obtained on only very few cores
at any moment, not nearly all cores).

Go to Host in GUI and check its statistics, you will see - take into
account any overprovisioning applied

Cheers

On Thu, Dec 13, 2018, 20:19 Rafael Weingärtner  You have a four core CPU, with 8 threads each? Then, each thread is 2.4
> GHz. So, 2.4 * 4 * 8.
>
> On Thu, Dec 13, 2018 at 6:14 PM Asai  wrote:
>
> > Greetings,
> >
> > I have a simple question regarding Cloudstack resource allocation.
> >
> > In the dashboard, I’m seeing that our Memory is 25 out of 30 GB and the
> > circle is colored red.  # of CPUs is 20 out of 24 and is colored red.
> But
> > CPU is 33 GHz out of 76 GHz.  We have a single Xeon E5-2620 v3 @ 2.40GHz.
> > This is a 4 core / 8 thread CPU.  How is that we have 76 GHz available?
> It
> > seems like we’d have around 57 or so GHz available.
> >
> > Thanks for your assistance.
> >
> > Asai
> >
> >
> >
>
> --
> Rafael Weingärtner
>

Re: How to Reorganise Storage Tags

[Andrija Panic]

Hi Melanie,

I did change it, yes (tags on existing offerings) and no need to restart
mgmt, just I once had to wait for a minute or two, but Im sure it was me
messed up something at that specific moment.

Tags are evaluated during creation of the volume only (and obviously when
changing offering as you can see) and not relevant later for the volume -
vs. i.e. cache mode (writeback etc.) which is read during starting VM
(attaching volume to VM in boot process).

Let me know if I can help more.

Cheers

On Thu, Dec 13, 2018, 18:41 Melanie Desaive  Hi andrija,
>
> thanks a lot for your answer.
>
> Indeed is absolutely sufficient for me to know that I may change
> disk_offering_id for a volume. I would assume it is not necessary to shut
> down/restart the VM or restart management service, but will try tomorrow.
>
> I will figure out a suitable path to migrate the volumes to their
> destination pools and also change the offering to those with the desired
> tags that way. Absolutely ok for me to do it in two or more steps.
>
> Anyone ever changed disk_offering.tags manually?
>
> Anyway, happy to see a solution for my task and looking forward to try it
> out tomorrow.
>
> Greetings,
>
> Melanie
>
> ⁣Gesendet mit BlueMail ​
>
> Am 13. Dez. 2018, 17:32, um 17:32, Andrija Panic 
> schrieb:
> >Hi Melanie,
> >
> >when  moving volume to new storage, when you want to change disk
> >offering
> >(or compute offering for ROOT disk...), ACS doesn't allow that - it
> >lists
> >only offerings that have same tag as current offering (not good...)
> >
> >We have inhouse patch, so that you CAN do that, by making sure to list
> >all
> >offergins that have TAG that matches the TAG of the new destination
> >pool of
> >the volume (hope Im clear here).
> >
> >All volumes read tag from their offering - so just either change
> >disk_offering_id filed for each moved/migrated volume  to point to same
> >sized offering on new storage - and then normally change it once more
> >via
> >UI to a new once etc - or manualy change to smaller disk offering (DB
> >edit)
> >and later via UI/API to correct (same size) disk offering (or bigger if
> >you
> >want to really resize)
> >
> >I can try to share a patch in a non-developer, copy/paste way - in case
> >you
> >want to patch your ACS to support this (as explained at the begining of
> >the
> >email...)
> >
> >Hope that helps
> >
> >Cheers
> >
> >On Thu, 13 Dec 2018 at 13:50, Melanie Desaive
> >
> >wrote:
> >
> >> Hi all,
> >>
> >> we are currently reorganizing our SAN Setup and I would like to
> >> introduce new storage tags on my existing volumes.
> >>
> >> I was naively assuming to simply change the tags or offering by GUI
> >or
> >> API calls.
> >>
> >> Does not seem to work. Only way to change the tags, seems to be by
> >using
> >> a new disk offering, which is denied, when the tags between old and
> >new
> >> offering differ. :( Or am I missing something?
> >>
> >> I had a look into the cloud database, and the storage tags, seem to
> >be
> >> only stored in
> >>
> >>   disk_offering.tags
> >> and
> >>   storage_pool_tags.tag
> >>
> >> Would it be a valid option for me to update disk_offering.tags by SQL
> >to
> >> the desired value or could that break some deeper logic?
> >>
> >> Or is there even a better way to change the storage tags for existing
> >> volumes. (With or without downtime for the VMs)
> >>
> >> Looking forward to any advice!
> >>
> >> Greetings,
> >>
> >> Melanie
> >> --
> >> --
> >>
> >> Heinlein Support GmbH
> >> Linux: Akademie - Support - Hosting
> >>
> >> http://www.heinlein-support.de
> >> Tel: 030 / 40 50 51 - 0
> >> Fax: 030 / 40 50 51 - 19
> >>
> >> Zwangsangaben lt. §35a GmbHG:
> >> HRB 93818 B / Amtsgericht Berlin-Charlottenburg,
> >> Geschäftsführer: Peer Heinlein  -- Sitz: Berlin
> >>
> >>
> >
> >--
> >
> >Andrija Panić
>

Re: How to Reorganise Storage Tags

[Andrija Panic]

Hi Melanie,

when  moving volume to new storage, when you want to change disk offering
(or compute offering for ROOT disk...), ACS doesn't allow that - it lists
only offerings that have same tag as current offering (not good...)

We have inhouse patch, so that you CAN do that, by making sure to list all
offergins that have TAG that matches the TAG of the new destination pool of
the volume (hope Im clear here).

All volumes read tag from their offering - so just either change
disk_offering_id filed for each moved/migrated volume  to point to same
sized offering on new storage - and then normally change it once more via
UI to a new once etc - or manualy change to smaller disk offering (DB edit)
and later via UI/API to correct (same size) disk offering (or bigger if you
want to really resize)

I can try to share a patch in a non-developer, copy/paste way - in case you
want to patch your ACS to support this (as explained at the begining of the
email...)

Hope that helps

Cheers

On Thu, 13 Dec 2018 at 13:50, Melanie Desaive 
wrote:

> Hi all,
>
> we are currently reorganizing our SAN Setup and I would like to
> introduce new storage tags on my existing volumes.
>
> I was naively assuming to simply change the tags or offering by GUI or
> API calls.
>
> Does not seem to work. Only way to change the tags, seems to be by using
> a new disk offering, which is denied, when the tags between old and new
> offering differ. :( Or am I missing something?
>
> I had a look into the cloud database, and the storage tags, seem to be
> only stored in
>
>   disk_offering.tags
> and
>   storage_pool_tags.tag
>
> Would it be a valid option for me to update disk_offering.tags by SQL to
> the desired value or could that break some deeper logic?
>
> Or is there even a better way to change the storage tags for existing
> volumes. (With or without downtime for the VMs)
>
> Looking forward to any advice!
>
> Greetings,
>
> Melanie
> --
> --
>
> Heinlein Support GmbH
> Linux: Akademie - Support - Hosting
>
> http://www.heinlein-support.de
> Tel: 030 / 40 50 51 - 0
> Fax: 030 / 40 50 51 - 19
>
> Zwangsangaben lt. §35a GmbHG:
> HRB 93818 B / Amtsgericht Berlin-Charlottenburg,
> Geschäftsführer: Peer Heinlein  -- Sitz: Berlin
>
>

-- 

Andrija Panić

Re: URGENT: Unable to schedule async job for command com.cloud.vm.VmWorkStart, unexpected exception

[Andrija Panic]

Done it once (8h old DB)...make sure to kill any new VMs that were created
in that period of time (and volumes/templates on the end storage).

You might run into situation that some VMs are i.e. powered on during that
last 8h, so DB state for that VM will not be in sync with reality, but that
should be synced shorty after.

I.e. hope that a very few changes/creations of any resources have happened
during that period - and expect some stuff will need to be manually
synced/fixed/repeated - i.e. user created PF rule on VR, but now you have
restored DB so - that might be a challenge to fix without pissing off users
:)

Fingers crossed!

On Thu, Dec 13, 2018, 14:33 Ugo Vasi  Hi all,
> is there anyone who tried to restore the database a day ago and
> restarted cloudstack manager?
>
> The two situations differ for the status of the VM / VR and I do not
> know what to expect and bearing in mind that the problem could recur.
>
> The statistics don't interest me.
>
>
> Il 13/12/18 10:39, Ugo Vasi ha scritto:
> > We have verified that the problem is not tied to a vm but it also
> > concerns the other VMs that are stopped. Trying to restart them gives
> > the same error message.
> >
> > Restoring the previous day's database which problems might imply?
> >
> >
> > Il 13/12/18 09:01, Ugo Vasi ha scritto:
> >> Hi all,
> >> I'm trying to reboot a vm after the host it ran on crashed and
> >> restarted from the HA system. All the VMs running on the rebooted
> >> host were restarted on other hosts except one.
> >> In the web interface and using cloudmonkey I get this message:
> >>   "Unable to schedule async job for command com.cloud.vm.VmWorkStart,
> >> unexpected exception."
> >>
> >> In the management-server.log file there would seem to be a problem
> >> when creating an element that is duplicated (Duplicate entry ''
> >> for key 'PRIMARY'):
> >>
> >> 2018-12-13 08:44:06,659 DEBUG [c.c.a.ApiServlet]
> >> (qtp1096283470-445:ctx-6c065e06) (logid:87edf8d7) ===START===
> >> 10.80.0.6 -- GET
> >>
> command=startVirtualMachine&response=json&id=dde566b2-ef2c-4f86-a82b-c8286f0c24f7&_=1544687047015
> >> 2018-12-13 08:44:06,665 DEBUG [c.c.a.ApiServer]
> >> (qtp1096283470-445:ctx-6c065e06 ctx-c8b27e4a) (logid:87edf8d7) CIDRs
> >> from which account 'Acct[26e597f2-b5ca-11e8-a619-c8cbb8cb15cd-admin]'
> >> is allowed to perform API calls: 0.0.0.0/0,::/0
> >> 2018-12-13 08:44:06,693 INFO  [o.a.c.f.j.i.AsyncJobMonitor]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:7e0c4dc9) Add
> >> job-1343 into job monitoring
> >> 2018-12-13 08:44:06,698 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >> (qtp1096283470-445:ctx-6c065e06 ctx-c8b27e4a) (logid:87edf8d7) submit
> >> async job-1343, details: AsyncJobVO {id:1343, userId: 2, accountId:
> >> 2, instanceType: VirtualMachine, instanceId: 8, cmd:
> >> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin,
> >> cmdInfo:
> >>
> {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"6170","id":"dde566b2-ef2c-4f86-a82b-c8286f0c24f7","ctxDetails":"{\"interface
>
> >>
> com.cloud.vm.VirtualMachine\":\"dde566b2-ef2c-4f86-a82b-c8286f0c24f7\"}","ctxAccountId":"2","uuid":"dde566b2-ef2c-4f86-a82b-c8286f0c24f7","cmdEventType":"VM.START","_":"1544687047015"},
>
> >> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> >> result: null, initMsid: 220777304233416, completeMsid: null,
> >> lastUpdated: null, lastPolled: null, created: null}
> >> 2018-12-13 08:44:06,700 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Executing
> >> AsyncJobVO {id:1343, userId: 2, accountId: 2, instanceType:
> >> VirtualMachine, instanceId: 8, cmd:
> >> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin,
> >> cmdInfo:
> >>
> {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"6170","id":"dde566b2-ef2c-4f86-a82b-c8286f0c24f7","ctxDetails":"{\"interface
>
> >>
> com.cloud.vm.VirtualMachine\":\"dde566b2-ef2c-4f86-a82b-c8286f0c24f7\"}","ctxAccountId":"2","uuid":"dde566b2-ef2c-4f86-a82b-c8286f0c24f7","cmdEventType":"VM.START","_":"1544687047015"},
>
> >> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> >> result: null, initMsid: 220777304233416, completeMsid: null,
> >> lastUpdated: null, lastPolled: null, created: null}
> >> 2018-12-13 08:44:06,705 DEBUG [c.c.a.ApiServlet]
> >> (qtp1096283470-445:ctx-6c065e06 ctx-c8b27e4a) (logid:87edf8d7)
> >> ===END===  10.80.0.6 -- GET
> >>
> command=startVirtualMachine&response=json&id=dde566b2-ef2c-4f86-a82b-c8286f0c24f7&_=1544687047015
> >> 2018-12-13 08:44:06,745 DEBUG [c.c.n.NetworkModelImpl]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >> (logid:6e9a71c5) Service SecurityGroup is not supported in the
> >> network id=205
> >> 2018-12-13 08:44:06,752 DEBUG [c.c.n.NetworkModelImpl]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >> (logid:6e9a71c5) Service SecurityGroup is not supported in the
> >> network i

Re: new committer: Boris Stoyanov (AKA Bobby)

[Andrija Panic]

Congrats Bobby !

On Thu, 13 Dec 2018 at 10:58, Sateesh Chodapuneedi <
sateesh.chodapune...@accelerite.com> wrote:

> Hearty Congratulations, Bobby!
> Well deserved.
> Bobby has been actively helping with validation of MRs under review. Also
> helping users in users list.
>
> Regards,
> Sateesh
>
>
> -Original Message-
> From: Paul Angus 
> Reply-To: "d...@cloudstack.apache.org" 
> Date: Thursday, 13 December 2018 at 14:59
> To: "users@cloudstack.apache.org" , "
> d...@cloudstack.apache.org" 
> Cc: Boris Stoyanov 
> Subject: new committer: Boris Stoyanov (AKA Bobby)
>
> Hi Everyone,
>
> The Project Management Committee (PMC) for Apache CloudStack
> has invited Boris Stoyanov to become a committer and we are pleased
> to announce that he has accepted.
>
> Please join me in congratulating Bobby!
>
>
> Being a committer enables easier contribution to the
> project since there is no need to go via the patch
> submission process. This should enable better productivity.
> Being a PMC member enables assistance with the management
> and to guide the direction of the project.
>
>
>
> paul.an...@shapeblue.com
> www.shapeblue.com
> Amadeus House, Floral Street, London  WC2E 9DPUK
> @shapeblue
>
>
>
>
>
>

-- 

Andrija Panić

Re: vSAN

[Andrija Panic]

Well, that requires some math I guess, but certainly not less than 10Gbps...

You need to be aware of client vs replication traffic between storage nodes
(hosts, whatever) and do some math...

With single NVMe, being able to push 1GB/s at minimum = 8 Gbps ( 8Gbps per
single NVMe device)...and enough clients, I would guess at least 40 Gbps
network makes sense to leverage couple of NVMe per host/storage node.

Again depends also on client IO requirements as well... Its more an art,
than a science, to make right decision.

Cheers


On Tue, Dec 11, 2018, 23:09 Fariborz Navidan  How much is minimum bandwidth requirement for storage network in which all
> underlying devices are NVMe SSD?
>
> On Wed, Dec 12, 2018 at 1:34 AM Andrija Panic 
> wrote:
>
> > ScaleIO is supposed to be better/more performant...yep.
> >
> > On Tue, 11 Dec 2018 at 23:00, Ivan Kudryavtsev  >
> > wrote:
> >
> > > Fariborz, you also can use ScaleIO exported through iSCSI if you are ok
> > to
> > > go with local storage.
> > >
> > > вт, 11 дек. 2018 г. в 16:51, Fariborz Navidan :
> > >
> > > > Something like VMware vSAN
> > > >
> > > > On Wed, Dec 12, 2018 at 1:15 AM Andrija Panic <
> andrija.pa...@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > CEPH ?
> > > > >
> > > > > On Tue, 11 Dec 2018 at 21:37, Fariborz Navidan <
> > mdvlinqu...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hello folks,
> > > > > > Do you know of any good vSAN software working with CloudStack?
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Andrija Panić
> > > > >
> > > >
> > >
> > >
> > > --
> > > With best regards, Ivan Kudryavtsev
> > > Bitworks LLC
> > > Cell RU: +7-923-414-1515
> > > Cell USA: +1-201-257-1512
> > > WWW: http://bitworks.software/ <http://bw-sw.com/>
> > >
> >
> >
> > --
> >
> > Andrija Panić
> >
>

Re: vSAN

[Andrija Panic]

ScaleIO is supposed to be better/more performant...yep.

On Tue, 11 Dec 2018 at 23:00, Ivan Kudryavtsev 
wrote:

> Fariborz, you also can use ScaleIO exported through iSCSI if you are ok to
> go with local storage.
>
> вт, 11 дек. 2018 г. в 16:51, Fariborz Navidan :
>
> > Something like VMware vSAN
> >
> > On Wed, Dec 12, 2018 at 1:15 AM Andrija Panic 
> > wrote:
> >
> > > CEPH ?
> > >
> > > On Tue, 11 Dec 2018 at 21:37, Fariborz Navidan 
> > > wrote:
> > >
> > > > Hello folks,
> > > > Do you know of any good vSAN software working with CloudStack?
> > > >
> > > > Thanks
> > > >
> > >
> > >
> > > --
> > >
> > > Andrija Panić
> > >
> >
>
>
> --
> With best regards, Ivan Kudryavtsev
> Bitworks LLC
> Cell RU: +7-923-414-1515
> Cell USA: +1-201-257-1512
> WWW: http://bitworks.software/ <http://bw-sw.com/>
>


-- 

Andrija Panić

Re: vSAN

[Andrija Panic]

CEPH ?

On Tue, 11 Dec 2018 at 21:37, Fariborz Navidan 
wrote:

> Hello folks,
> Do you know of any good vSAN software working with CloudStack?
>
> Thanks
>


-- 

Andrija Panić

Re: CPU/Memory Hotplug

[Andrija Panic]

Afaik, no.

On Tue, 11 Dec 2018 at 13:48, Fariborz Navidan 
wrote:

> Hello folks,
>
> Does CloudStack support CPU/Memory Hotplug on KVM?
>
> Kind Regards
>


-- 

Andrija Panić