Daan,
Can I track this issue somewhere?
Thank you!
Cristian
-----Original Message-----
From: Daan Hoogland [mailto:daan.hoogl...@gmail.com]
Sent: Thursday, October 12, 2017 9:46 AM
To: users@cloudstack.apache.org
Subject: Re: Upgrade Cloudstack 4.5 to 4.9 - Unable to upgrade the database
Cristian,
I am cross-posting this to dev@coudstack.a.o, sorry for the noise. I have heard
this before but am not sure if it was exactly the same. It sounds like a bug to
me. Can you enter one in jira?
Anybody else? Was this fixed in 4.10 or master yet?
On Wed, Oct 11, 2017 at 11:23 PM, Ciobanu Cristian <cristian.c@istream.today
> wrote:
> Hello,
>
> After I continued the investigation I found that router is not
> allowing the VM to fetch the password (port 8080) not sure what is
> wrong with this version but in order to fix I have added this rule "-A
> INPUT -p tcp -m tcp --dport 8080 -j ACCEPT" on virtual router..
>
> Any idea how to fix this, permanent? In case reinstall the router
> for sure I will have to do this again.
>
>
> Thank you.
> Cristian
>
> -----Original Message-----
> From: Ciobanu Cristian [mailto:cristian.c@istream.today]
> Sent: Wednesday, October 11, 2017 9:45 PM
> To: 'users@cloudstack.apache.org' <users@cloudstack.apache.org>
> Subject: RE: Upgrade Cloudstack 4.5 to 4.9 - Unable to upgrade the
> database
>
> I was able to update the virtual router only after I have restarted
> the iptables-persistend form virtual router side, not sure what is
> wrong with this router version but I don't get the right status if I
> don't restart the iptables from virtual router, I have did multiple tests :
>
> Example :
> - Reboot router
> - getting this error : job-3247/job-3248, cmd: CheckSshCommand)
> (logid:6b3de9fa) Trying to
> > connect to 149.56.xxx.xx ( I see this error in management log )
> - restart iptables from router side
> - works
>
> In case I destroy/restart the router I have to repeat the same
> thing, restart iptables...
>
> I also tested a new deploy with a template used before on this
> setup, looks like the password set for new deploy is not working...
>
>
> Any idea why I have those issue?
>
> Upgraded from 4.5.2 to 4.9.3 ( shapeblue packages )
>
>
> Regards,
> Cristian
>
>
> -----Original Message-----
> From: Daan Hoogland [mailto:daan.hoogl...@gmail.com]
> Sent: Wednesday, October 11, 2017 6:10 PM
> To: users@cloudstack.apache.org
> Subject: Re: Upgrade Cloudstack 4.5 to 4.9 - Unable to upgrade the
> database
>
> that is a tricky one. You can destroy the vm from VmWare and set the
> state to stopped in the DB. It should then start a new one.
> check if r-4-VM/ROOT-4-000001.vmdk has gone from the datastore.
>
> the cause is probably some synchronisation issue. We've seen and
> solved it before but are yet to find the root cause.
>
>
> On Wed, Oct 11, 2017 at 4:23 PM, Ciobanu Cristian
> <cristian.c@istream.today
> >
> wrote:
>
> > Hello,
> >
> > Was my mistake, forgot to add the system template from cloudstack
> > interface before starting the upgrade, now I have other issue, looks
> > like after I have enabled one of the zone and click on upgrade
> > virtual router I still see the state "starting" if I check the
> > router from vSphere side is up.
> >
> >
> > This is what is see :
> >
> > tting
> > catalina.out:INFO [c.c.h.v.r.VmwareResource]
> > (DirectAgent-381:ctx-5f16ff50 ns53xxxx.ip-149-56-xx.net,
> > job-3247/job-3248, cmd: StartCommand) (logid:6b3de9fa) Chose disk
> > controller for vol ROOT -> osdefault, based on root disk controller
> > settings at global configuration setting.
> > catalina.out:INFO [c.c.h.v.m.DatastoreMO]
> > (DirectAgent-381:ctx-5f16ff50 ns53xxxx.ip-149-56-xx.net,
> > job-3247/job-3248, cmd: StartCommand)
> > (logid:6b3de9fa) Folder r-4-VM exists on datastore catalina.out:INFO
> > [c.c.h.v.m.DatastoreMO] (DirectAgent-381:ctx-5f16ff50
> > ns53xxxx.ip-149-56-xx.net, job-3247/job-3248, cmd: StartCommand)
> > (logid:6b3de9fa) Search file ROOT-4-000001.vmdk on [datastore1]
> > catalina.out:INFO [c.c.h.v.m.DatastoreMO]
> > (DirectAgent-381:ctx-5f16ff50 ns53xxxx.ip-149-56-xx.net,
> > job-3247/job-3248, cmd: StartCommand)
> > (logid:6b3de9fa) File [datastore1] ROOT-4-000001.vmdk exists on
> > datastore catalina.out:INFO [c.c.h.v.m.DatastoreMO]
> > (DirectAgent-381:ctx-5f16ff50 ns53xxxx.ip-149-56-xx.net,
> > job-3247/job-3248, cmd: StartCommand)
> > (logid:6b3de9fa) Search file ROOT-4-000001-flat.vmdk on [datastore1]
> > catalina.out:INFO [c.c.h.v.m.DatastoreMO]
> > (DirectAgent-381:ctx-5f16ff50 ns53xxxx.ip-149-56-xx.net,
> > job-3247/job-3248, cmd: StartCommand)
> > (logid:6b3de9fa) File [datastore1] ROOT-4-000001-flat.vmdk exists on
> > datastore catalina.out:INFO [c.c.s.r.VmwareStorageLayoutHelper]
> > (DirectAgent-381:ctx-5f16ff50 ns53xxxx.ip-149-56-xx.net,
> > job-3247/job-3248, cmd: StartCommand) (logid:6b3de9fa) sync
> > [datastore1] ROOT-4-000001-flat.vmdk->[datastore1]
> > r-4-VM/ROOT-4-000001-flat.vmdk catalina.out:INFO
> > [c.c.h.v.m.DatastoreMO] (DirectAgent-381:ctx-5f16ff50
> > ns53xxxx.ip-149-56-xx.net, job-3247/job-3248, cmd: StartCommand)
> > (logid:6b3de9fa) Search file ROOT-4-000001-delta.vmdk on
> > [datastore1] catalina.out:INFO [c.c.h.v.m.DatastoreMO]
> > (DirectAgent-381:ctx-5f16ff50 ns53xxxx.ip-149-56-xx.net,
> > job-3247/job-3248, cmd: StartCommand)
> > (logid:6b3de9fa) File [datastore1] ROOT-4-000001-delta.vmdk does not
> > exist on datastore catalina.out:INFO [c.c.h.v.m.DatastoreMO]
> > (DirectAgent-381:ctx-5f16ff50 ns53xxxx.ip-149-56-xx.net,
> > job-3247/job-3248, cmd: StartCommand)
> > (logid:6b3de9fa) Search file ROOT-4-000001.vmdk on [datastore1]
> > catalina.out:INFO [c.c.h.v.m.DatastoreMO]
> > (DirectAgent-381:ctx-5f16ff50 ns53xxxx.ip-149-56-xx.net,
> > job-3247/job-3248, cmd: StartCommand)
> > (logid:6b3de9fa) File [datastore1] ROOT-4-000001.vmdk exists on
> > datastore catalina.out:INFO [c.c.s.r.VmwareStorageLayoutHelper]
> > (DirectAgent-381:ctx-5f16ff50 ns53xxxx.ip-149-56-xx.net,
> > job-3247/job-3248, cmd: StartCommand) (logid:6b3de9fa) sync
> > [datastore1] ROOT-4-000001.vmdk->[datastore1]
> > r-4-VM/ROOT-4-000001.vmdk catalina.out:INFO [c.c.h.v.m.DatastoreMO]
> > (DirectAgent-381:ctx-5f16ff50 ns53xxxx.ip-149-56-xx.net,
> > job-3247/job-3248, cmd: StartCommand)
> > (logid:6b3de9fa) Search file ROOT-4-000001.vmdk on [datastore1]
> > r-4-VM catalina.out:INFO [c.c.h.v.m.DatastoreMO]
> > (DirectAgent-381:ctx-5f16ff50 ns53xxxx.ip-149-56-xx.net,
> > job-3247/job-3248, cmd: StartCommand)
> > (logid:6b3de9fa) File [datastore1] r-4-VM/ROOT-4-000001.vmdk exists
> > on datastore catalina.out:INFO [c.c.u.n.n.p.NiciraNvpApiVersion]
> > (DirectAgent-381:ctx-5f16ff50 ns53xxxx.ip-149-56-xx.net,
> > job-3247/job-3248, cmd: StartCommand) (logid:6b3de9fa) NSX API VERSION:
> > NOT PRESENT
> > catalina.out:INFO [c.c.h.v.r.VmwareResource]
> > (DirectAgent-381:ctx-5f16ff50 ns53xxxx.ip-149-56-xx.net,
> > job-3247/job-3248, cmd: StartCommand) (logid:6b3de9fa) Prepare NIC
> > device based on NicTO: {"deviceId":0,"networkRateMbps":200,"
> > defaultNic":true,"pxeDisable":true,"nicUuid":"fc405400-6839-
> > 4eea-9b30-e5e14b375322","uuid":"41f42309-8eaa-425a-a4c5-
> > c8b5a88f92a2","ip":"149.56.xxx.xx","netmask":"255.255.
> > 255.224","gateway":"149.56.164.62","mac":"06:76:62:00:00:
> > 4b","dns1":"172.20.0.3","dns2":"8.8.8.8","broadcastType":"
> > Native","type":"Guest","broadcastUri":"vlan://untagged","
> > isSecurityGroupEnabled":false,"name":"vSwitch1"}
> > catalina.out:INFO [c.c.h.v.r.VmwareResource]
> > (DirectAgent-381:ctx-5f16ff50 ns53xxxx.ip-149-56-xx.net,
> > job-3247/job-3248, cmd: StartCommand) (logid:6b3de9fa) Prepare
> > network on vmwaresvs vSwitch1 with name prefix: cloud.guest
> > catalina.out:INFO [c.c.h.v.m.HypervisorHostHelper]
> > (DirectAgent-381:ctx-5f16ff50 ns53xxxx.ip-149-56-xx.net,
> > job-3247/job-3248, cmd: StartCommand) (logid:6b3de9fa) Network
> > cloud.guest.untagged.200.1-vSwitch1 is ready on vSwitch vSwitch1
> > catalina.out:INFO [c.c.h.v.m.HypervisorHostHelper]
> > (DirectAgent-381:ctx-5f16ff50 ns53xxxx.ip-149-56-xx.net,
> > job-3247/job-3248, cmd: StartCommand) (logid:6b3de9fa) Network
> > cloud.guest.untagged.200.1-vSwitch1 is ready on vSwitch vSwitch1
> > catalina.out:INFO [c.c.h.v.m.HypervisorHostHelper]
> > (DirectAgent-381:ctx-5f16ff50 ns53xxxx.ip-149-56-xx.net,
> > job-3247/job-3248, cmd: StartCommand) (logid:6b3de9fa) Network
> > cloud.guest.untagged.200.1-vSwitch1 is ready on vSwitch vSwitch1
> > catalina.out:INFO [c.c.h.v.r.VmwareResource]
> > (DirectAgent-381:ctx-5f16ff50 ns53xxxx.ip-149-56-xx.net,
> > job-3247/job-3248, cmd: StartCommand) (logid:6b3de9fa) Preparing NIC
> > device on network cloud.guest.untagged.200.1-vSwitch1
> > catalina.out:INFO [c.c.h.v.r.VmwareResource]
> > (DirectAgent-381:ctx-5f16ff50 ns53xxxx.ip-149-56-xx.net,
> > job-3247/job-3248, cmd: StartCommand) (logid:6b3de9fa) Prepare NIC
> > device based on NicTO: {"deviceId":1,"networkRateMbps":-1,"
> > defaultNic":false,"pxeDisable":true,"nicUuid":"fe3f7db4-
> > 9ff5-4b63-883b-e08bb6f7568b","uuid":"8ef0b547-f569-4987-
> > ac80-9ebf72f44fdd","ip":"0.0.0.0","netmask":"0.0.0.0","
> > gateway":"0.0.0.0","mac":"02:00:5c:9c:00:0a","
> broadcastType":"LinkLocal","
> > type":"Control","isSecurityGroupEnabled":false,"name":"vSwitch1"}
> > catalina.out:INFO [c.c.h.v.r.VmwareResource]
> > (DirectAgent-381:ctx-5f16ff50 ns53xxxx.ip-149-56-xx.net,
> > job-3247/job-3248, cmd: StartCommand) (logid:6b3de9fa) Prepare
> > network on vmwaresvs vSwitch1 with name prefix: cloud.private
> > catalina.out:WARN [c.c.h.v.r.VmwareResource]
> > (DirectAgent-381:ctx-5f16ff50 ns53xxxx.ip-149-56-xx.net,
> > job-3247/job-3248, cmd: StartCommand) (logid:6b3de9fa) Unrecognized
> > broadcast type in VmwareResource, type: LinkLocal. Use vlan info
> > from
> > labeling: untagged
> > catalina.out:INFO [c.c.h.v.m.HypervisorHostHelper]
> > (DirectAgent-381:ctx-5f16ff50 ns53xxxx.ip-149-56-xx.net,
> > job-3247/job-3248, cmd: StartCommand) (logid:6b3de9fa) Network
> > cloud.private.untagged.0.1-vSwitch1 is ready on vSwitch vSwitch1
> > catalina.out:INFO [c.c.h.v.r.VmwareResource]
> > (DirectAgent-381:ctx-5f16ff50 ns53xxxx.ip-149-56-xx.net,
> > job-3247/job-3248, cmd: StartCommand) (logid:6b3de9fa) Preparing NIC
> > device on network cloud.private.untagged.0.1-vSwitch1
> > catalina.out:INFO [c.c.h.v.r.VmwareResource]
> > (DirectAgent-381:ctx-5f16ff50 ns53xxxx.ip-149-56-xx.net,
> > job-3247/job-3248, cmd: StartCommand) (logid:6b3de9fa) Configure VNC
> > port for VM r-4-VM, port: 5939, host: ns53xxxx.ip-149-56-xx.net
> > catalina.out:INFO [c.c.h.v.r.VmwareResource]
> > (DirectAgent-381:ctx-5f16ff50 ns53xxxx.ip-149-56-xx.net,
> > job-3247/job-3248, cmd: StartCommand) (logid:6b3de9fa) Found
> > existing disk info from volume path: ROOT-4-000001 catalina.out:INFO
> > [c.c.h.v.r.VmwareResource]
> > (DirectAgent-381:ctx-5f16ff50 ns53xxxx.ip-149-56-xx.net,
> > job-3247/job-3248, cmd: CheckSshCommand) (logid:6b3de9fa) Trying to
> > connect to 149.56.xxx.xx catalina.out:WARN
> > [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-8baef180)
> > (logid:c6f5e75e) Task (job-3247) has been pending for 129 seconds
> > catalina.out:INFO [c.c.h.v.r.VmwareResource]
> > (DirectAgent-381:ctx-5f16ff50 ns53xxxx.ip-149-56-xx.net,
> > job-3247/job-3248, cmd: CheckSshCommand) (logid:6b3de9fa) Could not
> > connect to 149.56.xxx.xx due to java.net.ConnectException:
> > Connection timed out catalina.out:INFO [c.c.h.v.r.VmwareResource]
> > (DirectAgent-381:ctx-5f16ff50 ns53xxxx.ip-149-56-xx.net,
> > job-3247/job-3248, cmd: CheckSshCommand) (logid:6b3de9fa) Trying to
> > connect to 149.56.xxx.xx catalina.out:WARN
> > [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-3026cb48)
> > (logid:9ea6ca9e) Task (job-3247) has been pending for 189 seconds
> > catalina.out:INFO [c.c.h.v.r.VmwareResource]
> > (DirectAgent-381:ctx-5f16ff50 ns53xxxx.ip-149-56-xx.net,
> > job-3247/job-3248, cmd: CheckSshCommand) (logid:6b3de9fa) Could not
> > connect to 149.56.xxx.xx due to java.net.ConnectException:
> > Connection timed out catalina.out:INFO [c.c.h.v.r.VmwareResource]
> > (DirectAgent-381:ctx-5f16ff50 ns53xxxx.ip-149-56-xx.net,
> > job-3247/job-3248, cmd: CheckSshCommand) (logid:6b3de9fa) Trying to
> > connect to 149.56.xxx.xx
> >
> >
> > Any suggestion? Please.
> >
> > Regards,
> > Cristian
> >
> > -----Original Message-----
> > From: Daan Hoogland [mailto:daan.hoogl...@gmail.com]
> > Sent: Wednesday, October 11, 2017 3:41 PM
> > To: users@cloudstack.apache.org
> > Subject: Re: Upgrade Cloudstack 4.5 to 4.9 - Unable to upgrade the
> > database
> >
> > Christian,
> >
> > The chances are I have not done the exact same upgrade as you are
> > attempting. the generic upgrade of the database (assuming the right
> > template and other things) is quite straight forward. You will need
> > to give us at least the stacktrace and any related information. from
> > the log above it, if this is a reproducible error.
> >
> > On Wed, Oct 11, 2017 at 2:34 PM, Ciobanu Cristian
> > <cristian.c@istream.today
> > >
> > wrote:
> >
> > > I have restored back to 4.5.2, but if I upgrade again using the
> > > steps provide in documentation I will end with the same error..
> > > did you tested this? With tha configuration (acs 4.5.2, vmware,
> > > local
> storage) ?
> > >
> > > Regards,
> > > Cristian
> > >
> > > -----Original Message-----
> > > From: Daan Hoogland [mailto:daan.hoogl...@gmail.com]
> > > Sent: Wednesday, October 11, 2017 3:29 PM
> > > To: users@cloudstack.apache.org
> > > Subject: Re: Upgrade Cloudstack 4.5 to 4.9 - Unable to upgrade the
> > > database
> > >
> > > Hi the management server on starting up should take care of the
> upgrade.
> > > if you have a clustered MS make sure to start only one. In your
> > > case after the failure you should probably restore the latest
> > > backup before
> > continuing.
> > >
> > > On Wed, Oct 11, 2017 at 2:26 PM, Ciobanu Cristian
> > > <cristian.c@istream.today
> > > >
> > > wrote:
> > >
> > > > Hello,
> > > >
> > > >
> > > >
> > > > In order to upgrade cloudstack to 4.9 I have followed this
> > > > doecumentation
> > > > :
> > > > http://docs.cloudstack.apache.org/projects/cloudstack-
> > > > release-notes/en/4.9.2
> > > > .0/upgrade/upgrade-4.5.html and ended with this :
> > > >
> > > >
> > > >
> > > > management-server.log:2017-10-11 08:05:35,612 WARN
> > > > [o.a.c.s.m.c.ResourceApplicationContext] (main:null) (logid:)
> > > > Exception encountered during context initialization - cancelling
> > > > refresh
> > > attempt:
> > > > org.springframework.context.ApplicationContextException: Failed
> > > > to start bean 'cloudStackLifeCycle'; nested exception is
> > > > com.cloud.utils.exception.CloudRuntimeException: Unable to
> > > > upgrade the database
> > > >
> > > >
> > > >
> > > > Is there a valid upgrade procedure or script for : Cloudstack
> > > > 4.5.2 to CloudStack 4.9 ( VMware with Local Storage ) ?
> > > >
> > > >
> > > >
> > > > (cloudstack was installed from shapeblue repo)
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > Thank you.
> > > >
> > > > Cristian
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > > --
> > > Daan
> > >
> > >
> >
> >
> > --
> > Daan
> >
> >
>
>
> --
> Daan
>
>
--
Daan
