Hi Andrija,
Bold formatting does not come trough on the dev list. :)
But u might need a bit more info.
At a certain point I see this line
2014-05-30 13:56:23,935 DEBUG [c.c.a.t.Request]
(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Seq 1-609104082: Sending {
Cmd , MgmtId: 161344838950, via: 1(cs1.xxxxx.net), Ver: v1, Flags: 100111,
[{"com.cloud.agent.api.StartCommand":{"vm":{"id":801,"name":"r-801-VM"...
This is where the information is passed on to the agent handles. For XS
this would initiate an agent handler on the management server but for KVM,
if I remember correctly, it passed the command on to the cloudstack agent
service on the hypervisor.
Can you check the cloud service log on the KVM hypervisor executing the
request? it's this server cs1.xxxxx.net and then search top down for
609104082 in the log. See if you can provide the log from the agent
handler thread started by that sequence.
Kind regards,
Joris van Lieshout
Schuberg Philis
Boeingavenue 271
1119 PD Schiphol-Rijk
schubergphilis.com
+31 20-7506672
+31 6-51428188
On 30/05/14 14:08, "Andrija Panic" <andrija.pa...@gmail.com> wrote:
>Hi Joris,
>
>here is the management log: http://pastebin.com/zxnKxFhk
>
>Interesting parts (to me): in bold
>
>2014-05-30 13:56:21,899 DEBUG [o.a.c.s.m.AncientDataMotionStrategy]
>(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) copyAsync inspecting src type
>TEMPLATE copyAsync inspecting dest type VOLUME
>2014-05-30 13:56:21,905 DEBUG [c.c.a.t.Request]
>(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Seq 4-1248669612: Sending {
>Cmd , MgmtId: 161344838950, via: 4(cs2.xxxxx.net), Ver: v1, Flags: 100011,
>[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apach
>e.cloudstack.storage.to.TemplateObjectTO":{"path":"1adc1d2e-56ae-4a0f-b0b4
>-5e351e7cae55","origUrl":"
>http://download.cloud.com/templates/4.3/systemvm64template-2014-01-14-mast
>er-kvm.qcow2.bz2
>","uuid":"1adc1d2e-56ae-4a0f-b0b4-5e351e7cae55","id":414,"format":"QCOW2",
>"accountId":2,"checksum":"85a1bed07bf43cbf022451cb2ecae4ff","
>*hvm":true*
>,"displayText":"systemvm-kvm-4.3","imageDataStore":{"org.apache.cloudstack
>.storage.to.PrimaryDataStoreTO":{"uuid":"5b93422e-1a66-353d-88a8-2203f79b1
>dc6","id":209,"poolType":"RBD","host":"
>cephmon.xxxxx.net","path":"cloudstack","port":6789,"url":"RBD://
>cephmon.xxxxx.net/cloudstack/?ROLE=Primary&STOREUUID=5b93422e-1a66-353d-88
>a8-2203f79b1dc6
>"}},"name":"414-2-ec331e74-5858-3153-91a9-1d706d9c533e","hypervisorType":"
>KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid"
>:"9c440d3b-cba5-4960-b8bf-dca90291cd2b","volumeType":"ROOT","dataStore":{"
>org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"5b93422e-1a6
>6-353d-88a8-2203f79b1dc6","id":209,"poolType":"RBD","host":"
>cephmon.xxxxx.net","path":"cloudstack","port":6789,"url":"RBD://
>cephmon.xxxxx.net/cloudstack/?ROLE=Primary&STOREUUID=5b93422e-1a66-353d-88
>a8-2203f79b1dc6"}},"name":"ROOT-801","size":2621440000,"volumeId":1064,"vm
>Name":"r-801-VM","accountId":11,"format":"RAW","id":1064,"deviceId":0,"hyp
>ervisorType":"KVM"}},"executeInSequence":false,"options":{},"wait":0}}]
>}
>2014-05-30 13:56:23,742 DEBUG [c.c.a.t.Request]
>(AgentManager-Handler-12:null) Seq 4-1248669612: Processing: { Ans: ,
>MgmtId: 161344838950, via: 4, Ver: v1, Flags: 10,
>[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.a
>pache.cloudstack.storage.to.VolumeObjectTO":{"size":2621440000,"path":"9c4
>40d3b-cba5-4960-b8bf-dca90291cd2b","accountId":0,"format":"RAW","id":0}},"
>result":true,"wait":0}}]
>}
>2014-05-30 13:56:23,742 DEBUG [c.c.a.t.Request]
>(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Seq 4-1248669612: Received: {
>Ans: , MgmtId: 161344838950, via: 4, Ver: v1, Flags: 10, { CopyCmdAnswer
>} }
>2014-05-30 13:56:23,773 DEBUG
>[c.c.n.r.VpcVirtualNetworkApplianceManagerImpl]
>(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Removing nic NicProfile[
>*1092-801-null*-46.232.xxx.246-vlan://untagged of type Public from the
>nics
>passed on vm start. The nic will be plugged later
>2014-05-30 13:56:23,773 DEBUG
>[c.c.n.r.VpcVirtualNetworkApplianceManagerImpl]
>(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Removing nic
>NicProfile[1093-801-cd9fd29a-0573-4715-8742-00ecb9f82c9d-10.0.1.1-vlan://4
>4
>of type Guest from the nics passed on vm start. The nic will be plugged
>later
>2014-05-30 13:56:23,773 DEBUG
>[c.c.n.r.VpcVirtualNetworkApplianceManagerImpl]
>(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Removing nic
>NicProfile[1094-801-cd9fd29a-0573-4715-8742-00ecb9f82c9d-10.0.3.1-vlan://4
>3
>of type Guest from the nics passed on vm start. The nic will be plugged
>later
>2014-05-30 13:56:23,773 DEBUG
>[c.c.n.r.VpcVirtualNetworkApplianceManagerImpl]
>(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Removing nic
>NicProfile[1095-801-cd9fd29a-0573-4715-8742-00ecb9f82c9d-10.0.4.1-vlan://3
>004
>of type Guest from the nics passed on vm start. The nic will be plugged
>later
>2014-05-30 13:56:23,773 DEBUG
>[c.c.n.r.VpcVirtualNetworkApplianceManagerImpl]
>(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Removing nic
>NicProfile[1095-801-cd9fd29a-0573-4715-8742-00ecb9f82c9d-10.0.4.1-vlan://3
>004
>of type Guest from the nics passed on vm start
>
>
>Thanks,
>
>
>On 30 May 2014 13:54, Joris van Lieshout <jvanliesh...@schubergphilis.com>
>wrote:
>
>> Hi Andrija,
>>
>> Just the start of the VR should be sufficient.
>>
>> Kind regards,
>> Joris van Lieshout
>>
>> Schuberg Philis
>> Boeingavenue 271
>> 1119 PD Schiphol-Rijk
>> schubergphilis.com
>>
>> +31 20-7506672
>> +31 6-51428188
>>
>>
>>
>>
>> On 30/05/14 13:48, "Andrija Panic" <andrija.pa...@gmail.com> wrote:
>>
>> >Hi Joris,
>> >
>> >just to be sure - you want me to capture the log from the moment I
>>reboot
>> >router - or you want me to stop it, then start capturing log, and
>>start it
>> >(and continue capture untill ethnull errors inside VR) ?
>> >
>> >Thanks,
>> >
>> >
>> >On 30 May 2014 13:39, Joris van Lieshout
>><jvanliesh...@schubergphilis.com
>> >
>> >wrote:
>> >
>> >> Hi Andrija,
>> >>
>> >> Thanks for the answers. In deed your situation is different so
>>PV/HVM is
>> >> not the issue.
>> >>
>> >> When reading back the log output you have provided I noted that the
>>VR
>> >> messages log indicates that it's waiting for ethnull to be up. This
>> >>raises
>> >> the question where null was introduced instead of 1. The ACS
>>management
>> >> log output you send was, what I think, later down the road where ACS
>> >>gives
>> >> up trying to wait for the VR to come up. If you would capture the
>> >> job-executor in the management log from startCommand till the
>>exception,
>> >> do you see anywhere a mention of ethnull? You might need to reed into
>> >>the
>> >> DirectAgent executing the startCommand to find a clue. The thing is
>> >>that I
>> >> only have experience with XS based environment so I cannot point you
>>to
>> >> the exact output to look for. On XS, at least, it is
>> >> "[c.c.h.x.r.CitrixResourceBase] (DirectAgent-351:ctx-4a51bb9e)
>>Created a
>> >> vif e4c362bd-764b-f651-dc9a-1abd5cb33c43 on 1"
>> >>
>> >> Kind regards,
>> >> Joris van Lieshout
>> >>
>> >> Schuberg Philis
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> On 30/05/14 10:48, "Andrija Panic" <andrija.pa...@gmail.com> wrote:
>> >>
>> >> >Hi Deen,
>> >> >no, in DB there is field "vlan_id" with value "untagged" - that
>> >> >"vlan://untagged" is shown from ACS gui, and is used in API call (or
>> >> >better
>> >> >said commands that are seen in management server logs).
>> >> >
>> >> >Best,
>> >> >Andrija
>> >> >
>> >> >
>> >> >On 30 May 2014 10:37, Daan Hoogland <daan.hoogl...@gmail.com> wrote:
>> >> >
>> >> >> Andrija,
>> >> >>
>> >> >> Do not just assign a second net vlan://500 You have one like that
>>and
>> >> >> you don't want conflicting nets using the same vlan. I am
>>wondering
>> >> >> why 'untagged' comes out as 'vlan://untagged'. I think that is the
>> >> >> bug. Did you find the string 'vlan://untagged' in your db?
>> >> >>
>> >> >> On Fri, May 30, 2014 at 10:20 AM, Andrija Panic
>> >> >><andrija.pa...@gmail.com>
>> >> >> wrote:
>> >> >> > Hi Joris,
>> >> >> >
>> >> >> > thank you for taking time to address this issue :)
>> >> >> >
>> >> >> > So...:
>> >> >> >
>> >> >> > - I'm on KVM (stock CentOS 6.2 patched by Inktank for CEPH
>> >>support),
>> >> >>OS
>> >> >> is
>> >> >> > Centos 6.5, libvirt 1.2.3 compiled.
>> >> >> > - ACS 4.3 having problems, ACS 4.2.1 was fine
>> >> >> > - not XS, so I guess no answers for this part :)
>> >> >> > - guest_os_id is 184 = Debian 7 x64
>> >> >> > - SVM = systemvm-kvm-4.3 = os type 184 = Debian 7 x64
>> >> >> >
>> >> >> > This worked previously on 4.2.1 = template was ofcourse
>> >> >>systemvm-kvm-4.2
>> >> >> -
>> >> >> > but that was also Debian 7 x64 type... so this should not be the
>> >> >>issues
>> >> >> > (guest not supported by host...)
>> >> >> >
>> >> >> > The only thing that might be out of "standard" = all SVMs are on
>> >>CEPH
>> >> >>-
>> >> >> > there are official docs on altering database to make some new
>> >>System
>> >> >> > Offering as default for SSVM and CPVM - what I did, I also have
>> >>done
>> >> >>same
>> >> >> > config in DB, to make VR use another System Offering as default
>>-
>> >> >>which
>> >> >> is
>> >> >> > NOT explained in the docs - you could use "Change Offering..."
>> >>button
>> >> >>on
>> >> >> > exiting, shutdown VR to change it per docs...
>> >> >> > But still this worked all fine on 4.2.1...
>> >> >> >
>> >> >> > - regarding /var/cache/cloud/cmdline the content is folowing at
>>the
>> >> >> moment
>> >> >> > root@r-801-VM:~# cat /var/cache/cloud/cmdline
>> >> >> > vpccidr=10.0.0.0/8 domain=cscloud.internal dns1=8.8.8.8 dns2=
>> >> >> template=domP
>> >> >> > name=r-801-VM eth0ip=169.254.0.75 eth0mask=255.255.0.0
>> >>type=vpcrouter
>> >> >> > disable_rp_filter=true
>> >> >> >
>> >> >> > Also please note that only eth1 does not have IP info, eth0
>> >>(control
>> >> >> > 169.xxx) and all other eh2 and up that are used for Tiers get IP
>> >>info
>> >> >> fine.
>> >> >> > I could also manually add IP for eth1 (public NIC) and start
>>ifup
>> >> >>eth1 -
>> >> >> > and it works fine, but adding new IP Port Forwarding etc does
>>not
>> >> >>work...
>> >> >> >
>> >> >> > Daan or somebody said it could be realted to my "Public" network
>> >>(in
>> >> >>the
>> >> >> > Zones, Physical Network, eth1 listing) is NOT tagged
>> >> >>(vlan://untagged)...
>> >> >> > Interestingly the only VR that does work fine is the VR used in
>> >>Shared
>> >> >> > network, but that VR is using IP from Guest IP range (also
>> >>efectively
>> >> >> > public IPs but on vlan 500)
>> >> >> >
>> >> >> > I was instructed to try to change Public IP range from untagged
>>to
>> >> >>vlan
>> >> >> > 500, but I'm not sure how to do this, if there is any way at all
>> >> >>(editing
>> >> >> > "vlan" table and changing to vlan 500 does not work, after
>> >>rebooting
>> >> >>VR
>> >> >> > from ACS gui).
>> >> >> >
>> >> >> > :)
>> >> >> >
>> >> >> > So, not sure what is roughly expected date for 4.4, but right
>>now,
>> >>I'm
>> >> >> > pretty stuck with a big problem of all VPC not operational at
>> >>all...
>> >> >> >
>> >> >> > Thanks,
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > On 30 May 2014 08:27, Joris van Lieshout <
>> >> >> jvanliesh...@schubergphilis.com>
>> >> >> > wrote:
>> >> >> >
>> >> >> >> Hi Andrija,
>> >> >> >>
>> >> >> >> Daan asked me to have a look at this as well. Looking at you
>> >>issue I
>> >> >> >> recall having seen something similar. Back then when upgrading
>> >>4.2.1
>> >> >>to
>> >> >> >> 4.3 I though it had to do with out own custom build svm
>>template.
>> >> >> >> Let me fire off some questions before explaining what the cause
>> >>was
>> >> >>in
>> >> >> our
>> >> >> >> case. :)
>> >> >> >>
>> >> >> >> - what hypervisor (and version) are you using?
>> >> >> >> - if XS, is the new VR a para-virtualised instance (PV) or
>> >>hardware
>> >> >> >> assisted (HVM)? Do a "xe vm-param-list" on the VR uuid and
>>check
>> >>that
>> >> >> >> param PV-args is set and HVM-boot-policy is unset.
>> >> >> >> - what is the OS type of the VR in ACS (guest_os_id in
>>vm_instance
>> >> >>table
>> >> >> >> and match with table guest_os)
>> >> >> >> - what is the OS type of the SVM template?
>> >> >> >>
>> >> >> >> Now for the explaining. :)
>> >> >> >> In our case the OS type of the new template was not supported
>>on
>> >>the
>> >> >> >> XenServer version we are running. Therefore the VR was started
>>by
>> >>XS
>> >> >>as
>> >> >> a
>> >> >> >> HVM guest. System vms on XS rely on the arguments passed to
>>them
>> >>in
>> >> >>the
>> >> >> >> PV-args param (ends up on the guest in /var/cache/cloud/cmdline
>> >> >>which in
>> >> >> >> turn is used by cloud-early-config) in order to work. cmdline
>> >> >>contains
>> >> >> the
>> >> >> >> NIC configuration information.
>> >> >> >> So, long story short, if a VR gets started as a HVM it will not
>> >>get
>> >> >>the
>> >> >> >> information needed to configure it's NICs.
>> >> >> >>
>> >> >> >> Workaround
>> >> >> >> We corrected the os_type_id in the DB (yes I know editing the
>>DB
>> >>is
>> >> >> >> something you usually don't want but there is no other way in
>>this
>> >> >>case)
>> >> >> >> of the existing VR's and of the systemvmtemplate to something
>> >> >>supported
>> >> >> by
>> >> >> >> XenServer.
>> >> >> >>
>> >> >> >> Kind regards,
>> >> >> >> Joris van Lieshout
>> >> >> >>
>> >> >> >> Schuberg Philis
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >> On 29/05/14 12:18, "Andrija Panic" <andrija.pa...@gmail.com>
>> >>wrote:
>> >> >> >>
>> >> >> >> >They are 2 traffic types on 1 physical net (that is both
>>tagged
>> >>vlan
>> >> >> 500,
>> >> >> >> >and untagged packets travel over same KVM bridge, and over
>>eth1
>> >>to
>> >> >> outside
>> >> >> >> >world)...
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >On 29 May 2014 12:04, Daan Hoogland <daan.hoogl...@gmail.com>
>> >> wrote:
>> >> >> >> >
>> >> >> >> >> Are these two traffic types in one physical net? or two
>> >>physical
>> >> >>nets
>> >> >> >> >> on the same interface (seems wrong).
>> >> >> >> >>
>> >> >> >> >> On Thu, May 29, 2014 at 11:35 AM, Jayapal Reddy Uradi
>> >> >> >> >> <jayapalreddy.ur...@citrix.com> wrote:
>> >> >> >> >> > I don't think editing DB table will work.
>> >> >> >> >> >
>> >> >> >> >> > -Jayapal
>> >> >> >> >> > On 29-May-2014, at 2:52 PM, Andrija Panic
>> >> >><andrija.pa...@gmail.com
>> >> >> >
>> >> >> >> >> wrote:
>> >> >> >> >> >
>> >> >> >> >> >> It's like this:
>> >> >> >> >> >>
>> >> >> >> >> >> I have public subnet /24.
>> >> >> >> >> >>
>> >> >> >> >> >> half is dedicated for Guest traffic (vlan 500) and the
>> >>second
>> >> >> half is
>> >> >> >> >> >> dedicated to Public traffic/network (no vlan tags, that
>>is
>> >> >> untagged
>> >> >> >> >> packets)
>> >> >> >> >> >>
>> >> >> >> >> >> Both vlan500 and untagged packets travel over physical
>>eth1
>> >> >> >> >>interface on
>> >> >> >> >> >> hypervisors and can reach Internet.
>> >> >> >> >> >>
>> >> >> >> >> >> Thanks,
>> >> >> >> >> >>
>> >> >> >> >> >>
>> >> >> >> >> >> On 29 May 2014 11:06, Daan Hoogland
>> >><daan.hoogl...@gmail.com>
>> >> >> wrote:
>> >> >> >> >> >>
>> >> >> >> >> >>> On Thu, May 29, 2014 at 10:57 AM, Andrija Panic <
>> >> >> >> >> andrija.pa...@gmail.com>
>> >> >> >> >> >>> wrote:
>> >> >> >> >> >>>> 500
>> >> >> >> >> >>>
>> >> >> >> >> >>>
>> >> >> >> >> >>> is 500 the vlan of your guestnetwork or your physical
>> >>network?
>> >> >> You
>> >> >> >> >> >>> wouldn't want to have two nets with vlan 500!
>> >> >> >> >> >>>
>> >> >> >> >> >>> --
>> >> >> >> >> >>> Daan
>> >> >> >> >> >>>
>> >> >> >> >> >>
>> >> >> >> >> >>
>> >> >> >> >> >>
>> >> >> >> >> >> --
>> >> >> >> >> >>
>> >> >> >> >> >> Andrija Panić
>> >> >> >> >> >> --------------------------------------
>> >> >> >> >> >> http://admintweets.com
>> >> >> >> >> >> --------------------------------------
>> >> >> >> >> >
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> --
>> >> >> >> >> Daan
>> >> >> >> >>
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >--
>> >> >> >> >
>> >> >> >> >Andrija Panić
>> >> >> >> >--------------------------------------
>> >> >> >> > http://admintweets.com
>> >> >> >> >--------------------------------------
>> >> >> >>
>> >> >> >>
>> >> >> >
>> >> >> >
>> >> >> > --
>> >> >> >
>> >> >> > Andrija Panić
>> >> >> > --------------------------------------
>> >> >> > http://admintweets.com
>> >> >> > --------------------------------------
>> >> >>
>> >> >>
>> >> >>
>> >> >> --
>> >> >> Daan
>> >> >>
>> >> >
>> >> >
>> >> >
>> >> >--
>> >> >
>> >> >Andrija Panić
>> >> >--------------------------------------
>> >> > http://admintweets.com
>> >> >--------------------------------------
>> >>
>> >>
>> >
>> >
>> >--
>> >
>> >Andrija Panić
>> >--------------------------------------
>> > http://admintweets.com
>> >--------------------------------------
>>
>>
>
>
>--
>
>Andrija Panić
>--------------------------------------
> http://admintweets.com
>--------------------------------------
