yes, correct, eth1 is present, and can be started by static IP configuration...
On 30 May 2014 17:25, Marcus <shadow...@gmail.com> wrote: > Let me make sure I understand... the 'Plug' of the nic works fine, as it > seems you do have an eth1 and can manually assign the IP to get it to work? > If that's the case then there's probably not an issue in BridgeVifDriver or > the XML. It is definitely in fetching/matching the eth device here > > vpc_ipassoc.sh 169.254.0.52 -A -l 46.232.xxx.246 -c ethnull -g > 46.232.xxx.1 -m 24 -n 46.232.xxx.0 > > > On Fri, May 30, 2014 at 8:33 AM, Andrija Panic <andrija.pa...@gmail.com> > wrote: > > > Nope, started, did check, it is reported now as Debian 5 VM, but still > > doesn't work... > > Rebooted VPC (destroyed VR, new one created...) > > > > $ virsh dumpxml r-812-VM > > ... > > <description>Debian GNU/Linux 5(64-bit)</description> > > ... > > <os> > > <type arch='x86_64' machine='rhel6.5.0'>hvm</type> > > <boot dev='cdrom'/> > > <boot dev='hd'/> > > </os> > > > > :( > > > > > > > > On 30 May 2014 16:24, Andrija Panic <andrija.pa...@gmail.com> wrote: > > > > > I confirm, the highest is Debian 5 64bit > > > > > > Per docs > > > > > > http://docs.cloudstack.apache.org/projects/cloudstack-release-notes/en/4.3/rnotes.html#upgrade-from-4-2-x-to-4-3 > > > , you should use Debian 7.0 64bit as OS type for system-kvm-4.3 > > template... > > > (same for xen and vmware templatest) > > > > > > Will change now DB to point to debian 5, and let you know. > > > > > > This is guest_os_id: 184 | 2 | NULL | > > > 986a0e98-39d6-11e3-8f93-0025904e4412 | Debian GNU/Linux 7(64-bit) > > > > > > > > > On 30 May 2014 15:45, Joris van Lieshout < > > jvanliesh...@schubergphilis.com> > > > wrote: > > > > > >> Andrija, > > >> > > >> The thing is I don't know who the os matching on KVM works. There must > > be > > >> a way to list supported os types. > > >> > > >> I also did some queuing on the guest_os_hypervisor table (ACS 4.3) > and I > > >> don't see Debian 7 for KVM listed. > > >> > > >> select * from guest_os join guest_os_hypervisor on > > >> guest_os.id=guest_os_hypervisor.guest_os_id where > > >> guest_os_hypervisor.hypervisor_type='KVM' and > > >> guest_os_hypervisor.guest_os_name like '%Debian%'; > > >> > > >> What was the guest_os_id you where using? Could you try id 72 (Debian > 5 > > >> 64-bit)? Adjust both os_type_id in vm_instance and vm_template (where > > >> type='SYSTEM' and hypervisor_type='KVM'). > > >> > > >> Kind regards, > > >> Joris van Lieshout > > >> > > >> Schuberg Philis > > >> > > >> > > >> > > >> > > >> > > >> On 30/05/14 15:33, "Andrija Panic" <andrija.pa...@gmail.com> wrote: > > >> > > >> >Joris, > > >> > > > >> >do you have recommendation on how in particular to try ? I'm not sure > > how > > >> >to fix that, except playing with editing systemvm-4.3 template to > > define > > >> >it > > >> >as another OS type... ? > > >> > > > >> >Thanks again, > > >> >Andrija > > >> > > > >> > > > >> >On 30 May 2014 15:30, Joris van Lieshout < > > >> jvanliesh...@schubergphilis.com> > > >> >wrote: > > >> > > > >> >> I've read back a bit in the code and if you look at > > >> BridgeVifDriver.java > > >> >> (this is where the log message with the nic profile is generated) > you > > >> >>can > > >> >> see that the nic information might be off already once ACS hits the > > >> >> LibvirtVMDef.InterfaceDef plug function. This leads be to believer > > that > > >> >> the HVM/PV OS mismatch issue might still be related. Try fixing > that > > >> >> first. At least it will allow us to exclude this from the list. > > >> >> > > >> >> > > >> >> Kind regards, > > >> >> Joris van Lieshout > > >> >> > > >> >> Schuberg Philis > > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > > >> >> On 30/05/14 15:26, "Andrija Panic" <andrija.pa...@gmail.com> > wrote: > > >> >> > > >> >> >OK, thanks Joris. > > >> >> > > > >> >> >I will try playing with OS version option, on the systemvm-kvm-4.3 > > >> >> >template... > > >> >> > > > >> >> >Let me know if I can help with anything more. > > >> >> > > > >> >> >Thanks. > > >> >> >Andrija > > >> >> > > > >> >> > > > >> >> >On 30 May 2014 15:19, Joris van Lieshout > > >> >><jvanliesh...@schubergphilis.com > > >> >> > > > >> >> >wrote: > > >> >> > > > >> >> >> Hi Andrija, > > >> >> >> > > >> >> >> That does sound familiar and in the start xml of KVM you can see > > >> >>"<type > > >> >> >> arch='x86_64' machine='pc'>hvm</type>". I don't know KVM+ACS > well > > >> >>enough > > >> >> >> to judge if this is the cause but I thing focusing on getting > the > > VR > > >> >> >> started as PV guest might be worth trying. On the other hand I > do > > >> see > > >> >> >> patchviasocket.pl being executed successfully... > > >> >> >> > > >> >> >> The other thing I see is, and now we're getting into java code, > is > > >> >>this: > > >> >> >> > > >> >> >> 2014-05-30 14:41:01,386{GMT} DEBUG > [kvm.resource.BridgeVifDriver] > > >> >> >> (agentRequest-Handler-3:) > > >> >> >>nic=[Nic:Public-46.232.xxx.246-vlan://untagged] > > >> >> >> 2014-05-30 14:41:01,502{GMT} DEBUG [cloud.agent.Agent] > > >> >> >> (agentRequest-Handler-3:) Processing command: > > >> >> >> com.cloud.agent.api.routing.IpAssocVpcCommand > > >> >> >> 2014-05-30 14:41:01,506{GMT} DEBUG > > >> >> >> [resource.virtualnetwork.VirtualRoutingResource] > > >> >> >>(agentRequest-Handler-3:) > > >> >> >> Executing: > > >> >> >> > /usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh > > >> >> >> vpc_ipassoc.sh 169.254.0.52 -A -l 46.232.xxx.246 -c ethnull -g > > >> >> >> 46.232.xxx.1 -m 24 -n 46.232.xxx.0 > > >> >> >> > > >> >> >> My suspicion is that somewhere in the translation from the nic > > >> >>profile > > >> >> >>to > > >> >> >> the actual route_proxy.sh command ACS failes to find the nic id > > and > > >> >> >> returns null. > > >> >> >> > > >> >> >> Let me dig a bit deeper and see what I can find but this is > where > > we > > >> >> >>might > > >> >> >> need some help from someone with knowledge of this pice of the > > code. > > >> >>:) > > >> >> >> > > >> >> >> > > >> >> >> Kind regards, > > >> >> >> Joris van Lieshout > > >> >> >> > > >> >> >> Schuberg Philis > > >> >> >> Boeingavenue 271 > > >> >> >> 1119 PD Schiphol-Rijk > > >> >> >> schubergphilis.com > > >> >> >> > > >> >> >> +31 20-7506672 > > >> >> >> +31 6-51428188 > > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> On 30/05/14 14:49, "Andrija Panic" <andrija.pa...@gmail.com> > > wrote: > > >> >> >> > > >> >> >> >Hi Joris, > > >> >> >> > > > >> >> >> >I have turned on DEBUG loging in agent.log on cs1.xxx/net host: > > >> >> >> > > > >> >> >> >So, management logs again: http://pastebin.com/F6BRf7Y9 > > >> >> >> >Agent logs on cs1.xxx: http://pastebin.com/BJauKbaC > > >> >> >> > > > >> >> >> >Not playing smart, but there is some error: > > >> >> >> >[kvm.resource.KVMGuestOsMapper] > > >> >> >> >(agentRequest-Handler-3:) Can't find the mapping of guest os: > > >> Debian > > >> >> >> >GNU/Linux 7(64-bit) > > >> >> >> > > > >> >> >> >Best, > > >> >> >> >Andrija > > >> >> >> > > > >> >> >> > > > >> >> >> >On 30 May 2014 14:26, Joris van Lieshout > > >> >> >><jvanliesh...@schubergphilis.com > > >> >> >> > > > >> >> >> >wrote: > > >> >> >> > > > >> >> >> >> Hi Andrija, > > >> >> >> >> > > >> >> >> >> Bold formatting does not come trough on the dev list. :) > > >> >> >> >> But u might need a bit more info. > > >> >> >> >> > > >> >> >> >> At a certain point I see this line > > >> >> >> >> > > >> >> >> >> 2014-05-30 13:56:23,935 DEBUG [c.c.a.t.Request] > > >> >> >> >> (Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Seq 1-609104082: > > >> >>Sending > > >> >> >> { > > >> >> >> >> Cmd , MgmtId: 161344838950, via: 1(cs1.xxxxx.net), Ver: v1, > > >> Flags: > > >> >> >> >>100111, > > >> >> >> >> > > >> >> >> > > >> >> > > >> > > >> > > > >>>>>>[{"com.cloud.agent.api.StartCommand":{"vm":{"id":801,"name":"r-801-VM > > >> >>>>>>". > > >> >> >>>>.. > > >> >> >> >> > > >> >> >> >> > > >> >> >> >> This is where the information is passed on to the agent > > handles. > > >> >>For > > >> >> >>XS > > >> >> >> >> this would initiate an agent handler on the management server > > but > > >> >>for > > >> >> >> >>KVM, > > >> >> >> >> if I remember correctly, it passed the command on to the > > >> >>cloudstack > > >> >> >> >>agent > > >> >> >> >> service on the hypervisor. > > >> >> >> >> > > >> >> >> >> Can you check the cloud service log on the KVM hypervisor > > >> >>executing > > >> >> >>the > > >> >> >> >> request? it's this server cs1.xxxxx.net and then search top > > down > > >> >>for > > >> >> >> >> 609104082 in the log. See if you can provide the log from the > > >> >>agent > > >> >> >> >> handler thread started by that sequence. > > >> >> >> >> > > >> >> >> >> Kind regards, > > >> >> >> >> Joris van Lieshout > > >> >> >> >> > > >> >> >> >> Schuberg Philis > > >> >> >> >> Boeingavenue 271 > > >> >> >> >> 1119 PD Schiphol-Rijk > > >> >> >> >> schubergphilis.com > > >> >> >> >> > > >> >> >> >> +31 20-7506672 > > >> >> >> >> +31 6-51428188 > > >> >> >> >> > > >> >> >> >> > > >> >> >> >> > > >> >> >> >> > > >> >> >> >> On 30/05/14 14:08, "Andrija Panic" <andrija.pa...@gmail.com> > > >> >>wrote: > > >> >> >> >> > > >> >> >> >> >Hi Joris, > > >> >> >> >> > > > >> >> >> >> >here is the management log: http://pastebin.com/zxnKxFhk > > >> >> >> >> > > > >> >> >> >> >Interesting parts (to me): in bold > > >> >> >> >> > > > >> >> >> >> >2014-05-30 13:56:21,899 DEBUG > > >> >>[o.a.c.s.m.AncientDataMotionStrategy] > > >> >> >> >> >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) copyAsync > > inspecting > > >> >>src > > >> >> >> >>type > > >> >> >> >> >TEMPLATE copyAsync inspecting dest type VOLUME > > >> >> >> >> >2014-05-30 13:56:21,905 DEBUG [c.c.a.t.Request] > > >> >> >> >> >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Seq > 4-1248669612: > > >> >> >>Sending > > >> >> >> >>{ > > >> >> >> >> >Cmd , MgmtId: 161344838950, via: 4(cs2.xxxxx.net), Ver: v1, > > >> >>Flags: > > >> >> >> >> 100011, > > >> >> >> >> > > >> >> >> > > >> >> > > >> > > >> > > > >>>>>>>[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org > > >> >>>>>>>.a > > >> >> >>>>>pa > > >> >> >> >>>ch > > >> >> >> >> > > >> >> >> > > >> >> > > >> > > >> > > > >>>>>>>e.cloudstack.storage.to.TemplateObjectTO":{"path":"1adc1d2e-56ae-4a0 > > >> >>>>>>>f- > > >> >> >>>>>b0 > > >> >> >> >>>b4 > > >> >> >> >> >-5e351e7cae55","origUrl":" > > >> >> >> >> > > > >> >> >> >> > > >> >> >> >> > > >> >> >> > > >> >> >> > > >> >> > > >> >> > > >> > > > http://download.cloud.com/templates/4.3/systemvm64template-2014-01-14-mas > > >> >> >> >>t > > >> >> >> >> >er-kvm.qcow2.bz2 > > >> >> >> >> > > >> >> >> > > >> >> > > >> > > >> > > > >>>>>>>","uuid":"1adc1d2e-56ae-4a0f-b0b4-5e351e7cae55","id":414,"format":"Q > > >> >>>>>>>CO > > >> >> >>>>>W2 > > >> >> >> >>>", > > >> >> >> >> > >"accountId":2,"checksum":"85a1bed07bf43cbf022451cb2ecae4ff"," > > >> >> >> >> >*hvm":true* > > >> >> >> >> > > >> >> >> > > >> >> > > >> > > >> > > > >>>>>>>,"displayText":"systemvm-kvm-4.3","imageDataStore":{"org.apache.clou > > >> >>>>>>>ds > > >> >> >>>>>ta > > >> >> >> >>>ck > > >> >> >> >> > > >> >> >> > > >> >> > > >> > > >> > > > >>>>>>>.storage.to.PrimaryDataStoreTO":{"uuid":"5b93422e-1a66-353d-88a8-220 > > >> >>>>>>>3f > > >> >> >>>>>79 > > >> >> >> >>>b1 > > >> >> >> >> >dc6","id":209,"poolType":"RBD","host":" > > >> >> >> >> >cephmon.xxxxx.net > > >> ","path":"cloudstack","port":6789,"url":"RBD:// > > >> >> >> >> > > > >> >> >> >> > > >> >> >> >> > > >> >> >> > > >> >> >> > > >> >> > > >> >> > > >> > > > cephmon.xxxxx.net/cloudstack/?ROLE=Primary&STOREUUID=5b93422e-1a66-353d-8 > > >> >> >> >>8 > > >> >> >> >> >a8-2203f79b1dc6 > > >> >> >> >> > > >> >> >> > > >> >> > > >> > > >> > > > >>>>>>>"}},"name":"414-2-ec331e74-5858-3153-91a9-1d706d9c533e","hypervisorT > > >> >>>>>>>yp > > >> >> >>>>>e" > > >> >> >> >>>:" > > >> >> >> >> > > >> >> >> > > >> >> > > >> > > >> > > > >>>>>>>KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{ > > >> >>>>>>>"u > > >> >> >>>>>ui > > >> >> >> >>>d" > > >> >> >> >> > > >> >> >> > > >> >> > > >> > > >> > > > >>>>>>>:"9c440d3b-cba5-4960-b8bf-dca90291cd2b","volumeType":"ROOT","dataSto > > >> >>>>>>>re > > >> >> >>>>>": > > >> >> >> >>>{" > > >> >> >> >> > > >> >> >> > > >> >> > > >> > > >> > > > >>>>>>>org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"5b9342 > > >> >>>>>>>2e > > >> >> >>>>>-1 > > >> >> >> >>>a6 > > >> >> >> >> >6-353d-88a8-2203f79b1dc6","id":209,"poolType":"RBD","host":" > > >> >> >> >> >cephmon.xxxxx.net > > >> ","path":"cloudstack","port":6789,"url":"RBD:// > > >> >> >> >> > > > >> >> >> >> > > >> >> >> >> > > >> >> >> > > >> >> >> > > >> >> > > >> >> > > >> > > > cephmon.xxxxx.net/cloudstack/?ROLE=Primary&STOREUUID=5b93422e-1a66-353d-8 > > >> >> >> >>8 > > >> >> >> >> > > >> >> >> > > >> >> > > >> > > >> > > > >>>>>>>a8-2203f79b1dc6"}},"name":"ROOT-801","size":2621440000,"volumeId":10 > > >> >>>>>>>64 > > >> >> >>>>>," > > >> >> >> >>>vm > > >> >> >> >> > > >> >> >> > > >> >> > > >> > > >> > > > >>>>>>>Name":"r-801-VM","accountId":11,"format":"RAW","id":1064,"deviceId": > > >> >>>>>>>0, > > >> >> >>>>>"h > > >> >> >> >>>yp > > >> >> >> >> > > >> >> > > >> > > >> > > > >>>>>ervisorType":"KVM"}},"executeInSequence":false,"options":{},"wait":0}} > > >> >>>>>] > > >> >> >> >> >} > > >> >> >> >> >2014-05-30 13:56:23,742 DEBUG [c.c.a.t.Request] > > >> >> >> >> >(AgentManager-Handler-12:null) Seq 4-1248669612: Processing: > > { > > >> >> >>Ans: , > > >> >> >> >> >MgmtId: 161344838950, via: 4, Ver: v1, Flags: 10, > > >> >> >> >> > > >> >> >> > > >> >> > > >> > > >> > > > >>>>>>>[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{ > > >> >>>>>>>"o > > >> >> >>>>>rg > > >> >> >> >>>.a > > >> >> >> >> > > >> >> >> > > >> >> > > >> > > >> > > > >>>>>>>pache.cloudstack.storage.to.VolumeObjectTO":{"size":2621440000,"path > > >> >>>>>>>": > > >> >> >>>>>"9 > > >> >> >> >>>c4 > > >> >> >> >> > > >> >> >> > > >> >> > > >> > > >> > > > >>>>>>>40d3b-cba5-4960-b8bf-dca90291cd2b","accountId":0,"format":"RAW","id" > > >> >>>>>>>:0 > > >> >> >>>>>}} > > >> >> >> >>>," > > >> >> >> >> >result":true,"wait":0}}] > > >> >> >> >> >} > > >> >> >> >> >2014-05-30 13:56:23,742 DEBUG [c.c.a.t.Request] > > >> >> >> >> >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Seq > 4-1248669612: > > >> >> >> >>Received: { > > >> >> >> >> >Ans: , MgmtId: 161344838950, via: 4, Ver: v1, Flags: 10, { > > >> >> >> >>CopyCmdAnswer > > >> >> >> >> >} } > > >> >> >> >> >2014-05-30 13:56:23,773 DEBUG > > >> >> >> >> >[c.c.n.r.VpcVirtualNetworkApplianceManagerImpl] > > >> >> >> >> >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Removing nic > > >> >>NicProfile[ > > >> >> >> >> >*1092-801-null*-46.232.xxx.246-vlan://untagged of type > Public > > >> >>from > > >> >> >>the > > >> >> >> >> >nics > > >> >> >> >> >passed on vm start. The nic will be plugged later > > >> >> >> >> >2014-05-30 13:56:23,773 DEBUG > > >> >> >> >> >[c.c.n.r.VpcVirtualNetworkApplianceManagerImpl] > > >> >> >> >> >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Removing nic > > >> >> >> >> > > >> >> >> > > >> >> > > >> > > >> > > > >>>>>>>NicProfile[1093-801-cd9fd29a-0573-4715-8742-00ecb9f82c9d-10.0.1.1-vl > > >> >>>>>>>an > > >> >> >>>>>:/ > > >> >> >> >>>/4 > > >> >> >> >> >4 > > >> >> >> >> >of type Guest from the nics passed on vm start. The nic will > > be > > >> >> >>plugged > > >> >> >> >> >later > > >> >> >> >> >2014-05-30 13:56:23,773 DEBUG > > >> >> >> >> >[c.c.n.r.VpcVirtualNetworkApplianceManagerImpl] > > >> >> >> >> >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Removing nic > > >> >> >> >> > > >> >> >> > > >> >> > > >> > > >> > > > >>>>>>>NicProfile[1094-801-cd9fd29a-0573-4715-8742-00ecb9f82c9d-10.0.3.1-vl > > >> >>>>>>>an > > >> >> >>>>>:/ > > >> >> >> >>>/4 > > >> >> >> >> >3 > > >> >> >> >> >of type Guest from the nics passed on vm start. The nic will > > be > > >> >> >>plugged > > >> >> >> >> >later > > >> >> >> >> >2014-05-30 13:56:23,773 DEBUG > > >> >> >> >> >[c.c.n.r.VpcVirtualNetworkApplianceManagerImpl] > > >> >> >> >> >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Removing nic > > >> >> >> >> > > >> >> >> > > >> >> > > >> > > >> > > > >>>>>>>NicProfile[1095-801-cd9fd29a-0573-4715-8742-00ecb9f82c9d-10.0.4.1-vl > > >> >>>>>>>an > > >> >> >>>>>:/ > > >> >> >> >>>/3 > > >> >> >> >> >004 > > >> >> >> >> >of type Guest from the nics passed on vm start. The nic will > > be > > >> >> >>plugged > > >> >> >> >> >later > > >> >> >> >> >2014-05-30 13:56:23,773 DEBUG > > >> >> >> >> >[c.c.n.r.VpcVirtualNetworkApplianceManagerImpl] > > >> >> >> >> >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Removing nic > > >> >> >> >> > > >> >> >> > > >> >> > > >> > > >> > > > >>>>>>>NicProfile[1095-801-cd9fd29a-0573-4715-8742-00ecb9f82c9d-10.0.4.1-vl > > >> >>>>>>>an > > >> >> >>>>>:/ > > >> >> >> >>>/3 > > >> >> >> >> >004 > > >> >> >> >> >of type Guest from the nics passed on vm start > > >> >> >> >> > > > >> >> >> >> > > > >> >> >> >> >Thanks, > > >> >> >> >> > > > >> >> >> >> > > > >> >> >> >> >On 30 May 2014 13:54, Joris van Lieshout > > >> >> >> >><jvanliesh...@schubergphilis.com > > >> >> >> >> > > > >> >> >> >> >wrote: > > >> >> >> >> > > > >> >> >> >> >> Hi Andrija, > > >> >> >> >> >> > > >> >> >> >> >> Just the start of the VR should be sufficient. > > >> >> >> >> >> > > >> >> >> >> >> Kind regards, > > >> >> >> >> >> Joris van Lieshout > > >> >> >> >> >> > > >> >> >> >> >> Schuberg Philis > > >> >> >> >> >> Boeingavenue 271 > > >> >> >> >> >> 1119 PD Schiphol-Rijk > > >> >> >> >> >> schubergphilis.com > > >> >> >> >> >> > > >> >> >> >> >> +31 20-7506672 > > >> >> >> >> >> +31 6-51428188 > > >> >> >> >> >> > > >> >> >> >> >> > > >> >> >> >> >> > > >> >> >> >> >> > > >> >> >> >> >> On 30/05/14 13:48, "Andrija Panic" < > andrija.pa...@gmail.com > > > > > >> >> >>wrote: > > >> >> >> >> >> > > >> >> >> >> >> >Hi Joris, > > >> >> >> >> >> > > > >> >> >> >> >> >just to be sure - you want me to capture the log from the > > >> >>moment > > >> >> >>I > > >> >> >> >> >>reboot > > >> >> >> >> >> >router - or you want me to stop it, then start capturing > > log, > > >> >>and > > >> >> >> >> >>start it > > >> >> >> >> >> >(and continue capture untill ethnull errors inside VR) ? > > >> >> >> >> >> > > > >> >> >> >> >> >Thanks, > > >> >> >> >> >> > > > >> >> >> >> >> > > > >> >> >> >> >> >On 30 May 2014 13:39, Joris van Lieshout > > >> >> >> >> >><jvanliesh...@schubergphilis.com > > >> >> >> >> >> > > > >> >> >> >> >> >wrote: > > >> >> >> >> >> > > > >> >> >> >> >> >> Hi Andrija, > > >> >> >> >> >> >> > > >> >> >> >> >> >> Thanks for the answers. In deed your situation is > > different > > >> >>so > > >> >> >> >> >>PV/HVM is > > >> >> >> >> >> >> not the issue. > > >> >> >> >> >> >> > > >> >> >> >> >> >> When reading back the log output you have provided I > > noted > > >> >>that > > >> >> >> >>the > > >> >> >> >> >>VR > > >> >> >> >> >> >> messages log indicates that it's waiting for ethnull to > > be > > >> >>up. > > >> >> >> >>This > > >> >> >> >> >> >>raises > > >> >> >> >> >> >> the question where null was introduced instead of 1. > The > > >> ACS > > >> >> >> >> >>management > > >> >> >> >> >> >> log output you send was, what I think, later down the > > road > > >> >> >>where > > >> >> >> >>ACS > > >> >> >> >> >> >>gives > > >> >> >> >> >> >> up trying to wait for the VR to come up. If you would > > >> >>capture > > >> >> >>the > > >> >> >> >> >> >> job-executor in the management log from startCommand > till > > >> >>the > > >> >> >> >> >>exception, > > >> >> >> >> >> >> do you see anywhere a mention of ethnull? You might > need > > to > > >> >> >>reed > > >> >> >> >>into > > >> >> >> >> >> >>the > > >> >> >> >> >> >> DirectAgent executing the startCommand to find a clue. > > The > > >> >> >>thing > > >> >> >> >>is > > >> >> >> >> >> >>that I > > >> >> >> >> >> >> only have experience with XS based environment so I > > cannot > > >> >> >>point > > >> >> >> >>you > > >> >> >> >> >>to > > >> >> >> >> >> >> the exact output to look for. On XS, at least, it is > > >> >> >> >> >> >> "[c.c.h.x.r.CitrixResourceBase] > > >> >>(DirectAgent-351:ctx-4a51bb9e) > > >> >> >> >> >>Created a > > >> >> >> >> >> >> vif e4c362bd-764b-f651-dc9a-1abd5cb33c43 on 1" > > >> >> >> >> >> >> > > >> >> >> >> >> >> Kind regards, > > >> >> >> >> >> >> Joris van Lieshout > > >> >> >> >> >> >> > > >> >> >> >> >> >> Schuberg Philis > > >> >> >> >> >> >> > > >> >> >> >> >> >> > > >> >> >> >> >> >> > > >> >> >> >> >> >> > > >> >> >> >> >> >> > > >> >> >> >> >> >> On 30/05/14 10:48, "Andrija Panic" < > > >> andrija.pa...@gmail.com> > > >> >> >> >>wrote: > > >> >> >> >> >> >> > > >> >> >> >> >> >> >Hi Deen, > > >> >> >> >> >> >> >no, in DB there is field "vlan_id" with value > > "untagged" - > > >> >> >>that > > >> >> >> >> >> >> >"vlan://untagged" is shown from ACS gui, and is used > in > > >> API > > >> >> >>call > > >> >> >> >>(or > > >> >> >> >> >> >> >better > > >> >> >> >> >> >> >said commands that are seen in management server > logs). > > >> >> >> >> >> >> > > > >> >> >> >> >> >> >Best, > > >> >> >> >> >> >> >Andrija > > >> >> >> >> >> >> > > > >> >> >> >> >> >> > > > >> >> >> >> >> >> >On 30 May 2014 10:37, Daan Hoogland > > >> >><daan.hoogl...@gmail.com> > > >> >> >> >> wrote: > > >> >> >> >> >> >> > > > >> >> >> >> >> >> >> Andrija, > > >> >> >> >> >> >> >> > > >> >> >> >> >> >> >> Do not just assign a second net vlan://500 You have > > one > > >> >>like > > >> >> >> >>that > > >> >> >> >> >>and > > >> >> >> >> >> >> >> you don't want conflicting nets using the same > vlan. I > > >> am > > >> >> >> >> >>wondering > > >> >> >> >> >> >> >> why 'untagged' comes out as 'vlan://untagged'. I > think > > >> >>that > > >> >> >>is > > >> >> >> >>the > > >> >> >> >> >> >> >> bug. Did you find the string 'vlan://untagged' in > your > > >> >>db? > > >> >> >> >> >> >> >> > > >> >> >> >> >> >> >> On Fri, May 30, 2014 at 10:20 AM, Andrija Panic > > >> >> >> >> >> >> >><andrija.pa...@gmail.com> > > >> >> >> >> >> >> >> wrote: > > >> >> >> >> >> >> >> > Hi Joris, > > >> >> >> >> >> >> >> > > > >> >> >> >> >> >> >> > thank you for taking time to address this issue :) > > >> >> >> >> >> >> >> > > > >> >> >> >> >> >> >> > So...: > > >> >> >> >> >> >> >> > > > >> >> >> >> >> >> >> > - I'm on KVM (stock CentOS 6.2 patched by Inktank > > for > > >> >>CEPH > > >> >> >> >> >> >>support), > > >> >> >> >> >> >> >>OS > > >> >> >> >> >> >> >> is > > >> >> >> >> >> >> >> > Centos 6.5, libvirt 1.2.3 compiled. > > >> >> >> >> >> >> >> > - ACS 4.3 having problems, ACS 4.2.1 was fine > > >> >> >> >> >> >> >> > - not XS, so I guess no answers for this part :) > > >> >> >> >> >> >> >> > - guest_os_id is 184 = Debian 7 x64 > > >> >> >> >> >> >> >> > - SVM = systemvm-kvm-4.3 = os type 184 = Debian 7 > > x64 > > >> >> >> >> >> >> >> > > > >> >> >> >> >> >> >> > This worked previously on 4.2.1 = template was > > >> ofcourse > > >> >> >> >> >> >> >>systemvm-kvm-4.2 > > >> >> >> >> >> >> >> - > > >> >> >> >> >> >> >> > but that was also Debian 7 x64 type... so this > > should > > >> >>not > > >> >> >>be > > >> >> >> >>the > > >> >> >> >> >> >> >>issues > > >> >> >> >> >> >> >> > (guest not supported by host...) > > >> >> >> >> >> >> >> > > > >> >> >> >> >> >> >> > The only thing that might be out of "standard" = > all > > >> >>SVMs > > >> >> >> >>are on > > >> >> >> >> >> >>CEPH > > >> >> >> >> >> >> >>- > > >> >> >> >> >> >> >> > there are official docs on altering database to > make > > >> >>some > > >> >> >>new > > >> >> >> >> >> >>System > > >> >> >> >> >> >> >> > Offering as default for SSVM and CPVM - what I > did, > > I > > >> >>also > > >> >> >> >>have > > >> >> >> >> >> >>done > > >> >> >> >> >> >> >>same > > >> >> >> >> >> >> >> > config in DB, to make VR use another System > Offering > > >> as > > >> >> >> >>default > > >> >> >> >> >>- > > >> >> >> >> >> >> >>which > > >> >> >> >> >> >> >> is > > >> >> >> >> >> >> >> > NOT explained in the docs - you could use "Change > > >> >> >> >>Offering..." > > >> >> >> >> >> >>button > > >> >> >> >> >> >> >>on > > >> >> >> >> >> >> >> > exiting, shutdown VR to change it per docs... > > >> >> >> >> >> >> >> > But still this worked all fine on 4.2.1... > > >> >> >> >> >> >> >> > > > >> >> >> >> >> >> >> > - regarding /var/cache/cloud/cmdline the content > is > > >> >> >>folowing > > >> >> >> >>at > > >> >> >> >> >>the > > >> >> >> >> >> >> >> moment > > >> >> >> >> >> >> >> > root@r-801-VM:~# cat /var/cache/cloud/cmdline > > >> >> >> >> >> >> >> > vpccidr=10.0.0.0/8 domain=cscloud.internal > > >> dns1=8.8.8.8 > > >> >> >> dns2= > > >> >> >> >> >> >> >> template=domP > > >> >> >> >> >> >> >> > name=r-801-VM eth0ip=169.254.0.75 > > eth0mask=255.255.0.0 > > >> >> >> >> >> >>type=vpcrouter > > >> >> >> >> >> >> >> > disable_rp_filter=true > > >> >> >> >> >> >> >> > > > >> >> >> >> >> >> >> > Also please note that only eth1 does not have IP > > info, > > >> >> >>eth0 > > >> >> >> >> >> >>(control > > >> >> >> >> >> >> >> > 169.xxx) and all other eh2 and up that are used > for > > >> >>Tiers > > >> >> >> >>get IP > > >> >> >> >> >> >>info > > >> >> >> >> >> >> >> fine. > > >> >> >> >> >> >> >> > I could also manually add IP for eth1 (public NIC) > > and > > >> >> >>start > > >> >> >> >> >>ifup > > >> >> >> >> >> >> >>eth1 - > > >> >> >> >> >> >> >> > and it works fine, but adding new IP Port > Forwarding > > >> >>etc > > >> >> >>does > > >> >> >> >> >>not > > >> >> >> >> >> >> >>work... > > >> >> >> >> >> >> >> > > > >> >> >> >> >> >> >> > Daan or somebody said it could be realted to my > > >> >>"Public" > > >> >> >> >>network > > >> >> >> >> >> >>(in > > >> >> >> >> >> >> >>the > > >> >> >> >> >> >> >> > Zones, Physical Network, eth1 listing) is NOT > tagged > > >> >> >> >> >> >> >>(vlan://untagged)... > > >> >> >> >> >> >> >> > Interestingly the only VR that does work fine is > the > > >> VR > > >> >> >>used > > >> >> >> >>in > > >> >> >> >> >> >>Shared > > >> >> >> >> >> >> >> > network, but that VR is using IP from Guest IP > range > > >> >>(also > > >> >> >> >> >> >>efectively > > >> >> >> >> >> >> >> > public IPs but on vlan 500) > > >> >> >> >> >> >> >> > > > >> >> >> >> >> >> >> > I was instructed to try to change Public IP range > > from > > >> >> >> >>untagged > > >> >> >> >> >>to > > >> >> >> >> >> >> >>vlan > > >> >> >> >> >> >> >> > 500, but I'm not sure how to do this, if there is > > any > > >> >>way > > >> >> >>at > > >> >> >> >>all > > >> >> >> >> >> >> >>(editing > > >> >> >> >> >> >> >> > "vlan" table and changing to vlan 500 does not > work, > > >> >>after > > >> >> >> >> >> >>rebooting > > >> >> >> >> >> >> >>VR > > >> >> >> >> >> >> >> > from ACS gui). > > >> >> >> >> >> >> >> > > > >> >> >> >> >> >> >> > :) > > >> >> >> >> >> >> >> > > > >> >> >> >> >> >> >> > So, not sure what is roughly expected date for > 4.4, > > >> but > > >> >> >>right > > >> >> >> >> >>now, > > >> >> >> >> >> >>I'm > > >> >> >> >> >> >> >> > pretty stuck with a big problem of all VPC not > > >> >> >>operational at > > >> >> >> >> >> >>all... > > >> >> >> >> >> >> >> > > > >> >> >> >> >> >> >> > Thanks, > > >> >> >> >> >> >> >> > > > >> >> >> >> >> >> >> > > > >> >> >> >> >> >> >> > > > >> >> >> >> >> >> >> > > > >> >> >> >> >> >> >> > > > >> >> >> >> >> >> >> > > > >> >> >> >> >> >> >> > > > >> >> >> >> >> >> >> > > > >> >> >> >> >> >> >> > > > >> >> >> >> >> >> >> > > > >> >> >> >> >> >> >> > > > >> >> >> >> >> >> >> > > > >> >> >> >> >> >> >> > > > >> >> >> >> >> >> >> > On 30 May 2014 08:27, Joris van Lieshout < > > >> >> >> >> >> >> >> jvanliesh...@schubergphilis.com> > > >> >> >> >> >> >> >> > wrote: > > >> >> >> >> >> >> >> > > > >> >> >> >> >> >> >> >> Hi Andrija, > > >> >> >> >> >> >> >> >> > > >> >> >> >> >> >> >> >> Daan asked me to have a look at this as well. > > Looking > > >> >>at > > >> >> >>you > > >> >> >> >> >> >>issue I > > >> >> >> >> >> >> >> >> recall having seen something similar. Back then > > when > > >> >> >> >>upgrading > > >> >> >> >> >> >>4.2.1 > > >> >> >> >> >> >> >>to > > >> >> >> >> >> >> >> >> 4.3 I though it had to do with out own custom > build > > >> >>svm > > >> >> >> >> >>template. > > >> >> >> >> >> >> >> >> Let me fire off some questions before explaining > > what > > >> >>the > > >> >> >> >>cause > > >> >> >> >> >> >>was > > >> >> >> >> >> >> >>in > > >> >> >> >> >> >> >> our > > >> >> >> >> >> >> >> >> case. :) > > >> >> >> >> >> >> >> >> > > >> >> >> >> >> >> >> >> - what hypervisor (and version) are you using? > > >> >> >> >> >> >> >> >> - if XS, is the new VR a para-virtualised > instance > > >> >>(PV) > > >> >> >>or > > >> >> >> >> >> >>hardware > > >> >> >> >> >> >> >> >> assisted (HVM)? Do a "xe vm-param-list" on the VR > > >> uuid > > >> >> >>and > > >> >> >> >> >>check > > >> >> >> >> >> >>that > > >> >> >> >> >> >> >> >> param PV-args is set and HVM-boot-policy is > unset. > > >> >> >> >> >> >> >> >> - what is the OS type of the VR in ACS > (guest_os_id > > >> in > > >> >> >> >> >>vm_instance > > >> >> >> >> >> >> >>table > > >> >> >> >> >> >> >> >> and match with table guest_os) > > >> >> >> >> >> >> >> >> - what is the OS type of the SVM template? > > >> >> >> >> >> >> >> >> > > >> >> >> >> >> >> >> >> Now for the explaining. :) > > >> >> >> >> >> >> >> >> In our case the OS type of the new template was > not > > >> >> >> >>supported > > >> >> >> >> >>on > > >> >> >> >> >> >>the > > >> >> >> >> >> >> >> >> XenServer version we are running. Therefore the > VR > > >> was > > >> >> >> >>started > > >> >> >> >> >>by > > >> >> >> >> >> >>XS > > >> >> >> >> >> >> >>as > > >> >> >> >> >> >> >> a > > >> >> >> >> >> >> >> >> HVM guest. System vms on XS rely on the arguments > > >> >>passed > > >> >> >>to > > >> >> >> >> >>them > > >> >> >> >> >> >>in > > >> >> >> >> >> >> >>the > > >> >> >> >> >> >> >> >> PV-args param (ends up on the guest in > > >> >> >> >>/var/cache/cloud/cmdline > > >> >> >> >> >> >> >>which in > > >> >> >> >> >> >> >> >> turn is used by cloud-early-config) in order to > > work. > > >> >> >> >>cmdline > > >> >> >> >> >> >> >>contains > > >> >> >> >> >> >> >> the > > >> >> >> >> >> >> >> >> NIC configuration information. > > >> >> >> >> >> >> >> >> So, long story short, if a VR gets started as a > HVM > > >> it > > >> >> >>will > > >> >> >> >>not > > >> >> >> >> >> >>get > > >> >> >> >> >> >> >>the > > >> >> >> >> >> >> >> >> information needed to configure it's NICs. > > >> >> >> >> >> >> >> >> > > >> >> >> >> >> >> >> >> Workaround > > >> >> >> >> >> >> >> >> We corrected the os_type_id in the DB (yes I know > > >> >>editing > > >> >> >> >>the > > >> >> >> >> >>DB > > >> >> >> >> >> >>is > > >> >> >> >> >> >> >> >> something you usually don't want but there is no > > >> other > > >> >> >>way > > >> >> >> >>in > > >> >> >> >> >>this > > >> >> >> >> >> >> >>case) > > >> >> >> >> >> >> >> >> of the existing VR's and of the systemvmtemplate > to > > >> >> >> >>something > > >> >> >> >> >> >> >>supported > > >> >> >> >> >> >> >> by > > >> >> >> >> >> >> >> >> XenServer. > > >> >> >> >> >> >> >> >> > > >> >> >> >> >> >> >> >> Kind regards, > > >> >> >> >> >> >> >> >> Joris van Lieshout > > >> >> >> >> >> >> >> >> > > >> >> >> >> >> >> >> >> Schuberg Philis > > >> >> >> >> >> >> >> >> > > >> >> >> >> >> >> >> >> > > >> >> >> >> >> >> >> >> > > >> >> >> >> >> >> >> >> > > >> >> >> >> >> >> >> >> On 29/05/14 12:18, "Andrija Panic" > > >> >> >><andrija.pa...@gmail.com > > >> >> >> > > > >> >> >> >> >> >>wrote: > > >> >> >> >> >> >> >> >> > > >> >> >> >> >> >> >> >> >They are 2 traffic types on 1 physical net (that > > is > > >> >>both > > >> >> >> >> >>tagged > > >> >> >> >> >> >>vlan > > >> >> >> >> >> >> >> 500, > > >> >> >> >> >> >> >> >> >and untagged packets travel over same KVM > bridge, > > >> and > > >> >> >>over > > >> >> >> >> >>eth1 > > >> >> >> >> >> >>to > > >> >> >> >> >> >> >> outside > > >> >> >> >> >> >> >> >> >world)... > > >> >> >> >> >> >> >> >> > > > >> >> >> >> >> >> >> >> > > > >> >> >> >> >> >> >> >> >On 29 May 2014 12:04, Daan Hoogland > > >> >> >> >><daan.hoogl...@gmail.com> > > >> >> >> >> >> >> wrote: > > >> >> >> >> >> >> >> >> > > > >> >> >> >> >> >> >> >> >> Are these two traffic types in one physical > net? > > >> or > > >> >> >>two > > >> >> >> >> >> >>physical > > >> >> >> >> >> >> >>nets > > >> >> >> >> >> >> >> >> >> on the same interface (seems wrong). > > >> >> >> >> >> >> >> >> >> > > >> >> >> >> >> >> >> >> >> On Thu, May 29, 2014 at 11:35 AM, Jayapal > Reddy > > >> >>Uradi > > >> >> >> >> >> >> >> >> >> <jayapalreddy.ur...@citrix.com> wrote: > > >> >> >> >> >> >> >> >> >> > I don't think editing DB table will work. > > >> >> >> >> >> >> >> >> >> > > > >> >> >> >> >> >> >> >> >> > -Jayapal > > >> >> >> >> >> >> >> >> >> > On 29-May-2014, at 2:52 PM, Andrija Panic > > >> >> >> >> >> >> >><andrija.pa...@gmail.com > > >> >> >> >> >> >> >> > > > >> >> >> >> >> >> >> >> >> wrote: > > >> >> >> >> >> >> >> >> >> > > > >> >> >> >> >> >> >> >> >> >> It's like this: > > >> >> >> >> >> >> >> >> >> >> > > >> >> >> >> >> >> >> >> >> >> I have public subnet /24. > > >> >> >> >> >> >> >> >> >> >> > > >> >> >> >> >> >> >> >> >> >> half is dedicated for Guest traffic (vlan > > 500) > > >> >>and > > >> >> >>the > > >> >> >> >> >> >>second > > >> >> >> >> >> >> >> half is > > >> >> >> >> >> >> >> >> >> >> dedicated to Public traffic/network (no > vlan > > >> >>tags, > > >> >> >> >>that > > >> >> >> >> >>is > > >> >> >> >> >> >> >> untagged > > >> >> >> >> >> >> >> >> >> packets) > > >> >> >> >> >> >> >> >> >> >> > > >> >> >> >> >> >> >> >> >> >> Both vlan500 and untagged packets travel > over > > >> >> >>physical > > >> >> >> >> >>eth1 > > >> >> >> >> >> >> >> >> >>interface on > > >> >> >> >> >> >> >> >> >> >> hypervisors and can reach Internet. > > >> >> >> >> >> >> >> >> >> >> > > >> >> >> >> >> >> >> >> >> >> Thanks, > > >> >> >> >> >> >> >> >> >> >> > > >> >> >> >> >> >> >> >> >> >> > > >> >> >> >> >> >> >> >> >> >> On 29 May 2014 11:06, Daan Hoogland > > >> >> >> >> >> >><daan.hoogl...@gmail.com> > > >> >> >> >> >> >> >> wrote: > > >> >> >> >> >> >> >> >> >> >> > > >> >> >> >> >> >> >> >> >> >>> On Thu, May 29, 2014 at 10:57 AM, Andrija > > >> >>Panic < > > >> >> >> >> >> >> >> >> >> andrija.pa...@gmail.com> > > >> >> >> >> >> >> >> >> >> >>> wrote: > > >> >> >> >> >> >> >> >> >> >>>> 500 > > >> >> >> >> >> >> >> >> >> >>> > > >> >> >> >> >> >> >> >> >> >>> > > >> >> >> >> >> >> >> >> >> >>> is 500 the vlan of your guestnetwork or > your > > >> >> >>physical > > >> >> >> >> >> >>network? > > >> >> >> >> >> >> >> You > > >> >> >> >> >> >> >> >> >> >>> wouldn't want to have two nets with vlan > > 500! > > >> >> >> >> >> >> >> >> >> >>> > > >> >> >> >> >> >> >> >> >> >>> -- > > >> >> >> >> >> >> >> >> >> >>> Daan > > >> >> >> >> >> >> >> >> >> >>> > > >> >> >> >> >> >> >> >> >> >> > > >> >> >> >> >> >> >> >> >> >> > > >> >> >> >> >> >> >> >> >> >> > > >> >> >> >> >> >> >> >> >> >> -- > > >> >> >> >> >> >> >> >> >> >> > > >> >> >> >> >> >> >> >> >> >> Andrija Panić > > >> >> >> >> >> >> >> >> >> >> -------------------------------------- > > >> >> >> >> >> >> >> >> >> >> http://admintweets.com > > >> >> >> >> >> >> >> >> >> >> -------------------------------------- > > >> >> >> >> >> >> >> >> >> > > > >> >> >> >> >> >> >> >> >> > > >> >> >> >> >> >> >> >> >> > > >> >> >> >> >> >> >> >> >> > > >> >> >> >> >> >> >> >> >> -- > > >> >> >> >> >> >> >> >> >> Daan > > >> >> >> >> >> >> >> >> >> > > >> >> >> >> >> >> >> >> > > > >> >> >> >> >> >> >> >> > > > >> >> >> >> >> >> >> >> > > > >> >> >> >> >> >> >> >> >-- > > >> >> >> >> >> >> >> >> > > > >> >> >> >> >> >> >> >> >Andrija Panić > > >> >> >> >> >> >> >> >> >-------------------------------------- > > >> >> >> >> >> >> >> >> > http://admintweets.com > > >> >> >> >> >> >> >> >> >-------------------------------------- > > >> >> >> >> >> >> >> >> > > >> >> >> >> >> >> >> >> > > >> >> >> >> >> >> >> > > > >> >> >> >> >> >> >> > > > >> >> >> >> >> >> >> > -- > > >> >> >> >> >> >> >> > > > >> >> >> >> >> >> >> > Andrija Panić > > >> >> >> >> >> >> >> > -------------------------------------- > > >> >> >> >> >> >> >> > http://admintweets.com > > >> >> >> >> >> >> >> > -------------------------------------- > > >> >> >> >> >> >> >> > > >> >> >> >> >> >> >> > > >> >> >> >> >> >> >> > > >> >> >> >> >> >> >> -- > > >> >> >> >> >> >> >> Daan > > >> >> >> >> >> >> >> > > >> >> >> >> >> >> > > > >> >> >> >> >> >> > > > >> >> >> >> >> >> > > > >> >> >> >> >> >> >-- > > >> >> >> >> >> >> > > > >> >> >> >> >> >> >Andrija Panić > > >> >> >> >> >> >> >-------------------------------------- > > >> >> >> >> >> >> > http://admintweets.com > > >> >> >> >> >> >> >-------------------------------------- > > >> >> >> >> >> >> > > >> >> >> >> >> >> > > >> >> >> >> >> > > > >> >> >> >> >> > > > >> >> >> >> >> >-- > > >> >> >> >> >> > > > >> >> >> >> >> >Andrija Panić > > >> >> >> >> >> >-------------------------------------- > > >> >> >> >> >> > http://admintweets.com > > >> >> >> >> >> >-------------------------------------- > > >> >> >> >> >> > > >> >> >> >> >> > > >> >> >> >> > > > >> >> >> >> > > > >> >> >> >> >-- > > >> >> >> >> > > > >> >> >> >> >Andrija Panić > > >> >> >> >> >-------------------------------------- > > >> >> >> >> > http://admintweets.com > > >> >> >> >> >-------------------------------------- > > >> >> >> >> > > >> >> >> >> > > >> >> >> > > > >> >> >> > > > >> >> >> >-- > > >> >> >> > > > >> >> >> >Andrija Panić > > >> >> >> >-------------------------------------- > > >> >> >> > http://admintweets.com > > >> >> >> >-------------------------------------- > > >> >> >> > > >> >> >> > > >> >> > > > >> >> > > > >> >> >-- > > >> >> > > > >> >> >Andrija Panić > > >> >> >-------------------------------------- > > >> >> > http://admintweets.com > > >> >> >-------------------------------------- > > >> >> > > >> >> > > >> > > > >> > > > >> >-- > > >> > > > >> >Andrija Panić > > >> >-------------------------------------- > > >> > http://admintweets.com > > >> >-------------------------------------- > > >> > > >> > > > > > > > > > -- > > > > > > Andrija Panić > > > -------------------------------------- > > > http://admintweets.com > > > -------------------------------------- > > > > > > > > > > > -- > > > > Andrija Panić > > -------------------------------------- > > http://admintweets.com > > -------------------------------------- > > > -- Andrija Panić -------------------------------------- http://admintweets.com --------------------------------------
