I believe I've found the issue. In 4.3, some changes were made to BroadcastDomainType, to standardize Broadcast URIs to prepend vlan://. The issue is that your IpAssocVpcCommand doesn't use this new format for the broadcastUri it passes, so it fails to map the plugged device into the broadcastUriToNicNum map, resulting in ethnull.
On Fri, May 30, 2014 at 9:25 AM, Marcus <shadow...@gmail.com> wrote: > Let me make sure I understand... the 'Plug' of the nic works fine, as it > seems you do have an eth1 and can manually assign the IP to get it to work? > If that's the case then there's probably not an issue in BridgeVifDriver or > the XML. It is definitely in fetching/matching the eth device here > > > vpc_ipassoc.sh 169.254.0.52 -A -l 46.232.xxx.246 -c ethnull -g > 46.232.xxx.1 -m 24 -n 46.232.xxx.0 > > > On Fri, May 30, 2014 at 8:33 AM, Andrija Panic <andrija.pa...@gmail.com> > wrote: > >> Nope, started, did check, it is reported now as Debian 5 VM, but still >> doesn't work... >> Rebooted VPC (destroyed VR, new one created...) >> >> $ virsh dumpxml r-812-VM >> ... >> <description>Debian GNU/Linux 5(64-bit)</description> >> ... >> <os> >> <type arch='x86_64' machine='rhel6.5.0'>hvm</type> >> <boot dev='cdrom'/> >> <boot dev='hd'/> >> </os> >> >> :( >> >> >> >> On 30 May 2014 16:24, Andrija Panic <andrija.pa...@gmail.com> wrote: >> >> > I confirm, the highest is Debian 5 64bit >> > >> > Per docs >> > >> http://docs.cloudstack.apache.org/projects/cloudstack-release-notes/en/4.3/rnotes.html#upgrade-from-4-2-x-to-4-3 >> > , you should use Debian 7.0 64bit as OS type for system-kvm-4.3 >> template... >> > (same for xen and vmware templatest) >> > >> > Will change now DB to point to debian 5, and let you know. >> > >> > This is guest_os_id: 184 | 2 | NULL | >> > 986a0e98-39d6-11e3-8f93-0025904e4412 | Debian GNU/Linux 7(64-bit) >> > >> > >> > On 30 May 2014 15:45, Joris van Lieshout < >> jvanliesh...@schubergphilis.com> >> > wrote: >> > >> >> Andrija, >> >> >> >> The thing is I don't know who the os matching on KVM works. There must >> be >> >> a way to list supported os types. >> >> >> >> I also did some queuing on the guest_os_hypervisor table (ACS 4.3) and >> I >> >> don't see Debian 7 for KVM listed. >> >> >> >> select * from guest_os join guest_os_hypervisor on >> >> guest_os.id=guest_os_hypervisor.guest_os_id where >> >> guest_os_hypervisor.hypervisor_type='KVM' and >> >> guest_os_hypervisor.guest_os_name like '%Debian%'; >> >> >> >> What was the guest_os_id you where using? Could you try id 72 (Debian 5 >> >> 64-bit)? Adjust both os_type_id in vm_instance and vm_template (where >> >> type='SYSTEM' and hypervisor_type='KVM'). >> >> >> >> Kind regards, >> >> Joris van Lieshout >> >> >> >> Schuberg Philis >> >> >> >> >> >> >> >> >> >> >> >> On 30/05/14 15:33, "Andrija Panic" <andrija.pa...@gmail.com> wrote: >> >> >> >> >Joris, >> >> > >> >> >do you have recommendation on how in particular to try ? I'm not sure >> how >> >> >to fix that, except playing with editing systemvm-4.3 template to >> define >> >> >it >> >> >as another OS type... ? >> >> > >> >> >Thanks again, >> >> >Andrija >> >> > >> >> > >> >> >On 30 May 2014 15:30, Joris van Lieshout < >> >> jvanliesh...@schubergphilis.com> >> >> >wrote: >> >> > >> >> >> I've read back a bit in the code and if you look at >> >> BridgeVifDriver.java >> >> >> (this is where the log message with the nic profile is generated) >> you >> >> >>can >> >> >> see that the nic information might be off already once ACS hits the >> >> >> LibvirtVMDef.InterfaceDef plug function. This leads be to believer >> that >> >> >> the HVM/PV OS mismatch issue might still be related. Try fixing that >> >> >> first. At least it will allow us to exclude this from the list. >> >> >> >> >> >> >> >> >> Kind regards, >> >> >> Joris van Lieshout >> >> >> >> >> >> Schuberg Philis >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> On 30/05/14 15:26, "Andrija Panic" <andrija.pa...@gmail.com> wrote: >> >> >> >> >> >> >OK, thanks Joris. >> >> >> > >> >> >> >I will try playing with OS version option, on the systemvm-kvm-4.3 >> >> >> >template... >> >> >> > >> >> >> >Let me know if I can help with anything more. >> >> >> > >> >> >> >Thanks. >> >> >> >Andrija >> >> >> > >> >> >> > >> >> >> >On 30 May 2014 15:19, Joris van Lieshout >> >> >><jvanliesh...@schubergphilis.com >> >> >> > >> >> >> >wrote: >> >> >> > >> >> >> >> Hi Andrija, >> >> >> >> >> >> >> >> That does sound familiar and in the start xml of KVM you can see >> >> >>"<type >> >> >> >> arch='x86_64' machine='pc'>hvm</type>". I don't know KVM+ACS well >> >> >>enough >> >> >> >> to judge if this is the cause but I thing focusing on getting >> the VR >> >> >> >> started as PV guest might be worth trying. On the other hand I do >> >> see >> >> >> >> patchviasocket.pl being executed successfully... >> >> >> >> >> >> >> >> The other thing I see is, and now we're getting into java code, >> is >> >> >>this: >> >> >> >> >> >> >> >> 2014-05-30 14:41:01,386{GMT} DEBUG [kvm.resource.BridgeVifDriver] >> >> >> >> (agentRequest-Handler-3:) >> >> >> >>nic=[Nic:Public-46.232.xxx.246-vlan://untagged] >> >> >> >> 2014-05-30 14:41:01,502{GMT} DEBUG [cloud.agent.Agent] >> >> >> >> (agentRequest-Handler-3:) Processing command: >> >> >> >> com.cloud.agent.api.routing.IpAssocVpcCommand >> >> >> >> 2014-05-30 14:41:01,506{GMT} DEBUG >> >> >> >> [resource.virtualnetwork.VirtualRoutingResource] >> >> >> >>(agentRequest-Handler-3:) >> >> >> >> Executing: >> >> >> >> /usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh >> >> >> >> vpc_ipassoc.sh 169.254.0.52 -A -l 46.232.xxx.246 -c ethnull -g >> >> >> >> 46.232.xxx.1 -m 24 -n 46.232.xxx.0 >> >> >> >> >> >> >> >> My suspicion is that somewhere in the translation from the nic >> >> >>profile >> >> >> >>to >> >> >> >> the actual route_proxy.sh command ACS failes to find the nic id >> and >> >> >> >> returns null. >> >> >> >> >> >> >> >> Let me dig a bit deeper and see what I can find but this is >> where we >> >> >> >>might >> >> >> >> need some help from someone with knowledge of this pice of the >> code. >> >> >>:) >> >> >> >> >> >> >> >> >> >> >> >> Kind regards, >> >> >> >> Joris van Lieshout >> >> >> >> >> >> >> >> Schuberg Philis >> >> >> >> Boeingavenue 271 >> >> >> >> 1119 PD Schiphol-Rijk >> >> >> >> schubergphilis.com >> >> >> >> >> >> >> >> +31 20-7506672 >> >> >> >> +31 6-51428188 >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> On 30/05/14 14:49, "Andrija Panic" <andrija.pa...@gmail.com> >> wrote: >> >> >> >> >> >> >> >> >Hi Joris, >> >> >> >> > >> >> >> >> >I have turned on DEBUG loging in agent.log on cs1.xxx/net host: >> >> >> >> > >> >> >> >> >So, management logs again: http://pastebin.com/F6BRf7Y9 >> >> >> >> >Agent logs on cs1.xxx: http://pastebin.com/BJauKbaC >> >> >> >> > >> >> >> >> >Not playing smart, but there is some error: >> >> >> >> >[kvm.resource.KVMGuestOsMapper] >> >> >> >> >(agentRequest-Handler-3:) Can't find the mapping of guest os: >> >> Debian >> >> >> >> >GNU/Linux 7(64-bit) >> >> >> >> > >> >> >> >> >Best, >> >> >> >> >Andrija >> >> >> >> > >> >> >> >> > >> >> >> >> >On 30 May 2014 14:26, Joris van Lieshout >> >> >> >><jvanliesh...@schubergphilis.com >> >> >> >> > >> >> >> >> >wrote: >> >> >> >> > >> >> >> >> >> Hi Andrija, >> >> >> >> >> >> >> >> >> >> Bold formatting does not come trough on the dev list. :) >> >> >> >> >> But u might need a bit more info. >> >> >> >> >> >> >> >> >> >> At a certain point I see this line >> >> >> >> >> >> >> >> >> >> 2014-05-30 13:56:23,935 DEBUG [c.c.a.t.Request] >> >> >> >> >> (Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Seq 1-609104082: >> >> >>Sending >> >> >> >> { >> >> >> >> >> Cmd , MgmtId: 161344838950, via: 1(cs1.xxxxx.net), Ver: v1, >> >> Flags: >> >> >> >> >>100111, >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>[{"com.cloud.agent.api.StartCommand":{"vm":{"id":801,"name":"r-801-VM >> >> >>>>>>". >> >> >> >>>>.. >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> This is where the information is passed on to the agent >> handles. >> >> >>For >> >> >> >>XS >> >> >> >> >> this would initiate an agent handler on the management server >> but >> >> >>for >> >> >> >> >>KVM, >> >> >> >> >> if I remember correctly, it passed the command on to the >> >> >>cloudstack >> >> >> >> >>agent >> >> >> >> >> service on the hypervisor. >> >> >> >> >> >> >> >> >> >> Can you check the cloud service log on the KVM hypervisor >> >> >>executing >> >> >> >>the >> >> >> >> >> request? it's this server cs1.xxxxx.net and then search top >> down >> >> >>for >> >> >> >> >> 609104082 in the log. See if you can provide the log from the >> >> >>agent >> >> >> >> >> handler thread started by that sequence. >> >> >> >> >> >> >> >> >> >> Kind regards, >> >> >> >> >> Joris van Lieshout >> >> >> >> >> >> >> >> >> >> Schuberg Philis >> >> >> >> >> Boeingavenue 271 >> >> >> >> >> 1119 PD Schiphol-Rijk >> >> >> >> >> schubergphilis.com >> >> >> >> >> >> >> >> >> >> +31 20-7506672 >> >> >> >> >> +31 6-51428188 >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> On 30/05/14 14:08, "Andrija Panic" <andrija.pa...@gmail.com> >> >> >>wrote: >> >> >> >> >> >> >> >> >> >> >Hi Joris, >> >> >> >> >> > >> >> >> >> >> >here is the management log: http://pastebin.com/zxnKxFhk >> >> >> >> >> > >> >> >> >> >> >Interesting parts (to me): in bold >> >> >> >> >> > >> >> >> >> >> >2014-05-30 13:56:21,899 DEBUG >> >> >>[o.a.c.s.m.AncientDataMotionStrategy] >> >> >> >> >> >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) copyAsync >> inspecting >> >> >>src >> >> >> >> >>type >> >> >> >> >> >TEMPLATE copyAsync inspecting dest type VOLUME >> >> >> >> >> >2014-05-30 13:56:21,905 DEBUG [c.c.a.t.Request] >> >> >> >> >> >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Seq 4-1248669612: >> >> >> >>Sending >> >> >> >> >>{ >> >> >> >> >> >Cmd , MgmtId: 161344838950, via: 4(cs2.xxxxx.net), Ver: v1, >> >> >>Flags: >> >> >> >> >> 100011, >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org >> >> >>>>>>>.a >> >> >> >>>>>pa >> >> >> >> >>>ch >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>e.cloudstack.storage.to.TemplateObjectTO":{"path":"1adc1d2e-56ae-4a0 >> >> >>>>>>>f- >> >> >> >>>>>b0 >> >> >> >> >>>b4 >> >> >> >> >> >-5e351e7cae55","origUrl":" >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> http://download.cloud.com/templates/4.3/systemvm64template-2014-01-14-mas >> >> >> >> >>t >> >> >> >> >> >er-kvm.qcow2.bz2 >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>","uuid":"1adc1d2e-56ae-4a0f-b0b4-5e351e7cae55","id":414,"format":"Q >> >> >>>>>>>CO >> >> >> >>>>>W2 >> >> >> >> >>>", >> >> >> >> >> >"accountId":2,"checksum":"85a1bed07bf43cbf022451cb2ecae4ff"," >> >> >> >> >> >*hvm":true* >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>,"displayText":"systemvm-kvm-4.3","imageDataStore":{"org.apache.clou >> >> >>>>>>>ds >> >> >> >>>>>ta >> >> >> >> >>>ck >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>.storage.to.PrimaryDataStoreTO":{"uuid":"5b93422e-1a66-353d-88a8-220 >> >> >>>>>>>3f >> >> >> >>>>>79 >> >> >> >> >>>b1 >> >> >> >> >> >dc6","id":209,"poolType":"RBD","host":" >> >> >> >> >> >cephmon.xxxxx.net >> >> ","path":"cloudstack","port":6789,"url":"RBD:// >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> cephmon.xxxxx.net/cloudstack/?ROLE=Primary&STOREUUID=5b93422e-1a66-353d-8 >> >> >> >> >>8 >> >> >> >> >> >a8-2203f79b1dc6 >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>"}},"name":"414-2-ec331e74-5858-3153-91a9-1d706d9c533e","hypervisorT >> >> >>>>>>>yp >> >> >> >>>>>e" >> >> >> >> >>>:" >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{ >> >> >>>>>>>"u >> >> >> >>>>>ui >> >> >> >> >>>d" >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>:"9c440d3b-cba5-4960-b8bf-dca90291cd2b","volumeType":"ROOT","dataSto >> >> >>>>>>>re >> >> >> >>>>>": >> >> >> >> >>>{" >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"5b9342 >> >> >>>>>>>2e >> >> >> >>>>>-1 >> >> >> >> >>>a6 >> >> >> >> >> >6-353d-88a8-2203f79b1dc6","id":209,"poolType":"RBD","host":" >> >> >> >> >> >cephmon.xxxxx.net >> >> ","path":"cloudstack","port":6789,"url":"RBD:// >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> cephmon.xxxxx.net/cloudstack/?ROLE=Primary&STOREUUID=5b93422e-1a66-353d-8 >> >> >> >> >>8 >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>a8-2203f79b1dc6"}},"name":"ROOT-801","size":2621440000,"volumeId":10 >> >> >>>>>>>64 >> >> >> >>>>>," >> >> >> >> >>>vm >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>Name":"r-801-VM","accountId":11,"format":"RAW","id":1064,"deviceId": >> >> >>>>>>>0, >> >> >> >>>>>"h >> >> >> >> >>>yp >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>ervisorType":"KVM"}},"executeInSequence":false,"options":{},"wait":0}} >> >> >>>>>] >> >> >> >> >> >} >> >> >> >> >> >2014-05-30 13:56:23,742 DEBUG [c.c.a.t.Request] >> >> >> >> >> >(AgentManager-Handler-12:null) Seq 4-1248669612: Processing: >> { >> >> >> >>Ans: , >> >> >> >> >> >MgmtId: 161344838950, via: 4, Ver: v1, Flags: 10, >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{ >> >> >>>>>>>"o >> >> >> >>>>>rg >> >> >> >> >>>.a >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>pache.cloudstack.storage.to.VolumeObjectTO":{"size":2621440000,"path >> >> >>>>>>>": >> >> >> >>>>>"9 >> >> >> >> >>>c4 >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>40d3b-cba5-4960-b8bf-dca90291cd2b","accountId":0,"format":"RAW","id" >> >> >>>>>>>:0 >> >> >> >>>>>}} >> >> >> >> >>>," >> >> >> >> >> >result":true,"wait":0}}] >> >> >> >> >> >} >> >> >> >> >> >2014-05-30 13:56:23,742 DEBUG [c.c.a.t.Request] >> >> >> >> >> >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Seq 4-1248669612: >> >> >> >> >>Received: { >> >> >> >> >> >Ans: , MgmtId: 161344838950, via: 4, Ver: v1, Flags: 10, { >> >> >> >> >>CopyCmdAnswer >> >> >> >> >> >} } >> >> >> >> >> >2014-05-30 13:56:23,773 DEBUG >> >> >> >> >> >[c.c.n.r.VpcVirtualNetworkApplianceManagerImpl] >> >> >> >> >> >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Removing nic >> >> >>NicProfile[ >> >> >> >> >> >*1092-801-null*-46.232.xxx.246-vlan://untagged of type Public >> >> >>from >> >> >> >>the >> >> >> >> >> >nics >> >> >> >> >> >passed on vm start. The nic will be plugged later >> >> >> >> >> >2014-05-30 13:56:23,773 DEBUG >> >> >> >> >> >[c.c.n.r.VpcVirtualNetworkApplianceManagerImpl] >> >> >> >> >> >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Removing nic >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>NicProfile[1093-801-cd9fd29a-0573-4715-8742-00ecb9f82c9d-10.0.1.1-vl >> >> >>>>>>>an >> >> >> >>>>>:/ >> >> >> >> >>>/4 >> >> >> >> >> >4 >> >> >> >> >> >of type Guest from the nics passed on vm start. The nic will >> be >> >> >> >>plugged >> >> >> >> >> >later >> >> >> >> >> >2014-05-30 13:56:23,773 DEBUG >> >> >> >> >> >[c.c.n.r.VpcVirtualNetworkApplianceManagerImpl] >> >> >> >> >> >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Removing nic >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>NicProfile[1094-801-cd9fd29a-0573-4715-8742-00ecb9f82c9d-10.0.3.1-vl >> >> >>>>>>>an >> >> >> >>>>>:/ >> >> >> >> >>>/4 >> >> >> >> >> >3 >> >> >> >> >> >of type Guest from the nics passed on vm start. The nic will >> be >> >> >> >>plugged >> >> >> >> >> >later >> >> >> >> >> >2014-05-30 13:56:23,773 DEBUG >> >> >> >> >> >[c.c.n.r.VpcVirtualNetworkApplianceManagerImpl] >> >> >> >> >> >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Removing nic >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>NicProfile[1095-801-cd9fd29a-0573-4715-8742-00ecb9f82c9d-10.0.4.1-vl >> >> >>>>>>>an >> >> >> >>>>>:/ >> >> >> >> >>>/3 >> >> >> >> >> >004 >> >> >> >> >> >of type Guest from the nics passed on vm start. The nic will >> be >> >> >> >>plugged >> >> >> >> >> >later >> >> >> >> >> >2014-05-30 13:56:23,773 DEBUG >> >> >> >> >> >[c.c.n.r.VpcVirtualNetworkApplianceManagerImpl] >> >> >> >> >> >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Removing nic >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>>>>>>NicProfile[1095-801-cd9fd29a-0573-4715-8742-00ecb9f82c9d-10.0.4.1-vl >> >> >>>>>>>an >> >> >> >>>>>:/ >> >> >> >> >>>/3 >> >> >> >> >> >004 >> >> >> >> >> >of type Guest from the nics passed on vm start >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> >Thanks, >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> >On 30 May 2014 13:54, Joris van Lieshout >> >> >> >> >><jvanliesh...@schubergphilis.com >> >> >> >> >> > >> >> >> >> >> >wrote: >> >> >> >> >> > >> >> >> >> >> >> Hi Andrija, >> >> >> >> >> >> >> >> >> >> >> >> Just the start of the VR should be sufficient. >> >> >> >> >> >> >> >> >> >> >> >> Kind regards, >> >> >> >> >> >> Joris van Lieshout >> >> >> >> >> >> >> >> >> >> >> >> Schuberg Philis >> >> >> >> >> >> Boeingavenue 271 >> >> >> >> >> >> 1119 PD Schiphol-Rijk >> >> >> >> >> >> schubergphilis.com >> >> >> >> >> >> >> >> >> >> >> >> +31 20-7506672 >> >> >> >> >> >> +31 6-51428188 >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> On 30/05/14 13:48, "Andrija Panic" < >> andrija.pa...@gmail.com> >> >> >> >>wrote: >> >> >> >> >> >> >> >> >> >> >> >> >Hi Joris, >> >> >> >> >> >> > >> >> >> >> >> >> >just to be sure - you want me to capture the log from the >> >> >>moment >> >> >> >>I >> >> >> >> >> >>reboot >> >> >> >> >> >> >router - or you want me to stop it, then start capturing >> log, >> >> >>and >> >> >> >> >> >>start it >> >> >> >> >> >> >(and continue capture untill ethnull errors inside VR) ? >> >> >> >> >> >> > >> >> >> >> >> >> >Thanks, >> >> >> >> >> >> > >> >> >> >> >> >> > >> >> >> >> >> >> >On 30 May 2014 13:39, Joris van Lieshout >> >> >> >> >> >><jvanliesh...@schubergphilis.com >> >> >> >> >> >> > >> >> >> >> >> >> >wrote: >> >> >> >> >> >> > >> >> >> >> >> >> >> Hi Andrija, >> >> >> >> >> >> >> >> >> >> >> >> >> >> Thanks for the answers. In deed your situation is >> different >> >> >>so >> >> >> >> >> >>PV/HVM is >> >> >> >> >> >> >> not the issue. >> >> >> >> >> >> >> >> >> >> >> >> >> >> When reading back the log output you have provided I >> noted >> >> >>that >> >> >> >> >>the >> >> >> >> >> >>VR >> >> >> >> >> >> >> messages log indicates that it's waiting for ethnull to >> be >> >> >>up. >> >> >> >> >>This >> >> >> >> >> >> >>raises >> >> >> >> >> >> >> the question where null was introduced instead of 1. The >> >> ACS >> >> >> >> >> >>management >> >> >> >> >> >> >> log output you send was, what I think, later down the >> road >> >> >> >>where >> >> >> >> >>ACS >> >> >> >> >> >> >>gives >> >> >> >> >> >> >> up trying to wait for the VR to come up. If you would >> >> >>capture >> >> >> >>the >> >> >> >> >> >> >> job-executor in the management log from startCommand >> till >> >> >>the >> >> >> >> >> >>exception, >> >> >> >> >> >> >> do you see anywhere a mention of ethnull? You might >> need to >> >> >> >>reed >> >> >> >> >>into >> >> >> >> >> >> >>the >> >> >> >> >> >> >> DirectAgent executing the startCommand to find a clue. >> The >> >> >> >>thing >> >> >> >> >>is >> >> >> >> >> >> >>that I >> >> >> >> >> >> >> only have experience with XS based environment so I >> cannot >> >> >> >>point >> >> >> >> >>you >> >> >> >> >> >>to >> >> >> >> >> >> >> the exact output to look for. On XS, at least, it is >> >> >> >> >> >> >> "[c.c.h.x.r.CitrixResourceBase] >> >> >>(DirectAgent-351:ctx-4a51bb9e) >> >> >> >> >> >>Created a >> >> >> >> >> >> >> vif e4c362bd-764b-f651-dc9a-1abd5cb33c43 on 1" >> >> >> >> >> >> >> >> >> >> >> >> >> >> Kind regards, >> >> >> >> >> >> >> Joris van Lieshout >> >> >> >> >> >> >> >> >> >> >> >> >> >> Schuberg Philis >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> On 30/05/14 10:48, "Andrija Panic" < >> >> andrija.pa...@gmail.com> >> >> >> >> >>wrote: >> >> >> >> >> >> >> >> >> >> >> >> >> >> >Hi Deen, >> >> >> >> >> >> >> >no, in DB there is field "vlan_id" with value >> "untagged" - >> >> >> >>that >> >> >> >> >> >> >> >"vlan://untagged" is shown from ACS gui, and is used in >> >> API >> >> >> >>call >> >> >> >> >>(or >> >> >> >> >> >> >> >better >> >> >> >> >> >> >> >said commands that are seen in management server logs). >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >Best, >> >> >> >> >> >> >> >Andrija >> >> >> >> >> >> >> > >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >On 30 May 2014 10:37, Daan Hoogland >> >> >><daan.hoogl...@gmail.com> >> >> >> >> >> wrote: >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> Andrija, >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> Do not just assign a second net vlan://500 You have >> one >> >> >>like >> >> >> >> >>that >> >> >> >> >> >>and >> >> >> >> >> >> >> >> you don't want conflicting nets using the same vlan. >> I >> >> am >> >> >> >> >> >>wondering >> >> >> >> >> >> >> >> why 'untagged' comes out as 'vlan://untagged'. I >> think >> >> >>that >> >> >> >>is >> >> >> >> >>the >> >> >> >> >> >> >> >> bug. Did you find the string 'vlan://untagged' in >> your >> >> >>db? >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> On Fri, May 30, 2014 at 10:20 AM, Andrija Panic >> >> >> >> >> >> >> >><andrija.pa...@gmail.com> >> >> >> >> >> >> >> >> wrote: >> >> >> >> >> >> >> >> > Hi Joris, >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> > thank you for taking time to address this issue :) >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> > So...: >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> > - I'm on KVM (stock CentOS 6.2 patched by Inktank >> for >> >> >>CEPH >> >> >> >> >> >> >>support), >> >> >> >> >> >> >> >>OS >> >> >> >> >> >> >> >> is >> >> >> >> >> >> >> >> > Centos 6.5, libvirt 1.2.3 compiled. >> >> >> >> >> >> >> >> > - ACS 4.3 having problems, ACS 4.2.1 was fine >> >> >> >> >> >> >> >> > - not XS, so I guess no answers for this part :) >> >> >> >> >> >> >> >> > - guest_os_id is 184 = Debian 7 x64 >> >> >> >> >> >> >> >> > - SVM = systemvm-kvm-4.3 = os type 184 = Debian 7 >> x64 >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> > This worked previously on 4.2.1 = template was >> >> ofcourse >> >> >> >> >> >> >> >>systemvm-kvm-4.2 >> >> >> >> >> >> >> >> - >> >> >> >> >> >> >> >> > but that was also Debian 7 x64 type... so this >> should >> >> >>not >> >> >> >>be >> >> >> >> >>the >> >> >> >> >> >> >> >>issues >> >> >> >> >> >> >> >> > (guest not supported by host...) >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> > The only thing that might be out of "standard" = >> all >> >> >>SVMs >> >> >> >> >>are on >> >> >> >> >> >> >>CEPH >> >> >> >> >> >> >> >>- >> >> >> >> >> >> >> >> > there are official docs on altering database to >> make >> >> >>some >> >> >> >>new >> >> >> >> >> >> >>System >> >> >> >> >> >> >> >> > Offering as default for SSVM and CPVM - what I >> did, I >> >> >>also >> >> >> >> >>have >> >> >> >> >> >> >>done >> >> >> >> >> >> >> >>same >> >> >> >> >> >> >> >> > config in DB, to make VR use another System >> Offering >> >> as >> >> >> >> >>default >> >> >> >> >> >>- >> >> >> >> >> >> >> >>which >> >> >> >> >> >> >> >> is >> >> >> >> >> >> >> >> > NOT explained in the docs - you could use "Change >> >> >> >> >>Offering..." >> >> >> >> >> >> >>button >> >> >> >> >> >> >> >>on >> >> >> >> >> >> >> >> > exiting, shutdown VR to change it per docs... >> >> >> >> >> >> >> >> > But still this worked all fine on 4.2.1... >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> > - regarding /var/cache/cloud/cmdline the content is >> >> >> >>folowing >> >> >> >> >>at >> >> >> >> >> >>the >> >> >> >> >> >> >> >> moment >> >> >> >> >> >> >> >> > root@r-801-VM:~# cat /var/cache/cloud/cmdline >> >> >> >> >> >> >> >> > vpccidr=10.0.0.0/8 domain=cscloud.internal >> >> dns1=8.8.8.8 >> >> >> >> dns2= >> >> >> >> >> >> >> >> template=domP >> >> >> >> >> >> >> >> > name=r-801-VM eth0ip=169.254.0.75 >> eth0mask=255.255.0.0 >> >> >> >> >> >> >>type=vpcrouter >> >> >> >> >> >> >> >> > disable_rp_filter=true >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> > Also please note that only eth1 does not have IP >> info, >> >> >> >>eth0 >> >> >> >> >> >> >>(control >> >> >> >> >> >> >> >> > 169.xxx) and all other eh2 and up that are used for >> >> >>Tiers >> >> >> >> >>get IP >> >> >> >> >> >> >>info >> >> >> >> >> >> >> >> fine. >> >> >> >> >> >> >> >> > I could also manually add IP for eth1 (public NIC) >> and >> >> >> >>start >> >> >> >> >> >>ifup >> >> >> >> >> >> >> >>eth1 - >> >> >> >> >> >> >> >> > and it works fine, but adding new IP Port >> Forwarding >> >> >>etc >> >> >> >>does >> >> >> >> >> >>not >> >> >> >> >> >> >> >>work... >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> > Daan or somebody said it could be realted to my >> >> >>"Public" >> >> >> >> >>network >> >> >> >> >> >> >>(in >> >> >> >> >> >> >> >>the >> >> >> >> >> >> >> >> > Zones, Physical Network, eth1 listing) is NOT >> tagged >> >> >> >> >> >> >> >>(vlan://untagged)... >> >> >> >> >> >> >> >> > Interestingly the only VR that does work fine is >> the >> >> VR >> >> >> >>used >> >> >> >> >>in >> >> >> >> >> >> >>Shared >> >> >> >> >> >> >> >> > network, but that VR is using IP from Guest IP >> range >> >> >>(also >> >> >> >> >> >> >>efectively >> >> >> >> >> >> >> >> > public IPs but on vlan 500) >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> > I was instructed to try to change Public IP range >> from >> >> >> >> >>untagged >> >> >> >> >> >>to >> >> >> >> >> >> >> >>vlan >> >> >> >> >> >> >> >> > 500, but I'm not sure how to do this, if there is >> any >> >> >>way >> >> >> >>at >> >> >> >> >>all >> >> >> >> >> >> >> >>(editing >> >> >> >> >> >> >> >> > "vlan" table and changing to vlan 500 does not >> work, >> >> >>after >> >> >> >> >> >> >>rebooting >> >> >> >> >> >> >> >>VR >> >> >> >> >> >> >> >> > from ACS gui). >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> > :) >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> > So, not sure what is roughly expected date for 4.4, >> >> but >> >> >> >>right >> >> >> >> >> >>now, >> >> >> >> >> >> >>I'm >> >> >> >> >> >> >> >> > pretty stuck with a big problem of all VPC not >> >> >> >>operational at >> >> >> >> >> >> >>all... >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> > Thanks, >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> > On 30 May 2014 08:27, Joris van Lieshout < >> >> >> >> >> >> >> >> jvanliesh...@schubergphilis.com> >> >> >> >> >> >> >> >> > wrote: >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> Hi Andrija, >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> Daan asked me to have a look at this as well. >> Looking >> >> >>at >> >> >> >>you >> >> >> >> >> >> >>issue I >> >> >> >> >> >> >> >> >> recall having seen something similar. Back then >> when >> >> >> >> >>upgrading >> >> >> >> >> >> >>4.2.1 >> >> >> >> >> >> >> >>to >> >> >> >> >> >> >> >> >> 4.3 I though it had to do with out own custom >> build >> >> >>svm >> >> >> >> >> >>template. >> >> >> >> >> >> >> >> >> Let me fire off some questions before explaining >> what >> >> >>the >> >> >> >> >>cause >> >> >> >> >> >> >>was >> >> >> >> >> >> >> >>in >> >> >> >> >> >> >> >> our >> >> >> >> >> >> >> >> >> case. :) >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> - what hypervisor (and version) are you using? >> >> >> >> >> >> >> >> >> - if XS, is the new VR a para-virtualised instance >> >> >>(PV) >> >> >> >>or >> >> >> >> >> >> >>hardware >> >> >> >> >> >> >> >> >> assisted (HVM)? Do a "xe vm-param-list" on the VR >> >> uuid >> >> >> >>and >> >> >> >> >> >>check >> >> >> >> >> >> >>that >> >> >> >> >> >> >> >> >> param PV-args is set and HVM-boot-policy is unset. >> >> >> >> >> >> >> >> >> - what is the OS type of the VR in ACS >> (guest_os_id >> >> in >> >> >> >> >> >>vm_instance >> >> >> >> >> >> >> >>table >> >> >> >> >> >> >> >> >> and match with table guest_os) >> >> >> >> >> >> >> >> >> - what is the OS type of the SVM template? >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> Now for the explaining. :) >> >> >> >> >> >> >> >> >> In our case the OS type of the new template was >> not >> >> >> >> >>supported >> >> >> >> >> >>on >> >> >> >> >> >> >>the >> >> >> >> >> >> >> >> >> XenServer version we are running. Therefore the VR >> >> was >> >> >> >> >>started >> >> >> >> >> >>by >> >> >> >> >> >> >>XS >> >> >> >> >> >> >> >>as >> >> >> >> >> >> >> >> a >> >> >> >> >> >> >> >> >> HVM guest. System vms on XS rely on the arguments >> >> >>passed >> >> >> >>to >> >> >> >> >> >>them >> >> >> >> >> >> >>in >> >> >> >> >> >> >> >>the >> >> >> >> >> >> >> >> >> PV-args param (ends up on the guest in >> >> >> >> >>/var/cache/cloud/cmdline >> >> >> >> >> >> >> >>which in >> >> >> >> >> >> >> >> >> turn is used by cloud-early-config) in order to >> work. >> >> >> >> >>cmdline >> >> >> >> >> >> >> >>contains >> >> >> >> >> >> >> >> the >> >> >> >> >> >> >> >> >> NIC configuration information. >> >> >> >> >> >> >> >> >> So, long story short, if a VR gets started as a >> HVM >> >> it >> >> >> >>will >> >> >> >> >>not >> >> >> >> >> >> >>get >> >> >> >> >> >> >> >>the >> >> >> >> >> >> >> >> >> information needed to configure it's NICs. >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> Workaround >> >> >> >> >> >> >> >> >> We corrected the os_type_id in the DB (yes I know >> >> >>editing >> >> >> >> >>the >> >> >> >> >> >>DB >> >> >> >> >> >> >>is >> >> >> >> >> >> >> >> >> something you usually don't want but there is no >> >> other >> >> >> >>way >> >> >> >> >>in >> >> >> >> >> >>this >> >> >> >> >> >> >> >>case) >> >> >> >> >> >> >> >> >> of the existing VR's and of the systemvmtemplate >> to >> >> >> >> >>something >> >> >> >> >> >> >> >>supported >> >> >> >> >> >> >> >> by >> >> >> >> >> >> >> >> >> XenServer. >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> Kind regards, >> >> >> >> >> >> >> >> >> Joris van Lieshout >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> Schuberg Philis >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> On 29/05/14 12:18, "Andrija Panic" >> >> >> >><andrija.pa...@gmail.com >> >> >> >> > >> >> >> >> >> >> >>wrote: >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >They are 2 traffic types on 1 physical net (that >> is >> >> >>both >> >> >> >> >> >>tagged >> >> >> >> >> >> >>vlan >> >> >> >> >> >> >> >> 500, >> >> >> >> >> >> >> >> >> >and untagged packets travel over same KVM bridge, >> >> and >> >> >> >>over >> >> >> >> >> >>eth1 >> >> >> >> >> >> >>to >> >> >> >> >> >> >> >> outside >> >> >> >> >> >> >> >> >> >world)... >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >On 29 May 2014 12:04, Daan Hoogland >> >> >> >> >><daan.hoogl...@gmail.com> >> >> >> >> >> >> >> wrote: >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> Are these two traffic types in one physical >> net? >> >> or >> >> >> >>two >> >> >> >> >> >> >>physical >> >> >> >> >> >> >> >>nets >> >> >> >> >> >> >> >> >> >> on the same interface (seems wrong). >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> On Thu, May 29, 2014 at 11:35 AM, Jayapal Reddy >> >> >>Uradi >> >> >> >> >> >> >> >> >> >> <jayapalreddy.ur...@citrix.com> wrote: >> >> >> >> >> >> >> >> >> >> > I don't think editing DB table will work. >> >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> > -Jayapal >> >> >> >> >> >> >> >> >> >> > On 29-May-2014, at 2:52 PM, Andrija Panic >> >> >> >> >> >> >> >><andrija.pa...@gmail.com >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> wrote: >> >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> >> It's like this: >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> I have public subnet /24. >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> half is dedicated for Guest traffic (vlan >> 500) >> >> >>and >> >> >> >>the >> >> >> >> >> >> >>second >> >> >> >> >> >> >> >> half is >> >> >> >> >> >> >> >> >> >> >> dedicated to Public traffic/network (no vlan >> >> >>tags, >> >> >> >> >>that >> >> >> >> >> >>is >> >> >> >> >> >> >> >> untagged >> >> >> >> >> >> >> >> >> >> packets) >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> Both vlan500 and untagged packets travel >> over >> >> >> >>physical >> >> >> >> >> >>eth1 >> >> >> >> >> >> >> >> >> >>interface on >> >> >> >> >> >> >> >> >> >> >> hypervisors and can reach Internet. >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> Thanks, >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> On 29 May 2014 11:06, Daan Hoogland >> >> >> >> >> >> >><daan.hoogl...@gmail.com> >> >> >> >> >> >> >> >> wrote: >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>> On Thu, May 29, 2014 at 10:57 AM, Andrija >> >> >>Panic < >> >> >> >> >> >> >> >> >> >> andrija.pa...@gmail.com> >> >> >> >> >> >> >> >> >> >> >>> wrote: >> >> >> >> >> >> >> >> >> >> >>>> 500 >> >> >> >> >> >> >> >> >> >> >>> >> >> >> >> >> >> >> >> >> >> >>> >> >> >> >> >> >> >> >> >> >> >>> is 500 the vlan of your guestnetwork or >> your >> >> >> >>physical >> >> >> >> >> >> >>network? >> >> >> >> >> >> >> >> You >> >> >> >> >> >> >> >> >> >> >>> wouldn't want to have two nets with vlan >> 500! >> >> >> >> >> >> >> >> >> >> >>> >> >> >> >> >> >> >> >> >> >> >>> -- >> >> >> >> >> >> >> >> >> >> >>> Daan >> >> >> >> >> >> >> >> >> >> >>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> Andrija Panić >> >> >> >> >> >> >> >> >> >> >> -------------------------------------- >> >> >> >> >> >> >> >> >> >> >> http://admintweets.com >> >> >> >> >> >> >> >> >> >> >> -------------------------------------- >> >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> >> >> >> >> >> >> >> Daan >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >-- >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >Andrija Panić >> >> >> >> >> >> >> >> >> >-------------------------------------- >> >> >> >> >> >> >> >> >> > http://admintweets.com >> >> >> >> >> >> >> >> >> >-------------------------------------- >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> > -- >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> > Andrija Panić >> >> >> >> >> >> >> >> > -------------------------------------- >> >> >> >> >> >> >> >> > http://admintweets.com >> >> >> >> >> >> >> >> > -------------------------------------- >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> >> >> >> >> >> Daan >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> > >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >-- >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >Andrija Panić >> >> >> >> >> >> >> >-------------------------------------- >> >> >> >> >> >> >> > http://admintweets.com >> >> >> >> >> >> >> >-------------------------------------- >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> > >> >> >> >> >> >> >-- >> >> >> >> >> >> > >> >> >> >> >> >> >Andrija Panić >> >> >> >> >> >> >-------------------------------------- >> >> >> >> >> >> > http://admintweets.com >> >> >> >> >> >> >-------------------------------------- >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> >-- >> >> >> >> >> > >> >> >> >> >> >Andrija Panić >> >> >> >> >> >-------------------------------------- >> >> >> >> >> > http://admintweets.com >> >> >> >> >> >-------------------------------------- >> >> >> >> >> >> >> >> >> >> >> >> >> >> > >> >> >> >> > >> >> >> >> >-- >> >> >> >> > >> >> >> >> >Andrija Panić >> >> >> >> >-------------------------------------- >> >> >> >> > http://admintweets.com >> >> >> >> >-------------------------------------- >> >> >> >> >> >> >> >> >> >> >> > >> >> >> > >> >> >> >-- >> >> >> > >> >> >> >Andrija Panić >> >> >> >-------------------------------------- >> >> >> > http://admintweets.com >> >> >> >-------------------------------------- >> >> >> >> >> >> >> >> > >> >> > >> >> >-- >> >> > >> >> >Andrija Panić >> >> >-------------------------------------- >> >> > http://admintweets.com >> >> >-------------------------------------- >> >> >> >> >> > >> > >> > -- >> > >> > Andrija Panić >> > -------------------------------------- >> > http://admintweets.com >> > -------------------------------------- >> > >> >> >> >> -- >> >> Andrija Panić >> -------------------------------------- >> http://admintweets.com >> -------------------------------------- >> > >
