Hi Andrija, That does sound familiar and in the start xml of KVM you can see "<type arch='x86_64' machine='pc'>hvm</type>". I don't know KVM+ACS well enough to judge if this is the cause but I thing focusing on getting the VR started as PV guest might be worth trying. On the other hand I do see patchviasocket.pl being executed successfully...
The other thing I see is, and now we're getting into java code, is this: 2014-05-30 14:41:01,386{GMT} DEBUG [kvm.resource.BridgeVifDriver] (agentRequest-Handler-3:) nic=[Nic:Public-46.232.xxx.246-vlan://untagged] 2014-05-30 14:41:01,502{GMT} DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:) Processing command: com.cloud.agent.api.routing.IpAssocVpcCommand 2014-05-30 14:41:01,506{GMT} DEBUG [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-3:) Executing: /usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh vpc_ipassoc.sh 169.254.0.52 -A -l 46.232.xxx.246 -c ethnull -g 46.232.xxx.1 -m 24 -n 46.232.xxx.0 My suspicion is that somewhere in the translation from the nic profile to the actual route_proxy.sh command ACS failes to find the nic id and returns null. Let me dig a bit deeper and see what I can find but this is where we might need some help from someone with knowledge of this pice of the code. :) Kind regards, Joris van Lieshout Schuberg Philis Boeingavenue 271 1119 PD Schiphol-Rijk schubergphilis.com +31 20-7506672 +31 6-51428188 On 30/05/14 14:49, "Andrija Panic" <andrija.pa...@gmail.com> wrote: >Hi Joris, > >I have turned on DEBUG loging in agent.log on cs1.xxx/net host: > >So, management logs again: http://pastebin.com/F6BRf7Y9 >Agent logs on cs1.xxx: http://pastebin.com/BJauKbaC > >Not playing smart, but there is some error: >[kvm.resource.KVMGuestOsMapper] >(agentRequest-Handler-3:) Can't find the mapping of guest os: Debian >GNU/Linux 7(64-bit) > >Best, >Andrija > > >On 30 May 2014 14:26, Joris van Lieshout <jvanliesh...@schubergphilis.com> >wrote: > >> Hi Andrija, >> >> Bold formatting does not come trough on the dev list. :) >> But u might need a bit more info. >> >> At a certain point I see this line >> >> 2014-05-30 13:56:23,935 DEBUG [c.c.a.t.Request] >> (Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Seq 1-609104082: Sending { >> Cmd , MgmtId: 161344838950, via: 1(cs1.xxxxx.net), Ver: v1, Flags: >>100111, >> >>[{"com.cloud.agent.api.StartCommand":{"vm":{"id":801,"name":"r-801-VM"... >> >> >> This is where the information is passed on to the agent handles. For XS >> this would initiate an agent handler on the management server but for >>KVM, >> if I remember correctly, it passed the command on to the cloudstack >>agent >> service on the hypervisor. >> >> Can you check the cloud service log on the KVM hypervisor executing the >> request? it's this server cs1.xxxxx.net and then search top down for >> 609104082 in the log. See if you can provide the log from the agent >> handler thread started by that sequence. >> >> Kind regards, >> Joris van Lieshout >> >> Schuberg Philis >> Boeingavenue 271 >> 1119 PD Schiphol-Rijk >> schubergphilis.com >> >> +31 20-7506672 >> +31 6-51428188 >> >> >> >> >> On 30/05/14 14:08, "Andrija Panic" <andrija.pa...@gmail.com> wrote: >> >> >Hi Joris, >> > >> >here is the management log: http://pastebin.com/zxnKxFhk >> > >> >Interesting parts (to me): in bold >> > >> >2014-05-30 13:56:21,899 DEBUG [o.a.c.s.m.AncientDataMotionStrategy] >> >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) copyAsync inspecting src >>type >> >TEMPLATE copyAsync inspecting dest type VOLUME >> >2014-05-30 13:56:21,905 DEBUG [c.c.a.t.Request] >> >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Seq 4-1248669612: Sending >>{ >> >Cmd , MgmtId: 161344838950, via: 4(cs2.xxxxx.net), Ver: v1, Flags: >> 100011, >> >>>[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apa >>>ch >> >>>e.cloudstack.storage.to.TemplateObjectTO":{"path":"1adc1d2e-56ae-4a0f-b0 >>>b4 >> >-5e351e7cae55","origUrl":" >> > >> >>http://download.cloud.com/templates/4.3/systemvm64template-2014-01-14-mas >>t >> >er-kvm.qcow2.bz2 >> >>>","uuid":"1adc1d2e-56ae-4a0f-b0b4-5e351e7cae55","id":414,"format":"QCOW2 >>>", >> >"accountId":2,"checksum":"85a1bed07bf43cbf022451cb2ecae4ff"," >> >*hvm":true* >> >>>,"displayText":"systemvm-kvm-4.3","imageDataStore":{"org.apache.cloudsta >>>ck >> >>>.storage.to.PrimaryDataStoreTO":{"uuid":"5b93422e-1a66-353d-88a8-2203f79 >>>b1 >> >dc6","id":209,"poolType":"RBD","host":" >> >cephmon.xxxxx.net","path":"cloudstack","port":6789,"url":"RBD:// >> > >> >>cephmon.xxxxx.net/cloudstack/?ROLE=Primary&STOREUUID=5b93422e-1a66-353d-8 >>8 >> >a8-2203f79b1dc6 >> >>>"}},"name":"414-2-ec331e74-5858-3153-91a9-1d706d9c533e","hypervisorType" >>>:" >> >>>KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uui >>>d" >> >>>:"9c440d3b-cba5-4960-b8bf-dca90291cd2b","volumeType":"ROOT","dataStore": >>>{" >> >>>org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"5b93422e-1 >>>a6 >> >6-353d-88a8-2203f79b1dc6","id":209,"poolType":"RBD","host":" >> >cephmon.xxxxx.net","path":"cloudstack","port":6789,"url":"RBD:// >> > >> >>cephmon.xxxxx.net/cloudstack/?ROLE=Primary&STOREUUID=5b93422e-1a66-353d-8 >>8 >> >>>a8-2203f79b1dc6"}},"name":"ROOT-801","size":2621440000,"volumeId":1064," >>>vm >> >>>Name":"r-801-VM","accountId":11,"format":"RAW","id":1064,"deviceId":0,"h >>>yp >> >ervisorType":"KVM"}},"executeInSequence":false,"options":{},"wait":0}}] >> >} >> >2014-05-30 13:56:23,742 DEBUG [c.c.a.t.Request] >> >(AgentManager-Handler-12:null) Seq 4-1248669612: Processing: { Ans: , >> >MgmtId: 161344838950, via: 4, Ver: v1, Flags: 10, >> >>>[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org >>>.a >> >>>pache.cloudstack.storage.to.VolumeObjectTO":{"size":2621440000,"path":"9 >>>c4 >> >>>40d3b-cba5-4960-b8bf-dca90291cd2b","accountId":0,"format":"RAW","id":0}} >>>," >> >result":true,"wait":0}}] >> >} >> >2014-05-30 13:56:23,742 DEBUG [c.c.a.t.Request] >> >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Seq 4-1248669612: >>Received: { >> >Ans: , MgmtId: 161344838950, via: 4, Ver: v1, Flags: 10, { >>CopyCmdAnswer >> >} } >> >2014-05-30 13:56:23,773 DEBUG >> >[c.c.n.r.VpcVirtualNetworkApplianceManagerImpl] >> >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Removing nic NicProfile[ >> >*1092-801-null*-46.232.xxx.246-vlan://untagged of type Public from the >> >nics >> >passed on vm start. The nic will be plugged later >> >2014-05-30 13:56:23,773 DEBUG >> >[c.c.n.r.VpcVirtualNetworkApplianceManagerImpl] >> >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Removing nic >> >>>NicProfile[1093-801-cd9fd29a-0573-4715-8742-00ecb9f82c9d-10.0.1.1-vlan:/ >>>/4 >> >4 >> >of type Guest from the nics passed on vm start. The nic will be plugged >> >later >> >2014-05-30 13:56:23,773 DEBUG >> >[c.c.n.r.VpcVirtualNetworkApplianceManagerImpl] >> >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Removing nic >> >>>NicProfile[1094-801-cd9fd29a-0573-4715-8742-00ecb9f82c9d-10.0.3.1-vlan:/ >>>/4 >> >3 >> >of type Guest from the nics passed on vm start. The nic will be plugged >> >later >> >2014-05-30 13:56:23,773 DEBUG >> >[c.c.n.r.VpcVirtualNetworkApplianceManagerImpl] >> >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Removing nic >> >>>NicProfile[1095-801-cd9fd29a-0573-4715-8742-00ecb9f82c9d-10.0.4.1-vlan:/ >>>/3 >> >004 >> >of type Guest from the nics passed on vm start. The nic will be plugged >> >later >> >2014-05-30 13:56:23,773 DEBUG >> >[c.c.n.r.VpcVirtualNetworkApplianceManagerImpl] >> >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Removing nic >> >>>NicProfile[1095-801-cd9fd29a-0573-4715-8742-00ecb9f82c9d-10.0.4.1-vlan:/ >>>/3 >> >004 >> >of type Guest from the nics passed on vm start >> > >> > >> >Thanks, >> > >> > >> >On 30 May 2014 13:54, Joris van Lieshout >><jvanliesh...@schubergphilis.com >> > >> >wrote: >> > >> >> Hi Andrija, >> >> >> >> Just the start of the VR should be sufficient. >> >> >> >> Kind regards, >> >> Joris van Lieshout >> >> >> >> Schuberg Philis >> >> Boeingavenue 271 >> >> 1119 PD Schiphol-Rijk >> >> schubergphilis.com >> >> >> >> +31 20-7506672 >> >> +31 6-51428188 >> >> >> >> >> >> >> >> >> >> On 30/05/14 13:48, "Andrija Panic" <andrija.pa...@gmail.com> wrote: >> >> >> >> >Hi Joris, >> >> > >> >> >just to be sure - you want me to capture the log from the moment I >> >>reboot >> >> >router - or you want me to stop it, then start capturing log, and >> >>start it >> >> >(and continue capture untill ethnull errors inside VR) ? >> >> > >> >> >Thanks, >> >> > >> >> > >> >> >On 30 May 2014 13:39, Joris van Lieshout >> >><jvanliesh...@schubergphilis.com >> >> > >> >> >wrote: >> >> > >> >> >> Hi Andrija, >> >> >> >> >> >> Thanks for the answers. In deed your situation is different so >> >>PV/HVM is >> >> >> not the issue. >> >> >> >> >> >> When reading back the log output you have provided I noted that >>the >> >>VR >> >> >> messages log indicates that it's waiting for ethnull to be up. >>This >> >> >>raises >> >> >> the question where null was introduced instead of 1. The ACS >> >>management >> >> >> log output you send was, what I think, later down the road where >>ACS >> >> >>gives >> >> >> up trying to wait for the VR to come up. If you would capture the >> >> >> job-executor in the management log from startCommand till the >> >>exception, >> >> >> do you see anywhere a mention of ethnull? You might need to reed >>into >> >> >>the >> >> >> DirectAgent executing the startCommand to find a clue. The thing >>is >> >> >>that I >> >> >> only have experience with XS based environment so I cannot point >>you >> >>to >> >> >> the exact output to look for. On XS, at least, it is >> >> >> "[c.c.h.x.r.CitrixResourceBase] (DirectAgent-351:ctx-4a51bb9e) >> >>Created a >> >> >> vif e4c362bd-764b-f651-dc9a-1abd5cb33c43 on 1" >> >> >> >> >> >> Kind regards, >> >> >> Joris van Lieshout >> >> >> >> >> >> Schuberg Philis >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> On 30/05/14 10:48, "Andrija Panic" <andrija.pa...@gmail.com> >>wrote: >> >> >> >> >> >> >Hi Deen, >> >> >> >no, in DB there is field "vlan_id" with value "untagged" - that >> >> >> >"vlan://untagged" is shown from ACS gui, and is used in API call >>(or >> >> >> >better >> >> >> >said commands that are seen in management server logs). >> >> >> > >> >> >> >Best, >> >> >> >Andrija >> >> >> > >> >> >> > >> >> >> >On 30 May 2014 10:37, Daan Hoogland <daan.hoogl...@gmail.com> >> wrote: >> >> >> > >> >> >> >> Andrija, >> >> >> >> >> >> >> >> Do not just assign a second net vlan://500 You have one like >>that >> >>and >> >> >> >> you don't want conflicting nets using the same vlan. I am >> >>wondering >> >> >> >> why 'untagged' comes out as 'vlan://untagged'. I think that is >>the >> >> >> >> bug. Did you find the string 'vlan://untagged' in your db? >> >> >> >> >> >> >> >> On Fri, May 30, 2014 at 10:20 AM, Andrija Panic >> >> >> >><andrija.pa...@gmail.com> >> >> >> >> wrote: >> >> >> >> > Hi Joris, >> >> >> >> > >> >> >> >> > thank you for taking time to address this issue :) >> >> >> >> > >> >> >> >> > So...: >> >> >> >> > >> >> >> >> > - I'm on KVM (stock CentOS 6.2 patched by Inktank for CEPH >> >> >>support), >> >> >> >>OS >> >> >> >> is >> >> >> >> > Centos 6.5, libvirt 1.2.3 compiled. >> >> >> >> > - ACS 4.3 having problems, ACS 4.2.1 was fine >> >> >> >> > - not XS, so I guess no answers for this part :) >> >> >> >> > - guest_os_id is 184 = Debian 7 x64 >> >> >> >> > - SVM = systemvm-kvm-4.3 = os type 184 = Debian 7 x64 >> >> >> >> > >> >> >> >> > This worked previously on 4.2.1 = template was ofcourse >> >> >> >>systemvm-kvm-4.2 >> >> >> >> - >> >> >> >> > but that was also Debian 7 x64 type... so this should not be >>the >> >> >> >>issues >> >> >> >> > (guest not supported by host...) >> >> >> >> > >> >> >> >> > The only thing that might be out of "standard" = all SVMs >>are on >> >> >>CEPH >> >> >> >>- >> >> >> >> > there are official docs on altering database to make some new >> >> >>System >> >> >> >> > Offering as default for SSVM and CPVM - what I did, I also >>have >> >> >>done >> >> >> >>same >> >> >> >> > config in DB, to make VR use another System Offering as >>default >> >>- >> >> >> >>which >> >> >> >> is >> >> >> >> > NOT explained in the docs - you could use "Change >>Offering..." >> >> >>button >> >> >> >>on >> >> >> >> > exiting, shutdown VR to change it per docs... >> >> >> >> > But still this worked all fine on 4.2.1... >> >> >> >> > >> >> >> >> > - regarding /var/cache/cloud/cmdline the content is folowing >>at >> >>the >> >> >> >> moment >> >> >> >> > root@r-801-VM:~# cat /var/cache/cloud/cmdline >> >> >> >> > vpccidr=10.0.0.0/8 domain=cscloud.internal dns1=8.8.8.8 dns2= >> >> >> >> template=domP >> >> >> >> > name=r-801-VM eth0ip=169.254.0.75 eth0mask=255.255.0.0 >> >> >>type=vpcrouter >> >> >> >> > disable_rp_filter=true >> >> >> >> > >> >> >> >> > Also please note that only eth1 does not have IP info, eth0 >> >> >>(control >> >> >> >> > 169.xxx) and all other eh2 and up that are used for Tiers >>get IP >> >> >>info >> >> >> >> fine. >> >> >> >> > I could also manually add IP for eth1 (public NIC) and start >> >>ifup >> >> >> >>eth1 - >> >> >> >> > and it works fine, but adding new IP Port Forwarding etc does >> >>not >> >> >> >>work... >> >> >> >> > >> >> >> >> > Daan or somebody said it could be realted to my "Public" >>network >> >> >>(in >> >> >> >>the >> >> >> >> > Zones, Physical Network, eth1 listing) is NOT tagged >> >> >> >>(vlan://untagged)... >> >> >> >> > Interestingly the only VR that does work fine is the VR used >>in >> >> >>Shared >> >> >> >> > network, but that VR is using IP from Guest IP range (also >> >> >>efectively >> >> >> >> > public IPs but on vlan 500) >> >> >> >> > >> >> >> >> > I was instructed to try to change Public IP range from >>untagged >> >>to >> >> >> >>vlan >> >> >> >> > 500, but I'm not sure how to do this, if there is any way at >>all >> >> >> >>(editing >> >> >> >> > "vlan" table and changing to vlan 500 does not work, after >> >> >>rebooting >> >> >> >>VR >> >> >> >> > from ACS gui). >> >> >> >> > >> >> >> >> > :) >> >> >> >> > >> >> >> >> > So, not sure what is roughly expected date for 4.4, but right >> >>now, >> >> >>I'm >> >> >> >> > pretty stuck with a big problem of all VPC not operational at >> >> >>all... >> >> >> >> > >> >> >> >> > Thanks, >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > On 30 May 2014 08:27, Joris van Lieshout < >> >> >> >> jvanliesh...@schubergphilis.com> >> >> >> >> > wrote: >> >> >> >> > >> >> >> >> >> Hi Andrija, >> >> >> >> >> >> >> >> >> >> Daan asked me to have a look at this as well. Looking at you >> >> >>issue I >> >> >> >> >> recall having seen something similar. Back then when >>upgrading >> >> >>4.2.1 >> >> >> >>to >> >> >> >> >> 4.3 I though it had to do with out own custom build svm >> >>template. >> >> >> >> >> Let me fire off some questions before explaining what the >>cause >> >> >>was >> >> >> >>in >> >> >> >> our >> >> >> >> >> case. :) >> >> >> >> >> >> >> >> >> >> - what hypervisor (and version) are you using? >> >> >> >> >> - if XS, is the new VR a para-virtualised instance (PV) or >> >> >>hardware >> >> >> >> >> assisted (HVM)? Do a "xe vm-param-list" on the VR uuid and >> >>check >> >> >>that >> >> >> >> >> param PV-args is set and HVM-boot-policy is unset. >> >> >> >> >> - what is the OS type of the VR in ACS (guest_os_id in >> >>vm_instance >> >> >> >>table >> >> >> >> >> and match with table guest_os) >> >> >> >> >> - what is the OS type of the SVM template? >> >> >> >> >> >> >> >> >> >> Now for the explaining. :) >> >> >> >> >> In our case the OS type of the new template was not >>supported >> >>on >> >> >>the >> >> >> >> >> XenServer version we are running. Therefore the VR was >>started >> >>by >> >> >>XS >> >> >> >>as >> >> >> >> a >> >> >> >> >> HVM guest. System vms on XS rely on the arguments passed to >> >>them >> >> >>in >> >> >> >>the >> >> >> >> >> PV-args param (ends up on the guest in >>/var/cache/cloud/cmdline >> >> >> >>which in >> >> >> >> >> turn is used by cloud-early-config) in order to work. >>cmdline >> >> >> >>contains >> >> >> >> the >> >> >> >> >> NIC configuration information. >> >> >> >> >> So, long story short, if a VR gets started as a HVM it will >>not >> >> >>get >> >> >> >>the >> >> >> >> >> information needed to configure it's NICs. >> >> >> >> >> >> >> >> >> >> Workaround >> >> >> >> >> We corrected the os_type_id in the DB (yes I know editing >>the >> >>DB >> >> >>is >> >> >> >> >> something you usually don't want but there is no other way >>in >> >>this >> >> >> >>case) >> >> >> >> >> of the existing VR's and of the systemvmtemplate to >>something >> >> >> >>supported >> >> >> >> by >> >> >> >> >> XenServer. >> >> >> >> >> >> >> >> >> >> Kind regards, >> >> >> >> >> Joris van Lieshout >> >> >> >> >> >> >> >> >> >> Schuberg Philis >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> On 29/05/14 12:18, "Andrija Panic" <andrija.pa...@gmail.com> >> >> >>wrote: >> >> >> >> >> >> >> >> >> >> >They are 2 traffic types on 1 physical net (that is both >> >>tagged >> >> >>vlan >> >> >> >> 500, >> >> >> >> >> >and untagged packets travel over same KVM bridge, and over >> >>eth1 >> >> >>to >> >> >> >> outside >> >> >> >> >> >world)... >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> >On 29 May 2014 12:04, Daan Hoogland >><daan.hoogl...@gmail.com> >> >> >> wrote: >> >> >> >> >> > >> >> >> >> >> >> Are these two traffic types in one physical net? or two >> >> >>physical >> >> >> >>nets >> >> >> >> >> >> on the same interface (seems wrong). >> >> >> >> >> >> >> >> >> >> >> >> On Thu, May 29, 2014 at 11:35 AM, Jayapal Reddy Uradi >> >> >> >> >> >> <jayapalreddy.ur...@citrix.com> wrote: >> >> >> >> >> >> > I don't think editing DB table will work. >> >> >> >> >> >> > >> >> >> >> >> >> > -Jayapal >> >> >> >> >> >> > On 29-May-2014, at 2:52 PM, Andrija Panic >> >> >> >><andrija.pa...@gmail.com >> >> >> >> > >> >> >> >> >> >> wrote: >> >> >> >> >> >> > >> >> >> >> >> >> >> It's like this: >> >> >> >> >> >> >> >> >> >> >> >> >> >> I have public subnet /24. >> >> >> >> >> >> >> >> >> >> >> >> >> >> half is dedicated for Guest traffic (vlan 500) and the >> >> >>second >> >> >> >> half is >> >> >> >> >> >> >> dedicated to Public traffic/network (no vlan tags, >>that >> >>is >> >> >> >> untagged >> >> >> >> >> >> packets) >> >> >> >> >> >> >> >> >> >> >> >> >> >> Both vlan500 and untagged packets travel over physical >> >>eth1 >> >> >> >> >> >>interface on >> >> >> >> >> >> >> hypervisors and can reach Internet. >> >> >> >> >> >> >> >> >> >> >> >> >> >> Thanks, >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> On 29 May 2014 11:06, Daan Hoogland >> >> >><daan.hoogl...@gmail.com> >> >> >> >> wrote: >> >> >> >> >> >> >> >> >> >> >> >> >> >>> On Thu, May 29, 2014 at 10:57 AM, Andrija Panic < >> >> >> >> >> >> andrija.pa...@gmail.com> >> >> >> >> >> >> >>> wrote: >> >> >> >> >> >> >>>> 500 >> >> >> >> >> >> >>> >> >> >> >> >> >> >>> >> >> >> >> >> >> >>> is 500 the vlan of your guestnetwork or your physical >> >> >>network? >> >> >> >> You >> >> >> >> >> >> >>> wouldn't want to have two nets with vlan 500! >> >> >> >> >> >> >>> >> >> >> >> >> >> >>> -- >> >> >> >> >> >> >>> Daan >> >> >> >> >> >> >>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> >> >> >> >> >> >> >> >> >> >> >> Andrija Panić >> >> >> >> >> >> >> -------------------------------------- >> >> >> >> >> >> >> http://admintweets.com >> >> >> >> >> >> >> -------------------------------------- >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> >> >> >> Daan >> >> >> >> >> >> >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> >-- >> >> >> >> >> > >> >> >> >> >> >Andrija Panić >> >> >> >> >> >-------------------------------------- >> >> >> >> >> > http://admintweets.com >> >> >> >> >> >-------------------------------------- >> >> >> >> >> >> >> >> >> >> >> >> >> >> > >> >> >> >> > >> >> >> >> > -- >> >> >> >> > >> >> >> >> > Andrija Panić >> >> >> >> > -------------------------------------- >> >> >> >> > http://admintweets.com >> >> >> >> > -------------------------------------- >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> >> Daan >> >> >> >> >> >> >> > >> >> >> > >> >> >> > >> >> >> >-- >> >> >> > >> >> >> >Andrija Panić >> >> >> >-------------------------------------- >> >> >> > http://admintweets.com >> >> >> >-------------------------------------- >> >> >> >> >> >> >> >> > >> >> > >> >> >-- >> >> > >> >> >Andrija Panić >> >> >-------------------------------------- >> >> > http://admintweets.com >> >> >-------------------------------------- >> >> >> >> >> > >> > >> >-- >> > >> >Andrija Panić >> >-------------------------------------- >> > http://admintweets.com >> >-------------------------------------- >> >> > > >-- > >Andrija Panić >-------------------------------------- > http://admintweets.com >--------------------------------------