> > Just some more debugging data -- this is debuggng from haproxy....I marked > where the pause happens here as well: > > Available polling systems : > select : pref=150, test result OK > Total: 1 (1 usable), will use select. > Using select() as the polling mechanism. > 00000000:services.accept(0005)=0006 from [24.99.160.41:52838] > 00000001:http_proxy.accept(0003)=0008 from [24.99.160.41:52839] > 00000001:http_proxy.clireq[0008:ffff]: GET /intelliqueue/agentui/ HTTP/1.1 > 00000001:http_proxy.clihdr[0008:ffff]: Host: www.cfpolitical.com > 00000001:http_proxy.clihdr[0008:ffff]: User-Agent: Mozilla/5.0 (Windows; U; > Windows NT 6.0; en-US; rv:1.9.2.2) Gecko/20100316 Firefox/3.6.2 (.NET CLR > 3.5.30729) > 00000001:http_proxy.clihdr[0008:ffff]: Accept: > text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 > 00000001:http_proxy.clihdr[0008:ffff]: Accept-Language: en-us,en;q=0.5 > 00000001:http_proxy.clihdr[0008:ffff]: Accept-Encoding: gzip,deflate > 00000001:http_proxy.clihdr[0008:ffff]: Accept-Charset: > ISO-8859-1,utf-8;q=0.7,*;q=0.7 > 00000001:http_proxy.clihdr[0008:ffff]: Keep-Alive: 115 > 00000001:http_proxy.clihdr[0008:ffff]: Connection: keep-alive > 00000001:http_proxy.clihdr[0008:ffff]: Cookie: CFID=408; CFTOKEN=12213899; > JSESSIONID=b6302e0e4a8d377c4e6e > 00000001:http_proxy.srvrep[0008:0009]: HTTP/1.1 200 OK > 00000001:http_proxy.srvhdr[0008:0009]: Connection: close > 00000001:http_proxy.srvhdr[0008:0009]: Date: Tue, 30 Mar 2010 14:55:39 GMT > 00000001:http_proxy.srvhdr[0008:0009]: Server: Microsoft-IIS/6.0 > 00000001:http_proxy.srvhdr[0008:0009]: X-Powered-By: ASP.NET > 00000001:http_proxy.srvhdr[0008:0009]: Set-Cookie: > CFAUTHORIZATION_IntelliQueue=;expires=Mon, 30-Mar-2009 14:55:39 GMT;path=/ > 00000001:http_proxy.srvhdr[0008:0009]: Content-Type: text/html; > charset=UTF-8 > 00000000:services.srvcls[0006:0007] > 00000000:services.clicls[0006:0007] > 00000000:services.closed[0006:0007] > > <!--- 16 second pause ---> > > 00000001:http_proxy.srvcls[0008:0009] > 00000001:http_proxy.clicls[0008:0009] > 00000001:http_proxy.closed[0008:0009] > 00000002:http_proxy.accept(0003)=0006 from [24.99.160.41:52843] > 00000002:http_proxy.clireq[0006:ffff]: GET /intelliqueue/agentui/config.cfm > HTTP/1.1 > 00000002:http_proxy.clihdr[0006:ffff]: Host: www.cfpolitical.com > 00000002:http_proxy.clihdr[0006:ffff]: User-Agent: Mozilla/5.0 (Windows; U; > Windows NT 6.0; en-US; rv:1.9.2.2) Gecko/20100316 Firefox/3.6.2 (.NET CLR > 3.5.30729) > 00000002:http_proxy.clihdr[0006:ffff]: Accept: > text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 > 00000002:http_proxy.clihdr[0006:ffff]: Accept-Language: en-us,en;q=0.5 > 00000002:http_proxy.clihdr[0006:ffff]: Accept-Encoding: gzip,deflate > 00000002:http_proxy.clihdr[0006:ffff]: Accept-Charset: > ISO-8859-1,utf-8;q=0.7,*;q=0.7 > 00000002:http_proxy.clihdr[0006:ffff]: Keep-Alive: 115 > 00000002:http_proxy.clihdr[0006:ffff]: Connection: keep-alive > 00000002:http_proxy.clihdr[0006:ffff]: Cookie: CFID=408; CFTOKEN=12213899; > JSESSIONID=b6302e0e4a8d377c4e6e > 00000002:http_proxy.srvrep[0006:0007]: HTTP/1.1 200 OK > 00000002:http_proxy.srvhdr[0006:0007]: Connection: close > 00000002:http_proxy.srvhdr[0006:0007]: Date: Tue, 30 Mar 2010 14:55:56 GMT > 00000002:http_proxy.srvhdr[0006:0007]: Server: Microsoft-IIS/6.0 > 00000002:http_proxy.srvhdr[0006:0007]: X-Powered-By: ASP.NET > 00000002:http_proxy.srvhdr[0006:0007]: Set-Cookie: > CFAUTHORIZATION_IntelliQueue=;expires=Mon, 30-Mar-2009 14:55:56 GMT;path=/ > 00000002:http_proxy.srvhdr[0006:0007]: Content-Type: text/xml > 00000002:http_proxy.srvcls[0006:0007] > 00000002:http_proxy.clicls[0006:0007] > 00000002:http_proxy.closed[0006:0007] > 00000003:http_proxy.accept(0003)=0006 from [24.99.160.41:52844] > 00000003:http_proxy.clireq[0006:ffff]: GET > /intelliqueue/manage/get_logo.cfm?ui=TRUE HTTP/1.1 > 00000003:http_proxy.clihdr[0006:ffff]: Host: www.cfpolitical.com > 00000003:http_proxy.clihdr[0006:ffff]: User-Agent: Mozilla/5.0 (Windows; U; > Windows NT 6.0; en-US; rv:1.9.2.2) Gecko/20100316 Firefox/3.6.2 (.NET CLR > 3.5.30729) > 00000003:http_proxy.clihdr[0006:ffff]: Accept: > text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 > 00000003:http_proxy.clihdr[0006:ffff]: Accept-Language: en-us,en;q=0.5 > 00000003:http_proxy.clihdr[0006:ffff]: Accept-Encoding: gzip,deflate > 00000003:http_proxy.clihdr[0006:ffff]: Accept-Charset: > ISO-8859-1,utf-8;q=0.7,*;q=0.7 > 00000003:http_proxy.clihdr[0006:ffff]: Keep-Alive: 115 > 00000003:http_proxy.clihdr[0006:ffff]: Connection: keep-alive > 00000003:http_proxy.clihdr[0006:ffff]: Cookie: CFID=408; CFTOKEN=12213899; > JSESSIONID=b6302e0e4a8d377c4e6e > 00000003:http_proxy.srvrep[0006:0007]: HTTP/1.1 200 OK > 00000003:http_proxy.srvhdr[0006:0007]: Connection: close > 00000003:http_proxy.srvhdr[0006:0007]: Date: Tue, 30 Mar 2010 14:55:56 GMT > 00000003:http_proxy.srvhdr[0006:0007]: Server: Microsoft-IIS/6.0 > 00000003:http_proxy.srvhdr[0006:0007]: X-Powered-By: ASP.NET > 00000003:http_proxy.srvhdr[0006:0007]: Content-Type: image/png > 00000003:http_proxy.srvcls[0006:0007] > 00000003:http_proxy.clicls[0006:0007] > 00000003:http_proxy.closed[0006:0007] > 00000004:services.accept(0005)=0006 from [24.99.160.41:52846] > 00000004:services.srvcls[0006:0007] > 00000004:services.clicls[0006:0007] > 00000004:services.closed[0006:0007] > 00000005:services.accept(0005)=0006 from [24.99.160.41:52847] > > On Tue, Mar 30, 2010 at 10:19 AM, Amanda Machutta <[email protected]>wrote: > >> Hi Malte, >> >> I didn't compile HAProxy with any options. Just did a basic make. I am >> using iptables to control access to the machine, not sure about ip_conntrack >> - an lsmod shows I'm using nf_conntrack. Is there something I should check >> that would cause issues with HAProxy? >> >> Here are the results of a straight tcpdump on port 80 through one request >> of the webpage. As you can see there is a 16s pause in there, but I'm having >> a hard time tracking down why...any ideas? >> >> [r...@rs-globallb1 log]# tcpdump port 80 -v >> tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 96 >> bytes >> 14:12:43.538443 IP (tos 0x0, ttl 112, id 18932, offset 0, flags [DF], >> proto: TCP (6), length: 48) c-24-99-160-41.hsd1.ga.comcast.net.52283 > >> rs-globallb1.http: S, cksum 0xcd02 (correct), 1742211419:1742211419(0) win >> 8192 <mss 1460,nop,nop,sackOK> >> 14:12:43.538443 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto: >> TCP (6), length: 48) rs-globallb1.http > >> c-24-99-160-41.hsd1.ga.comcast.net.52283: S, cksum 0x1871 (correct), >> 2819233190:2819233190(0) ack 1742211420 win 5840 <mss 1460,nop,nop,sackOK> >> 14:12:43.568443 IP (tos 0x0, ttl 112, id 18933, offset 0, flags [DF], >> proto: TCP (6), length: 40) c-24-99-160-41.hsd1.ga.comcast.net.52283 > >> rs-globallb1.http: ., cksum 0x1795 (correct), ack 1 win 17520 >> 14:12:43.578443 IP (tos 0x0, ttl 112, id 18934, offset 0, flags [DF], >> proto: TCP (6), length: 491) c-24-99-160-41.hsd1.ga.comcast.net.52283 > >> rs-globallb1.http: P 1:452(451) ack 1 win 17520 >> 14:12:43.588443 IP (tos 0x0, ttl 64, id 45179, offset 0, flags [DF], >> proto: TCP (6), length: 293) rs-globallb1.http > >> c-24-99-160-41.hsd1.ga.comcast.net.52283: P 1:254(253) ack 452 win 6432 >> 14:12:43.588443 IP (tos 0x0, ttl 64, id 45180, offset 0, flags [DF], >> proto: TCP (6), length: 2960) rs-globallb1.http > >> c-24-99-160-41.hsd1.ga.comcast.net.52283: . 254:3174(2920) ack 452 win 6432 >> 14:12:43.638442 IP (tos 0x0, ttl 112, id 18936, offset 0, flags [DF], >> proto: TCP (6), length: 40) c-24-99-160-41.hsd1.ga.comcast.net.52283 > >> rs-globallb1.http: ., cksum 0x0f21 (correct), ack 1714 win 17520 >> 14:12:43.638442 IP (tos 0x0, ttl 64, id 45182, offset 0, flags [DF], >> proto: TCP (6), length: 1464) rs-globallb1.http > >> c-24-99-160-41.hsd1.ga.comcast.net.52283: P 3174:4598(1424) ack 452 win 6432 >> 14:12:43.638442 IP (tos 0x0, ttl 112, id 18937, offset 0, flags [DF], >> proto: TCP (6), length: 40) c-24-99-160-41.hsd1.ga.comcast.net.52283 > >> rs-globallb1.http: ., cksum 0x096d (correct), ack 3174 win 17520 >> 14:12:43.868438 IP (tos 0x0, ttl 112, id 18939, offset 0, flags [DF], >> proto: TCP (6), length: 40) c-24-99-160-41.hsd1.ga.comcast.net.52283 > >> rs-globallb1.http: ., cksum 0x096d (correct), ack 4598 win 16096 >> >> <!--- Always a 16s pause right here ---> >> >> 14:12:59.028186 IP (tos 0x0, ttl 64, id 45183, offset 0, flags [DF], >> proto: TCP (6), length: 576) rs-globallb1.http > >> c-24-99-160-41.hsd1.ga.comcast.net.52283: P 4598:5134(536) ack 452 win 6432 >> 14:12:59.028186 IP (tos 0x0, ttl 64, id 45184, offset 0, flags [DF], >> proto: TCP (6), length: 1088) rs-globallb1.http > >> c-24-99-160-41.hsd1.ga.comcast.net.52283: P 5134:6182(1048) ack 452 win 6432 >> 14:12:59.028186 IP (tos 0x0, ttl 64, id 45185, offset 0, flags [DF], >> proto: TCP (6), length: 1500) rs-globallb1.http > >> c-24-99-160-41.hsd1.ga.comcast.net.52283: . 6182:7642(1460) ack 452 win 6432 >> 14:12:59.028186 IP (tos 0x0, ttl 64, id 45186, offset 0, flags [DF], >> proto: TCP (6), length: 1428) rs-globallb1.http > >> c-24-99-160-41.hsd1.ga.comcast.net.52283: FP 7642:9030(1388) ack 452 win >> 6432 >> 14:12:59.068186 IP (tos 0x0, ttl 112, id 19023, offset 0, flags [DF], >> proto: TCP (6), length: 40) c-24-99-160-41.hsd1.ga.comcast.net.52283 > >> rs-globallb1.http: ., cksum 0xfdac (correct), ack 6182 win 17520 >> 14:12:59.068186 IP (tos 0x0, ttl 112, id 19024, offset 0, flags [DF], >> proto: TCP (6), length: 40) c-24-99-160-41.hsd1.ga.comcast.net.52283 > >> rs-globallb1.http: ., cksum 0xfdab (correct), ack 9031 win 14672 >> 14:12:59.068186 IP (tos 0x0, ttl 112, id 19025, offset 0, flags [DF], >> proto: TCP (6), length: 40) c-24-99-160-41.hsd1.ga.comcast.net.52283 > >> rs-globallb1.http: ., cksum 0xf7f7 (correct), ack 9031 win 16132 >> 14:12:59.068186 IP (tos 0x0, ttl 112, id 19026, offset 0, flags [DF], >> proto: TCP (6), length: 40) c-24-99-160-41.hsd1.ga.comcast.net.52283 > >> rs-globallb1.http: F, cksum 0xf7f6 (correct), 452:452(0) ack 9031 win 16132 >> 14:12:59.068186 IP (tos 0x0, ttl 64, id 45187, offset 0, flags [DF], >> proto: TCP (6), length: 40) rs-globallb1.http > >> c-24-99-160-41.hsd1.ga.comcast.net.52283: ., cksum 0x1ddb (correct), ack 453 >> win 6432 >> 14:12:59.718177 IP (tos 0x0, ttl 112, id 19032, offset 0, flags [DF], >> proto: TCP (6), length: 48) c-24-99-160-41.hsd1.ga.comcast.net.52289 > >> rs-globallb1.http: S, cksum 0x1c74 (correct), 2671280771:2671280771(0) win >> 8192 <mss 1460,nop,nop,sackOK> >> 14:12:59.718177 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto: >> TCP (6), length: 48) rs-globallb1.http > >> c-24-99-160-41.hsd1.ga.comcast.net.52289: S, cksum 0x94b2 (correct), >> 3075791243:3075791243(0) ack 2671280772 win 5840 <mss 1460,nop,nop,sackOK> >> 14:12:59.758177 IP (tos 0x0, ttl 112, id 19034, offset 0, flags [DF], >> proto: TCP (6), length: 40) c-24-99-160-41.hsd1.ga.comcast.net.52289 > >> rs-globallb1.http: ., cksum 0x93d6 (correct), ack 1 win 17520 >> 14:12:59.758177 IP (tos 0x0, ttl 112, id 19035, offset 0, flags [DF], >> proto: TCP (6), length: 501) c-24-99-160-41.hsd1.ga.comcast.net.52289 > >> rs-globallb1.http: P 1:462(461) ack 1 win 17520 >> 14:12:59.768177 IP (tos 0x0, ttl 64, id 30115, offset 0, flags [DF], >> proto: TCP (6), length: 385) rs-globallb1.http > >> c-24-99-160-41.hsd1.ga.comcast.net.52289: P 1:346(345) ack 462 win 6432 >> 14:12:59.768177 IP (tos 0x0, ttl 64, id 30116, offset 0, flags [DF], >> proto: TCP (6), length: 40) rs-globallb1.http > >> c-24-99-160-41.hsd1.ga.comcast.net.52289: F, cksum 0xbbff (correct), >> 346:346(0) ack 462 win 6432 >> 14:12:59.808176 IP (tos 0x0, ttl 112, id 19036, offset 0, flags [DF], >> proto: TCP (6), length: 40) c-24-99-160-41.hsd1.ga.comcast.net.52289 > >> rs-globallb1.http: ., cksum 0x9208 (correct), ack 347 win 17175 >> 14:12:59.808176 IP (tos 0x0, ttl 112, id 19037, offset 0, flags [DF], >> proto: TCP (6), length: 40) c-24-99-160-41.hsd1.ga.comcast.net.52289 > >> rs-globallb1.http: F, cksum 0x9207 (correct), 462:462(0) ack 347 win 17175 >> 14:12:59.808176 IP (tos 0x0, ttl 64, id 30117, offset 0, flags [DF], >> proto: TCP (6), length: 40) rs-globallb1.http > >> c-24-99-160-41.hsd1.ga.comcast.net.52289: ., cksum 0xbbfe (correct), ack 463 >> win 6432 >> 14:13:00.098171 IP (tos 0x0, ttl 112, id 19040, offset 0, flags [DF], >> proto: TCP (6), length: 48) c-24-99-160-41.hsd1.ga.comcast.net.52290 > >> rs-globallb1.http: S, cksum 0xfc80 (correct), 3940964039:3940964039(0) win >> 8192 <mss 1460,nop,nop,sackOK> >> 14:13:00.098171 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto: >> TCP (6), length: 48) rs-globallb1.http > >> c-24-99-160-41.hsd1.ga.comcast.net.52290: S, cksum 0x776b (correct), >> 3079853729:3079853729(0) ack 3940964040 win 5840 <mss 1460,nop,nop,sackOK> >> 14:13:00.138170 IP (tos 0x0, ttl 112, id 19041, offset 0, flags [DF], >> proto: TCP (6), length: 40) c-24-99-160-41.hsd1.ga.comcast.net.52290 > >> rs-globallb1.http: ., cksum 0x768f (correct), ack 1 win 17520 >> 14:13:00.148170 IP (tos 0x0, ttl 112, id 19042, offset 0, flags [DF], >> proto: TCP (6), length: 510) c-24-99-160-41.hsd1.ga.comcast.net.52290 > >> rs-globallb1.http: P 1:471(470) ack 1 win 17520 >> 14:13:00.168170 IP (tos 0x0, ttl 64, id 31479, offset 0, flags [DF], >> proto: TCP (6), length: 212) rs-globallb1.http > >> c-24-99-160-41.hsd1.ga.comcast.net.52290: P 1:173(172) ack 471 win 6432 >> 14:13:00.168170 IP (tos 0x0, ttl 64, id 31480, offset 0, flags [DF], >> proto: TCP (6), length: 2960) rs-globallb1.http > >> c-24-99-160-41.hsd1.ga.comcast.net.52290: . 173:3093(2920) ack 471 win 6432 >> 14:13:00.208169 IP (tos 0x0, ttl 112, id 19044, offset 0, flags [DF], >> proto: TCP (6), length: 40) c-24-99-160-41.hsd1.ga.comcast.net.52290 > >> rs-globallb1.http: ., cksum 0x6e59 (correct), ack 1633 win 17520 >> 14:13:00.208169 IP (tos 0x0, ttl 64, id 31482, offset 0, flags [DF], >> proto: TCP (6), length: 2960) rs-globallb1.http > >> c-24-99-160-41.hsd1.ga.comcast.net.52290: . 3093:6013(2920) ack 471 win 6432 >> 14:13:00.208169 IP (tos 0x0, ttl 64, id 31484, offset 0, flags [DF], >> proto: TCP (6), length: 1478) rs-globallb1.http > >> c-24-99-160-41.hsd1.ga.comcast.net.52290: F 6013:7451(1438) ack 471 win 6432 >> 14:13:00.208169 IP (tos 0x0, ttl 112, id 19045, offset 0, flags [DF], >> proto: TCP (6), length: 40) c-24-99-160-41.hsd1.ga.comcast.net.52290 > >> rs-globallb1.http: R, cksum 0xad11 (correct), 471:471(0) ack 3093 win 0 >> >> On Tue, Mar 30, 2010 at 9:37 AM, Malte Geierhos <[email protected]>wrote: >> >>> Hi Amanda, >>> >>> What about the rest of my questions ? >>> With what options did you compile HaProxy ? >>> Do you've run a tcpdump / Wireshark ? >>> I think this will help a lot to see where your slowdown comes from... >>> >>> And specially : what about iptables / ip_conntrack ? This can slow down >>> things a lot ... >>> kind regards, >>> Malte >>> >>> Hi Malte, >>> >>> Thanks for your response. We switched to mode tcp while trying to >>> troubleshoot the load times. Originally my configuration was set for http, >>> and switching it to mode tcp seemed to speed things up a bit. >>> >>> Some additional information: >>> -Running HAProxy version 1.4.1 on a CentOS 5.4 system. >>> -Nothing else is configured in the file. I posted the whole config file. >>> >>> Thanks, >>> Amanda >>> >>> On Tue, Mar 30, 2010 at 9:04 AM, Malte Geierhos <[email protected]>wrote: >>> >>>> Hello Amanda, >>>> >>>> I'm wondering why you're using "mode tcp" for http loadbalancing ? >>>> Besides that - i don't see anything unusual in your config. >>>> Of course if you've got a recent kernel (> 2.6.28) , you can try setting >>>> "option splice-response" if your haproxy is compiled with >>>> "USE_LINUX_SPLICE=1" >>>> But more interesting : >>>> Which HaProxy version do you use on what System ? >>>> How many Req/s do you expect ? >>>> Anything else configured on your loadbalancer ? >>>> Such as ip_conntrack enabled or similar ? >>>> Did you change something in your sysctl's ? >>>> >>>> Kind regards, >>>> Malte >>>> >>>> Am 30.03.10 14:42, schrieb Amanda Machutta: >>>> >>>> > Hello All, >>>> >>>> > >>>> >>>> > I've completely new to HAProxy and have come up with a basic >>>> >>>> > configuration for my haproxy server to test with. I have it >>>> >>>> > loadbalancing port 80 for our websites, which load flash >>>> >>>> > applications, as well as port 1312 which is used for some backend >>>> >>>> > services. In testing, I'm noticing that my webpage is loading >>>> really >>>> >>>> > slow. So slow that I had to kick up my timeouts just to get it to >>>> >>>> > finish loading. When I load directly through the webserver, I do >>>> not >>>> >>>> > see this issue. Here is a copy of my config. Any input would be >>>> >>>> > greatly appreciated. >>>> >>>> > >>>> >>>> > global >>>> >>>> > maxconn 4096 # Total Max Connections. This is >>>> dependent >>>> >>>> > on ulimit >>>> >>>> > daemon >>>> >>>> > nbproc 4 # Number of processing cores. Dual >>>> Dual-core >>>> >>>> > Opteron is 4 cores for example. >>>> >>>> > log 127.0.0.1 local0 debug >>>> >>>> > >>>> >>>> > defaults >>>> >>>> > mode http >>>> >>>> > clitimeout 150000 >>>> >>>> > srvtimeout 30000 >>>> >>>> > contimeout 4000 >>>> >>>> > log global >>>> >>>> > #option httpclose # Disable Keepalive >>>> >>>> > >>>> >>>> > listen http_proxy x.x.x.x:80 >>>> >>>> > mode tcp >>>> >>>> > balance roundrobin >>>> >>>> > option httpchk >>>> >>>> > option tcpka >>>> >>>> > >>>> >>>> > ## Define your servers to balance >>>> >>>> > server rs-webserver1 x.x.x.215:80 >>>> >>>> > server rs-webserver2 x.x.x.216:80 >>>> >>>> > server rs-webserver3 x.x.x.136:80 >>>> >>>> > server rs-webserver4 x.x.x.220:80 >>>> >>>> > server rs-webserver5 x.x.x.126:80 >>>> >>>> > >>>> >>>> > listen services x.x.x.x:1312 >>>> >>>> > mode tcp >>>> >>>> > balance roundrobin >>>> >>>> > option tcpka >>>> >>>> > ## Define your servers to balance >>>> >>>> > server rs-webserver1 x.x.x.217:1312 >>>> >>>> > server rs-webserver2 x.x.x.216:1312 >>>> >>>> > server rs-webserver3 x.x.x.136:1312 >>>> >>>> > server rs-webserver4 x.x.x.220:1312 >>>> >>>> > server rs-webserver5 x.x.x.126:1312 >>>> >>>> > >>>> >>>> > Thanks, >>>> >>>> > Amanda >>>> >>>> > >>>> >>>> > >>>> >>>> > -- >>>> >>>> > ´¨) >>>> __o >>>> >>>> > .·´ .·´¨) ¸.·´¨) _'\< . >>>> >>>> > (¸.·´ (¸.·´ (¸.·´¨¨ Amanda ¨¨¨¨¨¨( * ) ( ) >>>> >>> >>> >> >> >> -- >> ´¨) __o >> .·´ .·´¨) ¸.·´¨) _'\< . >> (¸.·´ (¸.·´ (¸.·´¨¨ Amanda ¨¨¨¨¨¨( * ) ( ) >> > > > > -- > ´¨) __o > .·´ .·´¨) ¸.·´¨) _'\< . > (¸.·´ (¸.·´ (¸.·´¨¨ Amanda ¨¨¨¨¨¨( * ) ( ) >
-- ´¨) __o .·´ .·´¨) ¸.·´¨) _'\< . (¸.·´ (¸.·´ (¸.·´¨¨ Amanda ¨¨¨¨¨¨( * ) ( )
