[Bug 60283] Commons connectivity issue in Hong-Kong
https://bugzilla.wikimedia.org/show_bug.cgi?id=60283 Andre Klapper aklap...@wikimedia.org changed: What|Removed |Added Status|UNCONFIRMED |RESOLVED Resolution|--- |WORKSFORME --- Comment #32 from Andre Klapper aklap...@wikimedia.org --- Closing as WORKSFORME for the time being, as there isn't enough information to act here (see comment 29). Anybody please feel free to reopen once information is available. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 60283] Commons connectivity issue in Hong-Kong
https://bugzilla.wikimedia.org/show_bug.cgi?id=60283 --- Comment #31 from Deryck Chan deryckc...@gmail.com --- (In reply to Andre Klapper from comment #30) (In reply to Faidon Liambotis from comment #29) it'd be interesting to hear if Deryck still has issues (if ever). Deryck: ping? Haven't tried big uploads since February, sorry! Also I spent half a year in northeastern China (aka Manchuria) which had really really slow internet so Hong Kong suddenly felt very fast when I returned home. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 60283] Commons connectivity issue in Hong-Kong
https://bugzilla.wikimedia.org/show_bug.cgi?id=60283 --- Comment #30 from Andre Klapper aklap...@wikimedia.org --- (In reply to Faidon Liambotis from comment #29) it'd be interesting to hear if Deryck still has issues (if ever). Deryck: ping? -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 60283] Commons connectivity issue in Hong-Kong
https://bugzilla.wikimedia.org/show_bug.cgi?id=60283 --- Comment #27 from Kelson [Emmanuel Engelhart] kel...@kiwix.org --- Some feedback on this? or do you need more information from the field? -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 60283] Commons connectivity issue in Hong-Kong
https://bugzilla.wikimedia.org/show_bug.cgi?id=60283 --- Comment #28 from jeremyb bugzilla+org.wikime...@tuxmachine.com --- (In reply to Kelson [Emmanuel Engelhart] from comment #27) Some feedback on this? or do you need more information from the field? comment 25 (and 26 too) was a trace from a user's home connection. I focused too much on the quality of the trace tool and didn't ask if that was actually a connection with demonstrated slow uploads. Deryck, have you seen slow uploads recently? -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 60283] Commons connectivity issue in Hong-Kong
https://bugzilla.wikimedia.org/show_bug.cgi?id=60283 --- Comment #29 from Faidon Liambotis fai...@wikimedia.org --- I heard back from PCCW; the asymmetric peering in Ashburn is apparently per their policy so it won't change. It shouldn't matter for the purposes of this bug or at all, though. We also peered with HKBN directly since then, although this shouldn't matter much either, since our transits are far from being congested. Nevertheless, it'd be interesting to hear if Deryck still has issues (if ever). Other than that, nothing else I can comment on. Yes, I'd like more information from the field in the sense that I'd like to hear from users that actually experience the issue and gather information from them, as stated above (name of the ISP, traceroutes). -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 60283] Commons connectivity issue in Hong-Kong
https://bugzilla.wikimedia.org/show_bug.cgi?id=60283
--- Comment #26 from Deryck Chan deryckc...@gmail.com ---
As suggested by jeremyb: (done via HKBN domestic, same as above)
deryck@Xenon-RC:~$ for dc in ulsfo eqiad esams; do echo ${dc}:; mtr -rc 10
text-lb.${dc}.wikimedia.org; echo; done
ulsfo:
HOST: Xenon-RCLoss% Snt Last Avg Best Wrst StDev
1.|-- 014198068001.ctinets.com 0.0%101.3 1.3 0.8 2.8 0.6
2.|-- 061093141033.ctinets.com 0.0%100.8 1.0 0.7 2.9 0.7
3.|-- 014199252121.ctinets.com 0.0%10 11.0 8.1 3.3 12.8 3.6
4.|-- 014136128054.ctinets.com 0.0%102.0 1.8 1.6 2.0 0.1
5.|-- las-bb1-link.telia.net 0.0%10 180.2 180.3 180.0 181.1 0.3
6.|-- ntt-ic-151170-las-bb1.c.t 0.0%10 180.9 180.9 180.7 181.5 0.2
7.|-- ae-6.r21.lsanca03.us.bb.g 0.0%10 182.1 186.9 182.1 206.7 9.1
8.|-- ??? 100.0100.0 0.0 0.0 0.0 0.0
9.|-- ae-1.r06.snjsca04.us.bb.g 0.0%10 182.2 182.2 181.5 183.0 0.6
10.|-- xe-0-2-0-10.r06.snjsca04. 0.0%10 178.4 180.2 177.4 199.2 6.7
11.|-- text-lb.ulsfo.wikimedia.o 10.0%10 177.7 178.1 177.6 179.1 0.6
eqiad:
HOST: Xenon-RCLoss% Snt Last Avg Best Wrst StDev
1.|-- 014198068001.ctinets.com 0.0%101.0 1.1 0.8 1.7 0.3
2.|-- 061093141033.ctinets.com 0.0%100.9 2.5 0.7 17.4 5.2
3.|-- 014199252121.ctinets.com 0.0%10 10.1 6.2 2.4 10.1 3.1
4.|-- 014136128058.ctinets.com 0.0%101.8 3.7 1.7 20.2 5.8
5.|-- las-bb1-link.telia.net 0.0%10 176.4 176.6 176.4 176.7 0.1
6.|-- ntt-ic-143926-las-bb1.c.t 0.0%10 180.7 180.7 180.5 180.9 0.1
7.|-- ae-6.r21.lsanca03.us.bb.g 0.0%10 178.5 185.9 178.5 246.6 21.4
8.|-- ae-2.r20.asbnva02.us.bb.g 10.0%10 243.3 245.5 242.4 256.5 4.6
9.|-- ae-1.r04.asbnva02.us.bb.g 0.0%10 228.7 230.4 227.5 234.3 2.9
10.|-- xe-0-7-0-8.r04.asbnva02.u 10.0%10 236.3 236.8 233.8 242.1 3.1
11.|-- text-lb.eqiad.wikimedia.o 10.0%10 234.6 237.3 234.0 240.9 2.5
esams:
HOST: Xenon-RCLoss% Snt Last Avg Best Wrst StDev
1.|-- 014198068001.ctinets.com 0.0%100.9 1.4 0.9 5.0 1.3
2.|-- 061093141033.ctinets.com 0.0%101.0 0.8 0.6 1.0 0.1
3.|-- 014199252121.ctinets.com 0.0%10 10.4 7.7 3.8 12.5 3.3
4.|-- 014136128046.ctinets.com 20.0%101.9 1.8 1.6 2.0 0.1
5.|-- sjo-bb1-link.telia.net 0.0%10 143.0 142.9 142.7 143.1 0.1
6.|-- nyk-bb2-link.telia.net 0.0%10 223.6 241.6 223.6 322.3 37.5
7.|-- ldn-bb2-link.telia.net 0.0%10 289.3 289.2 289.1 289.5 0.1
8.|-- adm-bb4-link.telia.net 0.0%10 308.9 308.5 308.3 308.9 0.2
9.|-- adm-b5-link.telia.net 10.0%10 294.8 294.8 294.7 295.0 0.1
10.|-- wikimedia-ic-129908-adm-b 10.0%10 250.1 245.7 244.6 250.1 1.8
11.|-- text-lb.esams.wikimedia.o 10.0%10 246.6 244.1 243.3 246.6 1.0
--
You are receiving this mail because:
You are the assignee for the bug.
You are on the CC list for the bug.
___
Wikibugs-l mailing list
Wikibugs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 60283] Commons connectivity issue in Hong-Kong
https://bugzilla.wikimedia.org/show_bug.cgi?id=60283 --- Comment #14 from Faidon Liambotis fai...@wikimedia.org --- I won't comment if they or anyone else is trash and please let's avoid having this sort of discussion here (or at all -- do you really think it's good for our peering relationships to have public discussions about which vendor/peer X is trash?). This is a bug report, let's stick to the facts: please contribute data (pings, traceroutes, BGP AS paths, looking glasses that show problematic paths, error messages etc.) about the problem so that we can diagnose it and then evaluate solutions to fix/work around it. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 60283] Commons connectivity issue in Hong-Kong
https://bugzilla.wikimedia.org/show_bug.cgi?id=60283 --- Comment #15 from Pavel Selitskas [wizardist] p.selits...@gmail.com --- (In reply to comment #12) Pavel, paths to where and which lgs specifically? We peer directly with Hurricane Electric in all locations and we're one hop to PCCW via GTT (which carries it from Hong Kong to San Jose, then SFO). South Korea is long gone, but we serve Hong Kong via San Francisco (ulsfo) now. As for your question, yes, the file will go to the local caching center and then proxied silently (by Varnish) to Ashburn caches and appservers. I checked Hurricane Electric and PCCW (http://lookingglass.pccwglobal.com/). Hurricane Electric peers directly right like you said: 154 ms69 ms53 ms10ge3-1.core1.tyo1.he.net (184.105.222.105) 2150 ms150 ms150 ms10ge15-2.core1.lax2.he.net (184.105.223.105) 3168 ms159 ms165 ms10ge9-5.core1.sjc2.he.net (184.105.213.6) 4160 ms170 ms168 ms10ge5-2.core1.pao1.he.net (72.52.92.69) 5161 ms166 ms171 mseqix-sv9.wikimedia.org (198.32.176.214) 6161 ms164 ms161 mstext-lb.ulsfo.wikimedia.org (198.35.26.96) Looking through PCCW LG, I can see like they route requests to eqiad (although ulsfo would be the best choice for East Asia? - that is how they resolve commons.wikimedia.org through their DNS): traceroute to text-lb.eqiad.wikimedia.org (208.80.154.224), 64 hops max, 44 byte packets 1 bbs-1-250-0-210.on-nets.com (210.0.250.1) 0.373 ms 0.332 ms 0.320 ms 2 10.2.128.31 (10.2.128.31) 0.226 ms 0.228 ms 0.262 ms 3 203.131.243.121 (203.131.243.121) 1.251 ms 1.245 ms 2.749 ms 4 ae-3.r22.tkokhk01.hk.bb.gin.ntt.net (129.250.6.232) 22.838 ms 13.478 ms 1.011 ms 5 ae-12.r22.osakjp02.jp.bb.gin.ntt.net (129.250.6.234) 60.219 ms 58.072 ms 51.183 ms 6 ae-8.r21.osakjp02.jp.bb.gin.ntt.net (129.250.6.193) 46.586 ms 41.789 ms 44.033 ms 7 * * * 8 ae-8.r21.sttlwa01.us.bb.gin.ntt.net (129.250.6.141) 169.446 ms 193.944 ms 192.196 ms 9 ae-5.r21.asbnva02.us.bb.gin.ntt.net (129.250.4.181) 231.650 ms 221.329 ms 231.565 ms 10 ae-2.r04.asbnva02.us.bb.gin.ntt.net (129.250.4.207) 211.152 ms 247.144 ms 248.435 ms [ * * * repeats infinitely ] -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 60283] Commons connectivity issue in Hong-Kong
https://bugzilla.wikimedia.org/show_bug.cgi?id=60283 --- Comment #16 from Nemo federicol...@tiscali.it --- Yes sorry, I shouldn't have saved the comment above. Deryck and others in HK, let people know here or on IRC (e.g. #wikimedia-tech) if you need help posting more information of the kind requested. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 60283] Commons connectivity issue in Hong-Kong
https://bugzilla.wikimedia.org/show_bug.cgi?id=60283 vitu...@it.wiki vituzzu.w...@gmail.com changed: What|Removed |Added CC||vituzzu.w...@gmail.com --- Comment #17 from vitu...@it.wiki vituzzu.w...@gmail.com --- (In reply to comment #4) I disagree that this is an unsolvable problem: it needs to be properly debugged and measured. Hong Kong has a world top20 IX according to [[List of Internet exchange points by size]], we're not talking of the moon or an island in the middle of the ocean. It's surely necessary to attach some traceroute data from various machines/cities to the WMF servers, to start with, because there are several links: http://www.glif.is/publications/maps/GLIF_5-11_AP_2k.jpg (this website via multichill). We need to know which is/are being used and what's the latency. Then, can someone point out a procedure to measure speed/quality of down/up connections to WMF servers compared to, say, http://ftp.hk.debian.org/debian/ or whichever fast local server there is? Hosting a big IX doesn't imply almost a thing about overall infrastructure nor about the relevant (by our perspective) one. This kind of issue is actually pretty common (I've already experienced a horrible increase in delay from AS3269 towards Equinix) but as a final possibility I won't exclude BGP hijack. Vito -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 60283] Commons connectivity issue in Hong-Kong
https://bugzilla.wikimedia.org/show_bug.cgi?id=60283 --- Comment #18 from Pavel Selitskas [wizardist] p.selits...@gmail.com --- I'm sorry, that was a traceroute from HGC servers. PCCW is as follows (also resolves to Ashburn entry point), just to keep things properly: Tracing the route to text-lb.eqiad.wikimedia.org (208.80.154.224) 1 pos6-0.cr02.ams01.pccwbtn.net (63.218.64.102) [MPLS: Label 42 Exp 0] 4 msec pos1-0-1.cr02.ams01.pccwbtn.net (63.218.64.122) [MPLS: Label 42 Exp 0] 52 msec pos1-0.cr02.ams01.pccwbtn.net (63.218.64.98) [MPLS: Label 42 Exp 0] 0 msec 2 TenGE12-2.br02.ams01.pccwbtn.net (63.218.64.110) 20 msec 8 msec 0 msec 3 xe-0-1-0.cir1.amsterdam2-nh.nl.xo.net (195.69.145.200) 48 msec 4 msec 0 msec 4 te0-3-4-0.rar3.washington-dc.us.xo.net (207.88.13.198) [AS 2828] 92 msec 100 msec 88 msec 5 ae0d1.cir1.ashburn-va.us.xo.net (207.88.13.65) [AS 2828] 84 msec 84 msec 84 msec -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 60283] Commons connectivity issue in Hong-Kong
https://bugzilla.wikimedia.org/show_bug.cgi?id=60283 matanya mata...@foss.co.il changed: What|Removed |Added CC||mata...@foss.co.il --- Comment #19 from matanya mata...@foss.co.il --- Trace from equinix in HK: traceroute -A commons.wikimedia.org 1 202.177.199.133 (202.177.199.133) [AS17819/AS10138] 0.472 ms 0.477 ms 0.471 ms 2 ge-1-2-9.gw1502.hk1.ap.equinix.com (27.111.175.187) [AS17819] 0.437 ms 0.443 ms 0.437 ms 3 EQX-0022.gw1.hkg3.asianetcom.net (203.192.153.25) [AS3549] 1.009 ms 1.015 ms 1.011 ms 4 te0-4-0-1.wr1.hkg0.asianetcom.net (61.14.157.80) [AS10026] 5.194 ms 2.794 ms 2.791 ms 5 te0-1-0-3.gw1.lax3.asianetcom.net (202.147.61.193) [AS10026/AS3549/AS1221] 149.940 ms 150.086 ms 150.079 ms 6 be1.gw2.lax3.asianetcom.net (202.147.61.162) [AS10026/AS3549/AS1221] 150.042 ms 150.213 ms 150.277 ms 7 * * * -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 60283] Commons connectivity issue in Hong-Kong
https://bugzilla.wikimedia.org/show_bug.cgi?id=60283 Tei oscar.vi...@gmail.com changed: What|Removed |Added CC||oscar.vi...@gmail.com --- Comment #20 from Tei oscar.vi...@gmail.com --- I suggest solutions: - Allow the upload from third party servers. So people uploading a video can upload it to a local server, then give the url to wikimedia. There the wikimedia server can put the download in a queue, download at maybe very slow rate. - Allow the upload with a custom protocol. Maybe a java plugin could implement a upload mechanism for videos using a conexion that can survive low bandwith / high errors conditions?. The general idea is browsers suck for uploading large files. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 60283] Commons connectivity issue in Hong-Kong
https://bugzilla.wikimedia.org/show_bug.cgi?id=60283 --- Comment #21 from Pavel Selitskas [wizardist] p.selits...@gmail.com --- (In reply to comment #20) I suggest solutions: - Allow the upload from third party servers. So people uploading a video can upload it to a local server, then give the url to wikimedia. There the wikimedia server can put the download in a queue, download at maybe very slow rate. - Allow the upload with a custom protocol. Maybe a java plugin could implement a upload mechanism for videos using a conexion that can survive low bandwith / high errors conditions?. The general idea is browsers suck for uploading large files. We haven't had any [already requested by Andre Klapper] input on chunked uploads yet. Until we put heavy artillery like client-side Java on the battlefield, let's wait for folks from Hong Kong to check out chunked method first. To users facing the issue and willing to test the feature: go to your settings on Commons, pick the Upload Wizard tub, and enable Chunked uploads for files over 1MB in Upload Wizard. Then try to upload a large file and tell us how did it go. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 60283] Commons connectivity issue in Hong-Kong
https://bugzilla.wikimedia.org/show_bug.cgi?id=60283 --- Comment #22 from Faidon Liambotis fai...@wikimedia.org --- Pavel, thanks a lot for going through the trouble of collecting data via looking glasses, although I'd personally be more interested to get traceroutes from users that experience the issues. I doubt PCCW's backbone has bandwidth troubles with us. HK wasn't pointed to ulsfo due to lack of data from probes there, but it makes sense from a geographical/network sense, so I just did that. That being said, I don't expect huge differences in bandwidth, as the path was going via our carriers to Hong Kong, which I doubt that are going to be congested (i.e. if it enter's NTT's network in HK, it doesn't matter if it ends up to our eqiad or ulsfo node, from a bandwidth perspective). Oh, and finally, the third traceroute you sent was actually from Amsterdam -- first hop is ams, plus 84ms from HK to Ashburn is 2-3 times the speed of light or something :) -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 60283] Commons connectivity issue in Hong-Kong
https://bugzilla.wikimedia.org/show_bug.cgi?id=60283 --- Comment #23 from Pavel Selitskas [wizardist] p.selits...@gmail.com --- (In reply to comment #22) Pavel, thanks a lot for going through the trouble of collecting data via looking glasses, although I'd personally be more interested to get traceroutes from users that experience the issues. I doubt PCCW's backbone has bandwidth troubles with us. Yes, they can adjust priority for traffic coming from different consumers, but that would imply issues had Hong Kong only a couple of 1Gb links to the West Coast. :) Oh, and finally, the third traceroute you sent was actually from Amsterdam -- first hop is ams, plus 84ms from HK to Ashburn is 2-3 times the speed of light or something :) Yeah, sorry, my bad. 3 ge-7-0-0.hkg11.ip4.tinet.net (213.254.227.77) 4 msec 0 msec 4 msec 4 xe-11-0-1.was10.ip4.tinet.net (141.136.110.198) 232 msec 232 msec 236 msec Here's a part of a proper trace from PCCW. Nothing special at first glance: it moors to Seattle, although they route traffic through Tinet SpA, and it takes quite a large amount of time (~230ms vs standard ~150ms). If you look at their backbone network map, you can see that Tinet doesn't have submarine links coming from Asia directly to Washington - they're all terminated in California. I don't know why they make it this way, could their maps be just outdated :) Their ping result a half more than normal is not okay though. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 60283] Commons connectivity issue in Hong-Kong
https://bugzilla.wikimedia.org/show_bug.cgi?id=60283 --- Comment #24 from Faidon Liambotis fai...@wikimedia.org --- Tinet/GTT is our transit and Hong Kong to Ashburn is definitely not 150ms (it's 160ms+ from HK to the west coast + 70ms coast-to-coast). In fact, if you do an On-net ping from PCCW's lg from HKG to ASH you get 224ms, not that different at all. The path from PCCW to eqiad is suboptimal, though, since we peer directly with PCCW there, so they're probably filtering by mistake (the outbound path is correct and going directly, though). I'll ask them. This matters very little though, since I switched HK to ulsfo now, and we don't peer with PCCW there (since we can't -- they're not present in PAIX). We have two equivalent AS paths to PCCW from/to ulsfo and TiNet wins, and while I could traffic engineer it to go via NTT, I'd rather not, unless we have confirmed there's an issue and this would help. So far I have zero indications suggesting that, we're not even sure if the users having issues connect via PCCW. So, again, I'd like traceroutes from users that *actually* experience an issue, not just traceroute from large carriers' looking glasses. These are not very useful for the purpose of this bug report. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 60283] Commons connectivity issue in Hong-Kong
https://bugzilla.wikimedia.org/show_bug.cgi?id=60283 --- Comment #25 from Deryck Chan deryckc...@gmail.com --- tracepath from here (HKBN domestic): deryck@Xenon-RC:~$ tracepath commons.wikimedia.org 1: 014198068245.ctinets.com 0.211ms pmtu 1500 1: 014198068001.ctinets.com 2.332ms 1: 014198068001.ctinets.com 2.272ms 2: 061093141033.ctinets.com 2.123ms 3: 014199252121.ctinets.com 13.955ms 4: 014136128054.ctinets.com 3.554ms asymm 5 5: las-bb1-link.telia.net 165.887ms asymm 6 6: ntt-ic-143926-las-bb1.c.telia.net 182.466ms asymm 7 7: ae-6.r21.lsanca03.us.bb.gin.ntt.net 184.130ms asymm 8 8: no reply 9: ae-1.r06.snjsca04.us.bb.gin.ntt.net 184.650ms asymm 7 10: no reply 11: no reply 12: no reply 13: no reply [seems to loop indefinitely] -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 60283] Commons connectivity issue in Hong-Kong
https://bugzilla.wikimedia.org/show_bug.cgi?id=60283 Andre Klapper aklap...@wikimedia.org changed: What|Removed |Added Priority|Unprioritized |Low -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 60283] Commons connectivity issue in Hong-Kong
https://bugzilla.wikimedia.org/show_bug.cgi?id=60283 Faidon Liambotis fai...@wikimedia.org changed: What|Removed |Added CC||fai...@wikimedia.org --- Comment #8 from Faidon Liambotis fai...@wikimedia.org --- We have multiple 10Gbps links and plenty of surplus capacity with multiple Tier1/2 providers that have excellent connectivity to Hong Kong Asia in particular (some were explicitly procured with this criteria). Nevertheless, it could be some transient problem with some specific ISP that we could raise with them or our providers, or alter the path to go from one of our other carriers or to a different datacenter of ours. However, it's completely impossible for us to do any kind of troubleshooting without any data other than coming from Hong Kong. Please provide ISP names, IP addresses/networks that experience issues (anonymized will do, or pristine in private if you'd prefer that) as well as pings traceroutes to the Wikimedia sites in question. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 60283] Commons connectivity issue in Hong-Kong
https://bugzilla.wikimedia.org/show_bug.cgi?id=60283 --- Comment #9 from jeremyb bugzilla+org.wikime...@tuxmachine.com --- (In reply to comment #2) But, that commons fails (generate an error) or is too slow; this is IMO as bad. Could use some clarification about this as well. Is there an actual error message? Or is it just so slow you give up? (as Faidon says we can't do much with coming from Hong Kong we also can't do much with generates an error) Is it a consistent experience or are some times extra bad? For all of the above (error messages, slow uploads, slow page loads, pings, traceroutes, etc.), please also provide timestamps. (and timezone!) -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 60283] Commons connectivity issue in Hong-Kong
https://bugzilla.wikimedia.org/show_bug.cgi?id=60283 Pavel Selitskas [wizardist] p.selits...@gmail.com changed: What|Removed |Added CC||p.selits...@gmail.com --- Comment #10 from Pavel Selitskas [wizardist] p.selits...@gmail.com --- (In reply to comment #8) We have multiple 10Gbps links and plenty of surplus capacity with multiple Tier1/2 providers that have excellent connectivity to Hong Kong Asia in particular (some were explicitly procured with this criteria). Nevertheless, it could be some transient problem with some specific ISP that we could raise with them or our providers, or alter the path to go from one of our other carriers or to a different datacenter of ours. However, it's completely impossible for us to do any kind of troubleshooting without any data other than coming from Hong Kong. Please provide ISP names, IP addresses/networks that experience issues (anonymized will do, or pristine in private if you'd prefer that) as well as pings traceroutes to the Wikimedia sites in question. Public lg's in Hong Kong show appropriate paths (via Japan), but different providers show different perfomance (from Hurricane's 150ms to PCCW's 200+ms), not mentioning quite uncommon jitter. What's up with Wikimedia's infrastructure in South Korea? Is it still up? If so, is it utilized for HTML caches? What about uploading? (Just a question for the ops: as a user from Europe, does the file go to the European cluster and then being proxied silently to the US when I start uploading?) -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 60283] Commons connectivity issue in Hong-Kong
https://bugzilla.wikimedia.org/show_bug.cgi?id=60283 Max Semenik maxsem.w...@gmail.com changed: What|Removed |Added CC||maxsem.w...@gmail.com --- Comment #11 from Max Semenik maxsem.w...@gmail.com --- (In reply to comment #10) What's up with Wikimedia's infrastructure in South Korea? Is it still up? If so, is it utilized for HTML caches? Gone ages ago. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 60283] Commons connectivity issue in Hong-Kong
https://bugzilla.wikimedia.org/show_bug.cgi?id=60283 --- Comment #12 from Faidon Liambotis fai...@wikimedia.org --- Pavel, paths to where and which lgs specifically? We peer directly with Hurricane Electric in all locations and we're one hop to PCCW via GTT (which carries it from Hong Kong to San Jose, then SFO). South Korea is long gone, but we serve Hong Kong via San Francisco (ulsfo) now. As for your question, yes, the file will go to the local caching center and then proxied silently (by Varnish) to Ashburn caches and appservers. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 60283] Commons connectivity issue in Hong-Kong
https://bugzilla.wikimedia.org/show_bug.cgi?id=60283 --- Comment #13 from Nemo federicol...@tiscali.it --- (In reply to comment #12) We peer directly with Hurricane Electric in all locations and we're one hop to PCCW via GTT (which carries it from Hong Kong to San Jose, then SFO). Isn't HE known to be trash? Internet Archive was quite devastated due to HE till they added another carrier. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 60283] Commons connectivity issue in Hong-Kong
https://bugzilla.wikimedia.org/show_bug.cgi?id=60283 --- Comment #7 from Deryck Chan deryckc...@gmail.com --- Speedtests: (from my 10Mbps domestic connection in a building where most other households have 100Mbps) Hong Kong (STC) D 9.4Mbps, U 9.4Mbps, ping 10ms Tokyo (Alocac) D 9.5Mbps, U 7.5Mbps, ping 67ms London (Namesco) D 4.9Mbps, U 3.1Mbps, ping 220ms San Francisco (Unwired) D 9.3Mbps, U 7.3Mbps, ping 195ms Irvine, CA (Fireline) D 8.2Mbps, U 4.9Mbps, ping 235ms Melbourne (Telstra) D 9.2Mbps, U 8.9Mbps, ping 171ms [[User:Tsugiko]] suggested further that: - Accessing Wikimedia sites is significantly slower than similar services in the USA; - Much faster upload speeds are achieved when he routed all his traffic to Wikimedia sites via a VPN in Japan. The second point he made would suggest that some improvement is possible by reconfiguring WMF's IP routing tables. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 60283] Commons connectivity issue in Hong-Kong
https://bugzilla.wikimedia.org/show_bug.cgi?id=60283 Andre Klapper aklap...@wikimedia.org changed: What|Removed |Added Status|NEW |UNCONFIRMED Ever confirmed|1 |0 --- Comment #1 from Andre Klapper aklap...@wikimedia.org --- I don't see how this is fixable via a bug report - this feels like something to discuss with ops instead, maybe. impossible to upload videos to Commons Are there specific examples available of chunked uploads from that area *failing*? The email only implies that it is slow and takes a while. (Organizers could also send the videos via post to the datacenter, if all goes wrong.) -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 60283] Commons connectivity issue in Hong-Kong
https://bugzilla.wikimedia.org/show_bug.cgi?id=60283 Kelson [Emmanuel Engelhart] kel...@kiwix.org changed: What|Removed |Added CC||deryckc...@gmail.com --- Comment #2 from Kelson [Emmanuel Engelhart] kel...@kiwix.org --- I'm not the uploader myself and I don't live in HK. I can't provide other examples. But, that commons fails (generate an error) or is too slow; this is IMO as bad. The consequence is exactly the same: Wikimania videos are on Youtube and not on our own video platform. This is an availability incident and indeed something to treat with operations. If Bugzilla is not the right place to track this, what would be a better place? -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 60283] Commons connectivity issue in Hong-Kong
https://bugzilla.wikimedia.org/show_bug.cgi?id=60283 --- Comment #3 from Deryck Chan deryckc...@gmail.com --- This isn't a Wikimedia bug. The same problem applies when you try to upload something from *any* domestic connection in Hong Kong to *any* USA-based website which doesn't have its dedicated worldwide network backbone. Pre-2007 it was impossible to do en.wp vandalism work from the Far East because the connection was too slow to do anything useful. When WMF staff first visited Hong Kong for CWMC 2006 / around Wikimania 2007, the first thing they exclaimed was I'm surprised how slow it is to load Wikipedia here. If we had infinite budget, we could build our own network backbone like Google did (hence YouTube was a sensible uploading option when Commons isn't). But sending large files to a foreign country over the internet is in general a luxury that only North America and Western Europe can afford. It is a problem, but not a Wikimedia bug, that we can't do the same between HK and WMF datacenters. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 60283] Commons connectivity issue in Hong-Kong
https://bugzilla.wikimedia.org/show_bug.cgi?id=60283 Nemo federicol...@tiscali.it changed: What|Removed |Added Keywords||ops CC||bugzilla+org.wikimedia@tuxm ||achine.com, ||federicol...@tiscali.it, ||maar...@mdammers.nl --- Comment #4 from Nemo federicol...@tiscali.it --- I disagree that this is an unsolvable problem: it needs to be properly debugged and measured. Hong Kong has a world top20 IX according to [[List of Internet exchange points by size]], we're not talking of the moon or an island in the middle of the ocean. It's surely necessary to attach some traceroute data from various machines/cities to the WMF servers, to start with, because there are several links: http://www.glif.is/publications/maps/GLIF_5-11_AP_2k.jpg (this website via multichill). We need to know which is/are being used and what's the latency. Then, can someone point out a procedure to measure speed/quality of down/up connections to WMF servers compared to, say, http://ftp.hk.debian.org/debian/ or whichever fast local server there is? -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 60283] Commons connectivity issue in Hong-Kong
https://bugzilla.wikimedia.org/show_bug.cgi?id=60283 andrew@gmail.com changed: What|Removed |Added CC||andrew@gmail.com --- Comment #5 from andrew@gmail.com --- While I agree with Deryck that typically HK connections to the US are not speedy, it's not always a lost cause. When I was based there, you could get fast speeds to the US for certain servers/peers. So simply being in HK does not doom you to poor speeds. It's possible to find a triangle route that will give you fast uploads, or other upload sites that might be quick. Try Internet Archive first to see if that might be faster? -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 60283] Commons connectivity issue in Hong-Kong
https://bugzilla.wikimedia.org/show_bug.cgi?id=60283 --- Comment #6 from Sam Reed (reedy) s...@reedyboy.net --- If you're struggling uploading files, if you can find somewhere else to upload them to (which is fast enough for you) and we can pull them over this way. Certainly testing elsewhere is a good start Do speed tests give sensible values for various servers around the world? http://www.speedtest.net/ etc -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
