=8<==[Editors note]=8<==
I'll be honest I'm 100% convinced mailinator will ruin
the little formatting I have done, I put a copy of this
email on https://developer.blender.org/P3014.
=8<====8<====8<====8<==
How much as I like the sentiment of "lets move to git will it
solve all these problems" lets be honest here, git.blender.org's
speed is nothing to write home about either it may or may not be
as glitchy as svn, but it still wouldn't be fast.
from my 300mbit connection in western Canada
Receiving objects: 6% (135931/2078159), 33.34 MiB | 458.00 KiB/s
total time for a clone off git.blender.org 27m39s
Cloning off https://github.com/blender/blender.git however
total time for a clone off github.com 1m 52s
Now amount of finger pointing to client settings will convince me
it's a client setting at this point, but I'll be honest, I am all
the way in western Canada and that could definitely be a factor.
sooooo, lets investigate!
Let’s spin up an AWS EC2 instance in London, I'd say that be close
enough to Amsterdam, and let’s rule out any other CPU or Bandwidth
related factors as well, I threw a few dollars at it and got a
cn5.18xlarge instance (72 CPU’s 192gb memory, 100gbit ethernet)
overkill to do a git/svn test? yes.. yes indeed
first to rule out the blender.org servers lets grab a ubuntu iso
off an .nl mirror (
https://mirror.nl.datapacket.com/ubuntu-releases/22.04/ )
2022-06-17 17:13:21 (232 MB/s) - ‘ubuntu-22.04-desktop-amd64.iso’ saved
[3654957056/3654957056]
all right bandwidth is NOT going to be an issue on this box!
On to blender stuff:
cloning from git.blender.org 2m32.895s
svn checkout of lib/win64_vc15 2m56.388s (iftop said 65Mbit peak rate)
yowza!
All right, so in London, safe to say all is well
lets move closer to my location, us-east-2 Ohio (same instance type and os)
grabbing ubuntu from that same mirror
2022-06-17 17:31:00 (29.5 MB/s) - ‘ubuntu-22.04-desktop-amd64.iso’ saved
[3654957056/3654957056]
all right, that has lost quite a bit of steam, but it's
still nothing to sneeze at, just to be sure lets grab
an iso off the Princeton university mirror 120MB/s ok..
good enough...
onto cloning blender
Receiving objects: 5% (104235/2078173), 25.44 MiB | 692.00 KiB/s
well... that's not looking promising... lets wait it out
cloning from git.blender.org 17m23.449s
sadface... lets try svn next, i lost patience and did not let
it finish.. iftop reported a top RX of 5.59Mbit though..
To summarize:
Ubuntu iso from mirror.nl.datapacket.com (Speed taken from wget)
My home - western Canada - 1.3 MB/s
AWS - London - 232 MB/S
AWS - Ohio - 29.5 MB/s
Clone of git.blender.org (time taken from linux time command)
My home - western Canada - 27m39
AWS - London - 2m32
AWS - Ohio - 17m23
SVN Clone: (peak RX bitrate taken from iftop)
My home - western Canada - 1.5 Mbit/s
AWS - London - 64 Mbit/s
AWS - Ohio - 5.59 Mbit/s
Think the only thing we really can conclude is
that being further from the server is leading
to an "unhappy" time for the developer. Given
the fact the EC2 instances were 100% identical
between London and Ohio, it's unlikely to be a
client configuration issue.
--Ray
On 2022-06-17 10:10 a.m., Dan McGrath via Bf-committers wrote:
Hi,
Really, I would need to see a capture from both sides to dig into it. I
need to see what is being sent, and what is received on each side. There
could be some weird firewall stuff happening that is causing
fragmentation
or blocking TCP options, or window scaling issues, etc.
As for the default settings for tortoisesvn, the connection stuff I was
referring to was about TCP/IP behaviour, rather than an application
setting. Either way, I need to see what is actually happening. The fact
that I can pull 5-8MB/sec (~40mbit) from Canada and not have a single
interruption is telling. Granted, the host only has around 100 or 200mbit
allocated, iirc. Multiply that by a few dozen people and you have a
problem.
For a simpler test, maybe try a large svn check, perhaps some old FreeBSD
ports svn repo if you can find one, something also in .NL, and see how it
goes, just to help eliminate your PC/firewall. Worst case scenario, you
can
also poke me in chat and we can try some server side and client side
captures.
TBH though, 1gig uplink just isn't a whole lot to serve our user base ;)
On Fri, Jun 17, 2022 at 11:45 AM Tom M <letter...@gmail.com> wrote:
On Fri, Jun 17, 2022 at 5:20 AM Dan McGrath <danmcgrath...@gmail.com>
wrote:
Hi Tom,
Long time no see :)
I did some tests from home and found that, aside from slow speeds, I
was
able to checkout at least 5 or 6 gig of the bf-blender repo without
error
or having to retransmit.
Yeah, looks like I was mistaking really slow download of the larger
(.7 GB) libraries as freezing. I was averaging about .3 MB/s. I used
a different SVN client and saw that it was indeed downloading (the
sliksvn doesn't provide any feedback on files in progress, only once a
file completes) - It was taking about 40 minutes for each of the
larger files. It did complete eventually after about 5 hours.
I took the liberty of packet capturing the stream from my PC's POV, and
placed them in the ftp download area[1], if you care to take a look.
The gist is that I was able to get around 5MB/sec for at least 5G of
the
checkout, before I cancelled it.
Interesting, I usually get up to 3 MB/s download here normally from
other sources, but was typically getting .3 MB/s, with occasional
bursts of higher speed to a max of .8 MB/s when it wasn't downloading
the large files.
Now, my IP is a bit of a special case, so I am wondering if you are
just
hitting the server so hard with http requests in separate sockets, or
something extreme, that is causing the system to block you for a period
of
time. For comparison, my PC only had about 4 sockets open to the server
for
the transfer. I would be curious to know if you are doing pipelining, or
sending each request in a separate connection.
Whatever the sliksvn and tortoisesvn defaults are, I'd assume they are
setup for typical usage.
If you can, the next time it happens, see if you are unable to visit
Phabricator or resume the svn checkout process. It may not be a firewall
issue, but without seeing some packet captures, I wouldn't be able to
really guess. Ideally, try to capture a few packets, until failure. At
the
very least, you will need to snap 20b+20b=40b for IP+TCP headers, but I
would suggest a tad higher, maybe 64b.
Cheers,
Dan
[1] - https://download.blender.org/ftp/dan/svn-checkout/
On Thu, Jun 16, 2022 at 2:43 PM Tom M via Bf-committers <
bf-committers@blender.org> wrote:
When checking out the libraries from SVN following the directions on
https://wiki.blender.org/wiki/Building_Blender/Windows
The svn checkout of the libraries fails with significant frequency,
if you google you might stumble across using make svnfix
https://www.mail-archive.com/bf-blender-cvs@blender.org/msg151996.html
Even after using it, I'm still having to do Ctrl C, and then make
svnfix repeatedly (it downloads some files, than randomly freezes,
sometimes after 20-50 files, sometimes after 100's of files).
Git doesn't seem to have any problems, though.
So might we consider migrating the libs to git, rather than using SVN?
LetterRip
_______________________________________________
Bf-committers mailing list
Bf-committers@blender.org
List details, subscription details or unsubscribe:
https://lists.blender.org/mailman/listinfo/bf-committers
--
Cheers,
Danny
----------------------------------------------------------
Danny McGrath - danmcgrath...@gmail.com
GPG key: EDF6 AFF5 2086 F93A 1F59 36A5 44B6 26F3 6968 71CA
_______________________________________________
Bf-committers mailing list
Bf-committers@blender.org
List details, subscription details or unsubscribe:
https://lists.blender.org/mailman/listinfo/bf-committers
