As a comparison, FWIW, I tested with my computer at home (nominally 250
Mbit/s). I cloned the consolidated repo in ~33 minutes, which
corresponds to an average speed of 800 kB/s or rougly 6 Mbit/s), given
that all that time was spent transmitting data (and that I've done the
calculations right).
That sounds like a transfer rate that's close to what I'd expect of an
arbitrary, non-speed-optimized, service on the Internet in general. (On
the other hand, one might argue that one would expect more of a service
like hg.openjdk.java.net.)
/Magnus
On 2017-09-07 12:23, Volker Simonis wrote:
Yes, I can also confirm Thomas' numbers. Tried from home on my 25Mbps
line with both company/private PC and couldn't get more than 350KB
download speed during cloning.
I've also tested a traditional repository with the get_source.sh
script. By default the script uses a concurrency of two (i.e. it
always processes two repositories of the forest in parallel). This
obviously starts two hg processes at a time and it seems that both of
them can use up to 350KB during downloading. This gets the cloning
time down to 12m (compared to 38m for the new, fat consolidated
repository).
The concurrency of get_source.sh can be increased by setting
HGFOREST_CONCURRENCY. If I set this to 8 the cloning time only drops
down to 11 minutes because it is bound by the time it takes to clone
the jdk repository which is by far the biggest one.
So it really seems that the hg.openjdk.java.net limits the download
speed per connection. Before, with the forest, this was not so much of
an issue, because cloning was done in parallel for each forest, but
now, with the new, single, fat, consolidated repository, this becomes
a real bottleneck.
Can somebody from Oracle and/or the hg.openjdk.java.net operations
team please comment on this and ideally fix it?
Thank you and best regards,
Volker
On Thu, Sep 7, 2017 at 10:07 AM, Doug Simon <[email protected]> wrote:
On 7 Sep 2017, at 07:06, Thomas Stüfe <[email protected]> wrote:
On Wed, Sep 6, 2017 at 5:35 PM, Volker Simonis <[email protected]>
wrote:
On Wed, Sep 6, 2017 at 2:30 PM, Thomas Stüfe <[email protected]>
wrote:
Erik, Volker,
On Tue, Sep 5, 2017 at 7:31 PM, Erik Joelsson <[email protected]>
wrote:
On 2017-09-05 11:38, Volker Simonis wrote:
Hi Joe,
generally looks good!
I don't know if this has discussed before, but in my opinion the
top-level src/ directory looks a little overloaded. Wouldn't it make
sense to place all the modules into their own 'src/modules/'
subdirectory?
I see what you mean, but it's no more overloaded than the jdk/src dir
used
to be. Also, the bsd, linux, solaris, demo and sample directories are
all
going away at some point in the (hopefully near) future. Left are
hotspot
and utils which I don't think warrant another directory level.
Should jdk10/consol-proto build out of t he box or are there any known
issues? I'd like to give it a tray on AIX and Linux/ppc64 but if there
are any known, generic problems I'll wait until they get fixed.
Yes, please try it. I have put a lot of work into maintaining a set of
patches and scripts to keep the generated consolidated repo working and
(near) equivalent with the current forest. You can review those changes
in
the prototype forest if you like.
I cloned http://hg.openjdk.java.net/jdk10/consol-proto and built AIX.
Build
runs fine.
Hi Thomas,
thanks for checking the AIX build.
However, I found cloning the repository painfully slow.
On my Linux box clone took ~90 minutes. On Windows, I could not
successfully
clone, as cygwin mercurial did hit timeouts. I really hope this is only
temporary and will improve?
I think you should ask your company and/or Internet provider :)
The new, consolidated repository is about 1.6GB in size (i.e. the size
of .hg) and the cloning speed is actually pretty much proportional to
the time you need to download this amount of data.
When using our http-proxy I measured a download speed of about 350
KB/s. The calculation is quite simple:
1600000 KB / 350 KB/s = 4571 s / 60 = 76 min :(
The actual cloning speed was slightly faster:
$ time hg clone http://hg.openjdk.java.net/jdk10/consol-proto
jdk10-cons-proto
requesting all changes
adding changesets
adding manifests
adding file changes
added 46937 changesets with 397578 changes to 162338 files
updating to branch default
57067 files updated, 0 files merged, 0 files removed, 0 files
unresolved
real 38m37.424s
user 5m46.620s
sys 0m33.976s
which is probably because the 1.6 GB from the .hg repo may be
compressed for the transport.
Without proxy (i.e. 'transparent proxy') the average download speed is
about 150 KB/s. I think this pretty much explains the ~90 minutes.
I doubt that these poor download speeds are caused by
hg.openjdk.java.net because then it would be the same for the proxy
vs. non-proxy case. Nevertheless it would be interesting to see what
amount of data hg.openjdk.java.net can actually serve to Europe. So if
somebody with a decent Internet connection can share his experience,
that would be interesting.
I'm on cable, with a download speed of 128Mbit. Cloning the new repo still
took 39min. That was on Linux, with SSDs.
I'm in Switzerland, get 85Mbps according to fast.com and it took 42min to clone
the new repo.
-Doug
Best Regards, Thomas
Regards,
Volker
Kind Regards, Thomas
/Erik
Thank you and best regards,
Volker
On Fri, Sep 1, 2017 at 8:36 PM, joe darcy <[email protected]>
wrote:
On 8/25/2017 1:52 PM, joe darcy wrote:
Hello,
A follow-up to the most recent update [1] on the JDK 10 repo
consolidation
efforts, September 2017 is almost upon us and that month remains the
target
to implement the repo consolidation.
First, a third generation prototype having tags from both JDK 9 and
JDK
10
will be published in the near future.
[snip]
Third generation prototype with tags from JDK 9 and JDK 10 available
for
browsing at:
http://hg.openjdk.java.net/jdk10/consol-proto/tags
Please sent comments by Wednesday, September 6.
Thanks,
-Joe
