Thanks for the analysis! Agree with dropping fsutil then; I committed
the addition of the logs with it just so that we can have concrete
numbers for comparison (the last build indicates there was no permission
issues in using it, otherwise it wouldn't have timed out but just failed
to find the file). NIO.2 requires Java 7 though, so looks like a good
reason to try and update to it (and Jetty 9 in the process, as Olivier
suggested), but maybe not for this version. That also makes me wonder if
the NIO.2 SPARSE option doesn't implictly require special privileges on
Windows (like it does for creating symbolic links).
In any case, the result of the Jenkins build with regard to the commit
are here
https://builds.apache.org/job/maven-wagon-jetty8-windows/org.apache.maven.wagon$wagon-http/3/consoleFull.
The 4 Go test file was created instantly (30 milliseconds), and it timed
out performing the first download; so the creation of the file isn't an
issue. I was following the growing size of the download in the target
directory as it happened and things didn't appear to work slowly.
The tests for wagon-http started at 11:47:28 and timed out after
11:53:04 when doing the first download for HugeFileDownloadTest; this
makes up for the 400 seconds (roughly 6-7 minutes) of the Surefire
timeout. Another breakdown is seen here (oddly without
HugeFileDownloadTest):
https://builds.apache.org/job/maven-wagon-jetty8-windows/org.apache.maven.wagon$wagon-http/3/testReport/org.apache.maven.wagon.providers.http/
All of this is actually caused by HttpsWagonTest,
HttpsWagonPreemptiveTest and HugeFileDownloadTest taking each a bit more
than 2 minutes. I don't think we can cut
<https://builds.apache.org/job/maven-wagon-jetty8-windows/org.apache.maven.wagon$wagon-http/3/testReport/org.apache.maven.wagon.providers.http/HttpsWagonTest/><https://builds.apache.org/job/maven-wagon-jetty8-windows/org.apache.maven.wagon$wagon-http/3/testReport/org.apache.maven.wagon.providers.http/HttpsWagonPreemptiveTest/>HugeFileDownloadTest
below 2 minutes (with the sparse file created instantly, I see no room
for improvements here), so focus should actually be made on those 2
other tests.
Guillaume
Le 31/12/2016 à 16:51, Michael Osipov a écrit :
Am 2016-12-31 um 12:13 schrieb Guillaume Boué:
Do you think I can add a dummy log before the creation of the test file
(and add the timestamps to the logs of wagon-http), to see how much time
it takes on the Windows Server 2012? I'd like to see the breakdown of
what takes time on the Jenkins machine, perhaps there is nothing we can
do better (except increase the timeout to 500 or so...).
Let me share new findings before I get to the answers. I tested this
branch on my machine at work which is part of an Active Directory
domain, I am local admin. It is a regular, 2-core i5 machine with 12
GiB RAM and medium-size HDD:
1. fsutil failed, domain policy requires elevated permissions to run
fsutil in command prompt and you even see it because stderr/stdout of
fsutil is swalled. I tried manually.
2. I have even raised the timeout to 800 seconds, it still fails.
Given this, I'd wouldn't really use fsutil. Rather NIO2 with
StandardOption.SPARSE. This might work faster.
To your questions: go head and use SLF4J, it is already available.
Print the log messages and you'll see how much creation really consumes.
I'd test this ony my machine at home and work.
Michael
Le 30/12/2016 à 22:46, Michael Osipov a écrit :
I just pushed another commit to the branch with your changes.
The job does not really work on Windows [1], it simply takes too long
to complete on the Windows Server 2012.
[1]
https://builds.apache.org/job/maven-wagon-jetty8-windows/2/org.apache.maven.wagon$wagon-http/console
Am 2016-12-29 um 19:10 schrieb Guillaume Boué:
I have a Toshiba SSHD (MQ02ABD100H). I think the issue is that the
Java
code should create a sparse file to have things faster. Using
setLength
on a random access file probably does it depending on the OS and
type of
drive, but it isn't creating one in my situation.
When run on Ubuntu, creating the test file is done practically
instantly
whereas it takes 60 seconds on my Windows machine. Downloading takes
around 30 seconds in Ubuntu, whereas it takes 150 seconds in Windows.
I changed the method creating the test file (makeHugeFile) to:
if ( Os.isFamily( Os.FAMILY_WINDOWS ) )
{
Process p = new ProcessBuilder( "fsutil", "file",
"createnew", hugeFile.getAbsolutePath(),
String.valueOf(
HUGE_FILE_SIZE ) ).start();
p.waitFor();
}
else
{
RandomAccessFile ra = new RandomAccessFile(
hugeFile.getPath(), "rw" );
ra.setLength( HUGE_FILE_SIZE + 1 );
ra.seek( HUGE_FILE_SIZE );
ra.write( 1 );
ra.close();
}
This matches with the timings I get on Ubuntu (both for the
creation of
the file, and the downloading via Wagon). I ran the build several
times
with this change without any timeout issues. Can you test this
change on
your side?
Side-note: I added "ra.close();" in the code above, because the random
access file wasn't closed otherwise, and then the file wasn't getting
deleted properly.
Guillaume
Le 29/12/2016 à 15:50, Michael Osipov a écrit :
Am 2016-12-29 um 12:24 schrieb Guillaume Boué:
I ran them at least 10 times, and there was the timeout issue each
time.
Yes, the timeout is Surefire waiting for the forked VM. I applied
the
patch (I had done similar tries also) but I still have the issue on
Windows 10.
You'll find the logs attached. It seems that the download is really
advancing (inside target, I can see the file huge<numbers>txt slowly
growing), but is just taking a long time. Additionally, the test
class
HugeFileDownloadTest is composed of 2 tests, downloading the same
file 2
times with different parameters: from the logs, it seems one of
them is
succeeding (taking 140 seconds of the 400 in total), and then it
times
out performing the second.
Also, part of the issue is probably with the time it takes to
create the
4 Go test file that is later downloaded through Wagon (maybe it
should
be done by calling the fsutil command when the test runs on
Windows).
I'll try to do more timings when running the tests with Ubuntu
and see
where the difference is.
Thanks for the analysis. I think the cheapest improvement we can
do is
to create the file only once (setUp(), tearDown(). It is created for
each test again. I do not think that this is really necessary.
What type of disks do you have? This test passes on my machine within
70 seconds on a Samsung SSD.
I will apply the patch to the branch because those fixes are
necessary
anyway.
Please keep me updated.
Michael
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org
---
L'absence de virus dans ce courrier électronique a été vérifiée par le
logiciel antivirus Avast.
https://www.avast.com/antivirus
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org
---
L'absence de virus dans ce courrier électronique a été vérifiée par le
logiciel antivirus Avast.
https://www.avast.com/antivirus
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org
---
L'absence de virus dans ce courrier électronique a été vérifiée par le logiciel
antivirus Avast.
https://www.avast.com/antivirus
