[
https://issues.apache.org/jira/browse/IVY-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13875378#comment-13875378
]
Loren Kratzke commented on IVY-1197:
------------------------------------
I have identified the root cause of the issue and I have a solution.
Problem:
JVM experiences java.lang.OutOfMemoryError during ivy:publish of a file greater
than a few hundred MB.
Cause:
HttpURLConnection object as configured in
org.apache.ivy.util.url.BasicURLHandler buffers the data before sending so as
to calculate the content-length header value. This alone is bad however is
likely aggravated by the default array resizing algorythm of the
ByteArrayOutputStream used to buffer the data.
The buffer starts small having a 32 byte capacity. When written to, it fills up
and eventually it reaches maximum capacity. At this time a new buffer of twice
the size is allocated. The content of the original buffer is copied to the new
buffer and the old buffer is garbage collected. Lather, rinse, repeat.
This works fine for small amounts of data however this is a major problem when
talking about medium or large amounts of data. As the buffer size approaches
50% of the amount of free heap space, there will no longer be enough free RAM
to allocate the new buffer. For example, if a 512MB requires even one more
byte of capacity, and there is one byte less than 1024MB free, then an attempt
by ByteArrayOutputStream to allocate a new 1024MB will fail with an OOME. This
is a tragic waste of memory, especially if the content-length is already known
and buffering is not even required (which is always the case for Ivy).
Partial/Failed Solution Using HttpClient:
For one reason or another, possibly to fix this issue, a reflection invocation
is made by org.apache.ivy.util.url.URLHandlerRegistry for
org.apache.commons.httpclient.HttpClient. Comments about Ivy-1197 instruct
users to place HttpClient jar on the classpath (lib dir of Ant) to solve the
OOME issue. Indeed, HttpClient (even the ancient version from 2005 used by Ivy)
does not buffer the content and thus could avoid the OOME entirely, however
there are three major problems with this solution.
The first problem is that the trial invocation of HttpClient issued to detect
availablility on the classpath fails rather silently by logging a vague
message, and only if Ant is invoked in verbose mode. So somebody might drop the
jar into the lib, see nothing happen, and wonder why.
The second problem is that the trial invocation of HttpClient does indeed fail
if all one does is drop the jar into the lib dir. This is because HttpClient
requires two additional jars to in order to instantiate, let alone function:
commons-logging and commons-codec. Adding these jars successfully triggers the
Ivy code which substitutes the Apache HttpClient based
org.apache.ivy.util.url.HttpClientHandler for the problematic
URLConnectionHandler based org.apache.ivy.util.url.BasicURLConnectionHandler.
But there is one more problem.
The third (and most unusual) problem is as follows: Apache docs in the
HttpClient performance guide describe how to stream a request using a custom
RequestEntity object. This object is capable of restarting the stream in the
event of an interruption or an authentication request. They even provide sample
code. This code was copied into the Ivy HttpClientHandler however the block
that writes directly to the OutputStream (without buffering) has been replaced
by a call to the same method that URLConnectionHandler calls which buffers all
of the data. The net effect is that the OOME persists because nothing about the
upload has changed - the data is still buffered in a ByteArrayOutputStream
despite the fact that HttpClient is available and an OOME is still thrown.
Complete/Successful Solution:
In HttpClientHandler.FileRequestEntity.writeRequest(OutputStream) replace this:
FileUtil.copy(instream, out, null, false);
with the original Apache sample code (slightly refactored to match the class):
int length;
byte[] buffer = new byte[64*1024];
while ((length = instream.read(buffer)) != -1) {
out.write(buffer, 0, length);
}
and recompile Ivy. I recompiled using JDK 1.6 because that is the oldest VM
around here.
This fix only works when all of the following libraries are located in
$ANT_HOME/lib directory:
commons-httpclient.jar
commons-logging.jar
commons-codec.jar
Those libraries are available in the apache-ivy-2.3.0-bin-with-deps
distribution.
> OutOfMemoryError duriong ivy:publish
> ------------------------------------
>
> Key: IVY-1197
> URL: https://issues.apache.org/jira/browse/IVY-1197
> Project: Ivy
> Issue Type: Bug
> Components: Core
> Affects Versions: 2.0
> Reporter: Michael Rumpf
> Attachments: ASF.LICENSE.NOT.GRANTED--clipboard.txt
>
>
> When publishing a large file, an OutOfMemoryError occurs.
> {code}
> [ivy:publish] published ppg to
> XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
> BUILD FAILED
> /export/build/hudson/jobs/ppg-rcp/workspace/ppg-rcp/com.daimler.ppg.rcp.builder/build-wrapper.xml:152:
> The following error occurred while executing this line:
> /export/build/hudson/jobs/ppg-rcp/workspace/ppg-rcp/com.daimler.ppg.rcp.builder/build-wrapper.xml:277:
> java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:2786)
> at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
> at sun.net.www.http.PosterOutputStream.write(PosterOutputStream.java:61)
> at org.apache.ivy.util.FileUtil.copy(FileUtil.java:168)
> at
> org.apache.ivy.util.url.BasicURLHandler.upload(BasicURLHandler.java:200)
> at
> org.apache.ivy.util.url.URLHandlerDispatcher.upload(URLHandlerDispatcher.java:82)
> at org.apache.ivy.util.FileUtil.copy(FileUtil.java:140)
> at
> org.apache.ivy.plugins.repository.url.URLRepository.put(URLRepository.java:85)
> at
> org.apache.ivy.plugins.repository.AbstractRepository.put(AbstractRepository.java:130)
> at
> org.apache.ivy.plugins.resolver.RepositoryResolver.put(RepositoryResolver.java:219)
> at
> org.apache.ivy.plugins.resolver.RepositoryResolver.publish(RepositoryResolver.java:209)
> at
> org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:282)
> at
> org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:261)
> at
> org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:170)
> at org.apache.ivy.Ivy.publish(Ivy.java:600)
> at org.apache.ivy.ant.IvyPublish.doExecute(IvyPublish.java:299)
> at org.apache.ivy.ant.IvyTask.execute(IvyTask.java:277)
> at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
> at sun.reflect.GeneratedMethodAccessor101.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
> at org.apache.tools.ant.Task.perform(Task.java:348)
> at org.apache.tools.ant.Target.execute(Target.java:390)
> at org.apache.tools.ant.Target.performTasks(Target.java:411)
> at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1397)
> at
> org.apache.tools.ant.helper.SingleCheckExecutor.executeTargets(SingleCheckExecutor.java:38)
> at org.apache.tools.ant.Project.executeTargets(Project.java:1249)
> at org.apache.tools.ant.taskdefs.Ant.execute(Ant.java:442)
> at org.apache.tools.ant.taskdefs.CallTarget.execute(CallTarget.java:105)
> at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> Total time: 14 minutes 24 seconds
> Finished: FAILURE
> {code}
> The size of the file that is being uploaded is: 687712714, so around
> 650-700MB.
> The publish task is part of a Hudson Ant build where the artefacts are
> published to an Artifactory repository at the end.
> I have given the Job 1300MB for the max heap size.
> It seems as if the whole file is loaded into memory for the upload.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
