Nikola Knezevic created JCLOUDS-626:
---------------------------------------
Summary: OutOfMemoryException with FileSystem provider, large
content from non-repeatable input streams
Key: JCLOUDS-626
URL: https://issues.apache.org/jira/browse/JCLOUDS-626
Project: jclouds
Issue Type: Bug
Components: jclouds-blobstore
Affects Versions: 1.7.3
Environment: Mac OS X 10.9.4, Oracle Java 1.7.0_55; RHEL 6.5, IBM Java
Reporter: Nikola Knezevic
The following (incomplete) piece of code dies on my systems with a stacktrace
(given below):
{code}
public void sendLargeFilesOnlyJclouds() throws Exception
{
LOG.info("Working with {}", p);
Environment env =
Environment.builder().setName("SomeName").addProperties(p).build();
String providerName = env.getConnectionModules().get(0).getName();
String provider = checkNotNull(env.getProperty(providerName +
".provider"), new ICStoreException("No provider specified"));
Iterable<? extends Module> modules =
ImmutableSet.of(
new SLF4JLoggingModule()
);
if(provider.equals("s3"))
{
provider = "aws-s3";
modules =
ImmutableSet.<Module>builder()
.addAll(modules)
.add(new ICStoreMultipartUploadModule())
.build();
}
Properties properties = new Properties();
// store provider credentials in properties
properties.setProperty(provider + ".identity", env.getProperty(
providerName + ".identity", ""));
properties.setProperty(provider + ".credential",
env.getProperty(providerName + ".credential", ""));
properties.putAll(env.getPropertiesForModule(providerName));
final BlobStoreContext blobStoreContext =
ContextBuilder.newBuilder(provider)
.overrides(properties)
.modules(modules)
.buildView(BlobStoreContext.class);
final BlobStore bs = blobStoreContext.getBlobStore();
bs.createContainerInLocation(null, "mycontainer");
for (int i = 0; i < num_invocations; i++)
{
InputStream fis = new FakeInputStream(FAKE_IS_SEED, blobSize);
Blob blob = bs.blobBuilder("myblob").payload(fis).build();
LOG.info("Invocation number {}", i);
long startTime = System.currentTimeMillis();
bs.putBlob("mycontainer", blob);
long stopTime = System.currentTimeMillis();
LOG.info("Running time for one blob is {}", stopTime - startTime);
}
}
{code}
Where {{FakeInputStream}} is an input stream that produces a deterministic
output (though method {{read()}}, of a given size, and {{Environment}} is just
a smarter container for {{Properties}}.
The resulting stack trace follows, and I get it during the first iteration:
{code}
2014-07-14@12:03:48.653 I [main ][ ]
soTest:sendLargeFilesOnlyJclouds - Invocation number 0
java.lang.OutOfMemoryError: Requested array size exceeds VM limit
Dumping heap to java_pid8693.hprof ...
Heap dump file created [2145617933 bytes in 85.677 secs]
Exception in thread "main" java.lang.OutOfMemoryError: Requested array size
exceeds VM limit
at java.util.Arrays.copyOf(Arrays.java:2271)
at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
at
java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
at com.google.common.io.ByteStreams.copy(ByteStreams.java:173)
at com.google.common.io.ByteStreams.toByteArray(ByteStreams.java:214)
at com.google.common.io.ByteSource.read(ByteSource.java:266)
at com.google.common.io.ByteStreams.toByteArray(ByteStreams.java:226)
at
org.jclouds.filesystem.strategy.internal.FilesystemStorageStrategyImpl.putBlob(FilesystemStorageStrategyImpl.java:205)
at
org.jclouds.blobstore.LocalAsyncBlobStore.putBlob(LocalAsyncBlobStore.java:397)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:37)
at com.sun.proxy.$Proxy45.putBlob(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
com.google.common.reflect.Invokable$MethodInvokable.invokeInternal(Invokable.java:197)
at com.google.common.reflect.Invokable.invoke(Invokable.java:102)
at
org.jclouds.rest.internal.InvokeAndCallGetOnFutures.apply(InvokeAndCallGetOnFutures.java:68)
at
org.jclouds.rest.internal.InvokeAndCallGetOnFutures.apply(InvokeAndCallGetOnFutures.java:40)
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
at com.sun.proxy.$Proxy46.putBlob(Unknown Source)
at
com.ibm.icstore.stress.LargeFilesTest.sendLargeFilesOnlyJclouds(LargeFilesTest.java:153)
at com.ibm.icstore.stress.LargeFilesTest.main(LargeFilesTest.java:163)
{code}
I suspect the problem is with the following piece of code in the method
{{putBlob}} in {{FileSystemStorageImpl.java}}:
{code}
else {
if (!payload.isRepeatable()) {
payload = Payloads.newPayload(ByteStreams.toByteArray(payload));
}
Files.copy(payload, outputFile);
}
{code}
Judging by the previous fix for the issue (didn't fix it), the culprit seems to
be:
{{Payloads.newPayload(ByteStreams.toByteArray(payload));}}
Why not completely ditch that line, and just go for {{Files.copy()}}, as it
accepts an InputStream, and doesn't require a repeatable IS?
--
This message was sent by Atlassian JIRA
(v6.2#6252)
