[
https://issues.apache.org/jira/browse/JCLOUDS-626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Gaul updated JCLOUDS-626:
--------------------------------
Description:
The following (incomplete) piece of code dies on my systems with a stacktrace
(given below):
{code}
public void sendLargeFilesOnlyJclouds() throws Exception
{
LOG.info("Working with {}", p);
Environment env =
Environment.builder().setName("SomeName").addProperties(p).build();
String providerName = env.getConnectionModules().get(0).getName();
String provider = checkNotNull(env.getProperty(providerName +
".provider"), new ICStoreException("No provider specified"));
Iterable<? extends Module> modules =
ImmutableSet.of(
new SLF4JLoggingModule()
);
if(provider.equals("s3"))
{
provider = "aws-s3";
modules =
ImmutableSet.<Module>builder()
.addAll(modules)
.add(new ICStoreMultipartUploadModule())
.build();
}
Properties properties = new Properties();
// store provider credentials in properties
properties.setProperty(provider + ".identity", env.getProperty(
providerName + ".identity", ""));
properties.setProperty(provider + ".credential",
env.getProperty(providerName + ".credential", ""));
properties.putAll(env.getPropertiesForModule(providerName));
final BlobStoreContext blobStoreContext =
ContextBuilder.newBuilder(provider)
.overrides(properties)
.modules(modules)
.buildView(BlobStoreContext.class);
final BlobStore bs = blobStoreContext.getBlobStore();
bs.createContainerInLocation(null, "mycontainer");
for (int i = 0; i < num_invocations; i++)
{
InputStream fis = new FakeInputStream(FAKE_IS_SEED, blobSize);
Blob blob = bs.blobBuilder("myblob").payload(fis).build();
LOG.info("Invocation number {}", i);
long startTime = System.currentTimeMillis();
bs.putBlob("mycontainer", blob);
long stopTime = System.currentTimeMillis();
LOG.info("Running time for one blob is {}", stopTime - startTime);
}
}
{code}
Where {{FakeInputStream}} is an input stream that produces a deterministic
output (though method {{read()}}, of a given size, and {{Environment}} is just
a smarter container for {{Properties}}.
The resulting stack trace follows, and I get it during the first iteration:
{code}
2014-07-14@12:03:48.653 I [main ][ ]
soTest:sendLargeFilesOnlyJclouds - Invocation number 0
java.lang.OutOfMemoryError: Requested array size exceeds VM limit
Dumping heap to java_pid8693.hprof ...
Heap dump file created [2145617933 bytes in 85.677 secs]
Exception in thread "main" java.lang.OutOfMemoryError: Requested array size
exceeds VM limit
at java.util.Arrays.copyOf(Arrays.java:2271)
at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
at
java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
at com.google.common.io.ByteStreams.copy(ByteStreams.java:173)
at com.google.common.io.ByteStreams.toByteArray(ByteStreams.java:214)
at com.google.common.io.ByteSource.read(ByteSource.java:266)
at com.google.common.io.ByteStreams.toByteArray(ByteStreams.java:226)
at
org.jclouds.filesystem.strategy.internal.FilesystemStorageStrategyImpl.putBlob(FilesystemStorageStrategyImpl.java:205)
at
org.jclouds.blobstore.LocalAsyncBlobStore.putBlob(LocalAsyncBlobStore.java:397)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:37)
at com.sun.proxy.$Proxy45.putBlob(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
com.google.common.reflect.Invokable$MethodInvokable.invokeInternal(Invokable.java:197)
at com.google.common.reflect.Invokable.invoke(Invokable.java:102)
at
org.jclouds.rest.internal.InvokeAndCallGetOnFutures.apply(InvokeAndCallGetOnFutures.java:68)
at
org.jclouds.rest.internal.InvokeAndCallGetOnFutures.apply(InvokeAndCallGetOnFutures.java:40)
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
at com.sun.proxy.$Proxy46.putBlob(Unknown Source)
at
com.ibm.icstore.stress.LargeFilesTest.sendLargeFilesOnlyJclouds(LargeFilesTest.java:153)
at com.ibm.icstore.stress.LargeFilesTest.main(LargeFilesTest.java:163)
{code}
I suspect the problem is with the following piece of code in the method
{{putBlob}} in {{FilesystemStorageStrategyImpl.java}}:
{code}
else {
if (!payload.isRepeatable()) {
payload = Payloads.newPayload(ByteStreams.toByteArray(payload));
}
Files.copy(payload, outputFile);
}
{code}
Judging by the previous fix for the issue (didn't fix it), the culprit seems to
be:
{{Payloads.newPayload(ByteStreams.toByteArray(payload));}}
Why not completely ditch that line, and just go for {{Files.copy()}}, as it
accepts an InputStream, and doesn't require a repeatable IS?
was:
The following (incomplete) piece of code dies on my systems with a stacktrace
(given below):
{code}
public void sendLargeFilesOnlyJclouds() throws Exception
{
LOG.info("Working with {}", p);
Environment env =
Environment.builder().setName("SomeName").addProperties(p).build();
String providerName = env.getConnectionModules().get(0).getName();
String provider = checkNotNull(env.getProperty(providerName +
".provider"), new ICStoreException("No provider specified"));
Iterable<? extends Module> modules =
ImmutableSet.of(
new SLF4JLoggingModule()
);
if(provider.equals("s3"))
{
provider = "aws-s3";
modules =
ImmutableSet.<Module>builder()
.addAll(modules)
.add(new ICStoreMultipartUploadModule())
.build();
}
Properties properties = new Properties();
// store provider credentials in properties
properties.setProperty(provider + ".identity", env.getProperty(
providerName + ".identity", ""));
properties.setProperty(provider + ".credential",
env.getProperty(providerName + ".credential", ""));
properties.putAll(env.getPropertiesForModule(providerName));
final BlobStoreContext blobStoreContext =
ContextBuilder.newBuilder(provider)
.overrides(properties)
.modules(modules)
.buildView(BlobStoreContext.class);
final BlobStore bs = blobStoreContext.getBlobStore();
bs.createContainerInLocation(null, "mycontainer");
for (int i = 0; i < num_invocations; i++)
{
InputStream fis = new FakeInputStream(FAKE_IS_SEED, blobSize);
Blob blob = bs.blobBuilder("myblob").payload(fis).build();
LOG.info("Invocation number {}", i);
long startTime = System.currentTimeMillis();
bs.putBlob("mycontainer", blob);
long stopTime = System.currentTimeMillis();
LOG.info("Running time for one blob is {}", stopTime - startTime);
}
}
{code}
Where {{FakeInputStream}} is an input stream that produces a deterministic
output (though method {{read()}}, of a given size, and {{Environment}} is just
a smarter container for {{Properties}}.
The resulting stack trace follows, and I get it during the first iteration:
{code}
2014-07-14@12:03:48.653 I [main ][ ]
soTest:sendLargeFilesOnlyJclouds - Invocation number 0
java.lang.OutOfMemoryError: Requested array size exceeds VM limit
Dumping heap to java_pid8693.hprof ...
Heap dump file created [2145617933 bytes in 85.677 secs]
Exception in thread "main" java.lang.OutOfMemoryError: Requested array size
exceeds VM limit
at java.util.Arrays.copyOf(Arrays.java:2271)
at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
at
java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
at com.google.common.io.ByteStreams.copy(ByteStreams.java:173)
at com.google.common.io.ByteStreams.toByteArray(ByteStreams.java:214)
at com.google.common.io.ByteSource.read(ByteSource.java:266)
at com.google.common.io.ByteStreams.toByteArray(ByteStreams.java:226)
at
org.jclouds.filesystem.strategy.internal.FilesystemStorageStrategyImpl.putBlob(FilesystemStorageStrategyImpl.java:205)
at
org.jclouds.blobstore.LocalAsyncBlobStore.putBlob(LocalAsyncBlobStore.java:397)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:37)
at com.sun.proxy.$Proxy45.putBlob(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
com.google.common.reflect.Invokable$MethodInvokable.invokeInternal(Invokable.java:197)
at com.google.common.reflect.Invokable.invoke(Invokable.java:102)
at
org.jclouds.rest.internal.InvokeAndCallGetOnFutures.apply(InvokeAndCallGetOnFutures.java:68)
at
org.jclouds.rest.internal.InvokeAndCallGetOnFutures.apply(InvokeAndCallGetOnFutures.java:40)
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
at com.sun.proxy.$Proxy46.putBlob(Unknown Source)
at
com.ibm.icstore.stress.LargeFilesTest.sendLargeFilesOnlyJclouds(LargeFilesTest.java:153)
at com.ibm.icstore.stress.LargeFilesTest.main(LargeFilesTest.java:163)
{code}
I suspect the problem is with the following piece of code in the method
{{putBlob}} in {{FileSystemStorageImpl.java}}:
{code}
else {
if (!payload.isRepeatable()) {
payload = Payloads.newPayload(ByteStreams.toByteArray(payload));
}
Files.copy(payload, outputFile);
}
{code}
Judging by the previous fix for the issue (didn't fix it), the culprit seems to
be:
{{Payloads.newPayload(ByteStreams.toByteArray(payload));}}
Why not completely ditch that line, and just go for {{Files.copy()}}, as it
accepts an InputStream, and doesn't require a repeatable IS?
> OutOfMemoryException with FileSystem provider, large content from
> non-repeatable input streams
> ----------------------------------------------------------------------------------------------
>
> Key: JCLOUDS-626
> URL: https://issues.apache.org/jira/browse/JCLOUDS-626
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-blobstore
> Affects Versions: 1.7.3
> Environment: Mac OS X 10.9.4, Oracle Java 1.7.0_55; RHEL 6.5, IBM
> Java
> Reporter: Nikola Knezevic
>
> The following (incomplete) piece of code dies on my systems with a stacktrace
> (given below):
> {code}
> public void sendLargeFilesOnlyJclouds() throws Exception
> {
> LOG.info("Working with {}", p);
> Environment env =
> Environment.builder().setName("SomeName").addProperties(p).build();
> String providerName = env.getConnectionModules().get(0).getName();
> String provider = checkNotNull(env.getProperty(providerName +
> ".provider"), new ICStoreException("No provider specified"));
> Iterable<? extends Module> modules =
> ImmutableSet.of(
> new SLF4JLoggingModule()
> );
> if(provider.equals("s3"))
> {
> provider = "aws-s3";
> modules =
> ImmutableSet.<Module>builder()
> .addAll(modules)
> .add(new ICStoreMultipartUploadModule())
> .build();
> }
> Properties properties = new Properties();
> // store provider credentials in properties
> properties.setProperty(provider + ".identity", env.getProperty(
> providerName + ".identity", ""));
> properties.setProperty(provider + ".credential",
> env.getProperty(providerName + ".credential", ""));
> properties.putAll(env.getPropertiesForModule(providerName));
> final BlobStoreContext blobStoreContext =
> ContextBuilder.newBuilder(provider)
> .overrides(properties)
> .modules(modules)
> .buildView(BlobStoreContext.class);
> final BlobStore bs = blobStoreContext.getBlobStore();
> bs.createContainerInLocation(null, "mycontainer");
> for (int i = 0; i < num_invocations; i++)
> {
> InputStream fis = new FakeInputStream(FAKE_IS_SEED, blobSize);
> Blob blob = bs.blobBuilder("myblob").payload(fis).build();
> LOG.info("Invocation number {}", i);
> long startTime = System.currentTimeMillis();
> bs.putBlob("mycontainer", blob);
> long stopTime = System.currentTimeMillis();
> LOG.info("Running time for one blob is {}", stopTime - startTime);
> }
> }
> {code}
> Where {{FakeInputStream}} is an input stream that produces a deterministic
> output (though method {{read()}}, of a given size, and {{Environment}} is
> just a smarter container for {{Properties}}.
> The resulting stack trace follows, and I get it during the first iteration:
> {code}
> 2014-07-14@12:03:48.653 I [main ][ ]
> soTest:sendLargeFilesOnlyJclouds - Invocation number 0
> java.lang.OutOfMemoryError: Requested array size exceeds VM limit
> Dumping heap to java_pid8693.hprof ...
> Heap dump file created [2145617933 bytes in 85.677 secs]
> Exception in thread "main" java.lang.OutOfMemoryError: Requested array size
> exceeds VM limit
> at java.util.Arrays.copyOf(Arrays.java:2271)
> at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
> at
> java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
> at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
> at com.google.common.io.ByteStreams.copy(ByteStreams.java:173)
> at com.google.common.io.ByteStreams.toByteArray(ByteStreams.java:214)
> at com.google.common.io.ByteSource.read(ByteSource.java:266)
> at com.google.common.io.ByteStreams.toByteArray(ByteStreams.java:226)
> at
> org.jclouds.filesystem.strategy.internal.FilesystemStorageStrategyImpl.putBlob(FilesystemStorageStrategyImpl.java:205)
> at
> org.jclouds.blobstore.LocalAsyncBlobStore.putBlob(LocalAsyncBlobStore.java:397)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:37)
> at com.sun.proxy.$Proxy45.putBlob(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> com.google.common.reflect.Invokable$MethodInvokable.invokeInternal(Invokable.java:197)
> at com.google.common.reflect.Invokable.invoke(Invokable.java:102)
> at
> org.jclouds.rest.internal.InvokeAndCallGetOnFutures.apply(InvokeAndCallGetOnFutures.java:68)
> at
> org.jclouds.rest.internal.InvokeAndCallGetOnFutures.apply(InvokeAndCallGetOnFutures.java:40)
> at
> org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
> at
> org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
> at com.sun.proxy.$Proxy46.putBlob(Unknown Source)
> at
> com.ibm.icstore.stress.LargeFilesTest.sendLargeFilesOnlyJclouds(LargeFilesTest.java:153)
> at com.ibm.icstore.stress.LargeFilesTest.main(LargeFilesTest.java:163)
> {code}
> I suspect the problem is with the following piece of code in the method
> {{putBlob}} in {{FilesystemStorageStrategyImpl.java}}:
> {code}
> else {
> if (!payload.isRepeatable()) {
> payload =
> Payloads.newPayload(ByteStreams.toByteArray(payload));
> }
> Files.copy(payload, outputFile);
> }
> {code}
> Judging by the previous fix for the issue (didn't fix it), the culprit seems
> to be:
> {{Payloads.newPayload(ByteStreams.toByteArray(payload));}}
> Why not completely ditch that line, and just go for {{Files.copy()}}, as it
> accepts an InputStream, and doesn't require a repeatable IS?
--
This message was sent by Atlassian JIRA
(v6.2#6252)
