[
https://issues.apache.org/jira/browse/JCLOUDS-626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14063627#comment-14063627
]
Andrew Gaul edited comment on JCLOUDS-626 at 7/16/14 5:27 PM:
--------------------------------------------------------------
[~knl] Resolving this as fixed since you no longer encounter OutOfMemoryError.
Can you open another issue for the poor performance that you encountered and
include a reproduceable test case? Note that filesystem getBlob the
implementation currently *recalculates* the MD5 checksum, doubling IO
requirements. We could either remove this or store this metadata in extended
attributes when jclouds 2.0 moves to Java 7. See
https://github.com/jclouds/jclouds/pull/443 for a possible workaround.
was (Author: gaul):
[~knl] Resolving this as fixed since you no longer encounter OutOfMemoryError.
Can you open another issue for the poor performance that you encountered and
include a reproduceable test case? Note that filesystem getBlob the
implementation currently *recalculates* the MD5 checksum, doubling IO
requirements. We could either remove this or store this metadata in extended
attributes when jclouds 2.0 moves to Java 7.
> OutOfMemoryError with FileSystem provider, large content from non-repeatable
> input streams
> ------------------------------------------------------------------------------------------
>
> Key: JCLOUDS-626
> URL: https://issues.apache.org/jira/browse/JCLOUDS-626
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-blobstore
> Affects Versions: 1.7.3
> Environment: Mac OS X 10.9.4, Oracle Java 1.7.0_55; RHEL 6.5, IBM
> Java
> Reporter: Nikola Knezevic
> Assignee: Andrew Gaul
> Fix For: 1.8.0
>
>
> The following (incomplete) piece of code dies on my systems with a stacktrace
> (given below):
> {code}
> public void sendLargeFilesOnlyJclouds() throws Exception
> {
> LOG.info("Working with {}", p);
> Environment env =
> Environment.builder().setName("SomeName").addProperties(p).build();
> String providerName = env.getConnectionModules().get(0).getName();
> String provider = checkNotNull(env.getProperty(providerName +
> ".provider"), new ICStoreException("No provider specified"));
> Iterable<? extends Module> modules =
> ImmutableSet.of(
> new SLF4JLoggingModule()
> );
> if(provider.equals("s3"))
> {
> provider = "aws-s3";
> modules =
> ImmutableSet.<Module>builder()
> .addAll(modules)
> .add(new ICStoreMultipartUploadModule())
> .build();
> }
> Properties properties = new Properties();
> // store provider credentials in properties
> properties.setProperty(provider + ".identity", env.getProperty(
> providerName + ".identity", ""));
> properties.setProperty(provider + ".credential",
> env.getProperty(providerName + ".credential", ""));
> properties.putAll(env.getPropertiesForModule(providerName));
> final BlobStoreContext blobStoreContext =
> ContextBuilder.newBuilder(provider)
> .overrides(properties)
> .modules(modules)
> .buildView(BlobStoreContext.class);
> final BlobStore bs = blobStoreContext.getBlobStore();
> bs.createContainerInLocation(null, "mycontainer");
> for (int i = 0; i < num_invocations; i++)
> {
> InputStream fis = new FakeInputStream(FAKE_IS_SEED, blobSize);
> Blob blob = bs.blobBuilder("myblob").payload(fis).build();
> LOG.info("Invocation number {}", i);
> long startTime = System.currentTimeMillis();
> bs.putBlob("mycontainer", blob);
> long stopTime = System.currentTimeMillis();
> LOG.info("Running time for one blob is {}", stopTime - startTime);
> }
> }
> {code}
> Where {{FakeInputStream}} is an input stream that produces a deterministic
> output (though method {{read()}}, of a given size, and {{Environment}} is
> just a smarter container for {{Properties}}.
> The resulting stack trace follows, and I get it during the first iteration:
> {code}
> 2014-07-14@12:03:48.653 I [main ][ ]
> soTest:sendLargeFilesOnlyJclouds - Invocation number 0
> java.lang.OutOfMemoryError: Requested array size exceeds VM limit
> Dumping heap to java_pid8693.hprof ...
> Heap dump file created [2145617933 bytes in 85.677 secs]
> Exception in thread "main" java.lang.OutOfMemoryError: Requested array size
> exceeds VM limit
> at java.util.Arrays.copyOf(Arrays.java:2271)
> at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
> at
> java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
> at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
> at com.google.common.io.ByteStreams.copy(ByteStreams.java:173)
> at com.google.common.io.ByteStreams.toByteArray(ByteStreams.java:214)
> at com.google.common.io.ByteSource.read(ByteSource.java:266)
> at com.google.common.io.ByteStreams.toByteArray(ByteStreams.java:226)
> at
> org.jclouds.filesystem.strategy.internal.FilesystemStorageStrategyImpl.putBlob(FilesystemStorageStrategyImpl.java:205)
> at
> org.jclouds.blobstore.LocalAsyncBlobStore.putBlob(LocalAsyncBlobStore.java:397)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:37)
> at com.sun.proxy.$Proxy45.putBlob(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> com.google.common.reflect.Invokable$MethodInvokable.invokeInternal(Invokable.java:197)
> at com.google.common.reflect.Invokable.invoke(Invokable.java:102)
> at
> org.jclouds.rest.internal.InvokeAndCallGetOnFutures.apply(InvokeAndCallGetOnFutures.java:68)
> at
> org.jclouds.rest.internal.InvokeAndCallGetOnFutures.apply(InvokeAndCallGetOnFutures.java:40)
> at
> org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
> at
> org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
> at com.sun.proxy.$Proxy46.putBlob(Unknown Source)
> at
> com.ibm.icstore.stress.LargeFilesTest.sendLargeFilesOnlyJclouds(LargeFilesTest.java:153)
> at com.ibm.icstore.stress.LargeFilesTest.main(LargeFilesTest.java:163)
> {code}
> I suspect the problem is with the following piece of code in the method
> {{putBlob}} in {{FilesystemStorageStrategyImpl.java}}:
> {code}
> else {
> if (!payload.isRepeatable()) {
> payload =
> Payloads.newPayload(ByteStreams.toByteArray(payload));
> }
> Files.copy(payload, outputFile);
> }
> {code}
> Judging by the previous fix for the issue (didn't fix it), the culprit seems
> to be:
> {{Payloads.newPayload(ByteStreams.toByteArray(payload));}}
> Why not completely ditch that line, and just go for {{Files.copy()}}, as it
> accepts an InputStream, and doesn't require a repeatable IS?
--
This message was sent by Atlassian JIRA
(v6.2#6252)
