Lukasz Gajowy created BEAM-6923:
-----------------------------------
Summary: OOM errors in jobServer when using GCS artifactDir
Key: BEAM-6923
URL: https://issues.apache.org/jira/browse/BEAM-6923
Project: Beam
Issue Type: Bug
Components: sdk-java-harness
Reporter: Lukasz Gajowy
When starting jobServer with artifactDir pointing to a GCS bucket:
{code:java}
./gradlew :beam-runners-flink_2.11-job-server:runShadow
-PflinkMasterUrl=localhost:8081 -PartifactsDir=gs://the-bucket{code}
and running a Java portable pipeline with the following, portability related
pipeline options:
{code:java}
--runner=PortableRunner --jobEndpoint=localhost:8099
--defaultEnvironmentType=DOCKER
--defaultEnvironmentConfig=gcr.io/<my-freshly-built-sdk-harness-image>/java:latest'{code}
I'm facing a series of OOM errors, like this:
{code:java}
Exception in thread "grpc-default-executor-3" java.lang.OutOfMemoryError: Java
heap space
at
com.google.api.client.googleapis.media.MediaHttpUploader.buildContentChunk(MediaHttpUploader.java:606)
at
com.google.api.client.googleapis.media.MediaHttpUploader.resumableUpload(MediaHttpUploader.java:408)
at
com.google.api.client.googleapis.media.MediaHttpUploader.upload(MediaHttpUploader.java:336)
at
com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:508)
at
com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
at
com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:549)
at
com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation.call(AbstractGoogleAsyncWriteChannel.java:301)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745){code}
This does not happen when I'm using a local filesystem for the artifact staging
location.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
