[
https://issues.apache.org/jira/browse/BEAM-6923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16936197#comment-16936197
]
Ankur Goenka commented on BEAM-6923:
------------------------------------
ACK:
I am able to reproduce it on MAC
Environment:
Java 1.8
Flink 1.5.6
goenka@goenka-macbookpro:~/d/work/beam/beam$ ./gradlew
runners:flink:1.5:job-server:runShadow -PflinkMasterUrl=localhost:8081
-PartifactsDir="gs://clouddfe-goenka/tmp/t0"
Configuration on demand is an incubating feature.
> Task :runners:flink:1.5:job-server:runShadow
Listening for transport dt_socket at address: 5005
[main] INFO org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver -
ArtifactStagingService started on localhost:8098
[main] INFO org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver -
Java ExpansionService started on localhost:8097
[main] INFO org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver -
JobService started on localhost:8099
Exception in thread "grpc-default-executor-70" java.lang.OutOfMemoryError: Java
heap space
at
com.google.api.client.googleapis.media.MediaHttpUploader.buildContentChunk(MediaHttpUploader.java:606)
at
com.google.api.client.googleapis.media.MediaHttpUploader.resumableUpload(MediaHttpUploader.java:408)
at
com.google.api.client.googleapis.media.MediaHttpUploader.upload(MediaHttpUploader.java:336)
at
com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:508)
at
com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
at
com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:549)
at
com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation.call(AbstractGoogleAsyncWriteChannel.java:301)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Exception in thread "grpc-default-executor-146" java.lang.OutOfMemoryError:
Java heap space
at
com.google.api.client.googleapis.media.MediaHttpUploader.buildContentChunk(MediaHttpUploader.java:606)
at
com.google.api.client.googleapis.media.MediaHttpUploader.resumableUpload(MediaHttpUploader.java:408)
at
com.google.api.client.googleapis.media.MediaHttpUploader.upload(MediaHttpUploader.java:336)
at
com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:508)
at
com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
at
com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:549)
at
com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation.call(AbstractGoogleAsyncWriteChannel.java:301)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Exception in thread "grpc-default-executor-50" java.lang.OutOfMemoryError: Java
heap space
Exception in thread "grpc-default-executor-23" java.lang.OutOfMemoryError: Java
heap space
> OOM errors in jobServer when using GCS artifactDir
> --------------------------------------------------
>
> Key: BEAM-6923
> URL: https://issues.apache.org/jira/browse/BEAM-6923
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-harness
> Reporter: Lukasz Gajowy
> Assignee: Ankur Goenka
> Priority: Major
> Attachments: Instance counts.png, Paths to GC root.png,
> Telemetries.png, beam6923-flink156.m4v, beam6923flink182.m4v, heapdump
> size-sorted.png
>
>
> When starting jobServer with artifactDir pointing to a GCS bucket:
> {code:java}
> ./gradlew :beam-runners-flink_2.11-job-server:runShadow
> -PflinkMasterUrl=localhost:8081 -PartifactsDir=gs://the-bucket{code}
> and running a Java portable pipeline with the following, portability related
> pipeline options:
> {code:java}
> --runner=PortableRunner --jobEndpoint=localhost:8099
> --defaultEnvironmentType=DOCKER
> --defaultEnvironmentConfig=gcr.io/<my-freshly-built-sdk-harness-image>/java:latest'{code}
>
> I'm facing a series of OOM errors, like this:
> {code:java}
> Exception in thread "grpc-default-executor-3" java.lang.OutOfMemoryError:
> Java heap space
> at
> com.google.api.client.googleapis.media.MediaHttpUploader.buildContentChunk(MediaHttpUploader.java:606)
> at
> com.google.api.client.googleapis.media.MediaHttpUploader.resumableUpload(MediaHttpUploader.java:408)
> at
> com.google.api.client.googleapis.media.MediaHttpUploader.upload(MediaHttpUploader.java:336)
> at
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:508)
> at
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
> at
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:549)
> at
> com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation.call(AbstractGoogleAsyncWriteChannel.java:301)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745){code}
>
> This does not happen when I'm using a local filesystem for the artifact
> staging location.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
