Oh, I had overlooked --diskSizeGb. I did read the CAUTION, but I did not know how to increase it.
Unfortunately, I still can't get it to work. Does it tell anything that the GCP console does not show the options --dumpHeapOnOOM --saveHeapDumpsToGcsPath of a running job under PipelineOptions (it does for diskSizeGb)? On Mon, Nov 18, 2019 at 11:59 AM Jeff Klukas <[email protected]> wrote: > Using default Dataflow workers, this is the set of options I passed: > > --dumpHeapOnOOM --saveHeapDumpsToGcsPath=$MYBUCKET/heapdump > --diskSizeGb=100 > > > On Mon, Nov 18, 2019 at 11:57 AM Jeff Klukas <[email protected]> wrote: > >> It sounds like you're generally doing the right thing. I've successfully >> used --saveHeapDumpsToGcsPath in a Java pipeline running on Dataflow and >> inspected the results in Eclipse MAT. >> >> I think that --saveHeapDumpsToGcsPath will automatically turn on >> --dumpHeapOnOOM but worth setting that explicitly too. >> >> Are your boot disks large enough to store the heap dumps? The docs for >> getSaveHeapDumpsToGcsPath [0] mention "CAUTION: This option implies >> dumpHeapOnOOM, and has similar caveats. Specifically, heap dumps can of >> comparable size to the default boot disk. Consider increasing the boot disk >> size before setting this flag to true." >> >> When I've done this in the past, I definitely had to increase boot disk >> size (though I forget now what the relevant Dataflow option was). >> >> [0] >> https://beam.apache.org/releases/javadoc/2.16.0/org/apache/beam/runners/dataflow/options/DataflowPipelineDebugOptions.html >> >> On Mon, Nov 18, 2019 at 11:35 AM Reynaldo Baquerizo < >> [email protected]> wrote: >> >>> Hi all, >>> >>> We are running into OOM issues with one of our pipelines. They are not >>> reproducible with DirectRunner, only with Dataflow. >>> I tried --saveHeapDumpsToGcsPath, but it does not save any heap dump >>> (MyOptions extends DataflowPipelineDebugOptions) >>> I looked at the java process inside the docker container and it has >>> remote jmx enabled through port 5555, but outside traffic is firewalled. >>> >>> Beam SDK: 2.15.0 >>> >>> Any ideas? >>> >>> Cheers, >>> -- >>> Reynaldo >>> >>
