Re: Troubleshooting live Java process in Dataflow

[Marián Dvorský]

The worker exports an HTTP server with a handler for "/threadz":

$ curl localhost:8081/threadz

gives the stack traces for all Java threads.

On Fri, Dec 8, 2017 at 2:00 AM, Jacob Marble  wrote:

> In this post, Reuven says:
> "we had to ssh to the VMs to get actual thread profiles from workers.
> After a bit of digging, we found threads were often stuck in the following
> stack trace"
>
> Can someone describe what tools you use to do this?
>
> I logged into a Dataflow runner, found that the gc is thrashing. Neat! Now
> I'm trying to get a thread dump. Looks like the Dataflow runner is actually
> running in a container within the VM, and the host VM doesn't have jmap or
> any j* utility installed.
>
> Tried kill -3, didn't seem to trigger a thread dump.
>
> Also found an open JMX port, but only hangs VisualVM and JConsole.
>
> Jacob
>

Troubleshooting live Java process in Dataflow

[Jacob Marble]

In this post, Reuven says:
"we had to ssh to the VMs to get actual thread profiles from workers. After
a bit of digging, we found threads were often stuck in the following stack
trace"

Can someone describe what tools you use to do this?

I logged into a Dataflow runner, found that the gc is thrashing. Neat! Now
I'm trying to get a thread dump. Looks like the Dataflow runner is actually
running in a container within the VM, and the host VM doesn't have jmap or
any j* utility installed.

Tried kill -3, didn't seem to trigger a thread dump.

Also found an open JMX port, but only hangs VisualVM and JConsole.

Jacob