[
https://issues.apache.org/jira/browse/TEZ-3154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15175078#comment-15175078
]
Hitesh Shah edited comment on TEZ-3154 at 3/2/16 5:46 AM:
----------------------------------------------------------
bq. Creating this ticket to explore the possibility of adding thread-dump on
periodic basis for specific tasks.
This seems like a debugging only feature which needs to be pre-setup and would
be a lot of noise if enabled by default. The current description seems to imply
that the dag would need to be configured before it is executed unless there is
a way to do this while a dag is running.
What would be more useful is to trigger a thread dump when certain conditions
are hit:
- if there is a task timeout - this would trigger a thread dump before
killing the container
- if the user defines a policy that if certain counter thresholds are hit (
GC counters, etc ) then trigger it.
- command-line tool to trigger thread dumps on an demand basis if the user
wants to see what is going on within a task
Also, please take a look at the support yarn is introducing or has introduced
for sending a signal to a container - this would be useful to trigger a jstack
dump if needed.
was (Author: hitesh):
bq. Creating this ticket to explore the possibility of adding thread-dump on
periodic basis for specific tasks.
This seems like a debugging only feature which needs to be pre-setup and would
be a lot of noise if enabled by default. The current description seems to imply
that the dag would need to be configured before it is executed unless there is
a way to do this while a dag is running.
What would be more useful is to trigger a thread dump when certain conditions
are hit:
- if there is a task timeout - this would trigger a thread dump before
killing the container
- if the user defines a policy that if certain counter thresholds are hit (
GC counters, etc )
- command-line tool to trigger thread dumps on an demand basis if the user
wants to see what is going on within a task
Also, please take a look at the support yarn is introducing or has introduced
for sending a signal to a container - this would be useful to trigger a jstack
dump if needed.
> Debuggability : Add an option to take threaddump from a specific vertex/task
> ----------------------------------------------------------------------------
>
> Key: TEZ-3154
> URL: https://issues.apache.org/jira/browse/TEZ-3154
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Rajesh Balamohan
>
> tez.task-specific.launch.cmd-opts and tez.task-specific.launch.cmd-opts.list
> (e.g "Map 1[10]", 10th task in map 1) options are available to add certain
> parameters to task specific command line options. It has been useful for
> launching profilers on specific tasks.
> There are scenarios in which taking threaddumps on periodic basis on specific
> tasks could be helpful. E.g
> - In certain clusters it could be difficult to add profilers.
> - There could be scenarios where the tasks are slow due apps using Tez (but
> the counters might indicate no issues in Tez). (e.g Parsing using
> SimpleDateFormat for every record could be time consuming)
> - In certain clusters, access might not be there to take threaddumps of tasks
> from NM. YARN's threadstack (in RM UI) is mainly for NM and doesn't work on
> task level.
> Creating this ticket to explore the possibility of adding thread-dump on
> periodic basis for specific tasks.
> High level e.g: "--hiveconf tez.task-specific.launch.cmd-opts="
> -DthreadDumpInterval=5 " --hiveconf
> tez.task-specific.launch.cmd-opts.list="Map 1[10,15]" - This should print
> thread-dumps in tasks 10, 15 in Map-1 every 5 seconds.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
