[ 
https://issues.apache.org/jira/browse/HIVE-30262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

László Bodor updated HIVE-30262:
--------------------------------
    Description: 
h3. Problem

The LLAP task scheduler plugin running in the Tez AM 
({{LlapTaskSchedulerService}}, plus its {{GuaranteedTasks}} WM logger) writes 
*8 INFO lines for every task attempt*, whether workload management is in use or 
not. On large DAGs that makes it one of the biggest contributors to the AM log, 
and the volume grows with the number of task attempts rather than with anything 
an operator usually needs to see.

h3. Measurement

Setup:
* 12 LLAP daemons × 12 executors, one standalone Tez AM.
* TPC-DS scale factor 10000 (10 TB, partitioned Parquet).
* One pass of 10 queries (q4, q14, q23, q51, q64, q67, q72, q75, q78, q95), 24 
minutes wall clock, *162,512 task attempts*.
* Default AM log4j2 configuration (root logger at INFO).

Whole AM log for the pass: *4,070,638 lines, 1216 MB* (about 3 GB per hour).

The 8 per-attempt messages from this class:

||Logger||Message||Thread||Lines||MB||
|LlapTaskSchedulerService|Received allocateRequest. task=..., priority=..., 
capability=...|hread|162,512|34.4|
|LlapTaskSchedulerService|PendingTasksInfo=numPriorityLevels=... (counts per 
priority level and queue)|TaskSchedulerEventHandlerThread, under 
writeLock|162,512|53.5|
|LlapTaskSchedulerService|Assigning {node} when looking for ...|LlapScheduler, 
under readLo
|LlapTaskSchedulerService|Assigned #N, task=TaskInfo{...} on node={...}, to 
container=...|LlapScheduler, under writeLock|162,512|70.7|
|LlapTaskSchedulerService|Processing de-allocate request for task=..., 
state=..., endReasondlerThread, under writeLock|162,512|33.1|
|GuaranteedTasks|Registering attempt_..., guaranteed: false|LlapScheduler, 
under writeLock|162,512|20.8|
|GuaranteedTasks|Received response for attempt_..., true|task communicator 
callback|162,432
|GuaranteedTasks|Unregistering attempt_...; 
false|TaskSchedulerEventHandlerThread, under writeLock|162,512|22.0|
|*Total*| | |*1,300,016*|*308*|

That is:
* *32 % of all AM log lines and 25 % of its bytes*, from a single scheduler 
plugin;
* *8 lines, about 2 KB, per task attempt*;
* about *900 lines/s, roughly 770 MB per hour*, while the cluster is busy.

All other lines of both loggers together came to 819 in the same pass: the 
10-second "Stats for current dag" summary, preemption, node disable/enable, DAG 
start/end, errors and warnings.

The volume is proportional to the number of task attempts, so it scales with 
data size and cluster width, not with query count.

Example sequence for one attempt (host names replaced):
{code}
INFO [TaskSchedulerEventHandlerThread] tezplugins.LlapTaskSchedulerService: 
Received allocateRequest. task=attempt_1794566684298_0015_1_06_000000_0, 
priority=80, capability=<memory:2304, vCores:1>
INFO [TaskSchedulerEventHandlerThread] tezplugins.LlapTaskSchedulerService: 
PendingTasksInf80,c=1]. runningTasks=0. totalPendingTasks=1. 
delayedTaskQueueSize=0. highPriTaskQueueSize=0. preemptTaskQueueSize=0. 
pendingPreemptions=0
INFO [LlapScheduler] tezplugins.LlapTaskSchedulerService: Assigning 
{host-1.example.com:150true, st=0, ac=22, commF=false, disabled=false} when 
looking for host-1.example.com.local=true FirstRequestedHost=true, 
#prefLocations=1
INFO [LlapScheduler] tezplugins.LlapTaskSchedulerService: Assigned #1, 
task=TaskInfo{task=a1_06_000000_0, priority=80, startTime=0, containerId=null, 
uniqueId=0,localityDelayTimeout=64146377} on node={host-1.example.com:15001, 
...}, to container=container_111101111_0015_01_000003
INFO [LlapScheduler] GuaranteedTasks: Registering 
attempt_1794566684298_0015_1_06_000000_0,
INFO [pool-17-thread-1] GuaranteedTasks: Received response for 
attempt_1794566684298_0015_1_06_000000_0, true
INFO [TaskSchedulerEventHandlerThread] GuaranteedTasks: Unregistering 
attempt_1794566684298
INFO [TaskSchedulerEventHandlerThread] tezplugins.LlapTaskSchedulerService: 
Processing de-allocate request for 
task=attempt_1794566684298_0015_1_06_000000_0, state=ASSIGNED, endReason=null
{code}

The per-attempt facts these lines carry are already in the AM log at INFO from 
Tez itself:
* TaskAttemptImpl "submitted. Is using containerId: [...] on NM: [host:port]" 
gives the node and container;
* HistoryEventHandler TASK_ATTEMPT_STARTED / TASK_ATTEMPT_FINISHED give the 
lifecycle;
* LlapTaskCommunicator "Successfully launched task" confirms the launch.

The aggregate view stays available through the periodic "Stats for current dag" 
line.

Secondary points:
* Five of the eight lines are written while holding the scheduler's 
{{writeLock}} (the {{Ret the LlapScheduler loop, deallocation, node 
enable/disable and the communicator callbacksall contend on).
* {{PendingTasksInfo}}: {{constructPendingTaskCountsLogMessage()}} walks every 
pending priossage. The call is evaluated eagerly, under that lock, even when 
INFO is disabled. The samemessage in {{schedulePendingTasks}} is already 
guarded by {{isDebugEnabled()}} and logged at DEBUG.
* With a console or file appender each line is a page-cache write, so the extra 
lock hold t measured). The main cost is log volume: disk, rotation, and log 
shipping/indexing.
* The "Assigning ... in consistent order" INFO line prints 
{{requestedHosts=null}} unless DEBUG is enabled, because 
{{requestedHostsDebugStr}} is only filled in inside an {{isDebugEnabled()}} 
block.

h3. Proposed change

Move the 8 per-attempt messages to DEBUG:
* use {{isDebugEnabled()}} guards where an argument is computed eagerly: 
{{constructPending{{NodeInfo.toShortString()}};
* switch the {{GuaranteedTasks}} string concatenations to parameterized logging.

Everything else stays at INFO:
* the periodic stats line;
* preemption and node state changes;
* the WM aggregate messages ("Distributing", "Revoking", "Setting unused", ...);
* DAG lifecycle, warnings and errors.

The detail stays one setting away. Because {{GuaranteedTasks}} is its own 
logger, it can be enabled independently for WM debugging:
{code}
logger.llapsched.name = 
org.apache.hadoop.hive.llap.tezplugins.LlapTaskSchedulerService
logger.llapsched.level = DEBUG
logger.wm.name = GuaranteedTasks
logger.wm.level = DEBUG
{code}

Workaround on current releases: set both loggers to WARN in the Tez AM's log4j2 
configuration. This also silences the non-per-task INFO lines listed above.

Expected effect on the measured pass: −1.3 M lines and −308 MB per 24 minutes 
in the AM log (−32 % lines, −25 % bytes), with no change to scheduling 
behaviour. 

>  LlapTaskSchedulerService writes 8 INFO lines per task attempt, a quarter of 
> the Tez AM log on large DAGs
> ---------------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-30262
>                 URL: https://issues.apache.org/jira/browse/HIVE-30262
>             Project: Hive
>          Issue Type: Improvement
>          Components: llap
>            Reporter: László Bodor
>            Assignee: László Bodor
>            Priority: Major
>
> h3. Problem
> The LLAP task scheduler plugin running in the Tez AM 
> ({{LlapTaskSchedulerService}}, plus its {{GuaranteedTasks}} WM logger) writes 
> *8 INFO lines for every task attempt*, whether workload management is in use 
> or not. On large DAGs that makes it one of the biggest contributors to the AM 
> log, and the volume grows with the number of task attempts rather than with 
> anything an operator usually needs to see.
> h3. Measurement
> Setup:
> * 12 LLAP daemons × 12 executors, one standalone Tez AM.
> * TPC-DS scale factor 10000 (10 TB, partitioned Parquet).
> * One pass of 10 queries (q4, q14, q23, q51, q64, q67, q72, q75, q78, q95), 
> 24 minutes wall clock, *162,512 task attempts*.
> * Default AM log4j2 configuration (root logger at INFO).
> Whole AM log for the pass: *4,070,638 lines, 1216 MB* (about 3 GB per hour).
> The 8 per-attempt messages from this class:
> ||Logger||Message||Thread||Lines||MB||
> |LlapTaskSchedulerService|Received allocateRequest. task=..., priority=..., 
> capability=...|hread|162,512|34.4|
> |LlapTaskSchedulerService|PendingTasksInfo=numPriorityLevels=... (counts per 
> priority level and queue)|TaskSchedulerEventHandlerThread, under 
> writeLock|162,512|53.5|
> |LlapTaskSchedulerService|Assigning {node} when looking for 
> ...|LlapScheduler, under readLo
> |LlapTaskSchedulerService|Assigned #N, task=TaskInfo{...} on node={...}, to 
> container=...|LlapScheduler, under writeLock|162,512|70.7|
> |LlapTaskSchedulerService|Processing de-allocate request for task=..., 
> state=..., endReasondlerThread, under writeLock|162,512|33.1|
> |GuaranteedTasks|Registering attempt_..., guaranteed: false|LlapScheduler, 
> under writeLock|162,512|20.8|
> |GuaranteedTasks|Received response for attempt_..., true|task communicator 
> callback|162,432
> |GuaranteedTasks|Unregistering attempt_...; 
> false|TaskSchedulerEventHandlerThread, under writeLock|162,512|22.0|
> |*Total*| | |*1,300,016*|*308*|
> That is:
> * *32 % of all AM log lines and 25 % of its bytes*, from a single scheduler 
> plugin;
> * *8 lines, about 2 KB, per task attempt*;
> * about *900 lines/s, roughly 770 MB per hour*, while the cluster is busy.
> All other lines of both loggers together came to 819 in the same pass: the 
> 10-second "Stats for current dag" summary, preemption, node disable/enable, 
> DAG start/end, errors and warnings.
> The volume is proportional to the number of task attempts, so it scales with 
> data size and cluster width, not with query count.
> Example sequence for one attempt (host names replaced):
> {code}
> INFO [TaskSchedulerEventHandlerThread] tezplugins.LlapTaskSchedulerService: 
> Received allocateRequest. task=attempt_1794566684298_0015_1_06_000000_0, 
> priority=80, capability=<memory:2304, vCores:1>
> INFO [TaskSchedulerEventHandlerThread] tezplugins.LlapTaskSchedulerService: 
> PendingTasksInf80,c=1]. runningTasks=0. totalPendingTasks=1. 
> delayedTaskQueueSize=0. highPriTaskQueueSize=0. preemptTaskQueueSize=0. 
> pendingPreemptions=0
> INFO [LlapScheduler] tezplugins.LlapTaskSchedulerService: Assigning 
> {host-1.example.com:150true, st=0, ac=22, commF=false, disabled=false} when 
> looking for host-1.example.com.local=true FirstRequestedHost=true, 
> #prefLocations=1
> INFO [LlapScheduler] tezplugins.LlapTaskSchedulerService: Assigned #1, 
> task=TaskInfo{task=a1_06_000000_0, priority=80, startTime=0, 
> containerId=null, uniqueId=0,localityDelayTimeout=64146377} on 
> node={host-1.example.com:15001, ...}, to 
> container=container_111101111_0015_01_000003
> INFO [LlapScheduler] GuaranteedTasks: Registering 
> attempt_1794566684298_0015_1_06_000000_0,
> INFO [pool-17-thread-1] GuaranteedTasks: Received response for 
> attempt_1794566684298_0015_1_06_000000_0, true
> INFO [TaskSchedulerEventHandlerThread] GuaranteedTasks: Unregistering 
> attempt_1794566684298
> INFO [TaskSchedulerEventHandlerThread] tezplugins.LlapTaskSchedulerService: 
> Processing de-allocate request for 
> task=attempt_1794566684298_0015_1_06_000000_0, state=ASSIGNED, endReason=null
> {code}
> The per-attempt facts these lines carry are already in the AM log at INFO 
> from Tez itself:
> * TaskAttemptImpl "submitted. Is using containerId: [...] on NM: [host:port]" 
> gives the node and container;
> * HistoryEventHandler TASK_ATTEMPT_STARTED / TASK_ATTEMPT_FINISHED give the 
> lifecycle;
> * LlapTaskCommunicator "Successfully launched task" confirms the launch.
> The aggregate view stays available through the periodic "Stats for current 
> dag" line.
> Secondary points:
> * Five of the eight lines are written while holding the scheduler's 
> {{writeLock}} (the {{Ret the LlapScheduler loop, deallocation, node 
> enable/disable and the communicator callbacksall contend on).
> * {{PendingTasksInfo}}: {{constructPendingTaskCountsLogMessage()}} walks 
> every pending priossage. The call is evaluated eagerly, under that lock, even 
> when INFO is disabled. The samemessage in {{schedulePendingTasks}} is already 
> guarded by {{isDebugEnabled()}} and logged at DEBUG.
> * With a console or file appender each line is a page-cache write, so the 
> extra lock hold t measured). The main cost is log volume: disk, rotation, and 
> log shipping/indexing.
> * The "Assigning ... in consistent order" INFO line prints 
> {{requestedHosts=null}} unless DEBUG is enabled, because 
> {{requestedHostsDebugStr}} is only filled in inside an {{isDebugEnabled()}} 
> block.
> h3. Proposed change
> Move the 8 per-attempt messages to DEBUG:
> * use {{isDebugEnabled()}} guards where an argument is computed eagerly: 
> {{constructPending{{NodeInfo.toShortString()}};
> * switch the {{GuaranteedTasks}} string concatenations to parameterized 
> logging.
> Everything else stays at INFO:
> * the periodic stats line;
> * preemption and node state changes;
> * the WM aggregate messages ("Distributing", "Revoking", "Setting unused", 
> ...);
> * DAG lifecycle, warnings and errors.
> The detail stays one setting away. Because {{GuaranteedTasks}} is its own 
> logger, it can be enabled independently for WM debugging:
> {code}
> logger.llapsched.name = 
> org.apache.hadoop.hive.llap.tezplugins.LlapTaskSchedulerService
> logger.llapsched.level = DEBUG
> logger.wm.name = GuaranteedTasks
> logger.wm.level = DEBUG
> {code}
> Workaround on current releases: set both loggers to WARN in the Tez AM's 
> log4j2 configuration. This also silences the non-per-task INFO lines listed 
> above.
> Expected effect on the measured pass: −1.3 M lines and −308 MB per 24 minutes 
> in the AM log (−32 % lines, −25 % bytes), with no change to scheduling 
> behaviour. 



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to