[ 
https://issues.apache.org/jira/browse/HIVE-30262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated HIVE-30262:
----------------------------------
    Labels: pull-request-available  (was: )

>  LlapTaskSchedulerService writes 8 INFO lines per task attempt, a quarter of 
> the Tez AM log on large DAGs
> ---------------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-30262
>                 URL: https://issues.apache.org/jira/browse/HIVE-30262
>             Project: Hive
>          Issue Type: Improvement
>          Components: llap
>            Reporter: László Bodor
>            Assignee: László Bodor
>            Priority: Major
>              Labels: pull-request-available
>
> Summary: LlapTaskSchedulerService writes 8 INFO lines per task attempt, a 
> quarter of the Tez AM log on large DAGs
> Component: llap
> Type: Improvement
> h3. Problem
> The LLAP task scheduler plugin running in the Tez AM 
> ({{LlapTaskSchedulerService}}, plus its {{GuaranteedTasks}} WM logger) writes 
> *8 INFO lines for every task attempt*, whether workload management is in use 
> or not. On large DAGs that makes it one of the biggest contributors to the AM 
> log, and the volume grows with the number of task attempts rather than with 
> anything an operator usually needs to see.
> h3. Measurement
> Setup:
> * 12 LLAP daemons × 12 executors, one standalone Tez AM.
> * TPC-DS scale factor 10000 (10 TB, partitioned Parquet).
> * One pass of 10 queries (q4, q14, q23, q51, q64, q67, q72, q75, q78, q95), 
> 24 minutes wall clock, *162,512 task attempts*.
> * Default AM log4j2 configuration (root logger at INFO).
> Whole AM log for the pass: *4,070,638 lines, 1216 MB* (about 3 GB per hour).
> The 8 per-attempt messages from this class:
> ||Logger||Message||Thread||Lines||MB||
> |LlapTaskSchedulerService|Received allocateRequest. task=..., priority=..., 
> capability=...|TaskSchedulerEventHandlerThread|162,512|34.4|
> |LlapTaskSchedulerService|PendingTasksInfo=numPriorityLevels=... (counts per 
> priority level and queue)|TaskSchedulerEventHandlerThread, under 
> writeLock|162,512|53.5|
> |LlapTaskSchedulerService|Assigning {node} when looking for 
> ...|LlapScheduler, under readLock|162,512|51.6|
> |LlapTaskSchedulerService|Assigned #N, task=TaskInfo{...} on node={...}, to 
> container=...|LlapScheduler, under writeLock|162,512|70.7|
> |LlapTaskSchedulerService|Processing de-allocate request for task=..., 
> state=..., endReason=...|TaskSchedulerEventHandlerThread, under 
> writeLock|162,512|33.1|
> |GuaranteedTasks|Registering attempt_..., guaranteed: false|LlapScheduler, 
> under writeLock|162,512|20.8|
> |GuaranteedTasks|Received response for attempt_..., true|task communicator 
> callback|162,432|20.8|
> |GuaranteedTasks|Unregistering attempt_...; 
> false|TaskSchedulerEventHandlerThread, under writeLock|162,512|22.0|
> |*Total*| | |*1,300,016*|*308*|
> That is:
> * *32 % of all AM log lines and 25 % of its bytes*, from a single scheduler 
> plugin;
> * *8 lines, about 2 KB, per task attempt*;
> * about *900 lines/s, roughly 770 MB per hour*, while the cluster is busy.
> All other lines of both loggers together came to 819 in the same pass: the 
> 10-second "Stats for current dag" summary, preemption, node disable/enable, 
> DAG start/end, errors and warnings.
> The volume is proportional to the number of task attempts, so it scales with 
> data size and cluster width, not with query count.
> Example sequence for one attempt (host names replaced):
> {code}
> INFO [TaskSchedulerEventHandlerThread] tezplugins.LlapTaskSchedulerService: 
> Received allocateRequest. task=attempt_1794566684298_0015_1_06_000000_0, 
> priority=80, capability=<memory:2304, vCores:1>
> INFO [TaskSchedulerEventHandlerThread] tezplugins.LlapTaskSchedulerService: 
> PendingTasksInfo=numPriorityLevels=1. [p=80,c=1]. runningTasks=0. 
> totalPendingTasks=1. delayedTaskQueueSize=0. highPriTaskQueueSize=0. 
> preemptTaskQueueSize=0. pendingPreemptions=0
> INFO [LlapScheduler] tezplugins.LlapTaskSchedulerService: Assigning 
> {host-1.example.com:15001, id=..., canAcceptTask=true, st=0, ac=22, 
> commF=false, disabled=false} when looking for host-1.example.com. local=true 
> FirstRequestedHost=true, #prefLocations=1
> INFO [LlapScheduler] tezplugins.LlapTaskSchedulerService: Assigned #1, 
> task=TaskInfo{task=attempt_1794566684298_0015_1_06_000000_0, priority=80, 
> startTime=0, containerId=null, uniqueId=0, localityDelayTimeout=64146377} on 
> node={host-1.example.com:15001, ...}, to 
> container=container_111101111_0015_01_000003
> INFO [LlapScheduler] GuaranteedTasks: Registering 
> attempt_1794566684298_0015_1_06_000000_0, guaranteed: false
> INFO [pool-17-thread-1] GuaranteedTasks: Received response for 
> attempt_1794566684298_0015_1_06_000000_0, true
> INFO [TaskSchedulerEventHandlerThread] GuaranteedTasks: Unregistering 
> attempt_1794566684298_0015_1_06_000000_0; false
> INFO [TaskSchedulerEventHandlerThread] tezplugins.LlapTaskSchedulerService: 
> Processing de-allocate request for 
> task=attempt_1794566684298_0015_1_06_000000_0, state=ASSIGNED, endReason=null
> {code}
> The per-attempt facts these lines carry are already in the AM log at INFO 
> from Tez itself:
> * TaskAttemptImpl "submitted. Is using containerId: [...] on NM: [host:port]" 
> gives the node and container;
> * HistoryEventHandler TASK_ATTEMPT_STARTED / TASK_ATTEMPT_FINISHED give the 
> lifecycle;
> * LlapTaskCommunicator "Successfully launched task" confirms the launch.
> The aggregate view stays available through the periodic "Stats for current 
> dag" line.
> Secondary points:
> * Five of the eight lines are written while holding the scheduler's 
> {{writeLock}} (the {{ReentrantReadWriteLock}} that the LlapScheduler loop, 
> deallocation, node enable/disable and the communicator callbacks all contend 
> on).
> * {{PendingTasksInfo}}: {{constructPendingTaskCountsLogMessage()}} walks 
> every pending priority level to build the message. The call is evaluated 
> eagerly, under that lock, even when INFO is disabled. The same message in 
> {{schedulePendingTasks}} is already guarded by {{isDebugEnabled()}} and 
> logged at DEBUG.
> * With a console or file appender each line is a page-cache write, so the 
> extra lock hold time per call is small (not measured). The main cost is log 
> volume: disk, rotation, and log shipping/indexing.
> * The "Assigning ... in consistent order" INFO line prints 
> {{requestedHosts=null}} unless DEBUG is enabled, because 
> {{requestedHostsDebugStr}} is only filled in inside an {{isDebugEnabled()}} 
> block.
> h3. Proposed change
> Move the 8 per-attempt messages to DEBUG:
> * use {{isDebugEnabled()}} guards where an argument is computed eagerly: 
> {{constructPendingTaskCountsLogMessage()}}, {{NodeInfo.toShortString()}};
> * switch the {{GuaranteedTasks}} string concatenations to parameterized 
> logging.
> Everything else stays at INFO:
> * the periodic stats line;
> * preemption and node state changes;
> * the WM aggregate messages ("Distributing", "Revoking", "Setting unused", 
> ...);
> * DAG lifecycle, warnings and errors.
> The detail stays one setting away. Because {{GuaranteedTasks}} is its own 
> logger, it can be enabled independently for WM debugging:
> {code}
> logger.llapsched.name = 
> org.apache.hadoop.hive.llap.tezplugins.LlapTaskSchedulerService
> logger.llapsched.level = DEBUG
> logger.wm.name = GuaranteedTasks
> logger.wm.level = DEBUG
> {code}
> Workaround on current releases: set both loggers to WARN in the Tez AM's 
> log4j2 configuration. This also silences the non-per-task INFO lines listed 
> above.
> Expected effect on the measured pass: −1.3 M lines and −308 MB per 24 minutes 
> in the AM log (−32 % lines, −25 % bytes), with no change to scheduling 
> behaviour.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to