Jonathan Eagles commented on YARN-4183:

Here are the requirements that users at scale need, and unfortunately the 
config design does not allow for this properly. Let me draw up what the 
requirements in my mind should be based my current knowledge. This is by no 
means an edict, but just a conversation starting point, so you know where I'm 
coming from.

# Jobs that make use of the timeline service, may have a hard or soft runtime 
on the timeline service
-- Jobs that interact directly with the timeline service (TimelineClient) 
should obtain delegation token to use the service and optionally allow for 
non-fatal runtime dependency (job is allowed to run, but no history is written)
-- Jobs that don't interact with the timeline service 
(EntityFileTimelineClient), should obtain HDFS delegation tokens, but should 
not obtain timeline service delegation tokens.
# Jobs that don't make user of the timeline service, should have no runtime 
dependency on the timeline service and should be allowed freely to submit and 
run jobs if the regardless of the timeline service status.
# YARN services that interact with the timeline server (Generic History 
Server), may have runtime dependency of the timeline service that does not 
disrupt job submission.

The issue regarding this jira is that putting yarn.timeline-service.enabled in 
the client xml (breaks #2 above) forces every job (both MR (not using timeline 
service) and Tez (using timeline service)) to have a runtime dependency on the 
timeline service. This places an artificial runtime dependency on the timeline 
service which is not highly available or highly scalable until v2.0.

The issue regarding putting the yarn.timeline-service.enabled in the resource 
manager  (breaks #3 above) is that every YarnClientImpl (used in job status, 
used in job submission) now reaches out to get a delegation token token. This 
places the timeline service (neither highly scalable or highly available until 
v2.0) as a runtime dependency for job submission and get many unnecessary 
delegation token for YarnClients that never intent to use them.

> Enabling generic application history forces every job to get a timeline 
> service delegation token
> ------------------------------------------------------------------------------------------------
>                 Key: YARN-4183
>                 URL: https://issues.apache.org/jira/browse/YARN-4183
>             Project: Hadoop YARN
>          Issue Type: Bug
>    Affects Versions: 2.7.1
>            Reporter: Mit Desai
>            Assignee: Mit Desai
>             Fix For: 3.0.0, 2.8.0, 2.7.2
>         Attachments: YARN-4183.1.patch
> When enabling just the Generic History Server and not the timeline server, 
> the system metrics publisher will not publish the events to the timeline 
> store as it checks if the timeline server and system metrics publisher are 
> enabled before creating a timeline client.
> To make it work, if the timeline service flag is turned on, it will force 
> every yarn application to get a delegation token.
> Instead of checking if timeline service is enabled, we should be checking if 
> application history server is enabled.

This message was sent by Atlassian JIRA

Reply via email to