Carlo Curino commented on YARN-1039:

Tossing some fire back on duration. I read your concerns of applications 
ability to provide good values, 
however, I'd rather have the app providing their best duration estimate (and 
the framework rounding it 
or bucketing it), than the app providing a coarse grained tag-based version in 
the first place. 

Changing cluster configurations and policies might turn what used to be a 
"short" task into something 
not that short, which we want to handle differently and so on. In a sense 
asking for "duration" prevent 
us to rely on what application will judge as "short/long" etc.. 

As another example, based on whatever mechanisms for log aggregation we will 
have in the future, 
we can change our mind about what are the "cut-points" for short/long etc.. For 
example, because a
new technique makes it very cheap and we want to provide much more frequent 
feedback to users.

Bottom line, I find duration a rather "neutral" thing to ask, vs something 
which is more "opinion-based",
and corner cases like never-ending services are easily handled with -1 or +inf 

I also agree that there are many other use cases for tags, that emerged in the 
discussion, which have
a clear value and are by no means covered by duration.

> Add parameter for YARN resource requests to indicate "long lived"
> -----------------------------------------------------------------
>                 Key: YARN-1039
>                 URL: https://issues.apache.org/jira/browse/YARN-1039
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>    Affects Versions: 3.0.0, 2.1.1-beta
>            Reporter: Steve Loughran
>            Assignee: Craig Welch
>         Attachments: YARN-1039.1.patch, YARN-1039.2.patch, YARN-1039.3.patch
> A container request could support a new parameter "long-lived". This could be 
> used by a scheduler that would know not to host the service on a transient 
> (cloud: spot priced) node.
> Schedulers could also decide whether or not to allocate multiple long-lived 
> containers on the same node

This message was sent by Atlassian JIRA

Reply via email to