[
https://issues.apache.org/jira/browse/YARN-1039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14310328#comment-14310328
]
Carlo Curino commented on YARN-1039:
------------------------------------
Tossing some fire back on duration. I read your concerns of applications
ability to provide good values,
however, I'd rather have the app providing their best duration estimate (and
the framework rounding it
or bucketing it), than the app providing a coarse grained tag-based version in
the first place.
Changing cluster configurations and policies might turn what used to be a
"short" task into something
not that short, which we want to handle differently and so on. In a sense
asking for "duration" prevent
us to rely on what application will judge as "short/long" etc..
As another example, based on whatever mechanisms for log aggregation we will
have in the future,
we can change our mind about what are the "cut-points" for short/long etc.. For
example, because a
new technique makes it very cheap and we want to provide much more frequent
feedback to users.
Bottom line, I find duration a rather "neutral" thing to ask, vs something
which is more "opinion-based",
and corner cases like never-ending services are easily handled with -1 or +inf
values.
I also agree that there are many other use cases for tags, that emerged in the
discussion, which have
a clear value and are by no means covered by duration.
> Add parameter for YARN resource requests to indicate "long lived"
> -----------------------------------------------------------------
>
> Key: YARN-1039
> URL: https://issues.apache.org/jira/browse/YARN-1039
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
> Affects Versions: 3.0.0, 2.1.1-beta
> Reporter: Steve Loughran
> Assignee: Craig Welch
> Attachments: YARN-1039.1.patch, YARN-1039.2.patch, YARN-1039.3.patch
>
>
> A container request could support a new parameter "long-lived". This could be
> used by a scheduler that would know not to host the service on a transient
> (cloud: spot priced) node.
> Schedulers could also decide whether or not to allocate multiple long-lived
> containers on the same node
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
