[jira] [Updated] (YARN-4081) Add support for multiple resource types in the Resource class
[ https://issues.apache.org/jira/browse/YARN-4081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-4081: Fix Version/s: 2.10.0 > Add support for multiple resource types in the Resource class > - > > Key: YARN-4081 > URL: https://issues.apache.org/jira/browse/YARN-4081 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Major > Fix For: 3.0.0, 3.1.0, 2.10.0 > > Attachments: YARN-4081-YARN-3926.001.patch, > YARN-4081-YARN-3926.002.patch, YARN-4081-YARN-3926.003.patch, > YARN-4081-YARN-3926.004.patch, YARN-4081-YARN-3926.005.patch, > YARN-4081-YARN-3926.006.patch, YARN-4081-YARN-3926.007.patch, > YARN-4081-YARN-3926.008.patch, YARN-4081.branch-3.0.001.patch > > > For adding support for multiple resource types, we need to add support for > this in the Resource class. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4081) Add support for multiple resource types in the Resource class
[ https://issues.apache.org/jira/browse/YARN-4081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-4081: --- Attachment: YARN-4081.branch-3.0.001.patch Adding branch-3.0 patch from resource-types branch prior to merge. > Add support for multiple resource types in the Resource class > - > > Key: YARN-4081 > URL: https://issues.apache.org/jira/browse/YARN-4081 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Fix For: 3.0.0, 3.1.0 > > Attachments: YARN-4081-YARN-3926.001.patch, > YARN-4081-YARN-3926.002.patch, YARN-4081-YARN-3926.003.patch, > YARN-4081-YARN-3926.004.patch, YARN-4081-YARN-3926.005.patch, > YARN-4081-YARN-3926.006.patch, YARN-4081-YARN-3926.007.patch, > YARN-4081-YARN-3926.008.patch, YARN-4081.branch-3.0.001.patch > > > For adding support for multiple resource types, we need to add support for > this in the Resource class. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4081) Add support for multiple resource types in the Resource class
[ https://issues.apache.org/jira/browse/YARN-4081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-4081: --- Fix Version/s: 3.0.0 > Add support for multiple resource types in the Resource class > - > > Key: YARN-4081 > URL: https://issues.apache.org/jira/browse/YARN-4081 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Fix For: 3.0.0, 3.1.0 > > Attachments: YARN-4081-YARN-3926.001.patch, > YARN-4081-YARN-3926.002.patch, YARN-4081-YARN-3926.003.patch, > YARN-4081-YARN-3926.004.patch, YARN-4081-YARN-3926.005.patch, > YARN-4081-YARN-3926.006.patch, YARN-4081-YARN-3926.007.patch, > YARN-4081-YARN-3926.008.patch > > > For adding support for multiple resource types, we need to add support for > this in the Resource class. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4081) Add support for multiple resource types in the Resource class
[ https://issues.apache.org/jira/browse/YARN-4081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-4081: - Fix Version/s: (was: YARN-3926) 3.1.0 > Add support for multiple resource types in the Resource class > - > > Key: YARN-4081 > URL: https://issues.apache.org/jira/browse/YARN-4081 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Fix For: 3.1.0 > > Attachments: YARN-4081-YARN-3926.001.patch, > YARN-4081-YARN-3926.002.patch, YARN-4081-YARN-3926.003.patch, > YARN-4081-YARN-3926.004.patch, YARN-4081-YARN-3926.005.patch, > YARN-4081-YARN-3926.006.patch, YARN-4081-YARN-3926.007.patch, > YARN-4081-YARN-3926.008.patch > > > For adding support for multiple resource types, we need to add support for > this in the Resource class. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4081) Add support for multiple resource types in the Resource class
[ https://issues.apache.org/jira/browse/YARN-4081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-4081: - Fix Version/s: (was: 2.8.0) YARN-3926 > Add support for multiple resource types in the Resource class > - > > Key: YARN-4081 > URL: https://issues.apache.org/jira/browse/YARN-4081 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Fix For: YARN-3926 > > Attachments: YARN-4081-YARN-3926.001.patch, > YARN-4081-YARN-3926.002.patch, YARN-4081-YARN-3926.003.patch, > YARN-4081-YARN-3926.004.patch, YARN-4081-YARN-3926.005.patch, > YARN-4081-YARN-3926.006.patch, YARN-4081-YARN-3926.007.patch, > YARN-4081-YARN-3926.008.patch > > > For adding support for multiple resource types, we need to add support for > this in the Resource class. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4081) Add support for multiple resource types in the Resource class
[ https://issues.apache.org/jira/browse/YARN-4081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-4081: Attachment: YARN-4081-YARN-3926.008.patch Attaching a new version of the patch without the web services changes. [~leftnoteasy] had concerns that we don't have existing tests to make sure teh web services changes won't break existing APIs. This will lead to failing unit tests which will be addressed in later patches(once we add unit tests to validate we won't break the REST API). > Add support for multiple resource types in the Resource class > - > > Key: YARN-4081 > URL: https://issues.apache.org/jira/browse/YARN-4081 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: YARN-4081-YARN-3926.001.patch, > YARN-4081-YARN-3926.002.patch, YARN-4081-YARN-3926.003.patch, > YARN-4081-YARN-3926.004.patch, YARN-4081-YARN-3926.005.patch, > YARN-4081-YARN-3926.006.patch, YARN-4081-YARN-3926.007.patch, > YARN-4081-YARN-3926.008.patch > > > For adding support for multiple resource types, we need to add support for > this in the Resource class. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4081) Add support for multiple resource types in the Resource class
[ https://issues.apache.org/jira/browse/YARN-4081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-4081: Attachment: YARN-4081-YARN-3926.007.patch Fix the whitespace issue and addressed some checkstyle issues. > Add support for multiple resource types in the Resource class > - > > Key: YARN-4081 > URL: https://issues.apache.org/jira/browse/YARN-4081 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: YARN-4081-YARN-3926.001.patch, > YARN-4081-YARN-3926.002.patch, YARN-4081-YARN-3926.003.patch, > YARN-4081-YARN-3926.004.patch, YARN-4081-YARN-3926.005.patch, > YARN-4081-YARN-3926.006.patch, YARN-4081-YARN-3926.007.patch > > > For adding support for multiple resource types, we need to add support for > this in the Resource class. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4081) Add support for multiple resource types in the Resource class
[
https://issues.apache.org/jira/browse/YARN-4081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Vasudev updated YARN-4081:
Attachment: YARN-4081-YARN-3926.006.patch
{quote}
1)
For NONE/UNBOUNDED, maybe we don't need to initialize a real map for it. For
example, when query resource information of NONE, it should return
ResourceInformation.value=0 for all given resource type. And the returned
getResources() map is not a real map too, it should return a fake map that
always returns ResourceInformation for any given resource type.
{quote}
For now, I've left it as it is. Once we have the units stuff figured out, we
can re-visit this. The specific case we need to figure out is what to return
for the getResources() call.
{quote}
2)
Is it possible to merge implementation of NONE/UNBOUNDED?
{quote}
Fixed.
{quote}
And I found ResourceRequestInfo changes aren't related to this patch, should
they be better moved to the other following patch? (such as support multiple
resource types in REST API).
{quote}
The problem is that without the ResourceRequestInfo class, the RM web services
tests fail. This is due to some JAXB JSON generation but I wasn't able to
figure out the root cause.
I've also renamed the ResourceMapEntry proto to ResourceInformationProto in the
latest patch to get rid of any confusion.
> Add support for multiple resource types in the Resource class
> -
>
> Key: YARN-4081
> URL: https://issues.apache.org/jira/browse/YARN-4081
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
> Attachments: YARN-4081-YARN-3926.001.patch,
> YARN-4081-YARN-3926.002.patch, YARN-4081-YARN-3926.003.patch,
> YARN-4081-YARN-3926.004.patch, YARN-4081-YARN-3926.005.patch,
> YARN-4081-YARN-3926.006.patch
>
>
> For adding support for multiple resource types, we need to add support for
> this in the Resource class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4081) Add support for multiple resource types in the Resource class
[ https://issues.apache.org/jira/browse/YARN-4081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-4081: Attachment: YARN-4081-YARN-3926.005.patch [~leftnoteasy] pointed out that the current patch allows NONE and UNBOUNDED to be modified. Uploaded a new patch to fix that. > Add support for multiple resource types in the Resource class > - > > Key: YARN-4081 > URL: https://issues.apache.org/jira/browse/YARN-4081 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: YARN-4081-YARN-3926.001.patch, > YARN-4081-YARN-3926.002.patch, YARN-4081-YARN-3926.003.patch, > YARN-4081-YARN-3926.004.patch, YARN-4081-YARN-3926.005.patch > > > For adding support for multiple resource types, we need to add support for > this in the Resource class. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4081) Add support for multiple resource types in the Resource class
[
https://issues.apache.org/jira/browse/YARN-4081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Vasudev updated YARN-4081:
Attachment: YARN-4081-YARN-3926.003.patch
{quote}
1) Uses String instead of URI for resource key? I think it maybe more efficient
to use String, it will be easier to construct when using it, uses less resource
(hasn't tested, but I think it will be true according to #fields in String and
URI). I can understand the motivation of solving conflicts of resource
namespace, but I think namespace conflict is not the major use case AND String
can define namespace as well.
{quote}
Fixed.
{quote}
2) Relationship between ResourceInformation and ResourceMapEntry: currently
it's 1-1 mapping, a ResourceInformation has value/unit from ResourceMapEntry,
they're kind of overlapping and also confusing. I think it's better to make
ResourceInformation to be one for each resource type. ResourceMapEntry contains
runtime information, and ResourceInformation contains configured information.
This will also avoid create ResourceInformation instance when invoking
Resource.getResourceInformation()
3) Resource unit: I like the design which can easily convert a internal value
to human-readable value. But I think maybe we don't need to support define unit
in ResourceMapEntry. There're some cons of it:
When we doing comparision of resources, we have to convert units, it's an
extra overhead.
It doesn't make a lot of sense to me that keep internal unit of resources:
We should handle it when constructing Resource (something like
Resource.newInstance("memory", 12, "GB")). And we will use the standard unit to
do internal computations.
We can define the standard unit in each "ResourceInformation" if you agree
with #2.
{quote}
I spoke with Wangda offline an we agree that it makes more sense to do
performance testing once we have the DRC changes in. Since this patch is going
in to a branch, there's no issue committing and running a full suite of
performance tests once we have the DRC changes in.
{quote}
4) Do you think it's better to have a global ResourceInformation map instead of
storing it in each Resource instance?
{quote}
For now, I'd like to keep it per-resource instance but if it becomes an
overhead, we can make it a global instance.
{quote}
5) Resource#compareTo/hashCode has debug logging.
{quote}
Fixed.
{quote}
6) It seems not necessary to instance ArrayList in Resource#compareTo. Just
traverse the set can avoid create the temporary ArrayList.
{quote}
Good point. I decided to use the size of the set itself as the sort order and
avoid the issue altogether.
> Add support for multiple resource types in the Resource class
> -
>
> Key: YARN-4081
> URL: https://issues.apache.org/jira/browse/YARN-4081
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
> Attachments: YARN-4081-YARN-3926.001.patch,
> YARN-4081-YARN-3926.002.patch, YARN-4081-YARN-3926.003.patch
>
>
> For adding support for multiple resource types, we need to add support for
> this in the Resource class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4081) Add support for multiple resource types in the Resource class
[ https://issues.apache.org/jira/browse/YARN-4081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-4081: Attachment: YARN-4081-YARN-3926.004.patch Uploaded a new patch to address the findbugs and checkstyle issues. The test failures are unrelated to the patch. They're due to some issues with unzipping some files. > Add support for multiple resource types in the Resource class > - > > Key: YARN-4081 > URL: https://issues.apache.org/jira/browse/YARN-4081 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: YARN-4081-YARN-3926.001.patch, > YARN-4081-YARN-3926.002.patch, YARN-4081-YARN-3926.003.patch, > YARN-4081-YARN-3926.004.patch > > > For adding support for multiple resource types, we need to add support for > this in the Resource class. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4081) Add support for multiple resource types in the Resource class
[ https://issues.apache.org/jira/browse/YARN-4081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-4081: Attachment: YARN-4081-YARN-3926.002.patch Uploaded a new patch to get rid of un-neccessary formatting changes. > Add support for multiple resource types in the Resource class > - > > Key: YARN-4081 > URL: https://issues.apache.org/jira/browse/YARN-4081 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: YARN-4081-YARN-3926.001.patch, > YARN-4081-YARN-3926.002.patch > > > For adding support for multiple resource types, we need to add support for > this in the Resource class. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4081) Add support for multiple resource types in the Resource class
[ https://issues.apache.org/jira/browse/YARN-4081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-4081: Attachment: YARN-4081-YARN-3926.001.patch Uploaded a patch with support for multiple resource types in the Resource class. Add support for multiple resource types in the Resource class - Key: YARN-4081 URL: https://issues.apache.org/jira/browse/YARN-4081 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Varun Vasudev Assignee: Varun Vasudev Attachments: YARN-4081-YARN-3926.001.patch For adding support for multiple resource types, we need to add support for this in the Resource class. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
