[
https://issues.apache.org/jira/browse/YARN-5124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15317575#comment-15317575
]
Arun Suresh edited comment on YARN-5124 at 6/7/16 12:47 AM:
------------------------------------------------------------
Uploading patch addressing most of [~curino]'s feedback :
bq. Can we wrap this datastructure in some object? Map<Priority, Map<String,
Map<ExecutionType, TreeMap<Resource, ResourceRequestInfo>>>> it is starting to
get a bit too complicated to follow, and we might eventually want to change
representations.
I agree, it looks ugly, but even if we pull it up in a new class, we would
still see it. In the latest patch I alleviated the issue by somewhat emulating
C++ typedefs as follows:
# I create *typed* subclasses of HashMap classes like so:
{noformat}
class CapabilityMap extends TreeMap<Resource, ResourceRequestInfo> {}
class ExecutionTypeMap extends HashMap<ExecutionType, CapabilityMap> {}
class LocationMap extends HashMap<String, ExecutionTypeMap> {}
class RemoteRequestsTable extends HashMap<Priority, LocationMap> {}
{noformat}
# The remoteRequestsTable then looks like this:
{noformat}
final RemoteRequestsTable remoteRequestsTable = new RemoteRequestsTable();
{noformat}
bq. Do you need the new constructor AMRMClientImpl(ApplicationMasterProtocol
protocol)? Can't you use mockito for the conf in tests and pass the
ApplicationMasterProtocol that way?
I can't use mockito, since I actually have a real AMRMClientImpl object. This
is a functional test that uses an actual MiniYARNCluster. So don't think there
is any other way to inject a custom ApplicationMasterProtocol. But I did add a
{{@VisibleForTesting}} as per [~kasha]'s suggestion.
bq. The ordering of where ExecutionType goes is not consistent, in
ResourceRequest is at the end, while in most other places is earlier. Strong
typing make this not a big deal, just looks cleaner if consistent.
The reason I had to put ExecutionType in the end for {{ContainerRequest}} and
{{ResourceRequest}} was that otherwise, would have to change many of the
methods marked {{@Stable}} in previous releases.. In the AMRMClientImpl, it
appears before the {{Resource}} in the composite key for reasons i mentioend
[here|https://issues.apache.org/jira/browse/YARN-5124?focusedCommentId=15303099&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15303099]
but thats an implementation detail and hidden from the user.
Hope this made sense..
was (Author: asuresh):
Uploading patch addressing most of [~curino]'s feedback :
bq. Can we wrap this datastructure in some object? Map<Priority, Map<String,
Map<ExecutionType, TreeMap<Resource, ResourceRequestInfo>>>> it is starting to
get a bit too complicated to follow, and we might eventually want to change
representations.
I agree, it looks ugly, but even if we pull it up in a new class, we would
still see it. In the latest patch I alleviated the issue by somewhat emulating
C++ typedefs as follows:
# I create *typed* subclasses of HashMap classes like so:
{noformat}
class CapabilityMap extends TreeMap<Resource, ResourceRequestInfo> {}
class ExecutionTypeMap extends HashMap<ExecutionType, CapabilityMap> {}
class LocationMap extends HashMap<String, ExecutionTypeMap> {}
class RemoteRequestsTable extends HashMap<Priority, LocationMap> {}
{noformat}
# The remoteRequestsTable then looks like this:
{noformat}
final RemoteRequestsTable remoteRequestsTable = new RemoteRequestsTable();
{noformat}
bq. Do you need the new constructor AMRMClientImpl(ApplicationMasterProtocol
protocol)? Can't you use mockito for the conf in tests and pass the
ApplicationMasterProtocol that way?
I can't use mockito, since I actually have a real AMRMClientImpl object. This
is a functional test that uses an actual MiniYARNCluster. So don't think there
is any other way to inject a custom ApplicationMasterProtocol. But I did add a
{{@VisibleForTesting}} as per [~kasha]'s suggestion.
bq. The ordering of where ExecutionType goes is not consistent, in
ResourceRequest is at the end, while in most other places is earlier. Strong
typing make this not a big deal, just looks cleaner if consistent.
The reason I had to put ExecutionType in the end for {{ContainerRequest}} and
{{ResourceRequest}} was that otherwise, would have to change many of the
methods marked {{@Stable}} in previous releases.. In the AMRMClientImpl, it
appears before the {{Resource}} in the composite key for reasons i mentioend
[here|https://issues.apache.org/jira/browse/YARN-5124?focusedCommentId=15303099&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15303099]
but thats an implementation detail and hidden from the user.
Hope this made sense..
> Modify AMRMClient to set the ExecutionType in the ResourceRequest
> -----------------------------------------------------------------
>
> Key: YARN-5124
> URL: https://issues.apache.org/jira/browse/YARN-5124
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Arun Suresh
> Assignee: Arun Suresh
> Attachments: YARN-5124.001.patch, YARN-5124.002.patch,
> YARN-5124.003.patch, YARN-5124.004.patch, YARN-5124.005.patch,
> YARN-5124.006.patch, YARN-5124.008.patch, YARN-5124.009.patch,
> YARN-5124.010.patch, YARN-5124_YARN-5180_combined.007.patch,
> YARN-5124_YARN-5180_combined.008.patch
>
>
> Currently the {{ContainerRequest}} allows the AM to set the {{ExecutionType}}
> in the AMRMClient, but it is not being set in the actual {{ResourceRequest}}
> that is sent to the RM
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
