[jira] [Commented] (MAPREDUCE-5654) blacklist is not propagated from AM to RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-5654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14069057#comment-14069057 ] Gourav Khaneja commented on MAPREDUCE-5654: --- Hi. Just FYI - I've used Blacklist with Yarn RM 2.2 and 2.3 in https://issues.apache.org/jira/browse/TWILL-87 and it seems to work. (i.e. blacklist is propagated from AM to RM). blacklist is not propagated from AM to RM - Key: MAPREDUCE-5654 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5654 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Reporter: Robert Grandl I was trying to blacklist some nodes. I added a set of hosts as strings into blacklistAdditions list and propagated into RMContainerRequestor#makeRemoteRequest to the RM. However the blacklist is received empty at RM. I logged the path for blacklist in AM and I found that in ApplicationMasterProtocolPBClientImpl#allocate, this list is lost. I print request.getResourceBlacklistRequest().getBlacklistAdditions().toString() at the beginning of ApplicationMasterProtocolPBClientImpl#allocate and the blacklisted additions are there. After AllocateRequestProto requestProto is created based on this request, and I print again requestProto.getBlacklistRequest().getBlacklistAdditionsList().toString(), the blacklist additions is empty now. I looked even further and log what happened. At some point in yarn-api, I was lost with my logging as that code was regenerated every time I recompiled yarn-api. Thanks, robert -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5654) blacklist is not propagated from AM to RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-5654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13837146#comment-13837146 ] Robert Grandl commented on MAPREDUCE-5654: -- Do you have any thoughts with this guys ? blacklist is not propagated from AM to RM - Key: MAPREDUCE-5654 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5654 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Reporter: Robert Grandl I was trying to blacklist some nodes. I added a set of hosts as strings into blacklistAdditions list and propagated into RMContainerRequestor#makeRemoteRequest to the RM. However the blacklist is received empty at RM. I logged the path for blacklist in AM and I found that in ApplicationMasterProtocolPBClientImpl#allocate, this list is lost. I print request.getResourceBlacklistRequest().getBlacklistAdditions().toString() at the beginning of ApplicationMasterProtocolPBClientImpl#allocate and the blacklisted additions are there. After AllocateRequestProto requestProto is created based on this request, and I print again requestProto.getBlacklistRequest().getBlacklistAdditionsList().toString(), the blacklist additions is empty now. I looked even further and log what happened. At some point in yarn-api, I was lost with my logging as that code was regenerated every time I recompiled yarn-api. Thanks, robert -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5654) blacklist is not propagated from AM to RM
[
https://issues.apache.org/jira/browse/MAPREDUCE-5654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13833233#comment-13833233
]
Robert Grandl commented on MAPREDUCE-5654:
--
I did some logging in ResourceBlacklistRequestPBImpl. In function
public ResourceBlacklistRequestProto.getProto()
{
mergeLocalProto();
LOG.info(“black_a=”+blacklistAdditions.toString()); // IT SHOWS CORRECTLY
proto = viaProto ? proto : builder.build();
viaProto = true;
LOG.info(“black_aa=”+proto.getBlacklistAdditionsList().toString()); // IS
EMPTY
return proto;
}
and viaProto seems to be true, so proto = proto which is a
ResourceBlacklistRequestProto. ResourceBlacklistRequestProto is defined in
hadoop-yarn-api/YarnProtos.java, where the source is regenerated so cannot log.
I believe the problem is somewhere here in encoding
blacklistAdditions/removals. I believe that also because release containers are
propagated correctly and seems they are not using any YarnProtos.java related
stuff.
blacklist is not propagated from AM to RM
-
Key: MAPREDUCE-5654
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5654
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster
Reporter: Robert Grandl
I was trying to blacklist some nodes. I added a set of hosts as strings into
blacklistAdditions list and propagated into
RMContainerRequestor#makeRemoteRequest to the RM.
However the blacklist is received empty at RM. I logged the path for
blacklist in AM and I found that in
ApplicationMasterProtocolPBClientImpl#allocate, this list is lost.
I print
request.getResourceBlacklistRequest().getBlacklistAdditions().toString() at
the beginning of ApplicationMasterProtocolPBClientImpl#allocate and the
blacklisted additions are there.
After AllocateRequestProto requestProto is created based on this request, and
I print again
requestProto.getBlacklistRequest().getBlacklistAdditionsList().toString(),
the blacklist additions is empty now.
I looked even further and log what happened. At some point in yarn-api, I was
lost with my logging as that code was regenerated every time I recompiled
yarn-api.
Thanks,
robert
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (MAPREDUCE-5654) blacklist is not propagated from AM to RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-5654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13832102#comment-13832102 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-5654: Let's investigate if it turns out to be a bug in YARN before moving this to MR. [~rgrandl], you are forcing the blacklist by changing MR code and running your own modified MR App? Also you are saying that you put logs in ApplicationMasterProtocolPBClientImpl? Can you put more logs in ResourceBlacklistRequestPBImpl and see where it is breaking? blacklist is not propagated from AM to RM - Key: MAPREDUCE-5654 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5654 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Reporter: Robert Grandl I was trying to blacklist some nodes. I added a set of hosts as strings into blacklistAdditions list and propagated into RMContainerRequestor#makeRemoteRequest to the RM. However the blacklist is received empty at RM. I logged the path for blacklist in AM and I found that in ApplicationMasterProtocolPBClientImpl#allocate, this list is lost. I print request.getResourceBlacklistRequest().getBlacklistAdditions().toString() at the beginning of ApplicationMasterProtocolPBClientImpl#allocate and the blacklisted additions are there. After AllocateRequestProto requestProto is created based on this request, and I print again requestProto.getBlacklistRequest().getBlacklistAdditionsList().toString(), the blacklist additions is empty now. I looked even further and log what happened. At some point in yarn-api, I was lost with my logging as that code was regenerated every time I recompiled yarn-api. Thanks, robert -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5654) blacklist is not propagated from AM to RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-5654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13832113#comment-13832113 ] Robert Grandl commented on MAPREDUCE-5654: -- Vinod, yes, I am forcing blacklisting by changing MR code. What I basically did, is whenever computeIgnoreBlacklisting() is first called, I ensure that blacklistAdditions.addAll(blacklistNodes) is called, where I enforce blacklistedNodes to have a host.(blacklistedNodes is a set of string, so I guess is nothing bad if I do a blacklistedNodes.add(host_name)). I was tracing before till yarn-api but I did not saved the log calls. I will do again and let you know guys till where I am able to trace. thanks. blacklist is not propagated from AM to RM - Key: MAPREDUCE-5654 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5654 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Reporter: Robert Grandl I was trying to blacklist some nodes. I added a set of hosts as strings into blacklistAdditions list and propagated into RMContainerRequestor#makeRemoteRequest to the RM. However the blacklist is received empty at RM. I logged the path for blacklist in AM and I found that in ApplicationMasterProtocolPBClientImpl#allocate, this list is lost. I print request.getResourceBlacklistRequest().getBlacklistAdditions().toString() at the beginning of ApplicationMasterProtocolPBClientImpl#allocate and the blacklisted additions are there. After AllocateRequestProto requestProto is created based on this request, and I print again requestProto.getBlacklistRequest().getBlacklistAdditionsList().toString(), the blacklist additions is empty now. I looked even further and log what happened. At some point in yarn-api, I was lost with my logging as that code was regenerated every time I recompiled yarn-api. Thanks, robert -- This message was sent by Atlassian JIRA (v6.1#6144)
