rmatharu commented on a change in pull request #1281: SAMZA-2378: Container
Placements support for Standby containers enabled jobs
URL: https://github.com/apache/samza/pull/1281#discussion_r382400064
##########
File path:
samza-core/src/main/java/org/apache/samza/clustermanager/ContainerManager.java
##########
@@ -136,7 +135,22 @@ boolean handleContainerLaunch(SamzaResourceRequest
request, String preferredHost
LOG.info("Waiting for running container to shutdown due to existing
ContainerPlacement action {}", actionMetaData);
return false;
} else if (actionStatus ==
ContainerPlacementMetadata.ContainerStatus.STOPPED) {
- allocator.runStreamProcessor(request, preferredHost);
+ // If the job has standby containers enabled, always check standby
constraints before issuing a start on container
+ // Note: Always check constraints against allocated resource, since
preferred host can be ANY_HOST as well
+ if (standbyContainerManager.isPresent() &&
!standbyContainerManager.get().checkStandbyConstraints(request.getProcessorId(),
allocatedResource.getHost())) {
+ LOG.info(
+ "Starting container {} on host {} does not meet standby
constraints, falling back to source host placement metadata: {}",
+ request.getProcessorId(), preferredHost, actionMetaData);
+ resourceRequestState.releaseUnstartableContainer(allocatedResource,
preferredHost);
+ resourceRequestState.cancelResourceRequest(request);
+ // Fallback to source host since the new allocated resource does not
meet standby constraints
+ allocator.requestResource(processorId,
actionMetaData.getSourceHost());
+ markContainerPlacementActionFailed(actionMetaData,
+ String.format("allocated resource %s does not meet standby
constraints now, falling back to source host", allocatedResource));
Review comment:
Update failedStandbyAllocations metric?
Also it may be possible to expose a method on standby-container-manager to
do the
resourceRequestState.releaseUnstartableContainer();
resourceRequestState.cancelResourceRequest(request);
containerAllocator.requestResource();
because it is done in standby-container-manager as a part of
checkStandbyConstraintsAndRunStreamProcessor method
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
