[CONF] Apache Samza > SEP-22: Container Placements in Samza

[Sanil Jain (Confluence)]

Title: Message Title

There's 2 inline comment updates and 1 new edit on this page

SEP-22: Container Placements in Samza 2 inline comment updates Sanil Jain resolved populated by the client How would the client know whether the uuid is unique? View comment Boris Shkolnik has a steady lag not clear what is the criteria here. Sanil Jain Criteria is standby container has bootstrapped completely and the lag (watermark) is not increasing (stready) for standby View comment • Like 1 new edit Sanil Jain edited this page Here's what changed: ... API placeContainer Description Active Container: Stops container process on source-host and starts it for  Stateless Job on either Destination-host (destination host can be source as well) Any host (destination-host = ANY_HOST) Stateful Job on either  Destination-host (if specified, destination host can be source as well) Standby Container (destination-host = STANDBY) Any host (destination-host = ANY_HOST) StandBy Container: Stops container process on source-host and starts it on: Destination-host (if specified & matches StandBy Constraints) Any host (otherwise which matches StandBy Constraints) Parameters uuiddeploymentId: unique identifier of a request, populated by the clientapplicationId: unique identifier of the deployed app for which the action is taken processor-id: Samza resource id of container e.g 0, 1, 2  destination-host: valid hostname / “ANY_HOST” / “STANDBY” request-expiry-timeout: [optional]: timeout for any resource request to the cluster manager  Status code CREATED, BAD_REQUEST, ACCEPTED, IN_PROGRESS, SUCCEEDED, FAILED Returns Since this is an ASYNC API nothing is returned, UUID for the client to query the status of the request can be queried by processorId Failure Scenarios There are following cases under which a request to place container might fail: When an active container stop fails, in this case, we mark the request failed When requested resources cannot be obtained from the cluster manager, in this case, we mark the request failed When stopped active container fails to start on destination host in that case we mark the request failed and attempt to start on the source host, failure to do so results in starting the same on ANY_HOST ... API containerStatus Description Gives the status & info of the container placement request, for ex is it running, stopped what control commands are issued on it Parameters processor-id: Samza resource id of container e.g 0, 1, 2  applicationIddeploymentId: unique identifier of the deployed app for which the action is taken uuid: unique identifier of a request Status code BAD_REQUEST Returns Status of the Container placement action  ... API controlStandBy Description Starts or Stops a standBy container for the active container Parameters processor-id: Samza resource id of container e.g 0, 1, 2  applicationIddeploymentId: unique identifier of the deployed app for which the action is taken uuid: unique identifier of a request Status code CREATED, BAD_REQUEST, ACCEPTED, IN_PROGRESS, SUCCEEDED, FAILED Returns  UUID for the client to query the status of the request Architecture For implementing a scalable container placement control system, the proposed solution is divided into two parts: ... Code Block language java title ContainerPlacementMessage.java linenumbers true /** * Encapsulates the request or response payload information between the ContainerPlacementHandler service and external * controllers issuing placement actions */ public abstract class ContainerPlacementMessage { public enum StatusCode { /** * Indicates that the container placement action is created */ CREATED, /** * Indicates that the container placement action was rejected because request was deemed invalid */ BAD_REQUEST, /** * Indicates that the container placement action is accepted and waiting to be processed */ ACCEPTED, /** * Indicates that the container placement action is in progress */ IN_PROGRESS, /** * Indicates that the container placement action is in progress */ SUCCEEDED, /** * Indicates that the container placement action is in failed */ FAILED; } /** * UUID attached to a message which helps in identifying duplicate request messages written to metastore and not * retake actions even if metastore is eventually consistent */ protected final UUID uuid; /** * Unique identifier for a deployment so messages can be invalidated across a job restarts * for ex yarn bases cluster manager should set this to app attempt id */ protected final String applicationIddeploymentId; // Logical container Id 0, 1, 2 protected final String processorId; // Destination host where container is desired to be moved protected final String destinationHost; // Optional request expiry which acts as a timeout for any resource request to cluster resource manager protected final Duration requestExpiry; // Status of the current request protected final StatusCode statusCode; // Timestamp of the request or response message protected final long timestamp; protected ContainerPlacementMessage(UUID uuid, String applicationIddeploymentId, String processorId, String destinationHost, Duration requestExpiry, StatusCode statusCode, long timestamp) {…} } ... Code Block language java title ContainerPlacementRequestMessage linenumbers true /** * Encapsulates the request sent from the external controller to the JobCoordinator to take a container placement action */ public class ContainerPlacementRequestMessage extends ContainerPlacementMessage { public ContainerPlacementRequestMessage(UUID uuid, String applicationIddeploymentId, String processorId, String destinationHost, Duration requestExpiry, long timestamp) {...} public ContainerPlacementRequestMessage(UUID uuid, String applicationIddeploymentId, String processorId, String destinationHost, long timestamp) {...} } ... Code Block language java title ContainerPlacementResponseMessage linenumbers true /** * Encapsulates the response sent from the JobCoordinator for a container placement action */ public class ContainerPlacementResponseMessage extends ContainerPlacementMessage { // Returned status of the request private String responseMessage; public ContainerPlacementResponseMessage(UUID uuid, String applicationIddeploymentId, String processorId, String destinationHost, Duration requestExpiry, StatusCode statusCode, String responseMessage, long timestamp) {...} public ContainerPlacementResponseMessage(UUID uuid, String applicationIddeploymentId, String processorId, String destinationHost, StatusCode statusCode, String responseMessage, long timestamp) {...} ... Go to page history View page

Stop watching space • Manage notifications This message was sent by Atlassian Confluence 7.1.2