prateekm commented on a change in pull request #1421:
URL: https://github.com/apache/samza/pull/1421#discussion_r475007343
##########
File path: samza-core/src/main/java/org/apache/samza/job/model/JobModel.java
##########
@@ -39,34 +36,14 @@
* </p>
*/
public class JobModel {
- private static final String EMPTY_STRING = "";
private final Config config;
private final Map<String, ContainerModel> containers;
- private final LocalityManager localityManager;
- private final Map<String, String> localityMappings;
-
public int maxChangeLogStreamPartitions;
public JobModel(Config config, Map<String, ContainerModel> containers) {
- this(config, containers, null);
- }
-
- public JobModel(Config config, Map<String, ContainerModel> containers,
LocalityManager localityManager) {
Review comment:
If it's not too much extra work, does it make sense to move JobModel to
samza-api and expose it in JobContext as part of this PR?
##########
File path:
samza-core/src/main/java/org/apache/samza/clustermanager/ContainerManager.java
##########
@@ -494,7 +498,9 @@ private String
getSourceHostForContainer(ContainerPlacementRequestMessage reques
processorId, currentResource.getContainerId(),
currentResource.getHost(), requestMessage);
sourceHost = currentResource.getHost();
} else {
- sourceHost =
samzaApplicationState.jobModelManager.jobModel().getContainerToHostValue(processorId,
SetContainerHostMapping.HOST_KEY);
+ sourceHost =
Optional.ofNullable(localityManager.readLocality().getHostLocality(processorId))
Review comment:
Can localityManager or readLocality results be null?
##########
File path: samza-api/src/main/java/org/apache/samza/job/model/HostLocality.java
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.samza.job.model;
+
+import com.google.common.base.Objects;
Review comment:
Nitpick: Can use java.util.Objects (here and other classes)?
##########
File path: samza-api/src/main/java/org/apache/samza/job/model/LocalityModel.java
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.samza.job.model;
+
+import com.google.common.base.Objects;
+import java.util.Map;
+
+/**
+ * A model to represent the locality mapping of an application.
Review comment:
Since this is in samza-api, let's describe what "locality mapping" etc.
mean for users unfamiliar with Samza terms.
##########
File path: samza-api/src/main/java/org/apache/samza/job/model/HostLocality.java
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.samza.job.model;
+
+import com.google.common.base.Objects;
+
+/**
+ * A class to represent the host locality information.
+ * Fields such as <i>jmxUrl</i> and <i>jmxTunnelingUrl</i> exist for backward
compatibility reasons as they were
+ * historically stored under the same name space as locality and surfaced
within the framework through the locality
+ * manager.
+ */
+public class HostLocality {
Review comment:
Should this be ContainerLocation or something? HostLocality sounds wrong
because the host doesn't have a location, it is the location.
##########
File path: samza-core/src/main/java/org/apache/samza/job/model/JobModel.java
##########
@@ -39,34 +36,14 @@
* </p>
*/
public class JobModel {
- private static final String EMPTY_STRING = "";
private final Config config;
private final Map<String, ContainerModel> containers;
- private final LocalityManager localityManager;
- private final Map<String, String> localityMappings;
-
public int maxChangeLogStreamPartitions;
public JobModel(Config config, Map<String, ContainerModel> containers) {
- this(config, containers, null);
- }
-
- public JobModel(Config config, Map<String, ContainerModel> containers,
LocalityManager localityManager) {
this.config = config;
this.containers = Collections.unmodifiableMap(containers);
- this.localityManager = localityManager;
-
- // initialize container localityMappings
- this.localityMappings = new HashMap<>();
Review comment:
Nice, thanks for cleaning all of this up!
##########
File path:
samza-core/src/main/java/org/apache/samza/coordinator/server/LocalityServlet.java
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.samza.coordinator.server;
+
+import java.io.IOException;
+import java.util.Optional;
+import javax.servlet.http.HttpServlet;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+import org.apache.samza.container.LocalityManager;
+import org.apache.samza.job.model.HostLocality;
+import org.apache.samza.job.model.LocalityModel;
+import org.apache.samza.serializers.model.SamzaObjectMapper;
+import org.codehaus.jackson.map.ObjectMapper;
+
+
+/**
+ * A servlet for locality information of a job. The servlet is hosted
alongside of the {@link JobServlet} which hosts
+ * job model & configuration. Historically, locality information was part of
job model but we extracted the locality
+ * as job model is static within the lifecycle of an application attempt while
locality changes in the event of container
+ * movements.
+ *
+ * This separation enables us to achieve performance benefits by caching job
model when it is served by the AM as it
+ * can incur significant penalty in the job start time for jobs with large
number of containers.
+ */
+public class LocalityServlet extends HttpServlet {
+ private static final String PROCESSOR_ID_PARAM = "processorId";
+ private final ObjectMapper mapper = SamzaObjectMapper.getObjectMapper();
+ private final LocalityManager localityManager;
+
+ public LocalityServlet(LocalityManager localityManager) {
+ this.localityManager = localityManager;
+ }
+
+ @Override
+ public void doGet(HttpServletRequest request, HttpServletResponse response)
throws IOException {
+ response.setContentType("application/json");
+ response.setStatus(HttpServletResponse.SC_OK);
+ LocalityModel localityModel = localityManager.readLocality();
+
+ if (request.getParameterMap().size() == 1) {
+ String processorId = request.getParameter(PROCESSOR_ID_PARAM);
+ HostLocality hostLocality =
Optional.ofNullable(localityModel.getHostLocality(processorId))
+ .orElse(new HostLocality(processorId, ""));
Review comment:
Should this return an HTTP error (404 etc.) instead?
##########
File path:
samza-core/src/main/scala/org/apache/samza/coordinator/JobModelManager.scala
##########
@@ -466,11 +464,6 @@ class JobModelManager(
debug("Stopping HTTP server.")
server.stop
info("Stopped HTTP server.")
- if (localityManager != null) {
- info("Stopping localityManager")
- localityManager.close()
Review comment:
Where is this called now?
##########
File path: samza-api/src/main/java/org/apache/samza/job/model/LocalityModel.java
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.samza.job.model;
+
+import com.google.common.base.Objects;
+import java.util.Map;
+
+/**
+ * A model to represent the locality mapping of an application.
+ * Currently the locality mapping represents the container host locality of an
application.
+ *
+ * We want to keep the locality mapping open and not tie it to a container to
potentially
Review comment:
What does this mean?
Phrasing: Instead of "we want to", maybe just describe what the
implementation does and why.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
