smengcl commented on code in PR #11962:
URL: https://github.com/apache/ozone/pull/11962#discussion_r4509152804


##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/DatanodeQueueMetrics.java:
##########
@@ -60,6 +61,16 @@ public final class DatanodeQueueMetrics implements 
MetricsSource {
       "ContainerActionQueue";
   public static final String PIPELINE_ACTION_QUEUE_PREFIX =
       "PipelineActionQueue";
+  public static final String COMMAND_STATUS_REPORT_QUEUE_PREFIX =
+      "CommandStatusReportQueue";
+  public static final String COMMAND_STATUS_REPORT_SENT_PREFIX =
+      "CommandStatusReportSent";
+  public static final String COMMAND_STATUS_REPORT_DEFERRED_PREFIX =

Review Comment:
   `deferredBytes` comes from `getCommandStatusReportQueueBytes(endpoint)` 
right after selection, so `CommandStatusReportDeferred` reports the same value 
as the `CommandStatusReportQueue` gauge, only sampled at a different time. 
Maybe drop one of them? That also removes two of the new maps.



##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsConfigKeys.java:
##########
@@ -73,6 +73,10 @@ public final class HddsConfigKeys {
       "hdds.heartbeat.report.max.limit";
   public static final int HDDS_HEARTBEAT_REPORT_MAX_LIMIT_DEFAULT =
       10000;
+  public static final String HDDS_HEARTBEAT_MAX_COMMAND_STATUS_REPORT_SIZE =
+      "hdds.heartbeat.max.command.status.report.size";

Review Comment:
   nit: the related keys put the limit last (`hdds.heartbeat.report.max.limit`, 
`hdds.container.action.max.limit`). How about 
`hdds.heartbeat.command.status.report.max.size` to match? Easy to change while 
this is still a draft.



##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/states/endpoint/HeartbeatEndpointTask.java:
##########
@@ -148,25 +170,34 @@ public EndpointStateMachine.EndPointStates call() throws 
Exception {
       requestBuilder = SCMHeartbeatRequestProto.newBuilder()
           .setDatanodeDetails(datanodeDetailsProto)
           .setDatanodeVersion(versionInfo);
-      addReports(requestBuilder);
+      int reportCount = addNonCommandStatusReports(requestBuilder);
       addContainerActions(requestBuilder);
       addPipelineActions(requestBuilder);
       addQueuedCommandCounts(requestBuilder);
+      commandStatusSelection = addCommandStatusReports(
+          requestBuilder, maxReportsPerHB - reportCount);
       SCMHeartbeatRequestProto request = requestBuilder.build();
       LOG.debug("Sending heartbeat message : {}", request);
       SCMHeartbeatResponseProto response = rpcEndpoint.getEndPoint()
           .sendHeartbeat(request);
       processResponse(response, datanodeDetailsProto);
       rpcEndpoint.setLastSuccessfulHeartbeat(ZonedDateTime.now());
       rpcEndpoint.zeroMissedCount();
-      if (context.hasPendingReports(rpcEndpoint.getAddress())) {
-        // addReports was capped at maxReportsPerHB and left reports queued. 
Trigger immediately.
-        context.getParent().setNextHB(Time.monotonicNow());
+      if (context.getQueueMetrics() != null) {

Review Comment:
   Removing the immediate follow-up heartbeat changes behavior for all queued 
reports, not only command statuses. An ICR backlog, or a full report left 
behind by the count cap, now drains one `hdds.heartbeat.report.max.limit` batch 
per heartbeat interval instead of right away, which is what HDDS-16723 added 
the follow-up for. Could we keep it here, or move this part to its own JIRA?
   
   If it stays, the `hdds.heartbeat.report.max.limit` description in 
ozone-default.xml still says reports beyond the limit "are sent in an immediate 
follow-up heartbeat", so it needs an update. Also `hasPendingReports()` and 
`getAllAvailableReports(endpoint, maxLimit)` would only be used by tests after 
this, so they could go instead of updating the javadoc.



##########
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/states/endpoint/TestHeartbeatEndpointTask.java:
##########
@@ -441,16 +451,169 @@ public void 
testHeartbeatCapsReportsAndSchedulesFollowup() throws Exception {
     // Only the capped number of ICRs is sent in this heartbeat.
     SCMHeartbeatRequestProto heartbeat = argument.getValue();
     assertEquals(2, heartbeat.getIncrementalContainerReportCount());
-    // The remainder stays queued and a follow-up heartbeat is scheduled now.
+    // The remainder stays queued for the next regular heartbeat.
     assertTrue(context.hasPendingReports(TEST_SCM_ENDPOINT));
-    verify(datanodeStateMachine).setNextHB(anyLong());
+    verify(datanodeStateMachine, never()).setNextHB(anyLong());
 
-    // The follow-ups drain the rest; the last one sends 1 ICR and schedules 
nothing.
+    // Subsequent regular heartbeats drain the rest.
     endpointTask.call();
     endpointTask.call();
     assertEquals(1, argument.getValue().getIncrementalContainerReportCount());
     assertFalse(context.hasPendingReports(TEST_SCM_ENDPOINT));
-    verify(datanodeStateMachine, times(2)).setNextHB(anyLong());
+    verify(datanodeStateMachine, never()).setNextHB(anyLong());
+  }
+
+  @Test
+  public void testHeartbeatCapsCommandStatusReportsByBytes()

Review Comment:
   Could we add a case where non command status reports use up the count limit? 
For example `hdds.heartbeat.report.max.limit=2` with 2 ICRs and 1 command 
status report should send no command status and keep it queued. That covers the 
`maxReportsPerHB - reportCount` path. The new metrics (sent, requeued, 
oversized bytes) are not checked by any test either.



##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/StateContext.java:
##########
@@ -392,6 +395,71 @@ public List<Message> getAllAvailableReports(
     return getAllAvailableReportsUpToLimit(endpoint, maxLimit);
   }
 
+  /**
+   * Returns available reports other than command status reports. Full reports
+   * are selected before incremental reports, and the total number returned is
+   * capped by {@code maxLimit}.
+   */
+  public List<Message> getAvailableNonCommandStatusReports(
+      HostAndPort endpoint, int maxLimit) {
+    List<Message> reports = getFullReports(endpoint, maxLimit);
+    reports.addAll(getIncrementalReportsExcludingCommandStatus(
+        endpoint, maxLimit - reports.size()));
+    return reports;
+  }
+
+  /**
+   * Selects command status reports within both the report count and serialized
+   * byte limits. An aggregate report may be split between complete command
+   * statuses, but an individual command status is never split.
+   */
+  public CommandStatusReportSelection getCommandStatusReports(
+      HostAndPort endpoint, int maxLimit, long maxBytes,
+      long maxConfiguredBytes) {
+    List<CommandStatusReportsProto> selected = new LinkedList<>();
+    long selectedBytes = 0;
+    long oversizedBytes = 0;
+    synchronized (incrementalReportsQueue) {
+      List<Message> reportsForEndpoint =
+          incrementalReportsQueue.get(endpoint);
+      if (reportsForEndpoint != null) {
+        ListIterator<Message> iterator = reportsForEndpoint.listIterator();
+        while (iterator.hasNext() && selected.size() < maxLimit) {
+          Message message = iterator.next();
+          if (!(message instanceof CommandStatusReportsProto)) {
+            continue;
+          }
+
+          CommandStatusReportsProto report =
+              (CommandStatusReportsProto) message;
+          long remainingBytes = maxBytes - selectedBytes;
+          long reportBytes = getCommandStatusReportWireSize(report);
+          if (reportBytes <= remainingBytes) {
+            selected.add(report);
+            selectedBytes += reportBytes;
+            iterator.remove();
+            continue;
+          }
+
+          CommandStatusReportSplit split =
+              splitCommandStatusReport(report, remainingBytes);
+          if (split.getSelected() != null) {
+            selected.add(split.getSelected());
+            selectedBytes +=
+                getCommandStatusReportWireSize(split.getSelected());
+            iterator.set(split.getRemainder());
+          } else if (getFirstCommandStatusWireSize(report)

Review Comment:
   When a single CommandStatus does not fit the budget, it stays at the head of 
the queue and the `break` holds back every command status behind it, on every 
heartbeat. `CommandStatusReportPublisher` keeps appending a report each 
interval, so the queue grows and SCM gets no command ACKs from this datanode 
until the config is raised.
   
   The oversized check also compares only against the configured limit. If the 
limit is set above 90% of `ipc.maximum.data.length`, a status larger than 
`safeHeartbeatSize` is stuck the same way but never reported as oversized.
   
   Would it work to send such a status alone when it still fits in the safe RPC 
size, and to check oversized against the effective cap (min of configured limit 
and safe size)?



##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsConfigKeys.java:
##########
@@ -73,6 +73,10 @@ public final class HddsConfigKeys {
       "hdds.heartbeat.report.max.limit";
   public static final int HDDS_HEARTBEAT_REPORT_MAX_LIMIT_DEFAULT =
       10000;
+  public static final String HDDS_HEARTBEAT_MAX_COMMAND_STATUS_REPORT_SIZE =
+      "hdds.heartbeat.max.command.status.report.size";
+  public static final String
+      HDDS_HEARTBEAT_MAX_COMMAND_STATUS_REPORT_SIZE_DEFAULT = "64MB";

Review Comment:
   nit: this fits on one line within the 120 char limit. Same for many other 
wrapped lines in the patch, such as `testHeartbeatCapsReports()` / `throws 
Exception` and the `addCommandStatusReports(requestBuilder, maxReportsPerHB - 
reportCount)` call.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to