Copilot commented on code in PR #11962:
URL: https://github.com/apache/ozone/pull/11962#discussion_r4643861140


##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/report/CommandStatusReportPublisher.java:
##########
@@ -63,21 +65,31 @@ protected long getReportFrequency() {
 
   @Override
   protected CommandStatusReportsProto getReport() {
+    return null;
+  }
+
+  @Override
+  protected List<CommandStatusReportsProto> getIncrementalReports() {
     Map<Long, CommandStatus> map = this.getContext()
         .getCommandStatusMap();
     Iterator<Long> iterator = map.keySet().iterator();
-    CommandStatusReportsProto.Builder builder = CommandStatusReportsProto
-        .newBuilder();
+    List<CommandStatusReportsProto> reports = new ArrayList<>();
 
-    iterator.forEachRemaining(key -> {
+    while (iterator.hasNext()) {
+      Long key = iterator.next();
       CommandStatus cmdStatus = map.get(key);
+      if (cmdStatus == null) {
+        continue;
+      }
+      reports.add(CommandStatusReportsProto.newBuilder()
+          .addCmdStatus(cmdStatus.getProtoBufMessage())
+          .build());
       // If status is still pending then don't remove it from map as
       // CommandHandler will change its status when it works on this command.
       if (!cmdStatus.getStatus().equals(Status.PENDING)) {
         map.remove(key);
       }

Review Comment:
   A command can finish between building this report and the subsequent 
live-status check. In that interleaving, the queued protobuf still contains 
PENDING, but the check sees EXECUTED and removes the command from the map, so 
its final delete ACK is never published. StateContext.updateCommandStatus 
updates through computeIfPresent; please serialize and decide whether to remove 
the entry atomically through the same per-key operation.



##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/states/endpoint/HeartbeatEndpointTask.java:
##########
@@ -256,26 +272,54 @@ private void addReports(SCMHeartbeatRequestProto.Builder 
requestBuilder) {
           } else {
             requestBuilder.setField(descriptor, report);
           }
+          heartbeatSize.add(descriptor.getNumber(), report);
           break;
         }
       }
     }
+    return reportCount;
+  }
+
+  private void addCommandStatusReports(
+      SCMHeartbeatRequestProto.Builder requestBuilder,
+      int remainingReportCount, long currentHeartbeatSize) {
+    long remainingSafeBytes = Math.max(
+        0, safeHeartbeatSize - currentHeartbeatSize);
+    CommandStatusReportSelection selection =
+        context.getCommandStatusReports(rpcEndpoint.getAddress(),
+            remainingReportCount, remainingSafeBytes, safeHeartbeatSize);

Review Comment:
   This applies only the remaining IPC budget; the proposed 
hdds.heartbeat.max.command.status.report.size setting is neither defined nor 
read. Consequently, configuring a smaller command-status limit has no effect, 
and the default permits roughly 115 MiB instead of the stated 64 MB when other 
heartbeat fields are small. Please add the setting with its default and pass 
the minimum of that limit and remainingSafeBytes to the selector.



##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/StateContext.java:
##########
@@ -393,38 +395,66 @@ public int clearCommandStatusReports(HostAndPort 
endpoint) {
    */
   public List<Message> getAllAvailableReports(
       HostAndPort endpoint) {
-    return getAllAvailableReportsUpToLimit(endpoint, Integer.MAX_VALUE);
+    List<Message> reports = getFullReports(endpoint, Integer.MAX_VALUE);
+    reports.addAll(getIncrementalReports(endpoint, Integer.MAX_VALUE - 
reports.size()));
+    return reports;
   }
 
   /**
-   * Returns up to {@code maxLimit} available reports for the endpoint,
-   * removing them from the queue. Any reports beyond the limit stay queued
-   * and are returned by a later call; callers can use {@link
-   * #hasPendingReports(HostAndPort)} to detect that case and schedule a
-   * follow-up heartbeat.
-   *
-   * @return List of reports
+   * Returns available reports other than command status reports. Full reports
+   * are selected before incremental reports, and the total number returned is
+   * capped by {@code maxLimit}.
    */
-  public List<Message> getAllAvailableReports(
+  public List<Message> getAvailableNonCommandStatusReports(
       HostAndPort endpoint, int maxLimit) {
-    return getAllAvailableReportsUpToLimit(endpoint, maxLimit);
+    List<Message> reports = getFullReports(endpoint, maxLimit);
+    reports.addAll(getIncrementalReportsExcludingCommandStatus(
+        endpoint, maxLimit - reports.size()));
+    return reports;
   }
 
   /**
-   * Returns true if any reports are still queued for the endpoint.
+   * Selects command status reports within both the report count and serialized
+   * byte limits.
    */
-  public boolean hasPendingReports(HostAndPort endpoint) {
+  public CommandStatusReportSelection getCommandStatusReports(
+      HostAndPort endpoint, int maxLimit, long maxBytes,
+      long oversizedThreshold) {
+    List<CommandStatusReportsProto> selected = new LinkedList<>();
+    long selectedBytes = 0;
+    long oversizedBytes = 0;
     synchronized (incrementalReportsQueue) {
-      List<Message> reportsForEndpoint = incrementalReportsQueue.get(endpoint);
-      if (reportsForEndpoint != null && !reportsForEndpoint.isEmpty()) {
-        return true;
+      List<Message> reportsForEndpoint =
+          incrementalReportsQueue.get(endpoint);
+      if (reportsForEndpoint != null) {
+        Iterator<Message> iterator = reportsForEndpoint.iterator();
+        while (iterator.hasNext() && selected.size() < maxLimit) {
+          Message message = iterator.next();
+          if (!(message instanceof CommandStatusReportsProto)) {
+            continue;
+          }
+
+          CommandStatusReportsProto report =
+              (CommandStatusReportsProto) message;
+          long remainingBytes = maxBytes - selectedBytes;
+          long reportBytes = getCommandStatusReportWireSize(report);
+          if (reportBytes <= remainingBytes) {
+            selected.add(report);
+            selectedBytes += reportBytes;
+            iterator.remove();
+            if (reportBytes > oversizedThreshold) {
+              oversizedBytes = reportBytes;
+            }
+            continue;
+          }
+          if (reportBytes > oversizedThreshold) {
+            oversizedBytes = reportBytes;
+          }
+          break;

Review Comment:
   A command-status report larger than safeHeartbeatSize can never fit, but 
this break leaves it queued and blocks all later command-status reports on 
every heartbeat. Since those heartbeats can succeed, the failure-triggered 
queue reset does not recover the backlog. Handle permanently unsendable entries 
explicitly, by bounding ACK size upstream or providing a recovery path that 
lets later reports progress, rather than indefinitely deferring them.



##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/states/endpoint/HeartbeatEndpointTask.java:
##########
@@ -153,21 +169,22 @@ public EndpointStateMachine.EndPointStates call() throws 
Exception {
       requestBuilder = SCMHeartbeatRequestProto.newBuilder()
           .setDatanodeDetails(datanodeDetailsProto)
           .setDatanodeVersion(versionInfo);
-      addReports(requestBuilder);
-      addContainerActions(requestBuilder);
-      addPipelineActions(requestBuilder);
-      addQueuedCommandCounts(requestBuilder);
+      SerializedSizeAccumulator heartbeatSize = new 
SerializedSizeAccumulator();
+      heartbeatSize.add(SCMHeartbeatRequestProto.DATANODEDETAILS_FIELD_NUMBER, 
datanodeDetailsProto);
+      heartbeatSize.add(SCMHeartbeatRequestProto.DATANODEVERSION_FIELD_NUMBER, 
versionInfo);
+      int reportCount = addNonCommandStatusReports(requestBuilder, 
heartbeatSize);
+      addContainerActions(requestBuilder, heartbeatSize);
+      addPipelineActions(requestBuilder, heartbeatSize);
+      addQueuedCommandCounts(requestBuilder, heartbeatSize);
+      addCommandStatusReports(
+          requestBuilder, maxReportsPerHB - reportCount, heartbeatSize.get());

Review Comment:
   Selecting all non-command reports first can indefinitely starve 
command-status reports: if ICRs keep filling maxReportsPerHB on every 
heartbeat, remainingReportCount is always zero, even for an ACK queued before 
those ICRs. Unlike the previous shared FIFO selection, later ICRs continually 
overtake the ACK, preventing SCM from observing completed deletions. Please 
reserve a fair share of report slots for queued command statuses (while still 
adding their fields last for byte budgeting), and cover repeated heartbeats 
with replenished ICRs.



##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/states/endpoint/HeartbeatEndpointTask.java:
##########
@@ -256,26 +272,54 @@ private void addReports(SCMHeartbeatRequestProto.Builder 
requestBuilder) {
           } else {
             requestBuilder.setField(descriptor, report);
           }
+          heartbeatSize.add(descriptor.getNumber(), report);

Review Comment:
   XceiverServerRatis.sendPipelineReport queues PipelineReportsProto as 
incremental messages, so several selected reports can overwrite the same 
non-repeated heartbeat field. The accumulator counts all of them even though 
only the last is serialized, which can defer command ACKs despite available 
space. Subtract the previous field's wire size before replacing it, or measure 
the assembled non-command payload.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to