sadanand48 commented on PR #11962:
URL: https://github.com/apache/ozone/pull/11962#issuecomment-6910954499

   I had a discussion with @sumitagrawl where he proposed approach 2 - the 
publisher-side alternative and [attached the 
patch](https://issues.apache.org/jira/secure/attachment/13086505/limit_at_report_publisher.patch)
 so we can compare both approaches before deciding.
   
   Approach 1: split while constructing the heartbeat (the current PR)
   
   CommandStatusReportPublisher remains unchanged and publishes one aggregate 
CommandStatusReportsProto.
   During heartbeat construction, the aggregate is split only between complete 
CommandStatus entries.
   The split uses the actual remaining heartbeat byte budget for that SCM 
endpoint.
   The unsent remainder replaces the original queued report and is sent in 
later heartbeats.
   Advantages:
   - Smaller, more localized change.
   - No change to the generic ReportPublisher API.
   - Uses the available heartbeat space efficiently.
   - Naturally handles different remaining budgets for different SCM endpoints.
   
   
   Trade-off:
   -StateContext needs splitting/remainder handling.
   -The aggregate may be split independently for each endpoint.
   
   Approach 2: split in CommandStatusReportPublisher
   
   - Change ReportPublisher#getReport() to getReports().
   - CommandStatusReportPublisher creates multiple reports bounded by actual 
serialized size.
   - All chunks are queued during the same publisher execution.
   - Heartbeat construction only admits complete queued reports that fit its 
remaining count and byte budgets.
   
   Advantages:
   - Clear separation: the publisher creates bounded reports, while heartbeat 
construction only performs admission.
   - Simpler StateContext selection logic with no protobuf remainder 
replacement.
   - Chunking is performed once before reports are added to endpoint queues.
   
   Trade-offs:
   - Broader API change affecting every report publisher.
   - It still requires heartbeat-side size accounting because FCRs, ICRs and 
actions consume the same RPC payload.
   - Less efficient packing: for example, a complete 30 MB report is deferred 
when only 20 MB remains, even if some statuses inside it could have fitted.
   It does not solve an individual CommandStatus exceeding the safe RPC size; 
both approaches have that limitation.
   Therefore, publisher-side splitting is mainly an ownership and 
simplification choice, not an additional wire-size guarantee
   
   @sumitagrawl @smengcl @aryangupta1998 could you please share your thoughts 
to conclude on this one. Thanks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to