sadanand48 commented on PR #11962: URL: https://github.com/apache/ozone/pull/11962#issuecomment-6910954499
I had a discussion with @sumitagrawl where he proposed approach 2 - the publisher-side alternative and [attached the patch](https://issues.apache.org/jira/secure/attachment/13086505/limit_at_report_publisher.patch) so we can compare both approaches before deciding. Approach 1: split while constructing the heartbeat (the current PR) CommandStatusReportPublisher remains unchanged and publishes one aggregate CommandStatusReportsProto. During heartbeat construction, the aggregate is split only between complete CommandStatus entries. The split uses the actual remaining heartbeat byte budget for that SCM endpoint. The unsent remainder replaces the original queued report and is sent in later heartbeats. Advantages: - Smaller, more localized change. - No change to the generic ReportPublisher API. - Uses the available heartbeat space efficiently. - Naturally handles different remaining budgets for different SCM endpoints. Trade-off: -StateContext needs splitting/remainder handling. -The aggregate may be split independently for each endpoint. Approach 2: split in CommandStatusReportPublisher - Change ReportPublisher#getReport() to getReports(). - CommandStatusReportPublisher creates multiple reports bounded by actual serialized size. - All chunks are queued during the same publisher execution. - Heartbeat construction only admits complete queued reports that fit its remaining count and byte budgets. Advantages: - Clear separation: the publisher creates bounded reports, while heartbeat construction only performs admission. - Simpler StateContext selection logic with no protobuf remainder replacement. - Chunking is performed once before reports are added to endpoint queues. Trade-offs: - Broader API change affecting every report publisher. - It still requires heartbeat-side size accounting because FCRs, ICRs and actions consume the same RPC payload. - Less efficient packing: for example, a complete 30 MB report is deferred when only 20 MB remains, even if some statuses inside it could have fitted. It does not solve an individual CommandStatus exceeding the safe RPC size; both approaches have that limitation. Therefore, publisher-side splitting is mainly an ownership and simplification choice, not an additional wire-size guarantee @sumitagrawl @smengcl @aryangupta1998 could you please share your thoughts to conclude on this one. Thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
