sjvanrossum commented on PR #39886: URL: https://github.com/apache/beam/pull/39886#issuecomment-6126192793
If I'm not mistaken, the default value of `maxPublishBatchByteSize` (7.5 MB) already factors in most of the expected overhead, but a batch with every field other than `data` pushed to their maximum limits may still exceed the 2.5 MB that's left. The most conservative safe limit on `data` alone is closer to 7.2 MB iirc. We could raise the default value of `maxPublishBatchByteSize` to 10 MB and use the validated message size, but you may still exceed the total payload size accepted by the API because gRPC adds more overhead to the attributes map during encoding. Not raising `maxPublishBatchByteSize` and merging this PR as is would mean that we're effectively capping the encoded batch payload size to 75% of what Pub/Sub supports. I recall that the Pub/Sub team was going to raise the maximum payload size from 10 MB to 10 MiB. If they did, then the unaccounted overhead above 2.5 MB should fit in the additional ~0.5 MB that you get out of 10 MiB. TL;DR The proposed change leaves 25% of the maximum accepted payload size unused and raising the default maximum accepted payload size may still lead to a rejection of the batch publish request. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
