[jira] [Created] (BEAM-8145) Pubsub message size limit not taking size increase from base64 encoding into account

[Michael Yzvenn Wolanski (Jira)]

Michael Yzvenn Wolanski created BEAM-8145:
-

 Summary: Pubsub message size limit not taking size increase from 
base64 encoding into account
 Key: BEAM-8145
 URL: https://issues.apache.org/jira/browse/BEAM-8145
 Project: Beam
  Issue Type: Bug
  Components: io-java-gcp
Reporter: Michael Yzvenn Wolanski


In the PubSubIO, the default max batch size is set to `10 * 1024 * 1024` bytes. 
This however does not take into account the size increase of base64 encoding 
the messages after the flush. Base64 encodes each set of three bytes into four 
bytes.

Therefore the 'true' size limit placed on the unencoded batch should be
x = ((10 * 1024 * 1024) / 4) * 3 = 7864320 bytes



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (BEAM-8145) Pubsub message size limit not taking size increase from base64 encoding into account

[Michael Yzvenn Wolanski (Jira)]

[ 
https://issues.apache.org/jira/browse/BEAM-8145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962529#comment-16962529
 ] 

Michael Yzvenn Wolanski commented on BEAM-8145:
---

Yup, it is. Thanks.

> Pubsub message size limit not taking size increase from base64 encoding into 
> account
> 
>
> Key: BEAM-8145
> URL: https://issues.apache.org/jira/browse/BEAM-8145
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Reporter: Michael Yzvenn Wolanski
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> In the PubSubIO, the default max batch size is set to `10 * 1024 * 1024` 
> bytes. This however does not take into account the size increase of base64 
> encoding the messages after the flush. Base64 encodes each set of three bytes 
> into four bytes.
> Therefore the 'true' size limit placed on the unencoded batch should be
> x = ((10 * 1024 * 1024) / 4) * 3 = 7864320 bytes



--
This message was sent by Atlassian Jira
(v8.3.4#803005)