[jira] [Work logged] (BEAM-8145) Pubsub message size limit not taking size increase from base64 encoding into account
[ https://issues.apache.org/jira/browse/BEAM-8145?focusedWorklogId=306581=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-306581 ] ASF GitHub Bot logged work on BEAM-8145: Author: ASF GitHub Bot Created on: 04/Sep/19 17:54 Start Date: 04/Sep/19 17:54 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #8184: [BEAM-8145] Max publish batch byte size set to a value which takes base64 encodin… URL: https://github.com/apache/beam/pull/8184 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 306581) Time Spent: 0.5h (was: 20m) > Pubsub message size limit not taking size increase from base64 encoding into > account > > > Key: BEAM-8145 > URL: https://issues.apache.org/jira/browse/BEAM-8145 > Project: Beam > Issue Type: Bug > Components: io-java-gcp >Reporter: Michael Yzvenn Wolanski >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > In the PubSubIO, the default max batch size is set to `10 * 1024 * 1024` > bytes. This however does not take into account the size increase of base64 > encoding the messages after the flush. Base64 encodes each set of three bytes > into four bytes. > Therefore the 'true' size limit placed on the unencoded batch should be > x = ((10 * 1024 * 1024) / 4) * 3 = 7864320 bytes -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-8145) Pubsub message size limit not taking size increase from base64 encoding into account
[ https://issues.apache.org/jira/browse/BEAM-8145?focusedWorklogId=306578=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-306578 ] ASF GitHub Bot logged work on BEAM-8145: Author: ASF GitHub Bot Created on: 04/Sep/19 17:52 Start Date: 04/Sep/19 17:52 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #8184: [BEAM-8145] Max publish batch byte size set to a value which takes base64 encodin… URL: https://github.com/apache/beam/pull/8184#issuecomment-528011932 Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 306578) Time Spent: 20m (was: 10m) > Pubsub message size limit not taking size increase from base64 encoding into > account > > > Key: BEAM-8145 > URL: https://issues.apache.org/jira/browse/BEAM-8145 > Project: Beam > Issue Type: Bug > Components: io-java-gcp >Reporter: Michael Yzvenn Wolanski >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > In the PubSubIO, the default max batch size is set to `10 * 1024 * 1024` > bytes. This however does not take into account the size increase of base64 > encoding the messages after the flush. Base64 encodes each set of three bytes > into four bytes. > Therefore the 'true' size limit placed on the unencoded batch should be > x = ((10 * 1024 * 1024) / 4) * 3 = 7864320 bytes -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-8145) Pubsub message size limit not taking size increase from base64 encoding into account
[ https://issues.apache.org/jira/browse/BEAM-8145?focusedWorklogId=305998=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-305998 ] ASF GitHub Bot logged work on BEAM-8145: Author: ASF GitHub Bot Created on: 04/Sep/19 00:59 Start Date: 04/Sep/19 00:59 Worklog Time Spent: 10m Work Description: Primevenn commented on issue #8184: [BEAM-8145] Max publish batch byte size set to a value which takes base64 encodin… URL: https://github.com/apache/beam/pull/8184#issuecomment-527695960 R: @chamikaramj Done - https://issues.apache.org/jira/browse/BEAM-8145. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 305998) Remaining Estimate: 0h Time Spent: 10m > Pubsub message size limit not taking size increase from base64 encoding into > account > > > Key: BEAM-8145 > URL: https://issues.apache.org/jira/browse/BEAM-8145 > Project: Beam > Issue Type: Bug > Components: io-java-gcp >Reporter: Michael Yzvenn Wolanski >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > In the PubSubIO, the default max batch size is set to `10 * 1024 * 1024` > bytes. This however does not take into account the size increase of base64 > encoding the messages after the flush. Base64 encodes each set of three bytes > into four bytes. > Therefore the 'true' size limit placed on the unencoded batch should be > x = ((10 * 1024 * 1024) / 4) * 3 = 7864320 bytes -- This message was sent by Atlassian Jira (v8.3.2#803003)
