[jira] [Commented] (BEAM-7883) PubsubIO (Java) write batch size can exceed request payload limit
[ https://issues.apache.org/jira/browse/BEAM-7883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16911473#comment-16911473 ] Yurii Atamanchuk commented on BEAM-7883: Thanks! Yeah, another option can be to add something like `public abstract byte[] encodePayload(byte[] payload)` into PubsubClient to get better estimation (esp for Json clinet) of message size, but probably it's not really worth it :) > PubsubIO (Java) write batch size can exceed request payload limit > - > > Key: BEAM-7883 > URL: https://issues.apache.org/jira/browse/BEAM-7883 > Project: Beam > Issue Type: Bug > Components: io-java-gcp >Affects Versions: 2.13.0 >Reporter: Yurii Atamanchuk >Priority: Minor > > In some (probably rare) cases PubsubIO write (in Batch mode) batch size can > exceed request payload limit of 10mb. PubsubIO ensures that batch size is > less than limit (10mb by default). But then PubsubJsonClient is used that > converts message payloads into URL-Safe Base64 encoding which can inflate > message size (in my case for json strings it was up to 25-30%). As result we > get 400 response (with 'Request payload size exceeds the limit: 10485760 > bytes' message), even though original batch had correct size. > Obvious workaround is to reduce batch size > (`PubsubIO.writeMessages().to(...).withMaxBatchBytesSize(... i.e. 5mb ...)`), > but it is a bit annoying. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (BEAM-7883) PubsubIO (Java) write batch size can exceed request payload limit
[ https://issues.apache.org/jira/browse/BEAM-7883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910649#comment-16910649 ] Chamikara Jayalath commented on BEAM-7883: -- Thanks. Batch sizing that is done at PubSub IO is just an estimations and there can be cases where actual messages go over the 10 MB hard limit. Using 'withMaxBatchBytesSize' is the correct workaround. > PubsubIO (Java) write batch size can exceed request payload limit > - > > Key: BEAM-7883 > URL: https://issues.apache.org/jira/browse/BEAM-7883 > Project: Beam > Issue Type: Bug > Components: io-java-gcp >Affects Versions: 2.13.0 >Reporter: Yurii Atamanchuk >Priority: Minor > > In some (probably rare) cases PubsubIO write (in Batch mode) batch size can > exceed request payload limit of 10mb. PubsubIO ensures that batch size is > less than limit (10mb by default). But then PubsubJsonClient is used that > converts message payloads into URL-Safe Base64 encoding which can inflate > message size (in my case for json strings it was up to 25-30%). As result we > get 400 response (with 'Request payload size exceeds the limit: 10485760 > bytes' message), even though original batch had correct size. > Obvious workaround is to reduce batch size > (`PubsubIO.writeMessages().to(...).withMaxBatchBytesSize(... i.e. 5mb ...)`), > but it is a bit annoying. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (BEAM-7883) PubsubIO (Java) write batch size can exceed request payload limit
[ https://issues.apache.org/jira/browse/BEAM-7883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16905011#comment-16905011 ] Ismaël Mejía commented on BEAM-7883: [~pabl...@google.com] can you take a look or pass to someone who can. > PubsubIO (Java) write batch size can exceed request payload limit > - > > Key: BEAM-7883 > URL: https://issues.apache.org/jira/browse/BEAM-7883 > Project: Beam > Issue Type: Bug > Components: io-java-gcp >Affects Versions: 2.13.0 >Reporter: Yurii Atamanchuk >Priority: Minor > > In some (probably rare) cases PubsubIO write (in Batch mode) batch size can > exceed request payload limit of 10mb. PubsubIO ensures that batch size is > less than limit (10mb by default). But then PubsubJsonClient is used that > converts message payloads into URL-Safe Base64 encoding which can inflate > message size (in my case for json strings it was up to 25-30%). As result we > get 400 response (with 'Request payload size exceeds the limit: 10485760 > bytes' message), even though original batch had correct size. > Obvious workaround is to reduce batch size > (`PubsubIO.writeMessages().to(...).withMaxBatchBytesSize(... i.e. 5mb ...)`), > but it is a bit annoying. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
