sjvanrossum commented on PR #31608: URL: https://github.com/apache/beam/pull/31608#issuecomment-2380592688
@scwhittle I just realized that the `PubSubClient` implementations used by the writer transforms do not apply flow control to publish calls. The throughput quota per ordering key is capped to 1MB/s and would only be enforced by server-side rejections if exceeded by this connector. The official Pub/Sub client library for Java provides client-side flow control and batching per ordering key as well as an executor service that sends batches for different ordering keys concurrently and sequentially sending batches for a single ordering key. Would it make sense to change the client implementation to the official client library or should we duplicate that functionality by extending the Beam clients to handle this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
