whats your pipeline look like?
1. dataflow -> 2. pubsub lite publish -> 3. pubsub lite subscribe -> 4.
[spark or dataflow or whatever]
then step 2 is where we think either publish requests are being throttled
or taking too long for pslite server to either process or ack the message
as you mentioned. Slow latency is also what we struggled with. Their
throughput should scale with more capacity units and partition counts but
it doesn't in practice. You can check throughput utilization metrics
however. If its above 100 that will also create back pressure.
With java pslite io you won't see much backpressuer in dataflow
because that client just gives up after 1 min and throws errors that 1000s
of messages just didn't got published.
On Thu, Dec 21, 2023 at 5:41 PM hsy...@gmail.com wrote:
> In my application, the pubsubliteio seems never ack the message and the
> data lateness is building up forever, my question is how does dataflow know
> when to ack the message, How does the engine even know when it is
> processed?
>
