Re: pubsubliteio ack problem

[Nirav Patel]

whats your pipeline look like?

1. dataflow -> 2. pubsub lite publish -> 3. pubsub lite subscribe -> 4.
[spark or dataflow or whatever]

then step 2 is where we think either publish requests are being throttled
or taking too long for pslite server to either process or ack the message
as you mentioned. Slow latency is also what we struggled with. Their
throughput should scale with more capacity units and partition counts but
it doesn't in practice. You can check throughput utilization metrics
however. If its above 100 that will also create back pressure.

With java pslite io you won't see much backpressuer in dataflow
because that client just gives up after 1 min and throws errors that 1000s
of messages just didn't got published.





On Thu, Dec 21, 2023 at 5:41 PM hsy...@gmail.com  wrote:

> In my application, the pubsubliteio seems never ack the message and the
> data lateness is building up forever, my question is how does dataflow know
> when to ack the message, How does the engine even know when it is
> processed?
>

pubsubliteio ack problem

[hsy...@gmail.com]

In my application, the pubsubliteio seems never ack the message and the
data lateness is building up forever, my question is how does dataflow know
when to ack the message, How does the engine even know when it is
processed?

Re: pubsubliteio is super slow

[Valentyn Tymofieiev via user]

Hello, I talked offline to a few folks about this,
 - NO_CLIENT_TOKENS means the client code is getting flow controlled.
 - we'd need  more information to debug further, such as the Dataflow job
id , or project id , the version of the Beam SDK you use.
 - Reaching out to GCP support sounds like a good next step to get help.
See: https://cloud.google.com/support-hub and
https://cloud.google.com/support/docs/issue-trackers .

On Tue, Dec 19, 2023 at 10:54 PM hsy...@gmail.com  wrote:

> Do you have a ticket?
>
> And in the pubsublite metrics it show NO_CLIENT_TOKENS
>
> On Tue, Dec 19, 2023 at 1:39 PM Nirav Patel  wrote:
>
>> we have. yes it is super slow.  I tested python, java IO version as well
>> besides beam IO. we reported to google about this problem.
>>
>> On Tue, Dec 19, 2023 at 10:17 AM hsy...@gmail.com 
>> wrote:
>>
>>> Any one is using pubsublite? I find it super slow 5 messages/sec and
>>> there is no options for me to tune the performance
>>>
>>