bschofield opened a new issue #437:
URL: https://github.com/apache/pulsar-client-go/issues/437
#### Observed behavior
When using `SendAsync()` together with a large `BatchingMaxPublishDelay`,
such that batch flushing is driven mainly by `BatchingMaxMessages`, send stalls
can occur.
I don't fully understand the cause of these, but increasing
`MaxPendingMessages` seems to make them go away.
It may be relevant that I am producing to a topic with several partitions.
#### Steps to reproduce
Create a producer with a large `BatchingMaxPublishDelay` and the other
values default, e.g.
pulsar.ProducerOptions{
Topic: topic,
CompressionType: pulsar.ZLib,
BatchingMaxPublishDelay: 100 * time.Second,
}
Enable debug logging and produce to a partitioned topic with a reasonable
number of partitions (in my case, six), using `SendAsync()`. Note that the
debug log will frequently stall after a `Received send request` message, and
pause until a flush initiated by the max publish delay occurs.
Increase `MaxPendingMessages` to 2000 and try again. The stalls now go away.
For optimal throughput, it _seems_ that I need to set `MaxPendingMessages`
to approx `BatchingMaxMessages` multiplied by the number of partitions in the
target topic.
#### System configuration
Pulsar version: 2.6.1
Client version: 71cc54f (current master)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
