Github user JasonMWhite commented on the issue:
https://github.com/apache/spark/pull/17774
I think @koeninger's suggestion is valid. `effectiveRateLimitPerPartition`
is the upper bound on the number of messages per partition per second, and
`maxMessagesPerPartition` sets an upper bound on the number of messages to be
retrieved per partition per batch window.
Making `effectiveRateLimitPerPartition` a float will allow it to handle
properly rates of < 1/partition/s, so this is definitely a good idea.
`maxMessagesPerPartition` must still be an integer, as you can't retrieve
partial messages. All agreed there.
Setting `maxMessagesPerPartition` to have a minimum of 1 message per window
is a good safe value to allow progress in all cases. If there isn't 1 message
to retrieve, `clamp` will prevent it from attempting to retrieve an invalid
message.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
