Github user akonopko commented on the issue:
https://github.com/apache/spark/pull/19431
Latest rate means rate of previous batch. Is it possible that in alive
system 0 events were processed? Only if there is no backlog and no new events
came during last batch. Completely possible.
This happens during first ran. And this parameter should limit it during
1st ran. Quote from docs:
This is the initial maximum receiving rate at which each receiver will
receive data for the first batch when the backpressure mechanism is enabled.
If it happened during system run, for example there is no backlog and no
new events came, we still need to limit system rate since with LatestRate = 0
it results in no limit, causing danger of overflowing the system.
If somehow cluster was so heavily loaded with other processes that could
process 0 events in Spark Streaming, this means that we might have huge backlog
after that. Which mean without this fix system has big chance of overflowing
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
