Hello, I would like to ask about the rationale behind the backpressure mechanism in Flink.
As I understand it, backpressure is for handling the problem of one operator (or source) producing records faster then the next operator can consume them. However, an alternative solution would be to have a potentially "infinite" buffer for the incoming records of an operator, by spilling the buffer to disk when it is getting too large. Could you tell me why is backpressure considered a better option? Also, I would be interested in whether going with backpressure was a streaming-specific decision, or do you think that having backpressure is also better in batch jobs? Thanks, Gábor
