Flink Version - 1.6.1 In our application, we consume from Kafka and sink to Cassandra in the end. We are trying to introduce a custom async function in front of the Sink to carry out some customized operations. In our testing, it appears that the Async function is not generating backpressure to slow down our Kafka Source when Cassandra becomes unhappy. Essentially compared to an almost identical job where the only difference is the lack of the Async function, Kafka source consumption speed is much higher under the same settings and identical Cassandra cluster. The experiment is like this.
Job 1 - without async function in front of Cassandra Job 2 - with async function in front of Cassandra Job 1 is backpressured because Cassandra cannot handle all the writes and eventually slows down the source rate to 6.5k/s. Job 2 is slightly backpressured but was able to run at 14k/s. Is the AsyncFunction somehow not reporting the backpressure correctly? Thanks, Seed
