Hello There isn't really a feedback mechanism based on load on the Kafka topic. When you say overrunning the topic do you mean that you don't want there to be a large lag between consumers and their current offset and if that grows you want NiFi to slow down? I dont believe there is anything inherent to the kafka producer protocol that would inform of us this. We could periodically poll for this information and optional back-off.
Is this what you have in mind? Thanks On Thu, Apr 4, 2019 at 10:34 AM l vic <lvic4...@gmail.com> wrote: > I have to ingest large (200,000 messages) data set into Kafka topic as > quickly as possible without overrunning topic... Right now I just use rate > limiter to do it but can be there some better "adaptive" way to do it? > Thank you... > -V >
