goosalex commented on pull request #4410: URL: https://github.com/apache/nifi/pull/4410#issuecomment-670169916
Thanks @pvillard31 for having a look at it. We were noticing too small Flowfiles compared to the "maxPollRecords" when handling production Flows ( up to over 250'000msg/s ), on a secured Kafka cluster in a non trivial network layout. The fixed 10ms timeout per single poll roundtrip led to undeterministic polling results, reporting "no more records" after ~200 messages on average, never mind "maxPollRecords" and "maxWatingMilllis" beeing very high. In a simple, unsecured Lab setting, we could not reproduce the issue. As I understand it, "maxWaitmillis" is the maximum time a Nifi-KafkaConsumer-Processor can use up to fill a FlowFile, whereas x in kafkaConsumer.poll(x) is the timeout for a single Kafka-transaction. Perhaps we should not put the user supplied-maWaitMillis directly here, but also a lower bound of the previously hard-coded 10ms ? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
