oh now i get it.. yes its a unique instance of the ConsumeKafka proc for each topic!
Kindly let me know how do i increase the flow controller thread pool size and timeout associated with any single kafka consumer? On Mon, Jun 11, 2018 at 11:17 AM Joe Witt <joe.w...@gmail.com> wrote: > So you have a unique instance of the ConsumeKafka proc for each topic > rhen, right? > > Id increase the flow controller thread pool size by quite a bit as well. > > On Sun, Jun 10, 2018, 10:13 PM Faisal Durrani <te04.0...@gmail.com> wrote: > >> Hi, >> >> Yes the kafka service is hosted on a single server while NIFI is on a >> cluster of 4 servers. I'm not entirely sure what wild carding of topics is >> but kafka is integrated with a Oracle golden gate and the topics are auto >> generated as soon as a new table is created in Oracle. >> >> "If you want that in a single instance you may need to alter the timeout >> associated with any single kafka consumer. " >> Can let me know which configuration is this? >> >> The consumer kafka processors are scheduled(Timer driven ) to run on all >> four nodes with 4 concurrent task so i assume there will be 16 threads.( >> the Maximum timer driven thread count is set to 40) >> >> We tested the whole data flow with about 15-20 consumers and everything >> worked fine with out any errors. We started getting all these wired errors >> as soon as we bumped up our load test with 150+ consume kafka processors. >> >> On Mon, Jun 11, 2018 at 10:59 AM Joe Witt <joe.w...@gmail.com> wrote: >> >>> Hello >>> >>> Is this a single instance with wildcarding of topics? Please share >>> config details. >>> >>> If you want that in a single instance you may need to alter the timeout >>> associated with any single kafka consumer. The assignment will be per >>> topic per partion. How many he threads for that processor? >>> >>> Finally, consider using ConsumeKafkaRecord and if you are using kaka 1 >>> or newer use the latest processor. >>> >>> thanks >>> >>> On Sun, Jun 10, 2018, 9:21 PM Faisal Durrani <te04.0...@gmail.com> >>> wrote: >>> >>>> Does anyone know about this error from Kafka? I am using Nifi 1.5.0 >>>> with ConsumerKafka processor. >>>> >>>> ConsumeKafka[id=34753ed3-9dd6-15ed-9c91-147026236eee] Failed to retain >>>> connection due to No current assignment for partition TEST_KAFKA_TOPIC: >>>> >>>> This is the first time we are testing Nifi to consume from over 200 >>>> topics and its failing terribly so far. When this error goes the other one >>>> comes up which is as below >>>> >>>> Was interrupted while trying to communicate with Kafka with lease >>>> org.apache.nifi.processors.kafka.pubsub.ConsumerPool$ >>>> SimpleConsumerLease@6cb8afba. Will roll back session and discard any >>>> partially received data. >>>> >>>
