Each queue has back-pressure settings on it which default to 10k flow
files or 1GB size of flow files. When one of these thresholds is
exceeded, the preceeding processor will not execute until the queue
goes back below the threshold.
Most likely if GenerateFlowFile has a Run Schedule of 0 seconds,
Actually, it's not Kafka topic but Nifi queue between "generateFlow" and
"PutKafka" gets overflown
On Thu, Apr 4, 2019 at 10:58 AM Joe Witt wrote:
> Can you share screenshots, logs, and a more detailed description of what
> you're doing, observing with nifi and the system and what you expect it
We need to define what "NiFi stops responding" means...
Are there tons of flow files queued up before before PublishKafka?
Are there back-pressure indicators on any of the queues?
Do the kafka related processors show active threads in the top right
corners of the processors?
Does NiFI crash?
On T
What's the concurrency for these processors? What's a global NiFi thread
pool size?
I wonder if you might be running out of available threads while they are
waiting for external system i/o under load.
Andrew
On Thu, Apr 4, 2019, 8:24 AM l vic wrote:
> What's this particular processing group do
What's this particular processing group does: writes large dataset to Kafka
topic, one consumer reads from topic and saves data to Hbase/PQS table,
another consumer writes to ES index
On Thu, Apr 4, 2019 at 10:58 AM Joe Witt wrote:
> Can you share screenshots, logs, and a more detailed descripti
Can you share screenshots, logs, and a more detailed description of what
you're doing, observing with nifi and the system and what you expect it to
be doing.
Thanks
On Thu, Apr 4, 2019 at 10:56 AM l vic wrote:
> No, actually what happens is - NiFi stops responding ( if I use it without
> rate c
No, actually what happens is - NiFi stops responding ( if I use it without
rate contol)
On Thu, Apr 4, 2019 at 10:42 AM Joe Witt wrote:
> Hello
>
> There isn't really a feedback mechanism based on load on the Kafka topic.
> When you say overrunning the topic do you mean that you don't want
Hello
There isn't really a feedback mechanism based on load on the Kafka topic.
When you say overrunning the topic do you mean that you don't want there to
be a large lag between consumers and their current offset and if that grows
you want NiFi to slow down? I dont believe there is anything inhe
I have to ingest large (200,000 messages) data set into Kafka topic as
quickly as possible without overrunning topic... Right now I just use rate
limiter to do it but can be there some better "adaptive" way to do it?
Thank you...
-V
