Hi Andrew/Bryan,
Thank you for your replies. From the initial functional tests it looks like
that upgrading the consume_kafka processor to 1_0 is working but I'll run
the load tests to be sure of that. I'll try implementing the rest of the
ideas and will come back in case of any issues.
Thanks
Hi Faisal,
Some observations and next steps.
1. Brian has a great point of using the more appropriate consumer that
will leverage the latest API.
2. This still certainly feels like a resource issue -- testing a single
Kafka broker first and not growing the cluster seems odd especially
I’m not sure if this will help, but you mentioned you are using Kafka
(1.0.0.3.1.0).
NiFi has processors that correspond to the Kafka version, and generally it is
best to use the client version that matches the broker version.
So you should be using ConsumeKafka_1_0 which uses the Kafka 1.0.0
Hi Andrew,
Thanks for your reply. PFB the answers.
1. Are all 200 topics hosted on a single Kafka broker (a single node)?
[ans] :Yes, the single node Kafka server is integrated with Golden gate and
we need to consume transactions from around 250 tables ,each of which has a
corresponding
Hi Faisal,
Sorry for the slow response. I do have a couple of questions:
1. Are all 200 topics hosted on a single Kafka broker (a single node)?
2. What is the number of partitions per topic?
3. Which Kafka consumer for NiFi are you using?
4. Can you share the configuration of that?
Hi Andrew,
I was wondering if you got the chance to review the information i sent. I
seem to have tried most of the options but i guess Nifi doesn't scale by
default much like other distributed systems that i have worked with (mainly
teradata) and needs to be configured alot before you can expect
hi Andrew,
The kafka broker is hosted on a single node and this particular topic has
just 1 partition. The consume kafka processor is scheduled to run only on
primary node with 1 concurrent processor. Everything works well with about
50 consumers consuming from 50 topics of the same nature. When
Hi Faisal,
How many partitions are there for that TEST_KAFKA_TOPIC topic?
On Thu, Jun 14, 2018 at 9:06 PM Faisal Durrani wrote:
> Hi Mark, The heap size is set to 4 gb and the time driven threas count is
> set to 32 since its a 8 core node( we have a cluster of 4 nifi nodes) .
> Despite of all
Hi Mark, The heap size is set to 4 gb and the time driven threas count is
set to 32 since its a 8 core node( we have a cluster of 4 nifi nodes) .
Despite of all this we keep getting these errors :( .
On Thu, 14 Jun 2018, 10:02 p.m. Mark Payne, wrote:
> Faisal,
>
> How much heap do you have
Faisal,
How much heap do you have allocated to your NiFi instance? In
conf/bootstrap.conf
the default value is 512 MB. If you haven't changed that, you could be just
running out of
heap.
Also, have you changed the maximum number of threads available to your NiFi
instance?
In the top-right
oh now i get it.. yes its a unique instance of the ConsumeKafka proc for
each topic!
Kindly let me know how do i increase the flow controller thread pool size
and timeout associated with any single kafka consumer?
On Mon, Jun 11, 2018 at 11:17 AM Joe Witt wrote:
> So you have a unique
So you have a unique instance of the ConsumeKafka proc for each topic rhen,
right?
Id increase the flow controller thread pool size by quite a bit as well.
On Sun, Jun 10, 2018, 10:13 PM Faisal Durrani wrote:
> Hi,
>
> Yes the kafka service is hosted on a single server while NIFI is on a
>
Hi,
Yes the kafka service is hosted on a single server while NIFI is on a
cluster of 4 servers. I'm not entirely sure what wild carding of topics is
but kafka is integrated with a Oracle golden gate and the topics are auto
generated as soon as a new table is created in Oracle.
"If you want that
Hello
Is this a single instance with wildcarding of topics? Please share config
details.
If you want that in a single instance you may need to alter the timeout
associated with any single kafka consumer. The assignment will be per
topic per partion. How many he threads for that processor?
Does anyone know about this error from Kafka? I am using Nifi 1.5.0 with
ConsumerKafka processor.
ConsumeKafka[id=34753ed3-9dd6-15ed-9c91-147026236eee] Failed to retain
connection due to No current assignment for partition TEST_KAFKA_TOPIC:
This is the first time we are testing Nifi to consume
15 matches
Mail list logo
