Chris Sampson created NIFI-8532:
-----------------------------------
Summary: ConsumeKinesisStream tuning/performance testing
Key: NIFI-8532
URL: https://issues.apache.org/jira/browse/NIFI-8532
Project: Apache NiFi
Issue Type: Improvement
Reporter: Chris Sampson
{{ConsumeKinesisStream}} was added in NIFI-2982 with some seemingly sensible
default property settings that are passed to the Kinesis Client Library.
Some performance testing/experimentation should be conducted for different NiFi
setups (e.g. single instance, remote instance, cluster) to determine whether
other settings should be used by default.
For example (from comments in PR
[#4822|https://github.com/apache/nifi/pull/4822]):
{quote}
[~turcsanyip] did some kind of performance tests with 1 Mio messages in 12
shards and running NiFi on [his] local machine (no cluster). It took 64-74
seconds with default settings (no additional dynamic properties).
The interesting thing that [he] could not see any difference when I set
Checkpoint Interval to 0 sec (that is checkpointing "synchronously" after each
bunch of messages received in IRecordProcessor.processRecords() callback). It
seems there is no significant overhead of checkpointing more frequently (and it
has the advantage of having fewer duplicated messages in case of restart).
It can be investigated further in a follow-up ticket with more sophisticated
performance tests (nifi cluster, non-local machine, tuning KCL properties, etc)
and the default can be adjusted if that is reasonable.
{quote}
(this is *that* ticket)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
