Alaksiej Ščarbaty created NIFI-16187:
----------------------------------------
Summary: Write messages with different schemas into same FlowFiles
in ConsumeKafka
Key: NIFI-16187
URL: https://issues.apache.org/jira/browse/NIFI-16187
Project: Apache NiFi
Issue Type: Improvement
Components: Extensions
Affects Versions: 2.11.0
Reporter: Alaksiej Ščarbaty
Assignee: Alaksiej Ščarbaty
h2. Problem
When _ConsumeKafka_ is configured with _RECORD_ processing strategy, it uses
nifi record reader and writer for serde operations. Incoming messages are
written into FlowFiles in batches, but only if their schemas are identical. In
case there is a schema mismatch, a new FlowFile is created.
This works poorly for semi-structured json messages with schema inference, as a
lot of tiny FlowFiles are created. Very often with a single record per
FlowFile, which negatively affects the performance of the downstream processors.
h2. Suggestion
Add a new {{_Schema Conflict Resolution_}} option to the processor.
Its default value is {{{}_Create New FlowFile_{}}}, which will represent the
current behavior of the processor. Nothing is going to be changes in this path
- no risk of introducing degradation to existing customers.
Another value would be {{{}Merge Schemas on Write{}}}. # The processor reads a
batch of incoming Kafka messages, which are already in memory, into a list.
# The processor iterates over the records and merges their read schemas into a
single write schema.
# The processor creates a single record writer with the merged schema, the
records are written into a single FlowFile using this schema.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
