[
https://issues.apache.org/jira/browse/NIFI-14696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dariusz Seweryn updated NIFI-14696:
-----------------------------------
Summary: KinesisRecordProcessorRecord mishandles record schema changes
(was: KinesisRecordProcessorRecord does not handle
IllegalTypeConversionException)
> KinesisRecordProcessorRecord mishandles record schema changes
> -------------------------------------------------------------
>
> Key: NIFI-14696
> URL: https://issues.apache.org/jira/browse/NIFI-14696
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
> Affects Versions: 2.4.0
> Reporter: Dariusz Seweryn
> Assignee: Dariusz Seweryn
> Priority: Major
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> {{KinesisRecordProcessorRecord#processRecord}} method may encounter an
> {{IllegalTypeConversionException}} while trying write the record. In this
> case the exception is caught in the
> {{AbstractKinesisRecordProcessor#attemptProcessRecord}} function.
> {{KinesisRecordProcessorRecord#processRecord}} apart from writing records is
> responsible for:
> # creating FlowFiles and associated objects
> # closing the the FlowFile at the last record
> # routing problematic records to the Failure Relationship
> Current implementation has several bugs:
> * in case the problematic record does not conform to writers schema, the
> exception is not handled in the class — the record is not routed properly to
> the Failure Relationship
> * in case the problematic record is the last record in the batch, the
> FlowFile is left unfinished, without arguments
> The main problem is that the writers schema is determined based on the first
> observed record. If the schema is inferred — it is possible a number field in
> the first record may get inferred as an Integer and some other as a Long. In
> such case, the writers schema should be widened.
> Fixes needed:
> # Handling of {{IllegalTypeConversionException}}
> # Closing FlowFile when last record processing failed
> # Implement writer schema widening on incompatible records — possibly output
> multiple FlowFiles where each next has wider schema
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
