[
https://issues.apache.org/jira/browse/NIFI-4008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16068355#comment-16068355
]
ASF GitHub Bot commented on NIFI-4008:
--------------------------------------
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1891#discussion_r124803429
--- Diff:
nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-0-10-processors/src/main/java/org/apache/nifi/processors/kafka/pubsub/ConsumerLease.java
---
@@ -418,82 +419,106 @@ private void writeDemarcatedData(final
ProcessSession session, final List<Consum
}
- private void writeRecordData(final ProcessSession session, final
List<ConsumerRecord<byte[], byte[]>> records, final TopicPartition
topicPartition) {
+ private void writeRecordData(final ProcessSession session, final
List<ConsumerRecord<byte[], byte[]>> messages, final TopicPartition
topicPartition) {
// In order to obtain a RecordReader from the RecordReaderFactory,
we need to give it a FlowFile.
// We don't want to create a new FlowFile for each record that we
receive, so we will just create
// a "temporary flowfile" that will be removed in the finally
block below and use that to pass to
// the createRecordReader method.
final FlowFile tempFlowFile = session.create();
RecordSetWriter writer = null;
+ final BiConsumer<ConsumerRecord<byte[], byte[]>, Exception>
handleParseFailure = (consumerRecord, e) -> {
--- End diff --
Unfortunately, I think this PR very much conflicts with NIFI-4046/PR 1906,
which I just pushed another commit to, in order to address your feedback.
Either PR could be merged first and then the other updated. We both refactored
this piece of code to make it re-usable. However, with the new refactoring for
4046, I think I prefer the method there, of having a separate method instead of
a lambda because we end up passing in an optional message. Thoughts?
> ConsumeKafkaRecord_0_10 assumes there is always one Record in a message
> -----------------------------------------------------------------------
>
> Key: NIFI-4008
> URL: https://issues.apache.org/jira/browse/NIFI-4008
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
> Affects Versions: 1.2.0
> Reporter: Koji Kawamura
> Assignee: Koji Kawamura
>
> ConsumeKafkaRecord_0_10 uses ConsumerLease underneath, and it [assumes there
> is one Record available in a consumed
> message|https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-0-10-processors/src/main/java/org/apache/nifi/processors/kafka/pubsub/ConsumerLease.java#L434]
> retrieved from a Kafka topic.
> But in fact, a message can contain 0 or more records in it. For example, with
> a record schema shown below:
> {code}
> {
> "type": "record",
> "name": "temp",
> "fields" : [
> {"name": "value", "type": "string"}
> ]
> }
> {code}
> Multiple records can be sent within a single message, e.g. using JSON:
> {code}
> [{"value": "a"}, {"value": "b"}, {"value": "c"}]
> {code}
> But ConsumeKafkaRecord only outputs the first record:
> {code}
> [{"value": "a"}]
> {code}
> Also, if a message doesn't contain any record in it, the processor fails with
> NullPointerException.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
