[
https://issues.apache.org/jira/browse/NIFI-14020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17933063#comment-17933063
]
ASF subversion and git services commented on NIFI-14020:
--------------------------------------------------------
Commit fb17c10b2fc2ebf78f96f7e9b105f2143c3b5235 in nifi's branch
refs/heads/main from Pierre Villard
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=fb17c10b2f ]
NIFI-14020 - Add Record and Demarcator support to ConsumeGCPubSub
This closes #9530.
Signed-off-by: Lehel Boer <[email protected]>
> Add Record and Demarcator support to ConsumeGCPubSub
> ----------------------------------------------------
>
> Key: NIFI-14020
> URL: https://issues.apache.org/jira/browse/NIFI-14020
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
> Reporter: Pierre Villard
> Assignee: Pierre Villard
> Priority: Major
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> At the moment, ConsumeGCPubSub will generate one FlowFile per consumed
> message (the Batch Size property is only specifying the maximum number of
> messages we may pull from the subscription with one API call). This can be
> extremely inefficient.
> Similarly to the Kafka processors, we should add the option to have multiple
> Processing Strategies:
> * Flow File - which is the current behavior - where one message is one
> FlowFile and FlowFile attributes will be used to store the attributes
> associated with the message as well as some information such as message ID,
> ack ID, etc.
> * Demarcator - where messages will be appended into a single FlowFile with a
> custom demarcator between each message. In this case specific attributes
> associated to messages will be lost. This however is the most efficient
> strategy if very high throughput is required and message format is allowing
> this approach.
> * Record - where a reader and writer can be specified to process the
> messages. This is useful if we want to change message format on the fly or if
> the message format is not allowing the demarcator strategy. In addition, an
> output strategy is available with two allowable values:
> ** Value - messages are all added in the same flowfile with the specified
> writer. In this case specific attributes associated to messages will be lost.
> ** Wrapper - in this case, we are overriding the schema of the writer to
> include the metadata of the message as well as a map of its attributes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
