[
https://issues.apache.org/jira/browse/NIFI-6515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17794388#comment-17794388
]
Bryan Bende commented on NIFI-6515:
-----------------------------------
[~larsfrancke] I don't think FetchParquet should be reading the whole file into
memory, it should be reading one record at a time and writing it to the output
stream of the flow file. I think the original reason for this Jira was to able
to parallelize the processing after this processor so that you don't have to do
another SplitRecord after this which would require re-reading the whole 23GB
again.
> FetchParquet max FlowFile size
> ------------------------------
>
> Key: NIFI-6515
> URL: https://issues.apache.org/jira/browse/NIFI-6515
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
> Reporter: Matt Gilman
> Priority: Major
>
> FetchParquet cannot transfer out multiple FlowFiles. We should introduce a
> new property to set the size of the outgoing FlowFile and then transfer as
> many FlowFiles as needed based on the fetched data.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
