[
https://issues.apache.org/jira/browse/NIFI-6515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17794201#comment-17794201
]
Lars Francke commented on NIFI-6515:
------------------------------------
We are stumbling across this as well now.
We have to read a 23GB Parquet file, currently using {{FetchParquet}} but it
seems as if that'll read the whole file in memory before passing it on.
It takes a long time and we get timeouts in various parts of the system.
We really would like the option of FetchParquet already splitting the data up
into batches (e.g. by size or count) as low as one FlowFile per record.
Has anyone here found a solution? If not I might consider expanding this
Processor.
> FetchParquet max FlowFile size
> ------------------------------
>
> Key: NIFI-6515
> URL: https://issues.apache.org/jira/browse/NIFI-6515
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
> Reporter: Matt Gilman
> Priority: Major
>
> FetchParquet cannot transfer out multiple FlowFiles. We should introduce a
> new property to set the size of the outgoing FlowFile and then transfer as
> many FlowFiles as needed based on the fetched data.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
