[
https://issues.apache.org/jira/browse/MINIFICPP-1062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joe Witt updated MINIFICPP-1062:
--------------------------------
Summary: Improve FlowFile repository performance by leveraging batching
(was: Flowfile repo performance is unacceptable)
> Improve FlowFile repository performance by leveraging batching
> --------------------------------------------------------------
>
> Key: MINIFICPP-1062
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1062
> Project: Apache NiFi MiNiFi C++
> Issue Type: Improvement
> Affects Versions: 0.6.0
> Reporter: Arpad Boda
> Assignee: Arpad Boda
> Priority: Blocker
> Fix For: 0.7.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In case a processor produces high number of flowfiles in big batches (50+ in
> one onTrigger call), MiNiFi performance is ridiculous, the max throughput is
> at about 100FF/s on a decent notebook.
> The bottleneck is related to the way ProcessSession and Connection handles
> new flow records: every record is added to flowfile repo in a new
> transaction.
> As rocksdb supports batching, flowfiles created in one batch should be added
> as a single transaction.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
