[
https://issues.apache.org/jira/browse/MINIFICPP-1062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joe Witt updated MINIFICPP-1062:
--------------------------------
Description:
The current ProcessSession mechanism handles every record in a single
transaction fashion even if there is a batch of records being updated. Since
the underlying mechanism is Rocksdb and it supports batching into a single
transaction we should leverage that. This will drive baseline performance on
modest hardware up significantly.
Right now under default settings we see 100FF/s on a modest notebook and we can
do much better.
was:
In case a processor produces high number of flowfiles in big batches (50+ in
one onTrigger call), MiNiFi performance is ridiculous, the max throughput is at
about 100FF/s on a decent notebook.
The bottleneck is related to the way ProcessSession and Connection handles new
flow records: every record is added to flowfile repo in a new transaction.
As rocksdb supports batching, flowfiles created in one batch should be added as
a single transaction.
> Improve FlowFile repository performance by leveraging batching
> --------------------------------------------------------------
>
> Key: MINIFICPP-1062
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1062
> Project: Apache NiFi MiNiFi C++
> Issue Type: Improvement
> Affects Versions: 0.6.0
> Reporter: Arpad Boda
> Assignee: Arpad Boda
> Priority: Blocker
> Fix For: 0.7.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The current ProcessSession mechanism handles every record in a single
> transaction fashion even if there is a batch of records being updated. Since
> the underlying mechanism is Rocksdb and it supports batching into a single
> transaction we should leverage that. This will drive baseline performance on
> modest hardware up significantly.
> Right now under default settings we see 100FF/s on a modest notebook and we
> can do much better.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
