[
https://issues.apache.org/jira/browse/NIFI-4775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brandon DeVries updated NIFI-4775:
----------------------------------
Description:
Currently, when a FlowFile is written to the FlowFile Repository, the repo can
either fsync or not, depending on nifi.properties. We should allow a third
option, of fsync only for CREATE events. In this case, if we receive new data
from a source we can fsync the update to the FlowFile Repository before ACK'ing
the data from the source. This allows us to guarantee data persistence without
the overhead of an fsync for every FlowFile Repository update.
It may make sense, though, to be a bit more selective about when do this. For
example if the source is a system that does not allow us to acknowledge the
receipt of data, such as a ListenUDP processor, this doesn't really buy us
much. In such a case, we could be smart about avoiding the high cost of an
fsync. However, for something like GetSFTP where we have to remove the file in
order to 'acknowledge receipt' we can ensure that we wait for the fsync before
proceeding.
NOTE: This functionality was ultimately provided in a new implementation backed
by RocksDB
was:
Currently, when a FlowFile is written to the FlowFile Repository, the repo can
either fsync or not, depending on nifi.properties. We should allow a third
option, of fsync only for CREATE events. In this case, if we receive new data
from a source we can fsync the update to the FlowFile Repository before ACK'ing
the data from the source. This allows us to guarantee data persistence without
the overhead of an fsync for every FlowFile Repository update.
It may make sense, though, to be a bit more selective about when do this. For
example if the source is a system that does not allow us to acknowledge the
receipt of data, such as a ListenUDP processor, this doesn't really buy us
much. In such a case, we could be smart about avoiding the high cost of an
fsync. However, for something like GetSFTP where we have to remove the file in
order to 'acknowledge receipt' we can ensure that we wait for the fsync before
proceeding.
> Create a FlowFile repo backed by RocksDB
> ----------------------------------------
>
> Key: NIFI-4775
> URL: https://issues.apache.org/jira/browse/NIFI-4775
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
> Reporter: Mark Payne
> Assignee: Brandon DeVries
> Priority: Major
> Fix For: 1.10.0
>
> Attachments: RocksDBFlowFileRepo.html, rocksdb-flowfile-repo.adoc
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Currently, when a FlowFile is written to the FlowFile Repository, the repo
> can either fsync or not, depending on nifi.properties. We should allow a
> third option, of fsync only for CREATE events. In this case, if we receive
> new data from a source we can fsync the update to the FlowFile Repository
> before ACK'ing the data from the source. This allows us to guarantee data
> persistence without the overhead of an fsync for every FlowFile Repository
> update.
> It may make sense, though, to be a bit more selective about when do this. For
> example if the source is a system that does not allow us to acknowledge the
> receipt of data, such as a ListenUDP processor, this doesn't really buy us
> much. In such a case, we could be smart about avoiding the high cost of an
> fsync. However, for something like GetSFTP where we have to remove the file
> in order to 'acknowledge receipt' we can ensure that we wait for the fsync
> before proceeding.
> NOTE: This functionality was ultimately provided in a new implementation
> backed by RocksDB
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
