[
https://issues.apache.org/jira/browse/NIFI-4775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16905523#comment-16905523
]
Joseph Witt commented on NIFI-4775:
-----------------------------------
[~devriesb]
* The title of this JIRA does not help describe the work done at all.
* The great docs on this JIRA are not present in the commit therefore nobody
would know how to use it. This should happen with the commit or be really darn
close behind.
* The community follows an RTC model. I do not believe this has been reviewed
by a committer (though the non committer input is helpful and encouraged).
That said, in reading our language around commits in our e-mail history and on
our wiki/contributor guide we don't really clarify. I could see this being
worth debating but in any event see my other concerns.
* I am surprised there are zero LICENSE and NOTICE impacts to bringing this
library in. There is a comment about being unsure but nothing indicating this
was verified.
* The change to the WriteAheadFlowFileRepository appears to have nothing to do
with your commit and possibly seems related to a (serious) bug being found. If
so this is super helpful and important and should definitely not be in this
JIRA/feature and should have its own JIRA/PR for resolution and tracking.
Can you please fix the above things and do the diligence necessary to ensure
the LICENSE/NOTICE work is done. I am re-opening as we would not want to
create an RC with this at this state.
> Allow FlowFile Repository to optionally perform fsync when writing CREATE
> events but not other events
> -----------------------------------------------------------------------------------------------------
>
> Key: NIFI-4775
> URL: https://issues.apache.org/jira/browse/NIFI-4775
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
> Reporter: Mark Payne
> Assignee: Brandon DeVries
> Priority: Major
> Fix For: 1.10.0
>
> Attachments: RocksDBFlowFileRepo.html, rocksdb-flowfile-repo.adoc
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Currently, when a FlowFile is written to the FlowFile Repository, the repo
> can either fsync or not, depending on nifi.properties. We should allow a
> third option, of fsync only for CREATE events. In this case, if we receive
> new data from a source we can fsync the update to the FlowFile Repository
> before ACK'ing the data from the source. This allows us to guarantee data
> persistence without the overhead of an fsync for every FlowFile Repository
> update.
> It may make sense, though, to be a bit more selective about when do this. For
> example if the source is a system that does not allow us to acknowledge the
> receipt of data, such as a ListenUDP processor, this doesn't really buy us
> much. In such a case, we could be smart about avoiding the high cost of an
> fsync. However, for something like GetSFTP where we have to remove the file
> in order to 'acknowledge receipt' we can ensure that we wait for the fsync
> before proceeding.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
