GitHub user markap14 opened a pull request:
https://github.com/apache/nifi/pull/2416
[WIP] NIFI 4774: Provide alternate implementation of Write-Ahead Log
Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number
you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [ ] Has your PR been rebased against the latest commit within the target
branch (typically master)?
- [ ] Is your initial contribution a single, squashed commit?
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file, including the main
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in
which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/markap14/nifi NIFI-4774
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/2416.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2416
----
commit e0588d023293bc78375c2d98638ab65dc1b5b411
Author: Mark Payne <markap14@...>
Date: 2018-01-17T19:24:04Z
NIFI-4774: Initial implementation. Not fully tested. Quite slow, so looking
at performance improvements first. If unable to make it at least as fast as
current implementation, then there's no point in wasting time on this alternate
implementation
commit 66e94968f6357785f953ec0fcca90103f2a6cae4
Author: Mark Payne <markap14@...>
Date: 2018-01-18T16:18:35Z
NIFI-4774: Experimented with some alternate approaches to writing the
updates to disk faster. The implementation was slow because of the large number
of small writes to the FlowFile Repo. The technique here was not to enter
synchronized block to write just the bytes. Instead, add the bytes to queue and
let another thread write the bytes, then notify us that it's complete. By doing
this, we were able to have the other thread combine many updates into a single
disk write, then notify several threads that the update was complete. This
gives far better performance, comparable to that of the
MinimalLockingWriteAheadLog.
commit 2424ee62e2998ff3a67d7d0e5c734f9eca334f9f
Author: Mark Payne <markap14@...>
Date: 2018-01-19T16:22:55Z
NIFI-4774: Updates to just use synchronized block for as little work as
possible; bug fixes
----
---
