fgerlits opened a new pull request, #1518:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1518
On a sample test data of 20,000 files on Windows, `onTrigger` now takes ~2.5
sec instead of ~7.5 sec in the case when all 20,000 files have been processed
already. The main changes are:
* except for the file name (with path) and modification time, only query
anything else (eg. file size) when and if we need it;
* check first if the file is new, and only later if it matches the selection
criteria;
* instead of collecting the file names in a vector first and then processing
them in a loop, we process the files directly in the `list_dir` callback
(without this, the `onTrigger` duration was ~3.5 sec).
NOTE: there are three changes in behavior:
1. When storing the list of files with the latest modification time, we used
to store the path only, which doesn't uniquely identify the file. I'm pretty
sure this was a bug, now we store path + filename, and there is a new unit test
to verify that this works.
2. If we are not able to find the modification time of the file (apparently,
this can happen on Windows), we used to skip the file. Now we list the file,
with the modification time set to the epoch of the `system_clock` .
3. If the `Path Filter` is set, we used to list all files in the `Input
Directory` root, plus all files in subdirectories the names of which match the
filter. Now we only list files in matching subdirectories, only including the
root if the filter regex includes it (eg., `"good_subdir|ugly_subdir|\."`).
Also, `Path Filter` is ignored if `Recurse Subdirectories` is false -- in this
case, all files in the Input Directory are listed, but none in the
subdirectories. The new behavior matches the current behavior of NiFi; I think
the old behavior matched the behavior of an earlier version of NiFi.
---
Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [x] Does your PR title start with MINIFICPP-XXXX where XXXX is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically main)?
- [ ] Is your initial contribution a single, squashed commit?
### For code changes:
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file?
- [ ] If applicable, have you updated the NOTICE file?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI
results for build issues and submit an update to your PR as soon as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
