[
https://issues.apache.org/jira/browse/NIFI-4069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16048829#comment-16048829
]
ASF GitHub Bot commented on NIFI-4069:
--------------------------------------
GitHub user ijokarumawak opened a pull request:
https://github.com/apache/nifi/pull/1915
NIFI-4069: Make ListXXX work with timestamp precision in seconds or m…
…inutes
- Refactored variable names to better represents what those are meant for.
- Added deterministic logic which detects target filesystem timestamp
precision and adjust lag time based on it.
- Changed from using System.nanoTime() to System.currentTimeMillis in test
because Java File API reports timestamp in milliseconds at the best
granularity. Also, System.nanoTime should not be used in mix with epoch
milliseconds because it uses arbitrary origin and measured differently.
Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [x] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number
you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?
- [x] Is your initial contribution a single, squashed commit?
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn
-Pcontrib-check clean install at the root nifi folder?
- [x] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file, including the main
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in
which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ijokarumawak/nifi nifi-4069
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/1915.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1915
----
commit 9a42e60b6c40cad2f324a1729a5aef068dd37913
Author: Koji Kawamura <[email protected]>
Date: 2017-06-14T06:21:01Z
NIFI-4069: Make ListXXX work with timestamp precision in seconds or minutes
- Refactored variable names to better represents what those are meant for.
- Added deterministic logic which detects target filesystem timestamp
precision and adjust lag time based on it.
- Changed from using System.nanoTime() to System.currentTimeMillis in test
because Java File API reports timestamp in milliseconds at the best
granularity. Also, System.nanoTime should not be used in mix with epoch
milliseconds because it uses arbitrary origin and measured differently.
----
> ListXXX processors can miss files those created while the processor is
> listing and filesystem does not provide timestamp milliseconds precision
> -----------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: NIFI-4069
> URL: https://issues.apache.org/jira/browse/NIFI-4069
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
> Affects Versions: 1.0.0
> Reporter: Koji Kawamura
> Assignee: Koji Kawamura
> Attachments: ListFilesWithoutMilliseconds.png
>
>
> For some filesystems such as Mac OS X HFS (Hierarchical File System) or EXT3
> are known that only support timestamp in seconds precision. Also some FTP
> server is reported that it can only provides timestamp precision in minutes.
> This can cause files to NOT be listed as ListXXX processors logic expects
> timestamps in milliseconds.
> Specifically, if generate several files in one second, not all files will be
> listened.
> Steps to reproduce:
> 1. start processor ListFile
> 2. generate 10000 zero size files with following command:
> {code}
> for i in {1..10000}; do touch ./test_$i; done
> {code}
> 3. see processor stats: out 3952 (0 bytes)
> Current AbstractListProcessor logic adopts LISTING_LAG_NANOS (100ms) and
> postponing the files those have the latest timestamp within a listing
> iteration to next iteration, however with those filesystem without
> milliseconds precision, these logics do not work as expected.
> This issue is originally reported at nifi-dev ML.
> http://apache-nifi-developer-list.39713.n7.nabble.com/processors-ListFile-ListSFTP-do-not-store-milliseconds-in-timestamp-td16037.html
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
