[
https://issues.apache.org/jira/browse/NIFI-6137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pierre Villard resolved NIFI-6137.
----------------------------------
Resolution: Feedback Received
Apache NiFi 1.x is no longer maintained and no new release is planned on the
1.x release line. Marking as resolved as part of a cleanup operation. Please
open a new one with an updated description if this is still relevant for NiFi
2.x.
> The Test for List File ListFile.filterAge fails
> ------------------------------------------------
>
> Key: NIFI-6137
> URL: https://issues.apache.org/jira/browse/NIFI-6137
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
> Affects Versions: 1.9.0
> Environment: My OS Version is:
>
> 18.2.0 Darwin Kernel Version 18.2.0: Fri Oct 5 19:41:49 PDT 2018;
> root:xnu-4903.221.2~2/RELEASE_X86_64 x86_64
> I noticed that my root file system is apfs via mount -v /dev/disk1s1 on /
> (apfs, local, journaled)
> Reporter: Ryan Withers
> Priority: Minor
> Labels: test
>
> Couple notes this is a unit test that fails consistently for me running on my
> local machine. I know there is so much variability in build machine and
> environment that it might not make sense to log an issue like this. I've
> asked the mailing list and had one person come back and verify that they have
> also encountered this similar issue. I'm logging this so that I can work on
> it after I'm finished with one that I have in progress. If I've made an
> error in logging the issue or it isn't consistent with community norms I'm
> happy to fix it. Please see below for additional information.
> I'm encountering an error in the TestListFile.java class and in the
> testFilterAge() specifically. This unit test and associated class can be
> found in the Nifi-Standard-Bundle. I've noticed a couple issues have been
> reported on ListFile before. My issue is most closely related to this one:
> https://issues.apache.org/jira/browse/NIFI-3831. After reading that one and
> considering the problem I'm thinking the problem I'm having is separate, but
> would be happy getting guidance on that.
> I have never seen this particular test pass before and I've attempted
> building 3 different versions of nifi now.
> In looking at the ListFiles.testFilterAge() and related methods I found a
> comment above the getTestModifiedTime talking about hfs+ only having
> granularity to one second. For APFS this no longer seems to be the case
> based on the the link tagged below. I've modified this method removing the
> minus 1 and it doesn't seem to have any affect on the failures I get. I have
> included a validation that I'm on APFS in the environment window above. I
> need to get closer to what is actually happening here.
> /*
> * HFS+, default for OS X, only has granularity to one second, accordingly,
> we go back in time to establish consistent test cases
> *
> * Provides "now" minus 1 second in millis
> */
> private static long getTestModifiedTime() {
> final long nowMillis = System.currentTimeMillis();
> // Subtract a second to avoid possible rounding issues
> final long nowSeconds = TimeUnit.SECONDS.convert(nowMillis,
> TimeUnit.MILLISECONDS) - 1;
> return TimeUnit.MILLISECONDS.convert(nowSeconds, TimeUnit.SECONDS);
> }
> https://www.gillware.com/data-recovery-lab/apfs-apple-file-system/
> "Namely, APFS addresses file checksum, nanosecond time stamps, volume
> snapshots, and other features which were lacking under HFS+."
> All the file ranges seem shifted, but I'm surprised that nobody else has
> encountered this same issue. I haven't raised this issue as a story yet
> because no one else has noticed the same problem. I would think it would be
> a lot more common.
> In trying to gain additional understanding I've been modifying the unit test
> for TestListFile.testFilterAge() and I found incrementing the boundaries by
> one time range unit has allowed me to build a passing test case.
> // exclude oldest
> runner.setProperty(ListFile.MIN_AGE, age0);
> - runner.setProperty(ListFile.MAX_AGE, age3);
> + runner.setProperty(ListFile.MAX_AGE, age4);
> runNext.apply(true);
> runner.assertAllFlowFilesTransferred(ListFile.REL_SUCCESS);
> final List<MockFlowFile> successFiles2 =
> runner.getFlowFilesForRelationship(ListFile.REL_SUCCESS);
> @@ -362,7 +362,7 @@ public class TestListFile {
> assertEquals(file1.getName(),
> successFiles2.get(1).getAttribute("filename"));
> // exclude newest
> - runner.setProperty(ListFile.MIN_AGE, age1);
> + runner.setProperty(ListFile.MIN_AGE, age2);
> runner.setProperty(ListFile.MAX_AGE, age5);
> runNext.apply(true);
> runner.assertAllFlowFilesTransferred(ListFile.REL_SUCCESS);
> @@ -372,8 +372,8 @@ public class TestListFile {
> assertEquals(file2.getName(),
> successFiles3.get(1).getAttribute("filename"));
> // exclude oldest and newest
> - runner.setProperty(ListFile.MIN_AGE, age1);
> - runner.setProperty(ListFile.MAX_AGE, age3);
> + runner.setProperty(ListFile.MIN_AGE, age2);
> + runner.setProperty(ListFile.MAX_AGE, age4);
> runNext.apply(true);
> runner.assertAllFlowFilesTransferred(ListFile.REL_SUCCESS);
> final List<MockFlowFile> successFiles4 =
> runner.getFlowFilesForRelationship(ListFile.REL_SUCCESS);
> It makes me wonder if the generated timing is conflicting with the following
> line in ListFile.java
> final long fileAge = System.currentTimeMillis() -
> attributes.lastModifiedTime().toMillis();
> That line can be found on or around line 622 in the ListFile class in the
> createFileFilter method. I'm thinking that utilizing
> System.currentTimeMillis() instead of providing some kind of shim where the
> current clock time can be reliably mocked is causing some inconsistencies
> here. I'm not 100% confident in this because I would think that others would
> be encountering the same or similar issues, as I discussed above.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
