Ryan Withers created NIFI-6137:
----------------------------------
Summary: The Test for List File ListFile.filterAge fails
Key: NIFI-6137
URL: https://issues.apache.org/jira/browse/NIFI-6137
Project: Apache NiFi
Issue Type: Bug
Components: Extensions
Affects Versions: 1.9.0
Environment: My OS Version is:
18.2.0 Darwin Kernel Version 18.2.0: Fri Oct 5 19:41:49 PDT 2018;
root:xnu-4903.221.2~2/RELEASE_X86_64 x86_64
I noticed that my root file system is apfs via mount -v /dev/disk1s1 on /
(apfs, local, journaled)
Reporter: Ryan Withers
Fix For: 1.10.0
Couple notes this is a unit test that fails consistently for me running on my
local machine. I know there is so much variability in build machine and
environment that it might not make sense to log an issue like this. I've asked
the mailing list and had one person come back and verify that they have also
encountered this similar issue. I'm logging this so that I can work on it
after I'm finished with one that I have in progress. If I've made an error in
logging the issue or it isn't consistent with community norms I'm happy to fix
it. Please see below for additional information.
I'm encountering an error in the TestListFile.java class and in the
testFilterAge() specifically. This unit test and associated class can be found
in the Nifi-Standard-Bundle. I've noticed a couple issues have been reported
on ListFile before. My issue is most closely related to this one:
https://issues.apache.org/jira/browse/NIFI-3831. After reading that one and
considering the problem I'm thinking the problem I'm having is separate, but
would be happy getting guidance on that.
I have never seen this particular test pass before and I've attempted building
3 different versions of nifi now.
In looking at the ListFiles.testFilterAge() and related methods I found a
comment above the getTestModifiedTime talking about hfs+ only having
granularity to one second. For APFS this no longer seems to be the case based
on the the link tagged below. I've modified this method removing the minus 1
and it doesn't seem to have any affect on the failures I get. I have included
a validation that I'm on APFS in the environment window above. I need to get
closer to what is actually happening here.
/*
* HFS+, default for OS X, only has granularity to one second, accordingly, we
go back in time to establish consistent test cases
*
* Provides "now" minus 1 second in millis
*/
private static long getTestModifiedTime() {
final long nowMillis = System.currentTimeMillis();
// Subtract a second to avoid possible rounding issues
final long nowSeconds = TimeUnit.SECONDS.convert(nowMillis,
TimeUnit.MILLISECONDS) - 1;
return TimeUnit.MILLISECONDS.convert(nowSeconds, TimeUnit.SECONDS);
}
https://www.gillware.com/data-recovery-lab/apfs-apple-file-system/
"Namely, APFS addresses file checksum, nanosecond time stamps, volume
snapshots, and other features which were lacking under HFS+."
All the file ranges seem shifted, but I'm surprised that nobody else has
encountered this same issue. I haven't raised this issue as a story yet
because no one else has noticed the same problem. I would think it would be a
lot more common.
In trying to gain additional understanding I've been modifying the unit test
for TestListFile.testFilterAge() and I found incrementing the boundaries by one
time range unit has allowed me to build a passing test case.
// exclude oldest
runner.setProperty(ListFile.MIN_AGE, age0);
- runner.setProperty(ListFile.MAX_AGE, age3);
+ runner.setProperty(ListFile.MAX_AGE, age4);
runNext.apply(true);
runner.assertAllFlowFilesTransferred(ListFile.REL_SUCCESS);
final List<MockFlowFile> successFiles2 =
runner.getFlowFilesForRelationship(ListFile.REL_SUCCESS);
@@ -362,7 +362,7 @@ public class TestListFile {
assertEquals(file1.getName(),
successFiles2.get(1).getAttribute("filename"));
// exclude newest
- runner.setProperty(ListFile.MIN_AGE, age1);
+ runner.setProperty(ListFile.MIN_AGE, age2);
runner.setProperty(ListFile.MAX_AGE, age5);
runNext.apply(true);
runner.assertAllFlowFilesTransferred(ListFile.REL_SUCCESS);
@@ -372,8 +372,8 @@ public class TestListFile {
assertEquals(file2.getName(),
successFiles3.get(1).getAttribute("filename"));
// exclude oldest and newest
- runner.setProperty(ListFile.MIN_AGE, age1);
- runner.setProperty(ListFile.MAX_AGE, age3);
+ runner.setProperty(ListFile.MIN_AGE, age2);
+ runner.setProperty(ListFile.MAX_AGE, age4);
runNext.apply(true);
runner.assertAllFlowFilesTransferred(ListFile.REL_SUCCESS);
final List<MockFlowFile> successFiles4 =
runner.getFlowFilesForRelationship(ListFile.REL_SUCCESS);
It makes me wonder if the generated timing is conflicting with the following
line in ListFile.java
final long fileAge = System.currentTimeMillis() -
attributes.lastModifiedTime().toMillis();
That line can be found on or around line 622 in the ListFile class in the
createFileFilter method. I'm thinking that utilizing
System.currentTimeMillis() instead of providing some kind of shim where the
current clock time can be reliably mocked is causing some inconsistencies here.
I'm not 100% confident in this because I would think that others would be
encountering the same or similar issues, as I discussed above.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
