I've seen similar issues building locally on OSx. I tried going back and building 1.8.0 and saw similar behavior as master.
Haven't been able to sort it out yet though. Thanks, Bryan On Thu, Mar 14, 2019 at 6:41 PM Ryan Withers <[email protected]> wrote: > Hello all, > > I'm encountering an error in the TestListFile.java class and in the > testFilterAge() specifically. This unit test and associated class can be > found in the Nifi-Standard-Bundle. I've noticed a couple issues have been > reported on ListFile before. My issue is most closely related to this one: > https://issues.apache.org/jira/browse/NIFI-3831, although it may be an > issue altogether separate. I have never seen this particular test pass > before and I've attempted building several versions of nifi now. > > I'm doing the nifi build on a new model mac. My OS Version is: > > 18.2.0 Darwin Kernel Version 18.2.0: Fri Oct 5 19:41:49 PDT 2018; > root:xnu-4903.221.2~2/RELEASE_X86_64 x86_64 > > I noticed that my root file system is apfs via mount -v */dev/disk1s1 on / > (apfs, local, journaled)* > > In looking at the ListFiles.testFilterAge() and related methods I found a > comment above the getTestModifiedTime talking about hfs+ only having > granularity to one second. For APFS this no longer seems to be the case > based on the the link tagged below. I've modified this method removing the > minus 1 and it doesn't seem to have any affect on the failures I get. > > /* > * HFS+, default for OS X, only has granularity to one second, > accordingly, we go back in time to establish consistent test cases > * > * Provides "now" minus 1 second in millis > */ > private static long getTestModifiedTime() { > final long nowMillis = System.currentTimeMillis(); > // Subtract a second to avoid possible rounding issues > final long nowSeconds = TimeUnit.SECONDS.convert(nowMillis, > TimeUnit.MILLISECONDS) - 1; > return TimeUnit.MILLISECONDS.convert(nowSeconds, TimeUnit.SECONDS); > } > > https://www.gillware.com/data-recovery-lab/apfs-apple-file-system/ > "Namely, APFS addresses file checksum, nanosecond time stamps, volume > snapshots, and other features which were lacking under HFS+." > > All the file ranges seem shifted, but I'm surprised that nobody else has > encountered this same issue. I haven't raised this issue as a story yet > because no one else has noticed the same problem. > > In trying to gain additional understanding I've been modifying the unit > test for testFilterAge and I found incrementing the boundaries by one time > range unit has allowed me to build a passing test case. > > // exclude oldest > runner.setProperty(ListFile.MIN_AGE, age0); > - runner.setProperty(ListFile.MAX_AGE, age3); > + runner.setProperty(ListFile.MAX_AGE, age4); > runNext.apply(true); > runner.assertAllFlowFilesTransferred(ListFile.REL_SUCCESS); > final List<MockFlowFile> successFiles2 = > runner.getFlowFilesForRelationship(ListFile.REL_SUCCESS); > @@ -362,7 +362,7 @@ public class TestListFile { > assertEquals(file1.getName(), > successFiles2.get(1).getAttribute("filename")); > > // exclude newest > - runner.setProperty(ListFile.MIN_AGE, age1); > + runner.setProperty(ListFile.MIN_AGE, age2); > runner.setProperty(ListFile.MAX_AGE, age5); > runNext.apply(true); > runner.assertAllFlowFilesTransferred(ListFile.REL_SUCCESS); > @@ -372,8 +372,8 @@ public class TestListFile { > assertEquals(file2.getName(), > successFiles3.get(1).getAttribute("filename")); > > // exclude oldest and newest > - runner.setProperty(ListFile.MIN_AGE, age1); > - runner.setProperty(ListFile.MAX_AGE, age3); > + runner.setProperty(ListFile.MIN_AGE, age2); > + runner.setProperty(ListFile.MAX_AGE, age4); > runNext.apply(true); > runner.assertAllFlowFilesTransferred(ListFile.REL_SUCCESS); > final List<MockFlowFile> successFiles4 = > runner.getFlowFilesForRelationship(ListFile.REL_SUCCESS); > > Has anyone else encountered this? Are there special platform specific > flags I need to pass when running on a mac? It makes me wonder if the > generated timing is conflicting with the following line in ListFile.java > > final long fileAge = System.currentTimeMillis() - > attributes.lastModifiedTime().toMillis(); > > > That line can be found on line 622 in the ListFile class in the > createFileFilter method. I'm thinking that utilizing > System.currentTImeMillis() instead of providing some kind of shim where the > current clock time can be reliably mocked is causing some inconsistencies > here. I'm not 100% confident in this because I would think that others > would be encountering the same or similar issues. > > Any help would be greatly appreciated. I'm also happy to clarify if > anything seems unclear. > > Thanks in advance, > > -- > Ryan Withers > Senior Software Developer / Analyst > > http://www.linkedin.com/in/ryanwithers >
