Bryan, That's what I was looking for. Someone else that had encountered the error in the ListFile.filterAge() unit test. That way I know I'm not crazy. I think I have a pretty good idea of what is causing the unpredictable behavior I'm going to continue verifying today.
Thanks for your help, Ryan On Thu, Mar 14, 2019 at 7:15 PM Bryan Rosander <[email protected]> wrote: > I've seen similar issues building locally on OSx. I tried going back and > building 1.8.0 and saw similar behavior as master. > > Haven't been able to sort it out yet though. > > Thanks, > Bryan > > On Thu, Mar 14, 2019 at 6:41 PM Ryan Withers <[email protected]> > wrote: > > > Hello all, > > > > I'm encountering an error in the TestListFile.java class and in the > > testFilterAge() specifically. This unit test and associated class can be > > found in the Nifi-Standard-Bundle. I've noticed a couple issues have > been > > reported on ListFile before. My issue is most closely related to this > one: > > https://issues.apache.org/jira/browse/NIFI-3831, although it may be an > > issue altogether separate. I have never seen this particular test pass > > before and I've attempted building several versions of nifi now. > > > > I'm doing the nifi build on a new model mac. My OS Version is: > > > > 18.2.0 Darwin Kernel Version 18.2.0: Fri Oct 5 19:41:49 PDT 2018; > > root:xnu-4903.221.2~2/RELEASE_X86_64 x86_64 > > > > I noticed that my root file system is apfs via mount -v */dev/disk1s1 > on / > > (apfs, local, journaled)* > > > > In looking at the ListFiles.testFilterAge() and related methods I found a > > comment above the getTestModifiedTime talking about hfs+ only having > > granularity to one second. For APFS this no longer seems to be the case > > based on the the link tagged below. I've modified this method removing > the > > minus 1 and it doesn't seem to have any affect on the failures I get. > > > > /* > > * HFS+, default for OS X, only has granularity to one second, > > accordingly, we go back in time to establish consistent test cases > > * > > * Provides "now" minus 1 second in millis > > */ > > private static long getTestModifiedTime() { > > final long nowMillis = System.currentTimeMillis(); > > // Subtract a second to avoid possible rounding issues > > final long nowSeconds = TimeUnit.SECONDS.convert(nowMillis, > > TimeUnit.MILLISECONDS) - 1; > > return TimeUnit.MILLISECONDS.convert(nowSeconds, TimeUnit.SECONDS); > > } > > > > https://www.gillware.com/data-recovery-lab/apfs-apple-file-system/ > > "Namely, APFS addresses file checksum, nanosecond time stamps, volume > > snapshots, and other features which were lacking under HFS+." > > > > All the file ranges seem shifted, but I'm surprised that nobody else has > > encountered this same issue. I haven't raised this issue as a story yet > > because no one else has noticed the same problem. > > > > In trying to gain additional understanding I've been modifying the unit > > test for testFilterAge and I found incrementing the boundaries by one > time > > range unit has allowed me to build a passing test case. > > > > // exclude oldest > > runner.setProperty(ListFile.MIN_AGE, age0); > > - runner.setProperty(ListFile.MAX_AGE, age3); > > + runner.setProperty(ListFile.MAX_AGE, age4); > > runNext.apply(true); > > runner.assertAllFlowFilesTransferred(ListFile.REL_SUCCESS); > > final List<MockFlowFile> successFiles2 = > > runner.getFlowFilesForRelationship(ListFile.REL_SUCCESS); > > @@ -362,7 +362,7 @@ public class TestListFile { > > assertEquals(file1.getName(), > > successFiles2.get(1).getAttribute("filename")); > > > > // exclude newest > > - runner.setProperty(ListFile.MIN_AGE, age1); > > + runner.setProperty(ListFile.MIN_AGE, age2); > > runner.setProperty(ListFile.MAX_AGE, age5); > > runNext.apply(true); > > runner.assertAllFlowFilesTransferred(ListFile.REL_SUCCESS); > > @@ -372,8 +372,8 @@ public class TestListFile { > > assertEquals(file2.getName(), > > successFiles3.get(1).getAttribute("filename")); > > > > // exclude oldest and newest > > - runner.setProperty(ListFile.MIN_AGE, age1); > > - runner.setProperty(ListFile.MAX_AGE, age3); > > + runner.setProperty(ListFile.MIN_AGE, age2); > > + runner.setProperty(ListFile.MAX_AGE, age4); > > runNext.apply(true); > > runner.assertAllFlowFilesTransferred(ListFile.REL_SUCCESS); > > final List<MockFlowFile> successFiles4 = > > runner.getFlowFilesForRelationship(ListFile.REL_SUCCESS); > > > > Has anyone else encountered this? Are there special platform specific > > flags I need to pass when running on a mac? It makes me wonder if the > > generated timing is conflicting with the following line in ListFile.java > > > > final long fileAge = System.currentTimeMillis() - > > attributes.lastModifiedTime().toMillis(); > > > > > > That line can be found on line 622 in the ListFile class in the > > createFileFilter method. I'm thinking that utilizing > > System.currentTImeMillis() instead of providing some kind of shim where > the > > current clock time can be reliably mocked is causing some inconsistencies > > here. I'm not 100% confident in this because I would think that others > > would be encountering the same or similar issues. > > > > Any help would be greatly appreciated. I'm also happy to clarify if > > anything seems unclear. > > > > Thanks in advance, > > > > -- > > Ryan Withers > > Senior Software Developer / Analyst > > > > http://www.linkedin.com/in/ryanwithers > > > -- Ryan Withers Senior Software Developer / Analyst http://www.linkedin.com/in/ryanwithers
