On Thu, Nov 7, 2013 at 4:13 PM, Chao Shi <[email protected]> wrote: > I think this failure may depend on the execution order of tests, because: > > 1) I can reproduce this failure on some source tree, but cannot on another > 2) I can only reproduce it with "mvn install", but "mvn test > -Dtest=org.apache.crunch.CheckpointIT" cannot > 3) I was trying to add some printf in the case, after saving the file (I > modified that CheckpointIT.java), I cannot reproduce it now. > > Does anyone have some clue?
I have a feeling that this isn't due to order of execution. If it is down to that, then as far as I can see it would mean that there is something wrong with the TemporaryPath class, as that seems to be the only thing that is shared between tests in that class. I'm actually wondering if this could be down to a timing issue in the checkpointing code. The FileTargetImpl class checks if the lastModified timestamp of the checkpointed file is greater than the lastModified timestamp on the input source. Based on a check I just did locally here on Mac OS X and on a Linux machine with ext4, the time granularity of the lastModified timestamp on files is only second-based, not millisecond-based. I'm thinking that this could be an issue of the checkpointed file being created less than a second after the startup of the test and creation of the source file (although that seems really quick), and so the checkpointing wouldn't think that the source file is older than the checkpointed file, so it would run the full pipeline again (therefore resulting in two execution stages instead of one. Does that sound plausible? If so, the only thing I can see that would keep the code correct and prevent the test from failing would be to add a Thread.sleep() into the test, but I hate having stuff like that in tests. - Gabriel
