dongjoon-hyun opened a new pull request #24904: [SPARK-28101][DSTREAM][TEST] Fix Flaky Test: `InputStreamsSuit.Modified files are correctly detected` in JDK9+ URL: https://github.com/apache/spark/pull/24904 ## What changes were proposed in this pull request? It seems that https://bugs.openjdk.java.net/browse/JDK-8068730 makes the test suite flaky. <img width="903" alt="error" src="https://user-images.githubusercontent.com/9700541/59727067-017eb780-91e9-11e9-8bb0-ac5f4c1bc44d.png";> As we can see the Jenkins result, this can be reproduced frequently with JDK9+. ``` $ build/sbt "streaming/testOnly *.InputStreamsSuite" [info] - Modified files are correctly detected. *** FAILED *** (134 milliseconds) [info] Set("renamed") did not equal Set() (InputStreamsSuite.scala:312) [info] org.scalatest.exceptions.TestFailedException: ``` The reason is the new file modification time is greater than the clock. And, Spark ignored it. ``` Getting new files for time 1560896662000, ignoring files older than 1560896659679 file:/Users/dongjoon/PRS/SPARK-STEAM-TEST/target/tmp/spark-f34c23f6-9fb4-4ded-87f3-8cdfb57d85a6/streaming/subdir/renamed.txt not selected as mod time 1560896662679 > current time 1560896662000 file:/Users/dongjoon/PRS/SPARK-STEAM-TEST/target/tmp/spark-f34c23f6-9fb4-4ded-87f3-8cdfb57d85a6/streaming/subdir/existing ignored as mod time 1560896657679 <= ignore time 1560896659679 Finding new files took 0 ms New files at time 1560896662000 ms: ``` ## How was this patch tested? Pass the Jenkins and manually test with JDK11.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
![https://user-images.githubusercontent.com/9700541/59727067-017eb780-91e9-11e9-8bb0-ac5f4c1bc44d.png"](https://user-images.githubusercontent.com/9700541/59727067-017eb780-91e9-11e9-8bb0-ac5f4c1bc44d.png"){kind=link}