Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/14731#discussion_r97363419 --- Diff: streaming/src/test/scala/org/apache/spark/streaming/InputStreamsSuite.scala --- @@ -235,18 +236,97 @@ class InputStreamsSuite extends TestSuiteBase with BeforeAndAfter { } // Over time, create files in the temp directory 1 val input1 = Seq(1, 2, 3, 4, 5) - input1.foreach(i => createFileAndAdvenceTime(i, testSubDir1)) + input1.foreach(i => createFileAndAdvanceTime(i, testSubDir1)) // Over time, create files in the temp directory 1 val input2 = Seq(6, 7, 8, 9, 10) - input2.foreach(i => createFileAndAdvenceTime(i, testSubDir2)) + input2.foreach(i => createFileAndAdvanceTime(i, testSubDir2)) // Verify that all the files have been read val expectedOutput = (input1 ++ input2).map(_.toString).toSet assert(outputQueue.asScala.flatten.toSet === expectedOutput) } } finally { - if (testDir != null) Utils.deleteRecursively(testDir) + if (testDir != null) { + Utils.deleteRecursively(testDir) + } + } + } + + /** + * Tests that renamed directories are included in new batches -but that only files created + * within the batch window are included. + * Uses the Hadoop APIs to verify consistent behavior with the operations used internally. + */ + test("renamed directories are scanned") { + val testDir = Utils.createTempDir() --- End diff -- That's in `SQLTestUtils`, and not in the streaming suite can pickup without adding spark-sql:test-jar to the test cp. Or: copy & paste in streaming (`org.apache.spark.streaming.TestSuiteBase`), copy & paste somwhere into spark-core:test-jar, which would be a bigger change. How about I add it to the spark streaming testbase and then move the other tests in InputStreamSuite to it, either in this patch or a successor? That way: cleaner tests
--- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org