Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/14731#discussion_r97363419
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/InputStreamsSuite.scala ---
@@ -235,18 +236,97 @@ class InputStreamsSuite extends TestSuiteBase with
BeforeAndAfter {
}
// Over time, create files in the temp directory 1
val input1 = Seq(1, 2, 3, 4, 5)
- input1.foreach(i => createFileAndAdvenceTime(i, testSubDir1))
+ input1.foreach(i => createFileAndAdvanceTime(i, testSubDir1))
// Over time, create files in the temp directory 1
val input2 = Seq(6, 7, 8, 9, 10)
- input2.foreach(i => createFileAndAdvenceTime(i, testSubDir2))
+ input2.foreach(i => createFileAndAdvanceTime(i, testSubDir2))
// Verify that all the files have been read
val expectedOutput = (input1 ++ input2).map(_.toString).toSet
assert(outputQueue.asScala.flatten.toSet === expectedOutput)
}
} finally {
- if (testDir != null) Utils.deleteRecursively(testDir)
+ if (testDir != null) {
+ Utils.deleteRecursively(testDir)
+ }
+ }
+ }
+
+ /**
+ * Tests that renamed directories are included in new batches -but that
only files created
+ * within the batch window are included.
+ * Uses the Hadoop APIs to verify consistent behavior with the
operations used internally.
+ */
+ test("renamed directories are scanned") {
+ val testDir = Utils.createTempDir()
--- End diff --
That's in `SQLTestUtils`, and not in the streaming suite can pickup without
adding spark-sql:test-jar to the test cp. Or: copy & paste in streaming
(`org.apache.spark.streaming.TestSuiteBase`), copy & paste somwhere into
spark-core:test-jar, which would be a bigger change.
How about I add it to the spark streaming testbase and then move the other
tests in InputStreamSuite to it, either in this patch or a successor? That way:
cleaner tests
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
