linhongliu-db commented on code in PR #59890:
URL: https://github.com/apache/spark/pull/59890#discussion_r4449583513
##########
sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:
##########
@@ -855,6 +856,115 @@ class CachedTableSuite extends SharedSparkSession
}
}
+ private def appendParquetWithoutRefresh(path: String, value: Long): Unit = {
+ withTempDir { staging =>
+ val output = new File(staging, "data")
+ spark.range(value, value + 1).write.parquet(output.getCanonicalPath)
+ // Move files externally so Spark does not refresh the destination
before refreshByPath.
+ output.listFiles().filter(_.getName.endsWith(".parquet")).foreach { file
=>
+ Files.move(file.toPath, new File(path, file.getName).toPath)
+ }
+ }
+ }
+
+ for (relationCount <- Seq(1, 2, 3)) {
+ test(s"refreshByPath refreshes all $relationCount file indexes in one
cached plan") {
+ withTempDir { dir =>
+ val paths = (0 until relationCount).map(i => new File(dir,
s"part$i").getCanonicalPath)
+ paths.foreach(path => spark.range(1).write.parquet(path))
+ val cached =
paths.map(spark.read.parquet(_)).reduce(_.union(_)).cache()
Review Comment:
Addressed in 11e9f37, included in the current 21319a6 head. The one-, two-,
and three-read regressions run with both V1 and V2 file-source configurations
and assert the analyzed relation types, including FileTable for V2. The
subsequent review confirmed the V2 coverage. CI passes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
