longvu-db commented on code in PR #55463:
URL: https://github.com/apache/spark/pull/55463#discussion_r3318295330
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryBaseTable.scala:
##########
@@ -97,6 +97,23 @@ abstract class InMemoryBaseTable(
tableVersion = version.toInt
}
+ /**
+ * Copies version and validated version from another table.
+ *
+ * Test catalogs that decorate tables (e.g.
[[NullColumnIdInMemoryTableCatalog]],
+ * [[NullTableIdAndNullColumnIdInMemoryTableCatalog]]) create new objects
that start
+ * at version 0. Without this call, [[V2TableRefreshUtil]] would see version
0 on
+ * every load and never detect that the table has changed, breaking
stale-table
+ * refresh for incrementally constructed queries (e.g. joining DataFrames
analyzed
+ * at different times).
Review Comment:
Done. Rephrased to focus on the monotonic-version assumption that downstream
consumers rely on, rather than claiming V2TableRefreshUtil compares versions to
skip work.
##########
sql/connect/server/src/test/scala/org/apache/spark/sql/connect/DataSourceV2DataFrameConnectSuite.scala:
##########
@@ -21,28 +21,37 @@ import scala.reflect.ClassTag
import org.apache.spark.SparkConf
import org.apache.spark.sql.{DataFrame, QueryTest, Row, SparkSession}
-import org.apache.spark.sql.connector.{DSv2CacheTableReadTests,
DSv2RepeatedTableAccessTests, DSv2TempViewWithStoredPlanTests}
-import org.apache.spark.sql.connector.catalog.{CachingInMemoryTableCatalog,
InMemoryTableCatalog, TableCatalog}
+import org.apache.spark.sql.connector.{DSv2CacheTableReadTests,
DSv2RepeatedTableAccessTests, DSv2TempViewWithStoredPlanTests,
DSv2IncrementallyConstructedQueryTests}
Review Comment:
Done. Fixed alphabetical ordering.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
