pan3793 commented on code in PR #54411:
URL: https://github.com/apache/spark/pull/54411#discussion_r2836069358
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryTable.scala:
##########
@@ -261,3 +263,71 @@ object InMemoryTable {
}
}
}
+
+/**
+ * A metadata table that returns snapshot (commit) information for a parent
table.
+ * Simulates data source metadata tables like Iceberg's db.table.snapshots.
+ */
+class InMemorySnapshotsTable(parentTable: InMemoryTable) extends Table with
SupportsRead {
+ override def name(): String = parentTable.name + ".snapshots"
+
+ override def schema(): StructType = StructType(Seq(
+ StructField("committed_at", LongType, nullable = false),
+ StructField("snapshot_id", LongType, nullable = false)
+ ))
+
+ override def capabilities(): util.Set[TableCapability] = {
Review Comment:
> By the way, what do you mean read only? (I dont think there's such DSV2
concept yet?)
Just ignore it, I might be getting bogged down in the implementation details
of Iceberg. It's an invalid question for a general multi-part namespace table
name,
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryTable.scala:
##########
@@ -261,3 +263,71 @@ object InMemoryTable {
}
}
}
+
+/**
+ * A metadata table that returns snapshot (commit) information for a parent
table.
+ * Simulates data source metadata tables like Iceberg's db.table.snapshots.
+ */
+class InMemorySnapshotsTable(parentTable: InMemoryTable) extends Table with
SupportsRead {
+ override def name(): String = parentTable.name + ".snapshots"
+
+ override def schema(): StructType = StructType(Seq(
+ StructField("committed_at", LongType, nullable = false),
+ StructField("snapshot_id", LongType, nullable = false)
+ ))
+
+ override def capabilities(): util.Set[TableCapability] = {
Review Comment:
a couple of questions that mgiht go beyond this topic:
- seems there is no "metadata table" concept in Spark so far, do we need a
new `TableCapability` to signal that a table supports that?
- ~~how should the permission check be done for metadata tables? suppose
they're read-only, does it mean users have read permission on metadata tables
as long as they are granted to read the base table?~~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
