aokolnychyi commented on a change in pull request #1784:
URL: https://github.com/apache/iceberg/pull/1784#discussion_r527371845
##########
File path: spark3/src/main/java/org/apache/iceberg/spark/Spark3Util.java
##########
@@ -548,4 +564,97 @@ private static String
sqlString(org.apache.iceberg.expressions.Literal<?> lit) {
}
}
}
+
+ /**
+ * Returns a Metadata Table Dataset if it can be loaded from a Spark V2
Catalog
+ *
+ * Because Spark does not allow more than 1 piece in the namespace for a
Session Catalog table, we circumvent
+ * the entire resolution path for tables and instead look up the table
directly ourselves. This lets us correctly
+ * get metadata tables for the SessionCatalog, if we didn't have to work
around this we could just use spark.table.
+ *
+ * @param spark SparkSession used for looking up catalog references and
tables
+ * @param name The multipart identifier of the base Iceberg table
+ * @param type The type of metadata table to load
+ * @return null if we cannot find the Metadata Table, a Dataset of rows
otherwise
+ */
+ private static Dataset<Row> loadCatalogMetadataTable(SparkSession spark,
String name, MetadataTableType type) {
+ try {
+ CatalogAndIdentifier catalogAndIdentifier = catalogAndIdentifier(spark,
name);
+ if (catalogAndIdentifier.catalog instanceof BaseCatalog) {
+ BaseCatalog catalog = (BaseCatalog) catalogAndIdentifier.catalog;
+ Identifier baseIdent = catalogAndIdentifier.identifier;
+ Identifier metaIdent =
Identifier.of(ArrayUtils.add(baseIdent.namespace(), baseIdent.name()),
type.name());
+ Table metaTable = catalog.loadTable(metaIdent);
+ return Dataset
+ .ofRows(spark, DataSourceV2Relation.create(metaTable,
Some.apply(catalog), Some.apply(metaIdent)));
Review comment:
nit: I'd put the relation into a separate var to keep this on one line.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
