RussellSpitzer commented on a change in pull request #1784:
URL: https://github.com/apache/iceberg/pull/1784#discussion_r527185680
##########
File path: spark3/src/main/java/org/apache/iceberg/spark/Spark3Util.java
##########
@@ -548,4 +565,83 @@ private static String
sqlString(org.apache.iceberg.expressions.Literal<?> lit) {
}
}
}
+
+ /*
+ * Because Spark does not allow more than 1 piece in the namespace for a
Session Catalog table, we circumvent
+ * the entire resolution path for tables and instead look up the table
directly ourselves. This lets us correctly
+ * get metadata tables for the SessionCatalog, if we didn't have to work
around this we could just use spark.table.
+ */
+ private static Dataset<Row> loadCatalogMetadataTable(SparkSession spark,
String name, MetadataTableType type)
+ throws CatalogNotFoundException, ParseException, NoSuchTableException {
+
+ CatalogAndIdentifier catalogAndIdentifier = catalogAndIdentifier(spark,
name);
+ if (catalogAndIdentifier.catalog instanceof BaseCatalog) {
+ BaseCatalog catalog = (BaseCatalog) catalogAndIdentifier.catalog;
+ Identifier baseIdent = catalogAndIdentifier.identifier;
+ Identifier metaIdent =
Identifier.of(ArrayUtils.add(baseIdent.namespace(), baseIdent.name()),
type.name());
+ Table metaTable = catalog.loadTable(metaIdent);
+ return Dataset.ofRows(spark, DataSourceV2Relation.create(metaTable,
Some.apply(catalog), Some.apply(metaIdent)));
Review comment:
This is how Spark would have made the relation from our metadata table
if it didn't think multiple pieces in the Namespace was a dealbreaker.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
