RussellSpitzer commented on a change in pull request #1784:
URL: https://github.com/apache/iceberg/pull/1784#discussion_r526411140
##########
File path: spark/src/main/java/org/apache/iceberg/actions/BaseSparkAction.java
##########
@@ -137,7 +138,19 @@
}
// Try catalog based name based resolution
try {
- return spark.table(tableName + "." + type);
+ if (tableName.startsWith("spark_catalog")) {
+ // Do to the design of Spark, we cannot pass multi element namespaces
to the session catalog
+ // We also don't know whether the Catalog is Hive or Hadoop Based, so
we will try to load it
+ // in the hive manner first, then fall back and try the location if we
have completely run out of options
+ // TODO remove this when we have Spark workaround for multipart
identifiers in SparkSessionCatalog
+ try {
+ return
noCatalogReader.load(tableName.replaceFirst("spark_catalog\\.", "") + "." +
type);
Review comment:
If we could extend out into BaseSpark3Actions this would be a good place
to diverge, since I think we could load the metadata table directly from the
catalog, and bypass the name resolution code. But we can't do that without
redoing the hierarchy or having the Rewrite Actions use reflection to determine
their "loadMetadataTable" Method
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
