rdblue commented on a change in pull request #1784:
URL: https://github.com/apache/iceberg/pull/1784#discussion_r526505018
##########
File path: spark/src/main/java/org/apache/iceberg/actions/BaseSparkAction.java
##########
@@ -128,16 +129,35 @@
return manifestDF.union(otherMetadataFileDF).union(manifestListDF);
}
+ private static Dataset<Row> loadMetadataTableFromCatalog(SparkSession spark,
String tableName, String tableLocation,
+ MetadataTableType
type) {
+ DataFrameReader dataFrameReader = spark.read().format("iceberg");
+ if (tableName.startsWith("spark_catalog")) {
+ // Do to the design of Spark, we cannot pass multi-element namespaces to
the session catalog.
+ // We also don't know whether the Catalog is Hive or Hadoop Based so we
can't just load one way or the other.
+ // Instead we will try to load the metadata table in the hive manner
first, then fall back and try the
+ // hadoop location method if that fails
+ // TODO remove this when we have Spark workaround for multipart
identifiers in SparkSessionCatalog
Review comment:
I think this is a bug in Spark. There isn't a work-around that I know of.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
