RussellSpitzer commented on a change in pull request #3089:
URL: https://github.com/apache/iceberg/pull/3089#discussion_r704841445
##########
File path: spark/src/main/java/org/apache/iceberg/spark/SparkTableUtil.java
##########
@@ -595,17 +595,25 @@ private static void deleteManifests(FileIO io,
List<ManifestFile> manifests) {
}
// Attempt to use Spark3 Catalog resolution if available on the path
- private static final DynMethods.UnboundMethod LOAD_CATALOG =
DynMethods.builder("loadCatalogMetadataTable")
- .hiddenImpl("org.apache.iceberg.spark.Spark3Util", SparkSession.class,
String.class, MetadataTableType.class)
+ private static final DynMethods.UnboundMethod LOAD_METADATA_TABLE =
DynMethods.builder("loadMetadataTable")
+ .hiddenImpl("org.apache.iceberg.spark.Spark3Util", SparkSession.class,
Table.class, MetadataTableType.class)
.orNoop()
.build();
- public static Dataset<Row> loadCatalogMetadataTable(SparkSession spark,
String tableName, MetadataTableType type) {
- Preconditions.checkArgument(!LOAD_CATALOG.isNoop(), "Cannot find
Spark3Util class but Spark3 is in use");
- return LOAD_CATALOG.asStatic().invoke(spark, tableName, type);
+ public static Dataset<Row> loadCatalogMetadataTable(SparkSession spark,
Table table, MetadataTableType type) {
+ Preconditions.checkArgument(!LOAD_METADATA_TABLE.isNoop(), "Cannot find
Spark3Util class but Spark3 is in use");
+ return LOAD_METADATA_TABLE.asStatic().invoke(spark, table, type);
}
public static Dataset<Row> loadMetadataTable(SparkSession spark, Table
table, MetadataTableType type) {
+ if (spark.version().startsWith("3")) {
Review comment:
We should probably take this class
https://github.com/apache/iceberg/blob/master/spark3/src/main/java/org/apache/iceberg/spark/Spark3VersionUtil.java
and move it to Spark module and extend it to also do 3 vs 2 checking, not
important for now. I can write that up as a low hanging fruit for someone to do
in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
