rdblue commented on a change in pull request #1920:
URL: https://github.com/apache/iceberg/pull/1920#discussion_r552137788
##########
File path:
mr/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java
##########
@@ -202,12 +208,31 @@ public static PartitionSpec spec(Configuration config) {
*/
@VisibleForTesting
static void put(Configuration config, Table table) {
+ // The Table contains a FileIO and the FileIO serializes the configuration
so we might end up recursively
+ // serializing the objects. To avoid this unset the values for now before
serializing.
+ config.unset(InputFormatConfig.SERIALIZED_TABLE);
+ config.unset(InputFormatConfig.FILE_IO);
+ config.unset(InputFormatConfig.LOCATION_PROVIDER);
+ config.unset(InputFormatConfig.ENCRYPTION_MANAGER);
+ config.unset(InputFormatConfig.TABLE_LOCATION);
+ config.unset(InputFormatConfig.TABLE_SCHEMA);
+ config.unset(InputFormatConfig.PARTITION_SPEC);
+
+ String base64Table = table instanceof Serializable ?
SerializationUtil.serializeToBase64(table) : null;
+ String base64Io = SerializationUtil.serializeToBase64(table.io());
+ String base64LocationProvider =
SerializationUtil.serializeToBase64(table.locationProvider());
+ String base64EncryptionManager =
SerializationUtil.serializeToBase64(table.encryption());
+
+ if (base64Table != null) {
+ config.set(InputFormatConfig.SERIALIZED_TABLE, base64Table);
+ }
+
+ config.set(InputFormatConfig.FILE_IO, base64Io);
+ config.set(InputFormatConfig.LOCATION_PROVIDER, base64LocationProvider);
+ config.set(InputFormatConfig.ENCRYPTION_MANAGER, base64EncryptionManager);
Review comment:
Shouldn't these be serialized only if the table is not?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
