pvary commented on a change in pull request #1920:
URL: https://github.com/apache/iceberg/pull/1920#discussion_r552396207
##########
File path:
mr/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java
##########
@@ -202,12 +208,31 @@ public static PartitionSpec spec(Configuration config) {
*/
@VisibleForTesting
static void put(Configuration config, Table table) {
+ // The Table contains a FileIO and the FileIO serializes the configuration
so we might end up recursively
+ // serializing the objects. To avoid this unset the values for now before
serializing.
+ config.unset(InputFormatConfig.SERIALIZED_TABLE);
+ config.unset(InputFormatConfig.FILE_IO);
+ config.unset(InputFormatConfig.LOCATION_PROVIDER);
+ config.unset(InputFormatConfig.ENCRYPTION_MANAGER);
+ config.unset(InputFormatConfig.TABLE_LOCATION);
+ config.unset(InputFormatConfig.TABLE_SCHEMA);
+ config.unset(InputFormatConfig.PARTITION_SPEC);
+
+ String base64Table = table instanceof Serializable ?
SerializationUtil.serializeToBase64(table) : null;
+ String base64Io = SerializationUtil.serializeToBase64(table.io());
+ String base64LocationProvider =
SerializationUtil.serializeToBase64(table.locationProvider());
+ String base64EncryptionManager =
SerializationUtil.serializeToBase64(table.encryption());
+
+ if (base64Table != null) {
+ config.set(InputFormatConfig.SERIALIZED_TABLE, base64Table);
+ }
+
+ config.set(InputFormatConfig.FILE_IO, base64Io);
+ config.set(InputFormatConfig.LOCATION_PROVIDER, base64LocationProvider);
+ config.set(InputFormatConfig.ENCRYPTION_MANAGER, base64EncryptionManager);
Review comment:
Let's see what we can do about the serialization of the other table
types. I would love to fully get rid of these serialized objects. Maybe even on
the expense of not supporting reading some types of the tables through mr.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
