rdblue commented on a change in pull request #3752:
URL: https://github.com/apache/iceberg/pull/3752#discussion_r771545816
##########
File path: core/src/main/java/org/apache/iceberg/SerializableTable.java
##########
@@ -107,7 +108,13 @@ private String metadataFileLocation(Table table) {
private FileIO fileIO(Table table) {
if (table.io() instanceof HadoopConfigurable) {
- ((HadoopConfigurable) table.io()).serializeConfWith(conf -> new
SerializableConfiguration(conf)::get);
+ HadoopConfigurable configurableIo = (HadoopConfigurable) table.io();
+ if
(configurableIo.getConf().getBoolean(ConfigProperties.CONFIG_SERIALIZATION_DISABLED,
false)) {
+ // do not serialize the Configuration object - configs should be set
on the deserializer-side
+ configurableIo.serializeConfWith(conf -> new
SkipSerializableConfiguration(conf)::get);
+ } else {
+ configurableIo.serializeConfWith(conf -> new
SerializableConfiguration(conf)::get);
+ }
Review comment:
Is it possible to do this in Hive rather than here? Spark uses its own
`SerializableConfiguration` that is compatible with Kryo and calls
`serializeConfWith` to customize this. I think ideally that is how Hive would
work as well. That avoids needing to have a general Hadoop config property.
Instead, we can have one just for Hive where it's okay to set things in the
Hadoop conf.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
