calvin-pietersen commented on issue #5414:
URL: https://github.com/apache/iceberg/issues/5414#issuecomment-1226689287
Also having issues with kryo serialization of `ImmutableMap` for `S3FileIO`.
Was trying to register a customer serializer for
`org.apache.iceberg.relocated.com.google.common.collect.SingletonImmutableBiMap`
as suggested above, however I was still getting kryo exceptions. Turns out,
there are other concrete implementations of ImmutableMap that could potentially
fail to serialize. So I just serialized all of them and it worked.
`
import com.esotericsoftware.kryo.Kryo;
import com.esotericsoftware.kryo.Serializer;
import com.esotericsoftware.kryo.io.Input;
import com.esotericsoftware.kryo.io.Output;
import com.google.common.collect.Maps;
import java.util.HashMap;
import java.util.Map;
import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
public class ImmutableRelocatedMapSerializer extends
Serializer<ImmutableMap> {
private static final boolean DOES_NOT_ACCEPT_NULL = true;
private static final boolean IMMUTABLE = true;
public ImmutableRelocatedMapSerializer() {
super(DOES_NOT_ACCEPT_NULL, IMMUTABLE);
}
@Override
public void write(Kryo kryo, Output output, ImmutableMap object) {
kryo.writeObject(output, Maps.newHashMap(object));
}
@Override
public ImmutableMap read(Kryo kryo, Input input, Class<ImmutableMap>
type) {
Map map = kryo.readObject(input, HashMap.class);
return ImmutableMap.copyOf(map);
}
public void register(Kryo kryo) {
try {
kryo.register(Class.forName("org.apache.iceberg.relocated.com.google.common.collect.ImmutableBiMap"),
this);
kryo.register(Class.forName("org.apache.iceberg.relocated.com.google.common.collect.RegularImmutableBiMap"),
this);
kryo.register(Class.forName("org.apache.iceberg.relocated.com.google.common.collect.JdkBackedImmutableBiMap"),
this);
kryo.register(Class.forName("org.apache.iceberg.relocated.com.google.common.collect.SingletonImmutableBiMap"),
this);
kryo.register(Class.forName("org.apache.iceberg.relocated.com.google.common.collect.RegularImmutableMap"),
this);
kryo.register(Class.forName("org.apache.iceberg.relocated.com.google.common.collect.ImmutableSortedMapFauxverideShim"),
this);
kryo.register(Class.forName("org.apache.iceberg.relocated.com.google.common.collect.ImmutableSortedMap"),
this);
kryo.register(Class.forName("org.apache.iceberg.relocated.com.google.common.collect.ImmutableEnumMap"),
this);
kryo.register(Class.forName("org.apache.iceberg.relocated.com.google.common.collect.JdkBackedImmutableMap"),
this);
kryo.register(Class.forName("org.apache.iceberg.relocated.com.google.common.collect.ImmutableBiMapFauxverideShim"),
this);
kryo.register(Class.forName("org.apache.iceberg.relocated.com.google.common.collect.ImmutableEnumMap"),
this);
} catch (Exception ex) {
// do nothing
}
}
}
`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
