jessiedanwang commented on issue #5414:
URL: https://github.com/apache/iceberg/issues/5414#issuecomment-1211295322
thanks for getting back. I have tried both spark 3.1 and spark 3.2 with
SingletonImmutableBiMap as follows, still same error, i guess we will have to
work with JavaSerializer for now
Add spark config
.config("spark.kryo.registrator", "xxx.CustomKryoRegistrator")
import com.esotericsoftware.kryo.Kryo;
import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
import org.apache.spark.serializer.KryoRegistrator;
public class CustomKryoRegistrator implements KryoRegistrator {
@Override
public void registerClasses(Kryo kryo){
final ImmutableRelocatedMapSerializer serializer = new
ImmutableRelocatedMapSerializer();
//kryo.register(java.util.HashMap.class);
try {
kryo.register(Class.forName("org.apache.iceberg.relocated.com.google.common.collect.SingletonImmutableBiMap"),
serializer);
} catch (Exception e) {
// do nothing
}
}
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
