singhpk234 commented on code in PR #5437:
URL: https://github.com/apache/iceberg/pull/5437#discussion_r943062912
##########
spark/v3.3/spark/src/test/java/org/apache/iceberg/TestFileIOSerialization.java:
##########
@@ -101,6 +104,60 @@ public void testHadoopFileIOJavaSerialization() throws
IOException, ClassNotFoun
Assert.assertEquals("Conf values must be present", "v2",
actualConf.get("k2"));
}
+ @Test
+ public void testResolvingFileIOKryoSerialization() throws IOException {
Review Comment:
This was a pre-exisitng test class `TestFileIOSerialization` checking the
ser/de for HadoopFileIO, hence added other fileIO's serialization tests also to
the same class.
> the module that defines each FileIO
I thing what we can do here is create the kryoHelper (which doesn't take
dependency on spark) in core and since integrations (i.e aws etc) take a
dependency on it, it would be available to all FileIO.
The only concern I had here was that these test can become outdated when
spark uses diff version of twitter-chill or uses diff lib like
[margo](https://github.com/magro/kryo-serializers).
##########
build.gradle:
##########
@@ -379,6 +379,7 @@ project(':iceberg-aws') {
testImplementation 'software.amazon.awssdk:iam'
testImplementation 'software.amazon.awssdk:s3control'
+ testImplementation 'com.twitter:chill-java'
Review Comment:
This is needed because kryo serializer by default is not self-sufficient to
de-serialize unmodifiable map, serializers for un-modifiable map comes from
twitter's chill library when spark tries to ser/de. Hence to mimic the same
behavior without making aws module taking a dependency on spark, I added this
dependency. (more details -
[here](https://spark.apache.org/docs/latest/tuning.html#data-serialization))
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
