singhpk234 commented on code in PR #5437:
URL: https://github.com/apache/iceberg/pull/5437#discussion_r943062912
##########
spark/v3.3/spark/src/test/java/org/apache/iceberg/TestFileIOSerialization.java:
##########
@@ -101,6 +104,60 @@ public void testHadoopFileIOJavaSerialization() throws
IOException, ClassNotFoun
Assert.assertEquals("Conf values must be present", "v2",
actualConf.get("k2"));
}
+ @Test
+ public void testResolvingFileIOKryoSerialization() throws IOException {
Review Comment:
This was a pre-exisitng test class `TestFileIOSerialization` checking the
ser/de for HadoopFileIO, hence added other fileIO's serialization tests also to
the same class.
> the module that defines each FileIO
I think what we can do here is create the kryoHelper (which doesn't take
dependency on spark) in core and since integrations (i.e aws etc) take a
dependency on it, it would be available to all FileIO.
The only concern I had here was that these test can become outdated when
spark uses diff version of twitter-chill or uses diff lib like
[margo](https://github.com/magro/kryo-serializers). Having these UT in spark
itself will help us in catching failures across spark versions.
Would love to know your thoughts regarding the same
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
