kbendick opened a new pull request #3667: URL: https://github.com/apache/iceberg/pull/3667
This PR replaces https://github.com/apache/iceberg/pull/3597, as @dubeme has not responded about some minor changes and we need this patch prior to the upcoming 0.13.0 release. I have added them as a co-author on this PR so that they still get credit for their contributions 😄 Issue #3586 shows situation where Kyro is unable to serialize the SparkPartition object (@rdblue points it out from the stack trace). This commit replaces the ImmutableMap with HashMap and adds a test for round trip Kryo serialization and Java serialization for `SparkPartition` objects. I also tested this by running the `add_files` test suite with Kryo serialization enabled. The tests failed until this patch was applied. Unfortunatley, I couldn't add a test there as the usual `withSQLConf` method did not pick up the change in `spark.serializer`. This patch covers Spark 3.2. I'll add the same changes for earlier versions once this is merged. Co-authored-by: Dubem Enyekwe <[email protected]> cc @rdblue @jackye1995 @dubeme -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
