aokolnychyi commented on a change in pull request #2362:
URL: https://github.com/apache/iceberg/pull/2362#discussion_r600719639
##########
File path: spark/src/main/java/org/apache/iceberg/spark/SparkUtil.java
##########
@@ -38,6 +39,13 @@
private SparkUtil() {
}
+ public static Table toStaticTable(Table table) {
+ // TODO: consider setting some fields as null to reduce the size
Review comment:
Not much, unfortunately. Spark cannot use Kryo on `FileIO` as it
contains a Hadoop conf. Even though Spark may serialize the Table object using
Kryo, we still use Java serialization for `FileIO` and that's the dominant part.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
