openinx edited a comment on pull request #2343: URL: https://github.com/apache/iceberg/pull/2343#issuecomment-810735680
@aokolnychyi I tried to run the unit tests by switching to use `DirectByteBuffer`, both spark/flink `TestDataFileSerialization` tests works fine (See following patch). So did I miss some thing in you last [comment](https://github.com/apache/iceberg/pull/2343#issuecomment-810685833) ? I guess kryo will use the Java serializer to serialize & deserialize the `SerializableByteBufferMap`, but I need to confirm this by checking the code. ```diff diff --git a/flink/src/test/java/org/apache/iceberg/flink/TestDataFileSerialization.java b/flink/src/test/java/org/apache/iceberg/flink/TestDataFileSerialization.java index ad67b181..4b43718d 100644 --- a/flink/src/test/java/org/apache/iceberg/flink/TestDataFileSerialization.java +++ b/flink/src/test/java/org/apache/iceberg/flink/TestDataFileSerialization.java @@ -196,6 +196,6 @@ public class TestDataFileSerialization { } private static ByteBuffer longToBuffer(long value) { - return ByteBuffer.allocate(8).order(ByteOrder.LITTLE_ENDIAN).putLong(0, value); + return ByteBuffer.allocateDirect(8).order(ByteOrder.LITTLE_ENDIAN).putLong(0, value); } } diff --git a/spark/src/test/java/org/apache/iceberg/TestDataFileSerialization.java b/spark/src/test/java/org/apache/iceberg/TestDataFileSerialization.java index f5c2990b..eb22873d 100644 --- a/spark/src/test/java/org/apache/iceberg/TestDataFileSerialization.java +++ b/spark/src/test/java/org/apache/iceberg/TestDataFileSerialization.java @@ -164,6 +164,6 @@ public class TestDataFileSerialization { } private static ByteBuffer longToBuffer(long value) { - return ByteBuffer.allocate(8).order(ByteOrder.LITTLE_ENDIAN).putLong(0, value); + return ByteBuffer.allocateDirect(8).order(ByteOrder.LITTLE_ENDIAN).putLong(0, value); } } ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
