Fokko commented on code in PR #8625:
URL: https://github.com/apache/iceberg/pull/8625#discussion_r1335224154
##########
core/src/main/java/org/apache/iceberg/avro/AvroEncoderUtil.java:
##########
@@ -54,7 +54,8 @@ public static <T> byte[] encode(T datum, Schema avroSchema)
throws IOException {
dataOut.writeUTF(avroSchema.toString());
// Encode the datum with avro schema.
- BinaryEncoder encoder = EncoderFactory.get().binaryEncoder(out, null);
+ BinaryEncoder encoder =
+ new EncoderFactory().configureBlockSize(1024 *
1024).blockingBinaryEncoder(out, null);
Review Comment:
How did you come up with the 1MB? Seems to me that we can make this bigger.
It is also nice when we chunk it up into blocks so that we can evaluate the
blocks in parallel.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
