zhengchenyu commented on code in PR #2202:
URL:
https://github.com/apache/incubator-uniffle/pull/2202#discussion_r1809790751
##########
common/src/main/java/org/apache/uniffle/common/serializer/writable/WritableSerializerInstance.java:
##########
@@ -47,19 +47,28 @@ public <T> T deserialize(DataInputBuffer buffer, Class
vClass) throws IOExceptio
}
@Override
- public <K, V> SerializationStream serializeStream(OutputStream output,
boolean raw) {
+ public <K, V> SerializationStream serializeStream(
+ SerOutputStream output, boolean raw, boolean shared) {
if (raw) {
- return new RawWritableSerializationStream(this, output);
+ if (shared) {
Review Comment:
In client side, the parsed record will be used by reduce, so we need a deep
copy instance, every record have their own buffer. if we use shared buffer,
error will occur.
But in server side, we write the record to the mergedblock immediately after
parsing the record, so there is no need for each record to have a separate
memory copy. For a segment/block, we can use only two shared buffer, this saves
more memory.
BTW, although this PR is about Netty, a lot of work has actually been done
on saving memory
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
