[
https://issues.apache.org/jira/browse/FLINK-10537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16649805#comment-16649805
]
ASF GitHub Bot commented on FLINK-10537:
----------------------------------------
zhijiangW commented on a change in pull request #6833: [FLINK-10537][network]
Fix network small performance degradation after merging [FLINK-9913]
URL: https://github.com/apache/flink/pull/6833#discussion_r225030697
##########
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/api/serialization/SpanningRecordSerializer.java
##########
@@ -92,20 +92,9 @@ public void serializeRecord(T record) throws IOException {
*/
@Override
public SerializationResult copyToBufferBuilder(BufferBuilder
targetBuffer) {
- boolean mustCommit = false;
- if (lengthBuffer.hasRemaining()) {
Review comment:
It is really not necessary to check the conditions for every record in
normal cases, and it may bring some overheads to do so.
But if we directly remove these conditions, in the case of spanning multiple
buffers because of large records or remaining small spaces in previous
`targetBuffer`, we may encounter another cost overhead. In this condition, the
length buffer is probably not remaining but only the data buffer is remaining,
and the cost of `targetBuffer.append(lengthBuffer)` would be higher than the
cost of `lengthBuffer.hasRemaining()`.
If we can check this condition on `RecordWriter` side as the following,
maybe we can check accordingly in `RecordSerializer`.
```
private boolean copyFromSerializerToTargetChannel(int targetChannel) {
serializer.reset();
boolean pruneTriggered = false;
BufferBuilder bufferBuilder = getBufferBuilder(targetChannel);
SerializationResult result =
serializer.copyToBufferBuilder(bufferBuilder, **false**);
while (result.isFullBuffer()) {
numBytesOut.inc(bufferBuilder.finish());
numBuffersOut.inc();
if (result.isFullRecord()) {
pruneTriggered = true;
bufferBuilders[targetChannel] =
Optional.empty();
break;
}
bufferBuilder = requestNewBufferBuilder(targetChannel);
result = serializer.copyToBufferBuilder(bufferBuilder,
**true**);
}
```
But I am not sure whether it is worth doing by giving this hint in
`copyToBufferBuilder` interface. What do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Network throughput performance regression after broadcast changes
> -----------------------------------------------------------------
>
> Key: FLINK-10537
> URL: https://issues.apache.org/jira/browse/FLINK-10537
> Project: Flink
> Issue Type: Bug
> Components: Network
> Affects Versions: 1.7.0
> Reporter: Piotr Nowojski
> Assignee: Piotr Nowojski
> Priority: Major
> Labels: pull-request-available
>
> There is a slight network throughput regression introduced in:
> https://issues.apache.org/jira/browse/FLINK-9913
> It is visible in the following benchmark:
> [http://codespeed.dak8s.net:8000/timeline/#/?exe=1&ben=networkThroughput.1,100ms&env=2&revs=200&equid=off&quarts=on&extr=on]
> (drop in the chart that happened since 21st September.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
