Ngone51 commented on a change in pull request #36000:
URL: https://github.com/apache/spark/pull/36000#discussion_r838294518
##########
File path:
common/network-common/src/main/java/org/apache/spark/network/server/OneForOneStreamManager.java
##########
@@ -51,6 +51,12 @@
// The channel associated to the stream
final Channel associatedChannel;
+ // Indicates whether the buffers is materialized, some buffers like
+ // ShuffleManagedBufferIterator, ShuffleChunkManagedBufferIterator,
ManagedBufferIterator are
+ // materialized only when the iterator is requested by the next method, so
we should not
+ // request and release them, to reduce I/O operations and improve
performance for the External
+ // Shuffle Service.
Review comment:
```suggestion
// Indicates whether the buffers is only materialized when next() is
called. Some buffers like
// ShuffleManagedBufferIterator, ShuffleChunkManagedBufferIterator,
ManagedBufferIterator are
// not materialized until the iterator is traversed by calling next().
We use it to decide whether buffers
// should be released at connectionTerminated() in order to avoid
unnecessary buffer materialization,
// which could be I/O based.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
