rmcyang commented on code in PR #37638: URL: https://github.com/apache/spark/pull/37638#discussion_r1067534717
########## docs/monitoring.md: ########## @@ -1421,6 +1421,21 @@ Note: applies to the shuffle service - shuffle-server.usedDirectMemory - shuffle-server.usedHeapMemory +Note: applies to the shuffle service when the server side flag +`spark.shuffle.push.server.mergedShuffleFileManagerImpl` set as the appropriate +org.apache.spark.network.shuffle.MergedShuffleFileManager implementation for the Push-Based Shuffle + +- blockAppendCollisions - the number of shuffle push blocks collided in shuffle services + as another block for the same reduce partition were being written +- lateBlockPushes - the number of shuffle push blocks that are received in shuffle service + after the specific shuffle merge has been finalized +- blockBytesWritten - the size of the pushed block data written to file in bytes +- deferredBlockBytes - the size of the current deferred block parts buffered in memory +- deferredBlocks - the number of the current deferred block parts buffered in memory +- staleBlockPushes - the number of stale shuffle block push requests +- ignoredBlockBytes - the size of the pushed block data that are ignored after the shuffle + file is finalized or when a request is for a duplicate block Review Comment: Thanks for the suggestion, I further put the `staleBlockPushes` to the middle of `deferredBlockBytes ` and `ignoredBlockBytes`, as a stale block push appears in the case of indeterminate stage retries, which I feel is less common to deferred block. Let me know if that's not right. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
