mridulm commented on code in PR #37638: URL: https://github.com/apache/spark/pull/37638#discussion_r1067505986
########## docs/monitoring.md: ########## @@ -1421,6 +1421,21 @@ Note: applies to the shuffle service - shuffle-server.usedDirectMemory - shuffle-server.usedHeapMemory +Note: applies to the shuffle service when the server side flag +`spark.shuffle.push.server.mergedShuffleFileManagerImpl` set as the appropriate +org.apache.spark.network.shuffle.MergedShuffleFileManager implementation for the Push-Based Shuffle + +- blockAppendCollisions - the number of shuffle push blocks collided in shuffle services + as another block for the same reduce partition were being written +- lateBlockPushes - the number of shuffle push blocks that are received in shuffle service + after the specific shuffle merge has been finalized +- blockBytesWritten - the size of the pushed block data written to file in bytes +- deferredBlockBytes - the size of the current deferred block parts buffered in memory +- deferredBlocks - the number of the current deferred block parts buffered in memory +- staleBlockPushes - the number of stale shuffle block push requests +- ignoredBlockBytes - the size of the pushed block data that are ignored after the shuffle + file is finalized or when a request is for a duplicate block Review Comment: nit: Can we reorder this so that we go from the common metrics to the more corner cases ? For example, something like: * blockBytesWritten * blockAppendCollisions * lateBlockPushes * staleBlockPushes * deferredBlocks * deferredBlockBytes * ignoredBlockBytes (If this is not the right order, please feel free to reorder them) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org