rmcyang commented on code in PR #37638: URL: https://github.com/apache/spark/pull/37638#discussion_r1066466470
########## docs/monitoring.md: ########## @@ -1421,6 +1421,20 @@ Note: applies to the shuffle service - shuffle-server.usedDirectMemory - shuffle-server.usedHeapMemory +- Notes: below shuffle service server-side metrics are specific to the Push-Based Shuffle +(with `spark.shuffle.push.enabled` set as true on the client side, and with the server side +flag `spark.shuffle.push.server.mergedShuffleFileManagerImpl` set as the appropriate +org.apache.spark.network.shuffle.MergedShuffleFileManager implementation for the Push-Based +Shuffle to be enabled) + - blockAppendCollisions - the number of shuffle push blocks collided in shuffle services + as another block for the same reduce partition were being written + - lateBlockPushes - the number of shuffle push blocks that are received in shuffle service + after the specific shuffle merge has been finalized + - blockBytesWritten - the size of the pushed block data written to file in bytes + - deferredBlockBytes - the size of the current deferred block parts buffered in memory + - deferredBlocks - the number of the current deferred block parts buffered in memory + - staleBlockPushes - the number of stale shuffle block push requests Review Comment: Fixed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
