shardulm94 commented on a change in pull request #33446:
URL: https://github.com/apache/spark/pull/33446#discussion_r780519077
##########
File path: core/src/main/scala/org/apache/spark/internal/config/package.scala
##########
@@ -1214,6 +1214,29 @@ package object config {
.checkValue(_ > 0, "The max no. of blocks in flight cannot be
non-positive.")
.createWithDefault(Int.MaxValue)
+ private[spark] val REDUCER_SHUFFLE_FETCH_SLOW_LOG_THRESHOLD_MS =
+ ConfigBuilder("spark.reducer.shuffleFetchSlowLogThreshold.time")
+ .doc("When fetching blocks from an external shuffle service is slower
than expected, the " +
+ "fetch will be logged to allow for subsequent investigation. A fetch
is determined " +
+ "to be slow if it has a total duration of at least this value, and a
transfer rate " +
+ // cannot reference val REDUCER_SHUFFLE_FETCH_SLOW_LOG_THRESHOLD_BPS
since its uninitialized
Review comment:
Both these confs reference each other in the doc string, so atleast one
of these confs will have to be referenced using a string.
##########
File path: docs/configuration.md
##########
@@ -2064,6 +2064,28 @@ Apart from these, the following properties are also
available, and may be useful
</td>
<td>1.2.0</td>
</tr>
+<tr>
+ <td><code>spark.reducer.shuffleFetchSlowLogThreshold.time</code></td>
+ <td>value of
<code>spark.reducer.shuffleFetchSlowLogThreshold.time</code></td>
Review comment:
Fixed
##########
File path: docs/configuration.md
##########
@@ -2064,6 +2064,28 @@ Apart from these, the following properties are also
available, and may be useful
</td>
<td>1.2.0</td>
</tr>
+<tr>
+ <td><code>spark.reducer.shuffleFetchSlowLogThreshold.time</code></td>
+ <td>value of
<code>spark.reducer.shuffleFetchSlowLogThreshold.time</code></td>
+ <td>
+ When fetching blocks from an external shuffle service is slower than
expected, the
+ fetch will be logged to allow for subsequent investigation. A fetch is
determined
+ to be slow if it has a total duration of at least this value, and a
transfer rate
+ less than
<code>spark.reducer.shuffleFetchSlowLogThreshold.bytesPerSec</code>.
+ </td>
+ <td>3.2.0</td>
+</tr>
+<tr>
+ <td><code>spark.reducer.shuffleFetchSlowLogThreshold.bytesPerSec</code></td>
+ <td>value of
<code>spark.reducer.shuffleFetchSlowLogThreshold.bytesPerSec</code></td>
Review comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
