Ohad created SPARK-33557:
----------------------------
Summary: spark.storage.blockManagerSlaveTimeoutMs default value
does not follow spark.network.timeout value when the latter was changed
Key: SPARK-33557
URL: https://issues.apache.org/jira/browse/SPARK-33557
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 3.0.1, 3.0.0
Reporter: Ohad
According to the documentation "spark.network.timeout" is the default timeout
for "spark.storage.blockManagerSlaveTimeoutMs" which implies that when the user
sets "spark.network.timeout" the effective value of
"spark.storage.blockManagerSlaveTimeoutMs" should also be changed if it was not
specifically changed.
However this is not the case since the default value of
"spark.storage.blockManagerSlaveTimeoutMs" is always the default value of
"spark.network.timeout" (120s)
"spark.storage.blockManagerSlaveTimeoutMs" is defined in the package object of
"org.apache.spark.internal.config" as follows:
{code:java}
private[spark] val STORAGE_BLOCKMANAGER_SLAVE_TIMEOUT =
ConfigBuilder("spark.storage.blockManagerSlaveTimeoutMs")
.version("0.7.0")
.timeConf(TimeUnit.MILLISECONDS)
.createWithDefaultString(Network.NETWORK_TIMEOUT.defaultValueString)
{code}
So it seems like the its default value is indeed "fixed" to
"spark.network.timeout" default value.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
