[
https://issues.apache.org/jira/browse/SPARK-31199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
runnings updated SPARK-31199:
-----------------------------
Description:
io.connectionTimeout only used for connection timeout for connection setup
while io.idleTimeout is used to control how long to kill the connection if it
seems to be idle([https://github.com/apache/spark/pull/5584])
These 2 timeouts could be quite different and shorten connectiontimeout could
help fast fail the connection related problem in some cases like when doing
shuffle, we could fast fail the task and retry.
was:
spark.shuffle.io.connectionTimeout only used for connection timeout for
connection setup while spark.shuffle.io.idleTimeout is used to control how long
to kill the connection if it seems to be
idle([https://github.com/apache/spark/pull/5584])
These 2 timeouts could be quite different and shorten connectiontimeout could
help fast fail the shuffle task in some cases
> Separate connection timeout and idle timeout for netty
> ------------------------------------------------------
>
> Key: SPARK-31199
> URL: https://issues.apache.org/jira/browse/SPARK-31199
> Project: Spark
> Issue Type: Improvement
> Components: Shuffle
> Affects Versions: 3.1.0
> Reporter: runnings
> Priority: Major
>
> io.connectionTimeout only used for connection timeout for connection setup
> while io.idleTimeout is used to control how long to kill the connection if it
> seems to be idle([https://github.com/apache/spark/pull/5584])
>
> These 2 timeouts could be quite different and shorten connectiontimeout could
> help fast fail the connection related problem in some cases like when doing
> shuffle, we could fast fail the task and retry.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
