[
https://issues.apache.org/jira/browse/SPARK-39752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
SHOBHIT SHUKLA updated SPARK-39752:
-----------------------------------
Fix Version/s: 3.0.2
> Spark job failed with 10M rows data with Broken pipe error
> ----------------------------------------------------------
>
> Key: SPARK-39752
> URL: https://issues.apache.org/jira/browse/SPARK-39752
> Project: Spark
> Issue Type: Bug
> Components: PySpark, Spark Core
> Affects Versions: 3.0.3, 3.2.1
> Reporter: SHOBHIT SHUKLA
> Priority: Major
> Fix For: 3.0.2
>
> Attachments: Failed_spark_job_3.0.3.txt, spark_job_success_3.0.2.txt
>
>
> Spark job failed with 10M rows data with Broken pipe error. Same spark job
> was working previously with the settings "executor_cores": 1,
> "executor_memory": 1, "driver_cores": 1, "driver_memory": 1. where as the
> same job is failing with spark settings in 3.0.3 and 3.2.1.
> Major symptoms (slowness, timeout, out of memory as examples): Spark job is
> failing with the error java.net.SocketException: Broken pipe (Write failed)
> Here are the spark settings information which is working on Spark 3.0.3 and
> 3.2.1 : "executor_cores": 4, "executor_memory": 4, "driver_cores": 4,
> "driver_memory": 4.. The spark job doesn't consistently works with the above
> settings. Some times, need to increase the cores and memory.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
