liupengcheng created SPARK-26660: ------------------------------------ Summary: Add warning logs for large taskBinary size Key: SPARK-26660 URL: https://issues.apache.org/jira/browse/SPARK-26660 Project: Spark Issue Type: Improvement Components: Spark Core Affects Versions: 2.4.0, 2.1.0 Environment: Spark2.1 Reporter: liupengcheng
In our production environment, we encountered a OOM problem in ML application. After carefully investigation, it was found that some ML library may generate large ml model, thus causing executor may not able to deserialize it and result in OOM failures. !screenshot-1.png! In order to facilitate the debuging of memory problem caused by large taskBinary broadcast, we may should add same warning logs for it. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org