GitHub user yaooqinn opened a pull request:
https://github.com/apache/spark/pull/22180
[SPARK-25174][YARN]Limit the size of diagnostic message for am to
unregister itself from rm
## What changes were proposed in this pull request?
When using older versions of spark releases, a use case generated a huge
code-gen file which hit the limitation `Constant pool has grown past JVM limit
of 0xFFFF`. In this situation, it should fail immediately. But the diagnosis
message sent to RM is too large, the ApplicationMaster suspended and RM's
ZKStateStore was crashed. For 2.3 or later spark releases the limitation of
code-gen has been removed, but maybe there are still some uncaught exceptions
that contain oversized error message will cause such a problem.
This PR is aim to cut down the diagnosis message size.
## How was this patch tested?
Please review http://spark.apache.org/contributing.html before opening a
pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/yaooqinn/spark SPARK-25174
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22180.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22180
----
commit 8f5b67a57f6f8e9237fbfcfd9f80a02ee73cfe5d
Author: Kent Yao <yaooqinn@...>
Date: 2018-08-22T02:01:28Z
limit the size for am to unregister itself from rm
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
