GitHub user squito opened a pull request:
https://github.com/apache/spark/pull/17140
[SPARK-19796][CORE] Fix serialization of long property values in
TaskDescription
## What changes were proposed in this pull request?
The properties that are serialized with a TaskDescription can have very
long values (eg. "spark.job.description" which is set to the full sql statement
with the thrift-server). DataOutputStream.writeUTF() does not work well for
long strings, so this changes the way those values are serialized to handle
longer strings.
## How was this patch tested?
Updated existing unit test to reproduce the issue. All unit tests via
jenkins.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/squito/spark SPARK-19796
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/17140.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #17140
----
commit 60220a9b4721b131aefaa4f34d175b46e6b65bc6
Author: Imran Rashid <[email protected]>
Date: 2017-03-02T16:05:13Z
[SPARK-19796][CORE] Fix serialization of long property values
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
