GitHub user kiszk opened a pull request:
https://github.com/apache/spark/pull/21064
[SPARK-23976][Core] Detect length overflow in
UTF8String.concat()/ByteArray.concat()
## What changes were proposed in this pull request?
This PR detects length overflow if total elements in inputs are not
acceptable.
For example, when the three inputs has `0x7FFF_FF00`, `0x7FFF_FF00`, and
`0xE00`, we should detect length overflow since we cannot allocate such a large
structure on `byte[]`.
On the other hand, the current algorithm can allocate the result structure
with `0x1000`-byte length due to integer sum overflow.
## How was this patch tested?
Existing UTs.
If we would create UTs, we need large heap (6-8GB). It may make test
environment unstable.
If it is necessary to create UTs, I will create them.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kiszk/spark SPARK-23976
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21064.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21064
----
commit 6ba94f3b6c4cc3e69e4916a1669e95ef71267a6a
Author: Kazuaki Ishizaki <ishizaki@...>
Date: 2018-04-13T12:00:39Z
initial commit
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
