GitHub user MaxGekk opened a pull request:
https://github.com/apache/spark/pull/23159
[SPARK-26191][SQL] Control truncation of Spark plans via maxFields parameter
## What changes were proposed in this pull request?
In the PR, I propose to add `maxFields` parameter to all functions involved
in creation of textual representation of spark plans such as `simpleString` and
`verboseString`. New parameter restrict number of fields produced to truncated
strings. Any elements beyond the limit will be dropped and replaced by a `"...
N more fields"` placeholder. The threshold is bumped up to `Int.MaxValue` for
`toFile()`.
## How was this patch tested?
Added a test to `QueryExecutionSuite` which checks `maxFields` impacts on
number of truncated fields in `LocalRelation`.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/MaxGekk/spark-1 to-file-max-fields
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/23159.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #23159
----
commit 756242132d665223738b177af0b838fc953c6a6b
Author: Maxim Gekk <max.gekk@...>
Date: 2018-11-27T21:34:58Z
Added a test
commit e9e789311b998d961aa8fcf76463307a410969ea
Author: Maxim Gekk <max.gekk@...>
Date: 2018-11-27T22:40:45Z
Adding maxField parameter
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
