GitHub user maropu opened a pull request:
https://github.com/apache/spark/pull/17733
[SPARK-20425][SQL] Support an extended display mode for Dataset.show
## What changes were proposed in this pull request?
This pr added an extended display mode for `Dataset.show`. In the current
master, when printing Dataset with many columns, the readability is low like;
```
scala> val df = spark.range(100).selectExpr((0 until 100).map(i => s"rand()
AS c$i"): _*)
scala> df.show(3, 0)
+------------------+------------------+------------------+-------------------+------------------+------------------+-------------------+------------------+------------------+------------------+------------------+-------------------+------------------+------------------+------------------+-------------------+-------------------+-------------------+------------------+------------------+-------------------+------------------+-------------------+------------------+-------------------+-------------------+-------------------+--------------------+-------------------+------------------+-------------------+--------------------+------------------+------------------+-------------------+-------------------+-------------------+------------------+------------------+-------------------+------------------+------------------+-------------------+-------------------+-------------------+------------------+--------------------+--------------------+-------------------+-------------------+-------------
------+-------------------+-------------------+-------------------+--------------------+-------------------+-------------------+-------------------+-------------------+------------------+------------------+-------------------+-------------------+------------------+-------------------+------------------+------------------+-----------------+-------------------+-------------------+------------------+-------------------+------------------+-------------------+-------------------+-------------------+------------------+-------------------+------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+------------------+-------------------+-------------------+------------------+------------------+------------------+-------------------+------------------+-------------------+------------------+-------------------+-------------------+-------------------+
|c0 |c1 |c2 |c3
|c4 |c5 |c6 |c7
|c8 |c9 |c10 |c11
|c12 |c13 |c14 |c15
|c16 |c17 |c18 |c19
|c20 |c21 |c22 |c23
|c24 |c25 |c26 |c27
|c28 |c29 |c30 |c31
|c32 |c33 |c34 |c35
|c36 |c37 |c38 |c39
|c40 |c41 |c42 |c43
|c44 |c45 |c46 |c47
|c48 |c49 |c50
|c51 |c52 |c53 |c54
|c55 |c56 |c57 |c58
|c59 |c60 |c61 |c62
|c63 |c64 |c65 |c66
|c67 |c68 |c69 |c70
|c71 |c72 |c73 |c74
|c75 |c76 |c77 |c78
|c79 |c80 |c81 |c82
|c83 |c84 |c85 |c86
|c87 |c88 |c89 |c90
|c91 |c92 |c93 |c94
|c95 |c96 |c97 |c98
|c99 |
+------------------+------------------+------------------+-------------------+------------------+------------------+-------------------+------------------+------------------+------------------+------------------+-------------------+------------------+------------------+------------------+-------------------+-------------------+-------------------+------------------+------------------+-------------------+------------------+-------------------+------------------+-------------------+-------------------+-------------------+--------------------+-------------------+------------------+-------------------+--------------------+------------------+------------------+-------------------+-------------------+-------------------+------------------+------------------+-------------------+------------------+------------------+-------------------+-------------------+-------------------+------------------+--------------------+--------------------+-------------------+-------------------+-------------
------+-------------------+-------------------+-------------------+--------------------+-------------------+-------------------+-------------------+-------------------+------------------+------------------+-------------------+-------------------+------------------+-------------------+------------------+------------------+-----------------+-------------------+-------------------+------------------+-------------------+------------------+-------------------+-------------------+-------------------+------------------+-------------------+------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+------------------+-------------------+-------------------+------------------+------------------+------------------+-------------------+------------------+-------------------+------------------+-------------------+-------------------+-------------------+
|0.6306087152476858|0.9174349686288383|0.5511324165035159|0.3320844128641819
|0.7738486877101489|0.2154915886962553|0.4754997600674299 |0.922780639280355
|0.7136894772661909|0.2277580838165979|0.5926874459847249|0.40311408392226633|0.467830264333843
|0.8330466896984213|0.1893258482389527|0.6320849515511165 |0.7530911056912044
|0.06700254871955424|0.370528597355559
|0.2755437445193154|0.23704391110980128|0.8067400174905822|0.13597793616251852|0.1708888820162453|0.01672725007605702|0.983118121881555
|0.25040195628629924|0.060537253723083384|0.20000530582637488|0.3400572407133511|0.9375689433322597
|0.057039316954370256|0.8053269714347623|0.5247817572228813|0.28419308820527944|0.9798908885194533
|0.31805988175678146|0.7034448027077574|0.5400575751346084|0.25336322371116216|0.9361634546853429|0.6118681368289798|0.6295081549153907
|0.13417468943957422|0.41617137072255794|0.7267230869252035|0.023792726137561115|0.5776157058356362
|0.04884204913195467|0.26728716103441275|0.64668037080
7925 |0.9782712690657244
|0.16434031314818154|0.20985522381321275|0.24739842475440077
|0.26335189682977334|0.19604841662422068|0.10742950487300651|0.20283136488091502|0.3100312319723688|0.886959006630645
|0.25157102269776244|0.34428775168410786|0.3500506818575777|0.3781142441912052
|0.8560316444386715|0.4737104888956839|0.735903101602148|0.02236617130529006|0.8769074095835873
|0.2001426662503153|0.5534032319238532
|0.7289496620397098|0.41955191309992157|0.9337700133660436
|0.34059094378451005|0.6419144759403556|0.08167496930341167|0.9947099478497635|0.48010888605366586|0.22314796858167918|0.17786598882331306|0.7351521162297135
|0.5422057170020095 |0.9521927872726792 |0.7459825486368227
|0.40907708791990627|0.8903819313311575|0.7251413746923618 |0.2977174938745204
|0.9515209660203555|0.9375968604766713|0.5087851740042524|0.4255237544908751
|0.8023768698664653|0.48003189618006703|0.1775841829745185|0.09050775629268382|0.6743909291138167
|0.2498415755876865 |
|0.6866473844170801|0.4774360641212433|0.631696201340726
|0.33979113021468343|0.5663049010847052|0.7280190472258865|0.41370958502324806|0.9977433873622218|0.7671957338989901|0.2788708556233931|0.3355106391656496|0.88478952319287
|0.0333974166999893|0.6061744715862606|0.9617779139652359|0.22484954822341863|0.12770906021550898|0.5577789629508672
|0.2877649024640704|0.5566577406549361|0.9334933255278052
|0.9166720585157266|0.9689249324600591 |0.6367502457478598|0.7993572745928459
|0.23213222324218108|0.11928284054154137|0.6173493362456599
|0.0505122058694798 |0.9050228629552983|0.17112767911121707|0.47395598348370005
|0.5820498657823081|0.6241124650645072|0.18587258258036776|0.14987593554122225|0.3079446253653946
|0.9414228822867968|0.8362276265462365|0.9155655305576353
|0.5121559807153562|0.8963362656525707|0.22765970274318037|0.8177039187132797
|0.8190326635933787 |0.5256005177032199|0.8167598457269669
|0.030936807130934496|0.6733006585281015 |0.4208049626816347 |0.24603085738
518538|0.22719198954208153|0.1622280557565281
|0.22217325159218038|0.014684419513742553|0.08987111517447499|0.2157764759142622
|0.8223414104088321 |0.4868624404491777
|0.4016191733088167|0.6169281906889263|0.15603611040433385|0.18289285085714913|0.9538408988218972|0.15037154865295121|0.5364516961987454|0.8077254873163031|0.712600478545675|0.7277477241003857
|0.19822912960348305|0.8305051199208777|0.18631911396566114|0.8909532487898342|0.3470409226992506
|0.35306974180587636|0.9107058868891469
|0.3321327206004986|0.48952332459050607|0.3630403307479373|0.5400046826340376
|0.5387377194310529
|0.42860539421837585|0.23214101630985995|0.21438968839794847|0.15370603160082352|0.04355605642700022|0.6096006707067466
|0.6933354157094292|0.06302172470859002|0.03174631856164001|0.664243581650643
|0.7833239547446621|0.696884598352864
|0.34626385933237736|0.9263495598791336|0.404818892816584
|0.2085585394755507|0.6150004897990109 |0.05391193524302473|0.28188484028329097|
+------------------+------------------+------------------+-------------------+------------------+------------------+-------------------+------------------+------------------+------------------+------------------+-------------------+------------------+------------------+------------------+-------------------+-------------------+-------------------+------------------+------------------+-------------------+------------------+-------------------+------------------+-------------------+-------------------+-------------------+--------------------+-------------------+------------------+-------------------+--------------------+------------------+------------------+-------------------+-------------------+-------------------+------------------+------------------+-------------------+------------------+------------------+-------------------+-------------------+-------------------+------------------+--------------------+--------------------+-------------------+-------------------+-------------
------+-------------------+-------------------+-------------------+--------------------+-------------------+-------------------+-------------------+-------------------+------------------+------------------+-------------------+-------------------+------------------+-------------------+------------------+------------------+-----------------+-------------------+-------------------+------------------+-------------------+------------------+-------------------+-------------------+-------------------+------------------+-------------------+------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+------------------+-------------------+-------------------+------------------+------------------+------------------+-------------------+------------------+-------------------+------------------+-------------------+-------------------+-------------------+
only showing top 2 rows
```
`psql`, CLI for PostgreSQL, supports an extended display mode for this case
like:
http://stackoverflow.com/questions/9604723/alternate-output-format-for-psql
```
-RECORD 0-------------------
c0 | 0.6306087152476858
c1 | 0.9174349686288383
c2 | 0.5511324165035159
...
c98 | 0.05391193524302473
c99 | 0.28188484028329097
-RECORD 1-------------------
c0 | 0.6866473844170801
c1 | 0.4774360641212433
c2 | 0.631696201340726
...
c98 | 0.05391193524302473
c99 | 0.28188484028329097
only showing top 2 rows
```
## How was this patch tested?
Added tests in `DataFrameSuite`.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/maropu/spark SPARK-20425
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/17733.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #17733
----
commit 901bcf8491d384b94fa19d6f9b96b54d395484d6
Author: Takeshi Yamamuro <[email protected]>
Date: 2017-04-21T07:33:46Z
Support an extended display mode in spark-shell
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
