Takeshi Yamamuro created SPARK-20425:
----------------------------------------
Summary: Support an extended display mode to print a column data
per line
Key: SPARK-20425
URL: https://issues.apache.org/jira/browse/SPARK-20425
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 2.1.0
Reporter: Takeshi Yamamuro
Priority: Minor
In the master, when printing Dataset with many columns, the readability is too
low like;
{code}
scala> val df = spark.range(100).selectExpr((0 until 100).map(i => s"rand() AS
c$i"): _*)
scala> df.show(3, 0)
+------------------+------------------+------------------+-------------------+------------------+------------------+-------------------+------------------+------------------+------------------+------------------+-------------------+------------------+------------------+------------------+-------------------+-------------------+-------------------+------------------+------------------+-------------------+------------------+-------------------+------------------+-------------------+-------------------+-------------------+--------------------+-------------------+------------------+-------------------+--------------------+------------------+------------------+-------------------+-------------------+-------------------+------------------+------------------+-------------------+------------------+------------------+-------------------+-------------------+-------------------+------------------+--------------------+--------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+--------------------+-------------------+-------------------+-------------------+-------------------+------------------+------------------+-------------------+-------------------+------------------+-------------------+------------------+------------------+-----------------+-------------------+-------------------+------------------+-------------------+------------------+-------------------+-------------------+-------------------+------------------+-------------------+------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+------------------+-------------------+-------------------+------------------+------------------+------------------+-------------------+------------------+-------------------+------------------+-------------------+-------------------+-------------------+
|c0 |c1 |c2 |c3
|c4 |c5 |c6 |c7
|c8 |c9 |c10 |c11
|c12 |c13 |c14 |c15
|c16 |c17 |c18 |c19
|c20 |c21 |c22 |c23
|c24 |c25 |c26 |c27
|c28 |c29 |c30 |c31
|c32 |c33 |c34 |c35
|c36 |c37 |c38 |c39
|c40 |c41 |c42 |c43
|c44 |c45 |c46 |c47
|c48 |c49 |c50 |c51
|c52 |c53 |c54 |c55
|c56 |c57 |c58 |c59
|c60 |c61 |c62 |c63
|c64 |c65 |c66 |c67
|c68 |c69 |c70 |c71
|c72 |c73 |c74 |c75
|c76 |c77 |c78 |c79
|c80 |c81 |c82 |c83
|c84 |c85 |c86 |c87
|c88 |c89 |c90 |c91
|c92 |c93 |c94 |c95
|c96 |c97 |c98 |c99 |
+------------------+------------------+------------------+-------------------+------------------+------------------+-------------------+------------------+------------------+------------------+------------------+-------------------+------------------+------------------+------------------+-------------------+-------------------+-------------------+------------------+------------------+-------------------+------------------+-------------------+------------------+-------------------+-------------------+-------------------+--------------------+-------------------+------------------+-------------------+--------------------+------------------+------------------+-------------------+-------------------+-------------------+------------------+------------------+-------------------+------------------+------------------+-------------------+-------------------+-------------------+------------------+--------------------+--------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+--------------------+-------------------+-------------------+-------------------+-------------------+------------------+------------------+-------------------+-------------------+------------------+-------------------+------------------+------------------+-----------------+-------------------+-------------------+------------------+-------------------+------------------+-------------------+-------------------+-------------------+------------------+-------------------+------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+------------------+-------------------+-------------------+------------------+------------------+------------------+-------------------+------------------+-------------------+------------------+-------------------+-------------------+-------------------+
|0.6306087152476858|0.9174349686288383|0.5511324165035159|0.3320844128641819
|0.7738486877101489|0.2154915886962553|0.4754997600674299 |0.922780639280355
|0.7136894772661909|0.2277580838165979|0.5926874459847249|0.40311408392226633|0.467830264333843
|0.8330466896984213|0.1893258482389527|0.6320849515511165 |0.7530911056912044
|0.06700254871955424|0.370528597355559
|0.2755437445193154|0.23704391110980128|0.8067400174905822|0.13597793616251852|0.1708888820162453|0.01672725007605702|0.983118121881555
|0.25040195628629924|0.060537253723083384|0.20000530582637488|0.3400572407133511|0.9375689433322597
|0.057039316954370256|0.8053269714347623|0.5247817572228813|0.28419308820527944|0.9798908885194533
|0.31805988175678146|0.7034448027077574|0.5400575751346084|0.25336322371116216|0.9361634546853429|0.6118681368289798|0.6295081549153907
|0.13417468943957422|0.41617137072255794|0.7267230869252035|0.023792726137561115|0.5776157058356362
|0.04884204913195467|0.26728716103441275|0.646680370807925
|0.9782712690657244
|0.16434031314818154|0.20985522381321275|0.24739842475440077
|0.26335189682977334|0.19604841662422068|0.10742950487300651|0.20283136488091502|0.3100312319723688|0.886959006630645
|0.25157102269776244|0.34428775168410786|0.3500506818575777|0.3781142441912052
|0.8560316444386715|0.4737104888956839|0.735903101602148|0.02236617130529006|0.8769074095835873
|0.2001426662503153|0.5534032319238532
|0.7289496620397098|0.41955191309992157|0.9337700133660436
|0.34059094378451005|0.6419144759403556|0.08167496930341167|0.9947099478497635|0.48010888605366586|0.22314796858167918|0.17786598882331306|0.7351521162297135
|0.5422057170020095 |0.9521927872726792 |0.7459825486368227
|0.40907708791990627|0.8903819313311575|0.7251413746923618 |0.2977174938745204
|0.9515209660203555|0.9375968604766713|0.5087851740042524|0.4255237544908751
|0.8023768698664653|0.48003189618006703|0.1775841829745185|0.09050775629268382|0.6743909291138167
|0.2498415755876865 |
|0.6866473844170801|0.4774360641212433|0.631696201340726
|0.33979113021468343|0.5663049010847052|0.7280190472258865|0.41370958502324806|0.9977433873622218|0.7671957338989901|0.2788708556233931|0.3355106391656496|0.88478952319287
|0.0333974166999893|0.6061744715862606|0.9617779139652359|0.22484954822341863|0.12770906021550898|0.5577789629508672
|0.2877649024640704|0.5566577406549361|0.9334933255278052
|0.9166720585157266|0.9689249324600591 |0.6367502457478598|0.7993572745928459
|0.23213222324218108|0.11928284054154137|0.6173493362456599
|0.0505122058694798 |0.9050228629552983|0.17112767911121707|0.47395598348370005
|0.5820498657823081|0.6241124650645072|0.18587258258036776|0.14987593554122225|0.3079446253653946
|0.9414228822867968|0.8362276265462365|0.9155655305576353
|0.5121559807153562|0.8963362656525707|0.22765970274318037|0.8177039187132797
|0.8190326635933787 |0.5256005177032199|0.8167598457269669
|0.030936807130934496|0.6733006585281015 |0.4208049626816347
|0.24603085738518538|0.22719198954208153|0.1622280557565281
|0.22217325159218038|0.014684419513742553|0.08987111517447499|0.2157764759142622
|0.8223414104088321 |0.4868624404491777
|0.4016191733088167|0.6169281906889263|0.15603611040433385|0.18289285085714913|0.9538408988218972|0.15037154865295121|0.5364516961987454|0.8077254873163031|0.712600478545675|0.7277477241003857
|0.19822912960348305|0.8305051199208777|0.18631911396566114|0.8909532487898342|0.3470409226992506
|0.35306974180587636|0.9107058868891469
|0.3321327206004986|0.48952332459050607|0.3630403307479373|0.5400046826340376
|0.5387377194310529
|0.42860539421837585|0.23214101630985995|0.21438968839794847|0.15370603160082352|0.04355605642700022|0.6096006707067466
|0.6933354157094292|0.06302172470859002|0.03174631856164001|0.664243581650643
|0.7833239547446621|0.696884598352864
|0.34626385933237736|0.9263495598791336|0.404818892816584
|0.2085585394755507|0.6150004897990109 |0.05391193524302473|0.28188484028329097|
+------------------+------------------+------------------+-------------------+------------------+------------------+-------------------+------------------+------------------+------------------+------------------+-------------------+------------------+------------------+------------------+-------------------+-------------------+-------------------+------------------+------------------+-------------------+------------------+-------------------+------------------+-------------------+-------------------+-------------------+--------------------+-------------------+------------------+-------------------+--------------------+------------------+------------------+-------------------+-------------------+-------------------+------------------+------------------+-------------------+------------------+------------------+-------------------+-------------------+-------------------+------------------+--------------------+--------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+--------------------+-------------------+-------------------+-------------------+-------------------+------------------+------------------+-------------------+-------------------+------------------+-------------------+------------------+------------------+-----------------+-------------------+-------------------+------------------+-------------------+------------------+-------------------+-------------------+-------------------+------------------+-------------------+------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+------------------+-------------------+-------------------+------------------+------------------+------------------+-------------------+------------------+-------------------+------------------+-------------------+-------------------+-------------------+
only showing top 2 rows
{code}
psql, CLI for PostgreSQL supports an extended display mode for this case like:
http://stackoverflow.com/questions/9604723/alternate-output-format-for-psql
{code}
-RECORD 0-------------------
c0 | 0.6306087152476858
c1 | 0.9174349686288383
c2 | 0.5511324165035159
...
c98 | 0.05391193524302473
c99 | 0.28188484028329097
-RECORD 1-------------------
c0 | 0.6866473844170801
c1 | 0.4774360641212433
c2 | 0.631696201340726
...
c98 | 0.05391193524302473
c99 | 0.28188484028329097
only showing top 2 rows
{code}
I feel it is useful to support this mode in spark-shell.
Thought?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
