[
https://issues.apache.org/jira/browse/SPARK-15505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15839386#comment-15839386
]
Hyukjin Kwon commented on SPARK-15505:
--------------------------------------
Ah, then, we should calculate the maximum length of that array after scanning
the data and then use that number.
{code}
val length = df.selectExpr("max(size(Col2))").head.getInt(0)
df.selectExpr((0 until length).map(n => s"Col2[$n]"): _*).show()
{code}
IMHO, this can simply be worked around as above.
if we have an API, then, it should cause a long hang to calculate the max
unless we know the max ahead and then users would get confused.
> Explode nested Array in DF Column into Multiple Columns
> --------------------------------------------------------
>
> Key: SPARK-15505
> URL: https://issues.apache.org/jira/browse/SPARK-15505
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 1.6.1
> Reporter: Jorge Machado
> Priority: Minor
>
> At the moment if we have a DF like this :
> {noformat}
> +------+---------+
> | Col1 | Col2 |
> +------+---------+
> | 1 |[2, 3, 4]|
> | 1 |[2, 3, 4]|
> +------+---------+
> {noformat}
> There is no way to directly transform it into :
> {noformat}
> +------+------+------+------+
> | Col1 | Col2 | Col3 | Col4 |
> +------+------+------+------+
> | 1 | 2 | 3 | 4 |
> | 1 | 2 | 3 | 4 |
> +------+------+------+------+
> {noformat}
> I think this should be easy to implement
> More infos here :
> http://stackoverflow.com/questions/37391241/explode-spark-columns/37392793#37392793
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
