[
https://issues.apache.org/jira/browse/SPARK-11283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen updated SPARK-11283:
------------------------------
Assignee: Maciej Szymkiewicz
> List column gets additional level of nesting when converted to Spark DataFrame
> ------------------------------------------------------------------------------
>
> Key: SPARK-11283
> URL: https://issues.apache.org/jira/browse/SPARK-11283
> Project: Spark
> Issue Type: Bug
> Components: SparkR
> Affects Versions: 1.6.0
> Environment: R 3.2.2, Spark build from master
> 487d409e71767c76399217a07af8de1bb0da7aa8
> Reporter: Maciej Szymkiewicz
> Assignee: Maciej Szymkiewicz
> Fix For: 1.6.0
>
>
> When input data frame contains list column there is an additional level of
> nesting in a Spark DataFrame and as a result collected data is no longer
> identical to input:
> {code}
> ldf <- data.frame(row.names=1:2)
> ldf$x <- list(list(1), list(2))
> sdf <- createDataFrame(sqlContext, ldf)
> printSchema(sdf)
> ## root
> ## |-- x: array (nullable = true)
> ## | |-- element: array (containsNull = true)
> ## | | |-- element: double (containsNull = true)
> identical(ldf, collect(sdf))
> ## [1] FALSE
> {code}
> Comparing structure:
> Local df
> {code}
> unclass(ldf)
> ## $x
> ## $x[[1]]
> ## $x[[1]][[1]]
> ## [1] 1
> ##
> ## $x[[2]]
> ## $x[[2]][[1]]
> ## [1] 2
> ##
> ## attr(,"row.names")
> ## [1] 1 2
> {code}
> Collected
> {code}
> unclass(collect(sdf))
> ## $x
> ## $x[[1]]
> ## $x[[1]][[1]]
> ## $x[[1]][[1]][[1]]
> ## [1] 1
> ##
> ## $x[[2]]
> ## $x[[2]][[1]]
> ## $x[[2]][[1]][[1]]
> ## [1] 2
> ##
> ## attr(,"row.names")
> ## [1] 1 2
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
