[jira] [Commented] (SPARK-11281) Issue with creating and collecting DataFrame using environments
[
https://issues.apache.org/jira/browse/SPARK-11281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007038#comment-15007038
]
Maciej Szymkiewicz commented on SPARK-11281:
[~shivaram] I've tested both current master and my PR for [SPARK-11086] and it
looks it is indeed resolved. I would like to add some tests but otherwise it
looks like it can be closed.
> Issue with creating and collecting DataFrame using environments
>
>
> Key: SPARK-11281
> URL: https://issues.apache.org/jira/browse/SPARK-11281
> Project: Spark
> Issue Type: Bug
> Components: SparkR
>Affects Versions: 1.6.0
> Environment: R 3.2.2, Spark build from master
> 487d409e71767c76399217a07af8de1bb0da7aa8
>Reporter: Maciej Szymkiewicz
> Fix For: 1.6.0
>
>
> It is not possible to to access Map field created from an environment.
> Assuming local data frame is created as follows:
> {code}
> ldf <- data.frame(row.names=1:2)
> ldf$x <- c(as.environment(list(a=1, b=2)), as.environment(list(c=3)))
> str(ldf)
> ## 'data.frame': 2 obs. of 1 variable:
> ## $ x:List of 2
> ## ..$ :
> ## ..$ :
> get("a", ldf$x[[1]])
> ## [1] 1
> get("c", ldf$x[[2]])
> ## [1] 3
> {code}
> It is possible to create a Spark data frame:
> {code}
> sdf <- createDataFrame(sqlContext, ldf)
> printSchema(sdf)
> ## root
> ## |-- x: array (nullable = true)
> ## ||-- element: map (containsNull = true)
> ## |||-- key: string
> ## |||-- value: double (valueContainsNull = true)
> {code}
> but it throws:
> {code}
> java.lang.IllegalArgumentException: Invalid array type e
> {code}
> on collect / head.
> Problem seems to be specific to environments and cannot be reproduced when
> Map comes for example from Cassandra table.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11281) Issue with creating and collecting DataFrame using environments
[
https://issues.apache.org/jira/browse/SPARK-11281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15006966#comment-15006966
]
Shivaram Venkataraman commented on SPARK-11281:
---
Does the example posted in the description work now or does it still not work ?
Sorry I'm just confused what the resolution to this bug was (i.e. if it was
fixed or we decided we won't fix etc.)
> Issue with creating and collecting DataFrame using environments
>
>
> Key: SPARK-11281
> URL: https://issues.apache.org/jira/browse/SPARK-11281
> Project: Spark
> Issue Type: Bug
> Components: SparkR
>Affects Versions: 1.6.0
> Environment: R 3.2.2, Spark build from master
> 487d409e71767c76399217a07af8de1bb0da7aa8
>Reporter: Maciej Szymkiewicz
>Assignee: Maciej Szymkiewicz
> Fix For: 1.6.0
>
>
> It is not possible to to access Map field created from an environment.
> Assuming local data frame is created as follows:
> {code}
> ldf <- data.frame(row.names=1:2)
> ldf$x <- c(as.environment(list(a=1, b=2)), as.environment(list(c=3)))
> str(ldf)
> ## 'data.frame': 2 obs. of 1 variable:
> ## $ x:List of 2
> ## ..$ :
> ## ..$ :
> get("a", ldf$x[[1]])
> ## [1] 1
> get("c", ldf$x[[2]])
> ## [1] 3
> {code}
> It is possible to create a Spark data frame:
> {code}
> sdf <- createDataFrame(sqlContext, ldf)
> printSchema(sdf)
> ## root
> ## |-- x: array (nullable = true)
> ## ||-- element: map (containsNull = true)
> ## |||-- key: string
> ## |||-- value: double (valueContainsNull = true)
> {code}
> but it throws:
> {code}
> java.lang.IllegalArgumentException: Invalid array type e
> {code}
> on collect / head.
> Problem seems to be specific to environments and cannot be reproduced when
> Map comes for example from Cassandra table.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11281) Issue with creating and collecting DataFrame using environments
[
https://issues.apache.org/jira/browse/SPARK-11281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15006955#comment-15006955
]
Shivaram Venkataraman commented on SPARK-11281:
---
[~sunrui] [~zero323] Is there a test case in
https://github.com/apache/spark/commit/d7d9fa0b8750166f8b74f9bc321df26908683a8b
that covers this ?
> Issue with creating and collecting DataFrame using environments
>
>
> Key: SPARK-11281
> URL: https://issues.apache.org/jira/browse/SPARK-11281
> Project: Spark
> Issue Type: Bug
> Components: SparkR
>Affects Versions: 1.6.0
> Environment: R 3.2.2, Spark build from master
> 487d409e71767c76399217a07af8de1bb0da7aa8
>Reporter: Maciej Szymkiewicz
> Fix For: 1.6.0
>
>
> It is not possible to to access Map field created from an environment.
> Assuming local data frame is created as follows:
> {code}
> ldf <- data.frame(row.names=1:2)
> ldf$x <- c(as.environment(list(a=1, b=2)), as.environment(list(c=3)))
> str(ldf)
> ## 'data.frame': 2 obs. of 1 variable:
> ## $ x:List of 2
> ## ..$ :
> ## ..$ :
> get("a", ldf$x[[1]])
> ## [1] 1
> get("c", ldf$x[[2]])
> ## [1] 3
> {code}
> It is possible to create a Spark data frame:
> {code}
> sdf <- createDataFrame(sqlContext, ldf)
> printSchema(sdf)
> ## root
> ## |-- x: array (nullable = true)
> ## ||-- element: map (containsNull = true)
> ## |||-- key: string
> ## |||-- value: double (valueContainsNull = true)
> {code}
> but it throws:
> {code}
> java.lang.IllegalArgumentException: Invalid array type e
> {code}
> on collect / head.
> Problem seems to be specific to environments and cannot be reproduced when
> Map comes for example from Cassandra table.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11281) Issue with creating and collecting DataFrame using environments
[
https://issues.apache.org/jira/browse/SPARK-11281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15006960#comment-15006960
]
Maciej Szymkiewicz commented on SPARK-11281:
[~shivaram] No, there isn't. I removed this one because there was nothing we
could test there.
> Issue with creating and collecting DataFrame using environments
>
>
> Key: SPARK-11281
> URL: https://issues.apache.org/jira/browse/SPARK-11281
> Project: Spark
> Issue Type: Bug
> Components: SparkR
>Affects Versions: 1.6.0
> Environment: R 3.2.2, Spark build from master
> 487d409e71767c76399217a07af8de1bb0da7aa8
>Reporter: Maciej Szymkiewicz
>Assignee: Maciej Szymkiewicz
> Fix For: 1.6.0
>
>
> It is not possible to to access Map field created from an environment.
> Assuming local data frame is created as follows:
> {code}
> ldf <- data.frame(row.names=1:2)
> ldf$x <- c(as.environment(list(a=1, b=2)), as.environment(list(c=3)))
> str(ldf)
> ## 'data.frame': 2 obs. of 1 variable:
> ## $ x:List of 2
> ## ..$ :
> ## ..$ :
> get("a", ldf$x[[1]])
> ## [1] 1
> get("c", ldf$x[[2]])
> ## [1] 3
> {code}
> It is possible to create a Spark data frame:
> {code}
> sdf <- createDataFrame(sqlContext, ldf)
> printSchema(sdf)
> ## root
> ## |-- x: array (nullable = true)
> ## ||-- element: map (containsNull = true)
> ## |||-- key: string
> ## |||-- value: double (valueContainsNull = true)
> {code}
> but it throws:
> {code}
> java.lang.IllegalArgumentException: Invalid array type e
> {code}
> on collect / head.
> Problem seems to be specific to environments and cannot be reproduced when
> Map comes for example from Cassandra table.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11281) Issue with creating and collecting DataFrame using environments
[
https://issues.apache.org/jira/browse/SPARK-11281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007064#comment-15007064
]
Shivaram Venkataraman commented on SPARK-11281:
---
Thats cool ! lets keep this open till we add tests and then close it as a part
of that PR
> Issue with creating and collecting DataFrame using environments
>
>
> Key: SPARK-11281
> URL: https://issues.apache.org/jira/browse/SPARK-11281
> Project: Spark
> Issue Type: Bug
> Components: SparkR
>Affects Versions: 1.6.0
> Environment: R 3.2.2, Spark build from master
> 487d409e71767c76399217a07af8de1bb0da7aa8
>Reporter: Maciej Szymkiewicz
> Fix For: 1.6.0
>
>
> It is not possible to to access Map field created from an environment.
> Assuming local data frame is created as follows:
> {code}
> ldf <- data.frame(row.names=1:2)
> ldf$x <- c(as.environment(list(a=1, b=2)), as.environment(list(c=3)))
> str(ldf)
> ## 'data.frame': 2 obs. of 1 variable:
> ## $ x:List of 2
> ## ..$ :
> ## ..$ :
> get("a", ldf$x[[1]])
> ## [1] 1
> get("c", ldf$x[[2]])
> ## [1] 3
> {code}
> It is possible to create a Spark data frame:
> {code}
> sdf <- createDataFrame(sqlContext, ldf)
> printSchema(sdf)
> ## root
> ## |-- x: array (nullable = true)
> ## ||-- element: map (containsNull = true)
> ## |||-- key: string
> ## |||-- value: double (valueContainsNull = true)
> {code}
> but it throws:
> {code}
> java.lang.IllegalArgumentException: Invalid array type e
> {code}
> on collect / head.
> Problem seems to be specific to environments and cannot be reproduced when
> Map comes for example from Cassandra table.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11281) Issue with creating and collecting DataFrame using environments
[
https://issues.apache.org/jira/browse/SPARK-11281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007272#comment-15007272
]
Apache Spark commented on SPARK-11281:
--
User 'zero323' has created a pull request for this issue:
https://github.com/apache/spark/pull/9743
> Issue with creating and collecting DataFrame using environments
>
>
> Key: SPARK-11281
> URL: https://issues.apache.org/jira/browse/SPARK-11281
> Project: Spark
> Issue Type: Bug
> Components: SparkR
>Affects Versions: 1.6.0
> Environment: R 3.2.2, Spark build from master
> 487d409e71767c76399217a07af8de1bb0da7aa8
>Reporter: Maciej Szymkiewicz
> Fix For: 1.6.0
>
>
> It is not possible to to access Map field created from an environment.
> Assuming local data frame is created as follows:
> {code}
> ldf <- data.frame(row.names=1:2)
> ldf$x <- c(as.environment(list(a=1, b=2)), as.environment(list(c=3)))
> str(ldf)
> ## 'data.frame': 2 obs. of 1 variable:
> ## $ x:List of 2
> ## ..$ :
> ## ..$ :
> get("a", ldf$x[[1]])
> ## [1] 1
> get("c", ldf$x[[2]])
> ## [1] 3
> {code}
> It is possible to create a Spark data frame:
> {code}
> sdf <- createDataFrame(sqlContext, ldf)
> printSchema(sdf)
> ## root
> ## |-- x: array (nullable = true)
> ## ||-- element: map (containsNull = true)
> ## |||-- key: string
> ## |||-- value: double (valueContainsNull = true)
> {code}
> but it throws:
> {code}
> java.lang.IllegalArgumentException: Invalid array type e
> {code}
> on collect / head.
> Problem seems to be specific to environments and cannot be reproduced when
> Map comes for example from Cassandra table.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
