[ https://issues.apache.org/jira/browse/SPARK-12776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15576739#comment-15576739 ]
Michael Armbrust commented on SPARK-12776: ------------------------------------------ I would love to see better support here, but I don't think anyone has taken the time to flesh out the API. Some suggestions I've heard are too support arbitrary objects in Datasets that are pickled and stored as {{ArrayType(BytesType)}}. If you want schema, you could also tell us that, either with a schema string (where we use the schema to extract columns out of the object) or by using something like a named tuple. This is all very rough though and I knew when we did a prototype we ran into issues caused by the batching we use when talking to the JVM (essentially things like {{df.count()}} broke). If someone wants to flesh out these proposals that would be great. > Implement Python API for Datasets > --------------------------------- > > Key: SPARK-12776 > URL: https://issues.apache.org/jira/browse/SPARK-12776 > Project: Spark > Issue Type: Improvement > Components: PySpark > Reporter: Kevin Cox > Priority: Minor > > Now that the Dataset API is in Scala and Java it would be awesome to see it > show up in PySpark. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org