Michael Armbrust commented on SPARK-12776:

I would love to see better support here, but I don't think anyone has taken the 
time to flesh out the API.  Some suggestions I've heard are too support 
arbitrary objects in Datasets that are pickled and stored as 
{{ArrayType(BytesType)}}.  If you want schema, you could also tell us that, 
either with a schema string (where we use the schema to extract columns out of 
the object) or by using something like a named tuple.  This is all very rough 
though and I knew when we did a prototype we ran into issues caused by the 
batching we use when talking to the JVM (essentially things like {{df.count()}} 
broke).  If someone wants to flesh out these proposals that would be great.

> Implement Python API for Datasets
> ---------------------------------
>                 Key: SPARK-12776
>                 URL: https://issues.apache.org/jira/browse/SPARK-12776
>             Project: Spark
>          Issue Type: Improvement
>          Components: PySpark
>            Reporter: Kevin Cox
>            Priority: Minor
> Now that the Dataset API is in Scala and Java it would be awesome to see it 
> show up in PySpark.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to