[
https://issues.apache.org/jira/browse/SPARK-44450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17760186#comment-17760186
]
Herman van Hövell commented on SPARK-44450:
-------------------------------------------
It is not really part of the public API, and we don't write user facing
documentation for that. The use case is main that we can encode directly
between user objects and arrow batches. We needed this to get rid of the
Catalyst dependency. You can check how it is integrated with the Spark Connect
Scala Client, look at SparkResult for deserialization, and look at
SparkSession.createDataset for serialization. Alternatively you can look at the
ArrowEncoder suite.
> Make direct Arrow encoding work with SQL/API
> --------------------------------------------
>
> Key: SPARK-44450
> URL: https://issues.apache.org/jira/browse/SPARK-44450
> Project: Spark
> Issue Type: New Feature
> Components: Connect
> Affects Versions: 3.4.1
> Reporter: Herman van Hövell
> Assignee: Herman van Hövell
> Priority: Major
> Fix For: 3.5.0
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
