[
https://issues.apache.org/jira/browse/SPARK-26696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16749518#comment-16749518
]
Simeon Simeonov commented on SPARK-26696:
-----------------------------------------
[PR with improvement|https://github.com/apache/spark/pull/23620]
> Dataset encoder should be publicly accessible
> ---------------------------------------------
>
> Key: SPARK-26696
> URL: https://issues.apache.org/jira/browse/SPARK-26696
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.4.0
> Reporter: Simeon Simeonov
> Priority: Major
> Labels: dataset, encoding
>
> As a platform, Spark should enable framework developers to accomplish outside
> of the Spark codebase much of what can be accomplished inside the Spark
> codebase. One of the obstacles to this is a historical pattern of excessive
> data hiding in Spark, e.g., {{expr}} in {{Column}} not being accessible. This
> issue is an example of this pattern when it comes to {{Dataset}}.
> Consider a transformation with the signature `def foo[A](ds: Dataset[A]):
> Dataset[A]`, which requires the use of {{toDF()}}. To get back to
> {{Dataset[A]}} would require calling {{.as[A]}}, which requires an implicit
> {{Encoder[A]}}. A naive approach would change the function signature to
> `foo[A : Encoder]` but this is poor API design that requires unnecessarily
> carrying of implicits from user code into framework code. We know
> `Encoder[A]` exists because we have access to an instance of `Dataset[A]`...
> but its `encoder` is not accessible.
> The solution is simple: make {{encoder}} a {{@transient val}} just as is the
> case with {{queryExecution}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
