[
https://issues.apache.org/jira/browse/SPARK-49960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17889274#comment-17889274
]
Chris Twiner commented on SPARK-49960:
--------------------------------------
nb - a patch is in progress.
> Support end DeveloperAPI provided AgnosticEncoders based on ExpressionEncoders
> ------------------------------------------------------------------------------
>
> Key: SPARK-49960
> URL: https://issues.apache.org/jira/browse/SPARK-49960
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.0.0
> Reporter: Chris Twiner
> Priority: Major
> Labels: Encoding
>
> In Spark versions prior to 4.0.0-preview2 ExpressionEncoders were used
> throughout the original api. SPARK-49025 removed re-use of
> ExpressionEncoders for derivation of tuples rendering user provided
> ExpressionEncoders such as Frameless incompatible with joinWith (which only
> uses the AgnosticEncoder hierarchy). There are other likely combinations
> where the encoder is not passed into the function.
> This patch aims to add an extension AgnosticExpressionPathEncoder trait with
> added:
> {code:java}
> def toCatalyst(input: Expression): Expression
> def fromCatalyst(inputPath: Expression): Expression {code}
> functions mirroring the underlying Des/SerializerBuildHelper approach (also
> used by Frameless). In turn the BuildHelper's delegate to these functions
> when the AgnosticEncoder is an AgnosticExpressionPathEncoder.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
