[
https://issues.apache.org/jira/browse/SPARK-49960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Twiner updated SPARK-49960:
---------------------------------
Summary: Support end DeveloperAPI provided AgnosticEncoders based on
ExpressionEncoders to allow migration (was: Support end DeveloperAPI provided
AgnosticEncoders based on ExpressionEncoders)
> Support end DeveloperAPI provided AgnosticEncoders based on
> ExpressionEncoders to allow migration
> -------------------------------------------------------------------------------------------------
>
> Key: SPARK-49960
> URL: https://issues.apache.org/jira/browse/SPARK-49960
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.0.0
> Reporter: Chris Twiner
> Priority: Major
> Labels: Encoding, pull-request-available
>
> In Spark versions prior to 4.0.0-preview2 ExpressionEncoders were used
> throughout the original api. SPARK-49025 removed re-use of
> ExpressionEncoders for derivation of tuples rendering user provided
> ExpressionEncoders such as Frameless incompatible with joinWith (which only
> uses the AgnosticEncoder hierarchy). There are other likely combinations
> where the encoder is not passed into the function.
> This patch aims to add an extension AgnosticExpressionPathEncoder trait with
> added:
> {code:java}
> def toCatalyst(input: Expression): Expression
> def fromCatalyst(inputPath: Expression): Expression {code}
> functions mirroring the underlying Des/SerializerBuildHelper approach (also
> used by Frameless). In turn the BuildHelper's delegate to these functions
> when the AgnosticEncoder is an AgnosticExpressionPathEncoder.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
