[
https://issues.apache.org/jira/browse/SPARK-49960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17889274#comment-17889274
]
Chris Twiner edited comment on SPARK-49960 at 2/27/25 11:10 AM:
----------------------------------------------------------------
PR 50023 is tested against Frameless and Quality
was (Author: JIRAUSER301163):
nb - a patch is in progress.
> Support end DeveloperAPI provided AgnosticEncoders based on
> ExpressionEncoders to allow migration
> -------------------------------------------------------------------------------------------------
>
> Key: SPARK-49960
> URL: https://issues.apache.org/jira/browse/SPARK-49960
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.0.0
> Reporter: Chris Twiner
> Priority: Major
> Labels: Encoding, pull-request-available
>
> In Spark versions prior to 4.0.0-preview2 ExpressionEncoders were used
> throughout the original api. SPARK-49025 removed re-use of
> ExpressionEncoders for derivation of tuples rendering user provided
> ExpressionEncoders such as Frameless incompatible with joinWith (which only
> uses the AgnosticEncoder hierarchy). There are other likely combinations
> where the encoder is not passed into the function.
> This patch aims to add an extension AgnosticExpressionPathEncoder trait with
> added:
> {code:java}
> def toCatalyst(input: Expression): Expression
> def fromCatalyst(inputPath: Expression): Expression {code}
> functions mirroring the underlying Des/SerializerBuildHelper approach (also
> used by Frameless). In turn the BuildHelper's delegate to these functions
> when the AgnosticEncoder is an AgnosticExpressionPathEncoder.
> Moreover it aims to fix AgnosticEncoder issues identified with moving
> Frameless to AgnosticEncoders instead of ExpressionEncoders allowing users to
> migrate via both the existing Expression codebase on 4.0 and future use of
> AgnosticEncoders (limited in the short term to Classic as serialization of
> derived codecs requires closures)
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
