[
https://issues.apache.org/jira/browse/SPARK-49960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Twiner updated SPARK-49960:
---------------------------------
Description:
In Spark versions prior to 4.0.0-preview2 ExpressionEncoders were used
throughout the original api. SPARK-49025 removed re-use of ExpressionEncoders
for derivation of tuples rendering user provided ExpressionEncoders such as
Frameless incompatible with joinWith (which only uses the AgnosticEncoder
hierarchy). There are other likely combinations where the encoder is not
passed into the function.
This patch aims to add an extension AgnosticExpressionPathEncoder trait with
added:
{code:java}
def toCatalyst(input: Expression): Expression
def fromCatalyst(inputPath: Expression): Expression {code}
functions mirroring the underlying Des/SerializerBuildHelper approach (also
used by Frameless). In turn the BuildHelper's delegate to these functions when
the AgnosticEncoder is an AgnosticExpressionPathEncoder.
Moreover it aims to fix AgnosticEncoder issues identified with moving Frameless
to AgnosticEncoders instead of ExpressionEncoders allowing users to migrate via
both the existing Expression codebase on 4.0 and future use of AgnosticEncoders
(limited in the short term to Classic as serialization of derived codecs
requires closures)
was:
In Spark versions prior to 4.0.0-preview2 ExpressionEncoders were used
throughout the original api. SPARK-49025 removed re-use of ExpressionEncoders
for derivation of tuples rendering user provided ExpressionEncoders such as
Frameless incompatible with joinWith (which only uses the AgnosticEncoder
hierarchy). There are other likely combinations where the encoder is not
passed into the function.
This patch aims to add an extension AgnosticExpressionPathEncoder trait with
added:
{code:java}
def toCatalyst(input: Expression): Expression
def fromCatalyst(inputPath: Expression): Expression {code}
functions mirroring the underlying Des/SerializerBuildHelper approach (also
used by Frameless). In turn the BuildHelper's delegate to these functions when
the AgnosticEncoder is an AgnosticExpressionPathEncoder.
> Support end DeveloperAPI provided AgnosticEncoders based on
> ExpressionEncoders to allow migration
> -------------------------------------------------------------------------------------------------
>
> Key: SPARK-49960
> URL: https://issues.apache.org/jira/browse/SPARK-49960
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.0.0
> Reporter: Chris Twiner
> Priority: Major
> Labels: Encoding, pull-request-available
>
> In Spark versions prior to 4.0.0-preview2 ExpressionEncoders were used
> throughout the original api. SPARK-49025 removed re-use of
> ExpressionEncoders for derivation of tuples rendering user provided
> ExpressionEncoders such as Frameless incompatible with joinWith (which only
> uses the AgnosticEncoder hierarchy). There are other likely combinations
> where the encoder is not passed into the function.
> This patch aims to add an extension AgnosticExpressionPathEncoder trait with
> added:
> {code:java}
> def toCatalyst(input: Expression): Expression
> def fromCatalyst(inputPath: Expression): Expression {code}
> functions mirroring the underlying Des/SerializerBuildHelper approach (also
> used by Frameless). In turn the BuildHelper's delegate to these functions
> when the AgnosticEncoder is an AgnosticExpressionPathEncoder.
> Moreover it aims to fix AgnosticEncoder issues identified with moving
> Frameless to AgnosticEncoders instead of ExpressionEncoders allowing users to
> migrate via both the existing Expression codebase on 4.0 and future use of
> AgnosticEncoders (limited in the short term to Classic as serialization of
> derived codecs requires closures)
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
