Chris Twiner created SPARK-49960:
------------------------------------
Summary: Support end DeveloperAPI provided AgnosticEncoders based
on ExpressionEncoders
Key: SPARK-49960
URL: https://issues.apache.org/jira/browse/SPARK-49960
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.0.0
Reporter: Chris Twiner
In Spark versions prior to 4.0.0-preview2 ExpressionEncoders were used
throughout the original api. SPARK-49025 removed re-use of ExpressionEncoders
for derivation of tuples rendering user provided ExpressionEncoders such as
Frameless incompatible with joinWith (which only uses the AgnosticEncoder
hierarchy). There are other likely combinations where the encoder is not
passed into the function.
This patch aims to add an extension AgnosticExpressionPathEncoder trait with
added:
{code:java}
def toCatalyst(input: Expression): Expression
def fromCatalyst(inputPath: Expression): Expression {code}
functions mirroring the underlying Des/SerializerBuildHelper approach (also
used by Frameless). In turn the BuildHelper's delegate to these functions when
the AgnosticEncoder is an AgnosticExpressionPathEncoder.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
