L. C. Hsieh created SPARK-62120:
-----------------------------------
Summary: Return candidate positions from
SupportsExternalLookup.selectLookupPredicates
Key: SPARK-62120
URL: https://issues.apache.org/jira/browse/SPARK-62120
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: L. C. Hsieh
SupportsExternalLookup.selectLookupPredicates (added in 4.4.0 for external
lookup join) returns Predicate[] and requires the connector to return the
offered instances unchanged; Spark matches them by object identity. A connector
that returns an equal copy of an offered predicate fails at query start with
DATA_SOURCE_INVALID_EXTERNAL_LOOKUP.UNOFFERED_SELECTED_PREDICATE, which is easy
to get wrong.
The method only lets a connector choose a subset of the offered candidates and
their order, so it can return positions in the candidates array (int[])
instead. Positions keep same-shaped candidates distinct and keep the
connector's order, which defines the parameter row layout, without depending on
object identity. The API is not released yet, so this is the cheapest time to
change it.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]