L. C. Hsieh created SPARK-62120:
-----------------------------------

             Summary: Return candidate positions from 
SupportsExternalLookup.selectLookupPredicates
                 Key: SPARK-62120
                 URL: https://issues.apache.org/jira/browse/SPARK-62120
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: L. C. Hsieh


SupportsExternalLookup.selectLookupPredicates (added in 4.4.0 for external 
lookup join) returns Predicate[] and requires the connector to return the 
offered instances unchanged; Spark matches them by object identity. A connector 
that returns an equal copy of an offered predicate fails at query start with 
DATA_SOURCE_INVALID_EXTERNAL_LOOKUP.UNOFFERED_SELECTED_PREDICATE, which is easy 
to get wrong.

The method only lets a connector choose a subset of the offered candidates and 
their order, so it can return positions in the candidates array (int[]) 
instead. Positions keep same-shaped candidates distinct and keep the 
connector's order, which defines the parameter row layout, without depending on 
object identity. The API is not released yet, so this is the cheapest time to 
change it.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to