[ 
https://issues.apache.org/jira/browse/SPARK-61817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wenchen Fan updated SPARK-61817:
--------------------------------
    Fix Version/s:     (was: 4.5.0)

> Accept metadata column keys in DataFrame sort-merge ASOF joins
> --------------------------------------------------------------
>
>                 Key: SPARK-61817
>                 URL: https://issues.apache.org/jira/browse/SPARK-61817
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 4.4.0
>            Reporter: Luka Zdravic
>            Assignee: Luka Zdravic
>            Priority: Minor
>              Labels: pull-request-available
>             Fix For: 4.4.0
>
>
> SPARK-60582 checks DataFrame ASOF join keys during analysis. It assigns each 
> key to an input by using the input's output plus only the hidden USING and 
> NATURAL join columns. So a key over a regular metadata column, such as 
> `_metadata.file_size` of a file source or the `index` column of a DSv2 table, 
> matches neither input, and the join fails.
> With `spark.sql.join.sortMergeAsOfJoin.enabled=true`:
> val left = spark.read.parquet(leftPath)
> val right = spark.read.parquet(rightPath)
> left.joinAsOf(right, left.col("_metadata.file_size"), 
> right.col("_metadata.file_size"),
> usingColumns = Seq.empty, joinType = "inner", tolerance = null,
> allowExactMatches = true, direction = "backward")
> fails with:
> [AS_OF_JOIN.KEY_REFERENCE] ... Got: "_metadata.file_size AS file_size" and 
> "_metadata.file_size AS file_size".
> Each key references only its own input, so the error is wrong. Regular joins 
> accept these keys, and the same ASOF join works with the flag off.
> Proposed fix: for DataFrame `joinAsOf`, match each key against the input's 
> output plus all of its metadata columns. SQL `MATCH_CONDITION` operands do 
> not change.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to