Luka Zdravic created SPARK-61817:
------------------------------------

             Summary: Accept metadata column keys in DataFrame sort-merge ASOF 
joins
                 Key: SPARK-61817
                 URL: https://issues.apache.org/jira/browse/SPARK-61817
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 4.4.0
            Reporter: Luka Zdravic


SPARK-60582 checks DataFrame ASOF join keys during analysis. It assigns each 
key to an input by using the input's output plus only the hidden USING and 
NATURAL join columns. So a key over a regular metadata column, such as 
`_metadata.file_size` of a file source or the `index` column of a DSv2 table, 
matches neither input, and the join fails.

With `spark.sql.join.sortMergeAsOfJoin.enabled=true`:

val left = spark.read.parquet(leftPath)
val right = spark.read.parquet(rightPath)
left.joinAsOf(right, left.col("_metadata.file_size"), 
right.col("_metadata.file_size"),
usingColumns = Seq.empty, joinType = "inner", tolerance = null,
allowExactMatches = true, direction = "backward")

fails with:

[AS_OF_JOIN.KEY_REFERENCE] ... Got: "_metadata.file_size AS file_size" and 
"_metadata.file_size AS file_size".

Each key references only its own input, so the error is wrong. Regular joins 
accept these keys, and the same ASOF join works with the flag off.

Proposed fix: for DataFrame `joinAsOf`, match each key against the input's 
output plus all of its metadata columns. SQL `MATCH_CONDITION` operands do not 
change.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to