Luka Zdravic created SPARK-61817:
------------------------------------
Summary: Accept metadata column keys in DataFrame sort-merge ASOF
joins
Key: SPARK-61817
URL: https://issues.apache.org/jira/browse/SPARK-61817
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.4.0
Reporter: Luka Zdravic
SPARK-60582 checks DataFrame ASOF join keys during analysis. It assigns each
key to an input by using the input's output plus only the hidden USING and
NATURAL join columns. So a key over a regular metadata column, such as
`_metadata.file_size` of a file source or the `index` column of a DSv2 table,
matches neither input, and the join fails.
With `spark.sql.join.sortMergeAsOfJoin.enabled=true`:
val left = spark.read.parquet(leftPath)
val right = spark.read.parquet(rightPath)
left.joinAsOf(right, left.col("_metadata.file_size"),
right.col("_metadata.file_size"),
usingColumns = Seq.empty, joinType = "inner", tolerance = null,
allowExactMatches = true, direction = "backward")
fails with:
[AS_OF_JOIN.KEY_REFERENCE] ... Got: "_metadata.file_size AS file_size" and
"_metadata.file_size AS file_size".
Each key references only its own input, so the error is wrong. Regular joins
accept these keys, and the same ASOF join works with the flag off.
Proposed fix: for DataFrame `joinAsOf`, match each key against the input's
output plus all of its metadata columns. SQL `MATCH_CONDITION` operands do not
change.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]