[
https://issues.apache.org/jira/browse/SPARK-61817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenchen Fan updated SPARK-61817:
--------------------------------
Fix Version/s: 4.4.0
> Accept metadata column keys in DataFrame sort-merge ASOF joins
> --------------------------------------------------------------
>
> Key: SPARK-61817
> URL: https://issues.apache.org/jira/browse/SPARK-61817
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.4.0
> Reporter: Luka Zdravic
> Assignee: Luka Zdravic
> Priority: Minor
> Labels: pull-request-available
> Fix For: 4.4.0, 4.5.0
>
>
> SPARK-60582 checks DataFrame ASOF join keys during analysis. It assigns each
> key to an input by using the input's output plus only the hidden USING and
> NATURAL join columns. So a key over a regular metadata column, such as
> `_metadata.file_size` of a file source or the `index` column of a DSv2 table,
> matches neither input, and the join fails.
> With `spark.sql.join.sortMergeAsOfJoin.enabled=true`:
> val left = spark.read.parquet(leftPath)
> val right = spark.read.parquet(rightPath)
> left.joinAsOf(right, left.col("_metadata.file_size"),
> right.col("_metadata.file_size"),
> usingColumns = Seq.empty, joinType = "inner", tolerance = null,
> allowExactMatches = true, direction = "backward")
> fails with:
> [AS_OF_JOIN.KEY_REFERENCE] ... Got: "_metadata.file_size AS file_size" and
> "_metadata.file_size AS file_size".
> Each key references only its own input, so the error is wrong. Regular joins
> accept these keys, and the same ASOF join works with the flag off.
> Proposed fix: for DataFrame `joinAsOf`, match each key against the input's
> output plus all of its metadata columns. SQL `MATCH_CONDITION` operands do
> not change.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]