Luka Zdravic created SPARK-59999:
------------------------------------
Summary: Metadata column _metadata cannot be resolved after a JOIN
... USING
Key: SPARK-59999
URL: https://issues.apache.org/jira/browse/SPARK-59999
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.4.0
Reporter: Luka Zdravic
_metadata cannot be read after JOIN ... USING. The same query with ON works.
{code:sql}
CREATE TABLE t (k INT, v INT) USING parquet;
INSERT INTO t VALUES (1, 10);
-- fails: [UNRESOLVED_COLUMN.WITH_SUGGESTION] `r`.`_metadata`.`file_name`
cannot be resolved.
SELECT r._metadata.file_name FROM (VALUES (1)) AS l(k) JOIN t AS r USING (k);
-- works:
SELECT r._metadata.file_name FROM (VALUES (1)) AS l(k) JOIN t AS r ON l.k =
r.k;
{code}
LEFT JOIN ... USING and LEFT ASOF JOIN ... USING fail the same way.
Cause: the Project added for USING hides the child's metadata columns
(commonNaturalJoinProcessing in Analyzer, and ResolveAsOfJoin).
Expected: _metadata works after USING, the same as after ON.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
