[
https://issues.apache.org/jira/browse/SPARK-59999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luka Zdravic updated SPARK-59999:
---------------------------------
Description:
_metadata cannot be read after JOIN ... USING. The same query with ON works.
{code:sql}
CREATE TABLE t (k INT, v INT) USING parquet;
INSERT INTO t VALUES (1, 10);
-- fails: [UNRESOLVED_COLUMN.WITH_SUGGESTION] `r`.`_metadata`.`file_name`
cannot be resolved.
SELECT r._metadata.file_name FROM (VALUES (1)) AS l(k) JOIN t AS r USING (k);
-- works:
SELECT r._metadata.file_name FROM (VALUES (1)) AS l(k) JOIN t AS r ON l.k =
r.k;
{code}
LEFT JOIN ... USING and LEFT ASOF JOIN ... USING fail the same way.
Cause: the Project added for USING hides the child's metadata columns
(commonNaturalJoinProcessing in Analyzer, and ResolveAsOfJoin).
Assigned to : [~lukazdravic]
Expected: _metadata works after USING, the same as after ON.
was:
_metadata cannot be read after JOIN ... USING. The same query with ON works.
{code:sql}
CREATE TABLE t (k INT, v INT) USING parquet;
INSERT INTO t VALUES (1, 10);
-- fails: [UNRESOLVED_COLUMN.WITH_SUGGESTION] `r`.`_metadata`.`file_name`
cannot be resolved.
SELECT r._metadata.file_name FROM (VALUES (1)) AS l(k) JOIN t AS r USING (k);
-- works:
SELECT r._metadata.file_name FROM (VALUES (1)) AS l(k) JOIN t AS r ON l.k =
r.k;
{code}
LEFT JOIN ... USING and LEFT ASOF JOIN ... USING fail the same way.
Cause: the Project added for USING hides the child's metadata columns
(commonNaturalJoinProcessing in Analyzer, and ResolveAsOfJoin).
Expected: _metadata works after USING, the same as after ON.
> Metadata column _metadata cannot be resolved after a JOIN ... USING
> -------------------------------------------------------------------
>
> Key: SPARK-59999
> URL: https://issues.apache.org/jira/browse/SPARK-59999
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.4.0
> Reporter: Luka Zdravic
> Priority: Minor
>
> _metadata cannot be read after JOIN ... USING. The same query with ON works.
> {code:sql}
> CREATE TABLE t (k INT, v INT) USING parquet;
> INSERT INTO t VALUES (1, 10);
> -- fails: [UNRESOLVED_COLUMN.WITH_SUGGESTION] `r`.`_metadata`.`file_name`
> cannot be resolved.
> SELECT r._metadata.file_name FROM (VALUES (1)) AS l(k) JOIN t AS r USING
> (k);
> -- works:
> SELECT r._metadata.file_name FROM (VALUES (1)) AS l(k) JOIN t AS r ON l.k =
> r.k;
> {code}
> LEFT JOIN ... USING and LEFT ASOF JOIN ... USING fail the same way.
> Cause: the Project added for USING hides the child's metadata columns
> (commonNaturalJoinProcessing in Analyzer, and ResolveAsOfJoin).
> Assigned to : [~lukazdravic]
> Expected: _metadata works after USING, the same as after ON.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
